Objective: Fires are disaster events with destructive power. In relation to fire-related accidents, fire monitoring is one of the effective measures to reduce the casualties and economic losses caused by such incidents. Compared to traditional methods in fire monitoring, target detection has shown its strengths in terms of cost and outcome. Many researchers have investigated various ways to improve the efficiency of target detection by proposing new algorithms. Thus, numerous algorithms suited for fire monitoring applications have been proposed. However, these typically lack the capacity to detect small targets, which is the main characteristic of flame targets in incipient fires. To enhance the capacity to detect small targets for fire target detection, this paper improved the YOLOv5 algorithm and trained a model based on it with corresponding datasets collected. Methods: First, a fire image dataset with small target scene conditions is prepared for model training and performance testing. In the validation set, eight sets of mutually exclusive sub-datasets of environmental conditions are divided for the purpose of performance testing. Second, three improvements are introduced to improve the YOLOv5 algorithm: a) expansion of the multiscale detection layer to improve its receptive resolution; b) enhancement of the multiscale feature extraction capability by embedding the Swin transformer module, thus reducing the cost of calculation in algorithm deployment; and c) optimization of the postprocessing function by replacing the original algorithm with soft-NMS algorithm to maintain more potential adjacent targets. Next, an improved model YOLOv5s-SSS (swin transformer with soft-NMS for small target) is proposed. To verify the effect of every improvement and their contributions to the final model, the new model is evaluated using four sets of ablation experiments. After parameter optimization, a set of fire images is inputted into the models in the ablation experiment to compare and verify their outputs. Results: The ablation experimental results indicate that, first, all the improvements introduced into the algorithm are valid. Furthermore, the average accuracy of the improved model is 16.3% higher than that of the original algorithm in flame image targets under challenging scene conditions and 5.9% higher in normal-sized image targets. The verification result shows, compared to the original model, the improved model has obvious improvements in terms of reducing the location range of fire targets, thus minimizing the missing detection of small-sized and densely-distributed fire targets and clearly dividing densely or overlapping distributed fire targets. Conclusions: The dataset prepared in this paper can effectively support the training and testing of the improved fire detection algorithm model. Furthermore, the proposed model improvement has been shown to work effectively, along with the reliable performance test, thus providing a new improvement scheme for fire image detection technology. It can also serve as a reference in improving efficiency in various applications, such as accurate positioning of fire points in incipient forest fires and remote sensing monitoring of large-scale fires. However, the overall accuracy of the improved model is relatively low, possibly due to the images in the validation set being deliberately limited to small targets to assess the model's improvement. In the future, more improvements should be introduced to enhance the model's detection ability under various scenarios, such as low-light conditions, so that it can be adequate for industrial applications.