Timely, comprehensive and accurate access to the status and progress of various activities on the construction site is essential for quality control, progress tracking and productivity analysis, and is also necessary for the full realization of fine management and intelligent construction. At present, the progress recording and quality control under the concrete pouring construction scenario are still mostly done manually, leading to problems such as insufficient timeliness, misreporting and omission. In this study, the semantic segmentation and object detection technology in the field of deep learning computer vision are applied to the field of engineering construction. Real-time construction progress is obtained by identifying formwork cover ratios and the unloading event of the bucket, and the overtime warning of layer coverage time with second-level accuracy is realized.