[Objective] Cable cranes are the main concrete transportation equipment used in arch dam construction. Productivity analysis of cable crane transportation is crucial for improving scheduling management, reducing operational costs, and controlling dam construction progress. However, the traditional manual recording method for analyzing cable crane productivity is time-consuming and labor-intensive. Moreover, existing monitoring methods, such as sensors and global navigation satellite systems, are susceptible to interference because of the challenging environment and complicated operating space at dam construction sites. Furthermore, they usually entail high installation and maintenance costs. Therefore, this study proposes an intelligent monitoring method based on visual tracking and pattern recognition for cable crane transportation in dam construction. [Methods] The proposed method initially tracks the process of cable crane transporting concrete using visual tracking technology to obtain the complete moving trajectory of crane buckets. Subsequently, it establishes a pattern recognition model to automatically identify the working states of cable cranes and calculate their productivity by analyzing the time-series features of the trajectory data. In the visual tracking of cable cranes, the main challenge is to address the similar appearance and occlusion problems of crane buckets. Therefore, we propose a new multiobject tracking framework by introducing a rematching mechanism based on tracklet features (segments of the entire object trajectory), which effectively reduces the occurrences of ID switches and enhances tracking accuracy. Additionally, You Only Look Once (YOLO) model is trained as the object detector of the tracking framework. Subsequently, trajectory data obtained by visual tracking is used as input for the pattern recognition model of cable crane working states, with the output being the pouring productivity. This pattern recognition model employs spline interpolation and Savitzky-Golay filters to solve the problems of missing values and noises in the trajectory data. A first-differential method is applied to statistically analyze the variation patterns of the trajectory data. This model can rapidly and accurately identify the working states and determine the key efficiency indicators of cable cranes. [Results] A testing experiment was conducted at an arch dam construction site to evaluate the monitoring performance using this approach. Experimental results are summarized as follows: 1) The proposed vision-based multiobject tracking method proves effective in detecting and tracking cable buckets in intricate construction scenes, thus achieving effective and complete tracking of moving trajectories of crane buckets; moreover, identity F1 score (IDF1) and multiple object tracking accuracy (MOTA) metrics reach 94.8% and 90.0%, respectively. 2) The proposed pattern recognition model can rapidly and accurately distinguish six working states in the cable crane transportation process, including horizontal transport, descent, unloading, ascent, horizontal return, and waiting for loading. 3) Key productivity indicators, such as duration of a single transporting cycle, number of transporting cycles, duration of each working state, and concrete pouring intensity, are accurately calculated and meet engineering management requirements. This also confirms the practicability, reliability, and accuracy of the proposed monitoring method. [Conclusions] Thus, this study successfully integrates vision-based tracking and pattern recognition technologies to develop an intelligent monitoring method, consequently achieving automatic and accurate calculation of cable crane productivity. Furthermore, it demonstrates a positive application effect at dam construction sites and provides innovative perspectives and technical support for construction management.
[1] 王飞,刘金飞,尹习双,等.高拱坝智能进度仿真理论与关键技术[J].清华大学学报(自然科学版), 2021, 61(7):756-767. WANG F, LIU J F, YIN X S, et al. Intelligent scheduling for high arch dams[J]. Journal of Tsinghua University (Science and Technology), 2021, 61(7):756-767.(in Chinese)
[2] 王仁超,王晨旭.混凝土坝浇筑仿真中的缆机干扰与联合问题研究[J].水资源与水工程学报, 2015, 26(2):195-199. WANG R C, WANG C X. Research on interference and union problem of cable machine in concrete-dam pouring simulation[J]. Journal of Water Resources and Water Engineering, 2015, 26(2):195-199.(in Chinese)
[3] 徐建江,陈文夫,谭尧升,等.特高拱坝混凝土运输智能化关键技术与应用[J].清华大学学报(自然科学版), 2021, 61(7):768-776. XU J J, CHEN W F, TAN Y S, et al. Intelligent concrete transport methods for super-high arch dams[J]. Journal of Tsinghua University (Science and Technology), 2021, 61(7):768-776.(in Chinese)
[4] 钟登华,关涛,任炳昱.基于改进重抽样法的高拱坝施工进度仿真研究[J].水利学报, 2016, 47(4):473-482. ZHONG D H, GUAN T, REN B Y. Construction simulation for high arch dams based on modified Bootstrap[J]. Journal of Hydraulic Engineering, 2016, 47(4):473-482.(in Chinese)
[5] WANG D, WANG X L, REN B Y, et al. Vision-based productivity analysis of cable crane transportation using augmented reality-based synthetic image[J]. Journal of Computing in Civil Engineering, 2022, 36(1):04021030.
[6] WU H, YIN Y, WANG S J, et al. Optimizing GPS-guidance transit route for cable crane collision avoidance using artificial immune algorithm[J]. GPS Solutions, 2017, 21(2):823-834.
[7] WU H, TAO J, LI X P, et al. A location based service approach for collision warning systems in concrete dam construction[J]. Safety Science, 2013, 51(1):338-346.
[8] 赵春菊,资瑛,周宜红,等.缆机吊重海量监测数据的吊运混凝土模式识别[J].水力发电学报, 2021, 40(12):119-128. ZHAO C J, ZI Y, ZHOU Y H, et al. Pattern recognition of concrete lifting modes based on massive monitoring data of cable crane[J]. Journal of Hydroelectric Engineering, 2021, 40(12):119-128.(in Chinese)
[9] SON H, KIM C. Integrated worker detection and tracking for the safe operation of construction machinery[J]. Automation in Construction, 2021, 126:103670.
[10] XIAO B, LIN Q, CHEN Y. A vision-based method for automatic tracking of construction machines at nighttime based on deep learning illumination enhancement[J]. Automation in Construction, 2021, 127:103721.
[11] XIAO B, XIAO H R, WANG J W, et al. Vision-based method for tracking workers by integrating deep learning instance segmentation in off-site construction[J]. Automation in Construction, 2022, 136:104148.
[12] 华夏,王新晴,芮挺,等.视觉感知的无人机端到端目标跟踪控制技术[J].浙江大学学报(工学版), 2022, 56(7):1464-1472. HUA X, WANG X Q, RUI T, et al. Vision-driven end-to-end maneuvering object tracking of UAV[J]. Journal of Zhejiang University (Engineering Science), 2022, 56(7):1464-1472.(in Chinese)
[13] 刘创,梁军.基于注意力机制的车辆运动轨迹预测[J].浙江大学学报(工学版), 2020, 54(6):1156-1163. LIU C, LIANG J. Vehicle motion trajectory prediction based on attention mechanism[J]. Journal of Zhejiang University (Engineering Science), 2020, 54(6):1156-1163.(in Chinese)
[14] SUN P Z, CAO J K, JIANG Y, et al. DanceTrack:Multi-object tracking in uniform appearance and diverse motion[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA:IEEE, 2022:20961-20970.
[15] ZHANG Y, WANG T C, ZHANG X Y. MOTRv2:Bootstrapping end-to-end multi-object tracking by pretrained object detectors[C]//Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada:IEEE, 2023:22056-22065.
[16] WOJKE N, BEWLEY A, PAULUS D. Simple online and realtime tracking with a deep association metric[C]//Proceedings of 2017 IEEE International Conference on Image Processing. Beijing, China:IEEE, 2017:3645-3649.
[17] GUO Y Y, HONG W H, WU J X, et al. Vision-based cow tracking and feeding monitoring for autonomous livestock farming:The YOLOv5s-CA+DeepSORT-Vision transformer[J]. IEEE Robotics&Automation Magazine, 2023, 30(4):68-76.
[18] ZHANG C, TANG Z Z, ZHANG M, et al. Developing a more reliable aerial photography-based method for acquiring freeway traffic data[J]. Remote Sensing, 2022, 14(9):2202.
[19] SUN Z H, CHEN J, CHAO L, et al. A survey of multiple pedestrian tracking based on tracking-by-detection framework[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(5):1819-1833.
[20] GAO T Y, WUSHOUER M, TUERHONG G. Small object detection method based on improved YOLOv5[C]//Proceedings of 2022 International Conference on Virtual Reality, Human-Computer Interaction and Artificial Intelligence. Changsha, China:IEEE, 2022:144-149.
[21] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once:Unified, real-time object detection[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA:IEEE, 2016.
[22] YANG Y Z. Drone-view object detection based on the improved YOLOv5[C]//Proceedings of 2022 IEEE International Conference on Electrical Engineering, Big Data and Algorithms. Changchun, China:IEEE, 2022:612-617.
[23] 金立生,华强,郭柏苍,等.基于优化DeepSort的前方车辆多目标跟踪[J].浙江大学学报(工学版), 2021, 55(6):1056-1064. JIN L S, HUA Q, GUO B C, et al. Multi-target tracking of vehicles based on optimized DeepSort[J]. Journal of Zhejiang University (Engineering Science), 2021, 55(6):1056-1064.(in Chinese)
[24] CHEN R X, FERREIRA V G, LI X H. Detecting moving vehicles from satellite-based videos by tracklet feature classification[J]. Remote Sensing, 2023, 15(1):34.
[25] ZHANG Y F, SUN P Z, JIANG Y, et al. ByteTrack:Multi-object tracking by associating every detection box[C]//Proceedings of the 17th European Conference on Computer Vision. Tel Aviv, Israel:Springer, 2022:1-21.
[26] 陈爽,张悦,张勤俭,等.一种基于B样条插值的机器人速度规划算法[J].应用基础与工程科学学报, 2018, 26(03):661-671.CHEN S, ZHANG Y, ZHANG Q J, et al. A kind of robot speed planning algorithm based on B-spline interpolations[J]. Journal of Basic Science and Engineering, 2018, 26(03):661-671.(in Chinese)
[27] 万欣,刘锡明,苗积臣,等.改进的多层螺旋CT线性插值算法[J].清华大学学报(自然科学版), 2016, 56(12):1346-1351.WAN X, LIU X M, MIAO J C, et al. Improved linear interpolation algorithm for multi-slice helical CT. Journal of Tsinghua University (Science and Technology), 2016, 56(12):1346-1351.(in Chinese)
[28] CHEN Y, CAO R Y, CHEN J, et al. A practical approach to reconstruct high-quality Landsat NDVI time-series data by gap filling and the Savitzky-Golay filter[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 180:174-190.
