Objective: In complex mountainous environments, unmanned aerial vehicle (UAV) coverage search tasks often encounter two core challenges: path redundancy and terrain obstructions. Although fixed-pattern search methods offer convenience and high efficiency in simple scenarios, they struggle to effectively avoid dead points and obstructures in complex terrains due to their rigid pre-planned trajectories. As a result, path repetition and reduced search efficiency become particularly prominent. To address the challenges of path redundancy and terrain obstructions in UAV coverage search tasks within complex mountainous environments, this study proposes a hybrid strategy that integrates traditional fixed-pattern search with an improved particle swarm optimization (PSO) algorithm. This strategy optimizes return path planning, minimizes path redundancy, and enhances adaptability in complex terrains. Methods: This research adopts a grid-based modeling approach to discretize complex terrains, constructing a simulation environment using real-world digital elevation model data from a specific area of Luding County, Sichuan Province, China. During data preprocessing, high-precision terrain data are converted into 3D surfaces via bi-linear interpolation, and threshold segmentation algorithms create binary representations of obstacle zones and passable areas. To address the challenge of dead points in fixed-pattern searches, this study introduces a hybrid backtracking mechanism that integrates queue-based and stack-based backtracking. When encountering dead points, an improved PSO algorithm with adaptive inertia weights is introduced to plan safe and efficient cross-regional paths. In the early iterations, the algorithm assigns larger inertia weights to enhance global exploration. Subsequently, these weights are reduced to refine local searches. In addition, path safety is ensured through various constraint functions, including mathematical models to avoid terrain blockages, maintain safe distances from obstacles, and ensure path continuity. Results: The experimental results indicate that the proposed hybrid strategy exhibits significant advantages in complex mountainous enviornments. This strategy, which combines queue-based backtracking and stack-based backtracking, reduces total path length by 0.66% and 21.1%, respectively. Path coverage gradually increases from initial levels to full coverage (100%), demonstrating robust performance across various terrain conditions. Notably, in highly complex environments, the improved PSO algorithm exhibits faster convergence speed and higher path-planning accuracy than the traditional PSO and the artificial bee colony algorithms. Comparative analysis reveals that stack-based backtracking performs better in complex terrains, whereas queue-based backtracking is more suitable for regions with greater local connectivity. Furthermore, this research is the first to demonstrate that the hybrid strategy can automatically adjust the number of backtrackings without prior information, ensuring flight safety while achieving optimal coverage. The overall optimization reaches 21.1%. Conclusions: This paper presents a hybrid-strategy-based UAV coverage search method for complex mountainous areas and validates its applicability and superiority across various terrain features through experiments. The findings reveal that the hybrid strategy maintains strong terrain adaptability while balancing efficiency and feasibility. In addition, the selection of backtracking methods directly influences the frequency of heuristic algorithm invocations and ultimately affects the quality of path planning. The successful application of the improved PSO algorithm demonstrates its potential for multi-objective optimization in complex environments, laying a foundation for further exploration of more intelligent and flexible UAV path planning technologies. This study holds significant implications for UAV applications in critical scenarios such as emergency rescue and disaster reconnaissance and provides new perspectives for autonomous UAV navigation.