Objective: In recent years, pretrained models have achieved remarkable progress in tasks such as speech recognition and speaker identification, significantly enhancing performance across a wide range of downstream applications, particularly when trained on large-scale speech datasets. However, the performance of pretrained models depends heavily on the quality of pseudolabels, and obtaining large quantities of audio data with true annotations is far more costly than acquiring unlabeled data. Therefore, improving the quality of pseudolabels generated from large-scale unlabeled data is critical. To address this challenge, this paper proposes a progressive clustering strategy to optimize pseudolabels produced by unsupervised clustering algorithms. Methods: The progressive clustering process involved the following steps. First, HuBERT was trained on an unlabeled dataset, and the trained model was used to extract speaker embeddings from the audio data. Next, unsupervised clustering methods were applied to these embeddings to generate initial pseudolabels for each audio sample based on the clustering results. For each pseudolabel category, the mean of all speaker embeddings within that category was calculated as its centroid. Cosine similarity was then computed between the centroids of different categories. When the cosine similarity met a predefined threshold, the corresponding pseudolabel categories were merged to obtain a final, high-quality pseudolabel set. Results: Four algorithms, namely K-means, agglomerative hierarchical clustering, Infomap, and Leiden, were used to cluster the unlabeled Librispeech dataset to generate pseudolabels. After applying the progressive clustering strategy, the F₁-score, normalized mutual information, and cluster accuracy metrics of the pseudolabel set all improved, while the Coverage metric decreased. These results indicated that the quality of the pseudolabels improved significantly, demonstrating the positive effect of pseudolabel merging on clustering optimization. A slight decrease in purity was observed, which was attributed to the merging process inevitably combining some incorrectly assigned categories, thereby increasing intraclass complexity. This suggested that, in pursuit of higher category consistency and label accuracy, and in improving overall pseudolabel quality, some loss in cluster purity was unavoidable. Based on this strategy, a pretrained model incorporating joint speaker information modeling—speaker-related HuBERT (SR-HuBERT)—was trained. Experimental results showed that the model achieved a speaker identification accuracy of 85.95% and a speaker diarization error rate of 4.39% on speech processing universal PERformance benchmark speaker-related tasks, outperforming publicly available pretrained models. To further validate the generalization capability of the progressive clustering strategy, additional experiments were conducted on the VoxCeleb2 dataset, where pseudolabel quality also improved significantly after applying the strategy. The speaker representations extracted by SR-HuBERT were evaluated on VoxCeleb1-O, where the equal error rate metric decreased from 12.82% to 12.73%. These results indicate that the proposed strategy effectively enabled the model to learn more discriminative speaker representations. Conclusions: This study proposes a progressive clustering strategy for unsupervised clustering to address the issue of pseudolabel quality in unlabeled audio data. The experimental results demonstrate that this strategy significantly improved pseudolabel quality. More importantly, under the guidance of higher-quality pseudolabels, model performance is further enhanced.