Objective: As a critical environmental perception sensor, millimeter-wave radar offers notable advantages such as privacy protection and immunity to lighting conditions, making it well-suited to complex environments. However, accurately detecting static targets, such as furniture and walls, remains a substantial challenge because static objects lack relative motion, causing their Doppler shifts to approach zero so that they are easily filtered out as noise. Furthermore, traditional radar scene reconstruction methods, such as radar Simultaneous Localization and Mapping (SLAM), rely heavily on static point cloud accumulation, which provides only rough geometric outlines and fails to extract semantic categories or fine-grained three-dimensional (3D) bounding boxes. Existing vision-based scene reconstruction methods depend largely on RGB-D data and remain constrained by privacy and lighting issues. Although recent studies have attempted to use human–object interaction for scene reasoning, applying these vision-driven methods directly to millimeter-wave radar is difficult. Radar point clouds are inherently sparse and accompanied by high-frequency jitter, and directly using traditional absolute global coordinate features introduces cascading errors. Therefore, exploring how to effectively use the dynamic human posture information captured by millimeter-wave radar to accurately infer static indoor scene layouts is critically important. Methods: To address the problems of sparse point clouds and unstable absolute pose features, this paper proposed a novel millimeter-wave radar-driven indoor scene reconstruction method that indirectly infers static scene layouts from human motion patterns. The proposed model consisted of three core modules: spatiotemporal feature extraction, a scene-aware voting mechanism, and Gaussian mixture decoding. Initially, human activity point clouds were captured, and a 3D skeletal posture sequence was extracted. In the feature extraction stage, a centroid displacement encoding scheme was designed. Using the relative displacement trajectory of the human centroid rather than absolute joint coordinates, the model captures the macroscopic dynamic features of human–object interactions while effectively filtering out local high-frequency noise. Subsequently, a spatial graph attention mechanism and one-dimensional temporal convolutions were encapsulated into a stacked spatiotemporal residual module to extract deep interactive representations. In the scene-aware voting phase, the human centroid served as a seed position, and a learnable offset function calculated the center votes for potential interactive objects; these votes were then clustered and weighted into stable voting clusters. Finally, considering the inherent uncertainty of predicting scenes from single-frame postures, a hybrid prediction module was introduced. It used a Gaussian mixture distribution to model the 3D bounding box parameters (center, size, and orientation), generating diverse and plausible scene hypotheses that are jointly optimized by classification and Huber regression losses. Results: Extensive experiments were conducted on a self-constructed real-world millimeter-wave radar dataset encompassing 6 indoor spatial layouts, 12 interacting object categories, and 20,000 human posture sequences. The quantitative results demonstrated that the proposed method achieved an overall mean average precision (mAP) of 50.81% in 3D scene reconstruction. This performance surpassed that of mainstream visual scene reconstruction models, achieving a 4.50 percentage point improvement over the best baseline. Notably, for highly challenging categories such as "sofa," the mAP reached 78.17%. Comprehensive ablation studies confirmed the necessity of each component: introducing the centroid displacement encoding raised the mAP from 8.66% to 33.89%, while integrating spatiotemporal encoding and the hybrid prediction module further improved the accuracy to the final 50.81%. Furthermore, cross-room generalization evaluations using minimal matching distance and total mutual diversity metrics under different data-splitting strategies (S1 and S2) showed that the proposed multimodal decoding effectively balances prediction accuracy and scene generation diversity. Qualitative visualization results further indicated that the generated 3D bounding boxes were highly consistent with real environments in terms of structural configuration and spatial rationality, with no severe unnatural penetrations. Conclusions: The proposed scene reconstruction method based on centroid displacement encoding reduces dependence on absolute posture coordinates and overcomes the instability caused by the sparsity and local jitter of millimeter-wave radar point clouds. By capturing macroscopic motion trends and employing a hybrid multimodal decoding mechanism, the method significantly improves both reconstruction accuracy and prediction stability for multiple typical human–object interaction targets. In addition, it demonstrates strong cross-scene generalization capabilities. Although the system relies on the accuracy of frontend human posture extraction and may be affected by severe multipath effects in complex metallic environments, it establishes an effective framework and offers a novel perspective for intelligent indoor environmental perception using millimeter-wave radar. Future work will focus on multisensor fusion and robust feature extraction under severe multipath conditions to further enhance the system's engineering applicability.