Objective: Nuclear medicine imaging is a dynamic imaging technique that enables researchers to analyze the physiological and pathological processes, especially in the brain. When imaging awake and unconstrained mice, the free movement of their heads can cause motion artifacts in the nuclear medicine images. These artifacts reduce image resolution, decrease the concentration of the tracer in the region of interest, and affect the quantification of the standard uptake value and the estimation of tracer kinetic model parameters. Therefore, the elimination of head motion artifacts is crucial for improving the quality of brain positron emission tomography (PET) images. In recent years, some researchers have been using markers attached to the mice's heads to track their movement. However, attaching markers to the mice's heads may cause discomfort and anxiety. In addition, the freedom of movement of the head during imaging can lead to relative sliding or detachment of the markers, resulting in incorrect motion estimation. Methods: In this study, we design a mouse head motion tracking system based on the you only live once (YOLO) v5 algorithm. This system can accurately monitor the position and posture of the mice's heads in real time, providing precise motion information for motion correction in nuclear medicine images. In contrast to traditional motion tracking systems, this system does not require markers attached to the mice's heads, effectively addressing the limitations of previous tracking methods. The proposed motion tracking system consists of three main stages, namely feature point recognition and positioning, three-dimensional reconstruction of feature points, and calculation of the rotation and translation parameters. First, the YOLO v5 algorithm automatically identifies and locates existing feature points on the mice's heads to obtain the pixel coordinates of each feature point. Then, using the parallax effect and triangulation principles, we reconstruct the three-dimensional coordinates of the feature points in the world coordinate system. Finally, we calculate the Euler angles of the mice's heads using the symmetry of the feature points and utilize inter-frame pose differential methods to compute the translation and rotation parameters of the head pose change between adjacent frames. Results: To verify the performance of the designed motion tracking system, we place a mouse phantom in a stationary position and measure the changes in its head position and posture angles using the designed system. The experimental results show that for the X, Y, and Z axes, the root-mean-square errors of the translational degrees of freedom are 0.04, 0.19, and 0.03 mm, whereas the root-mean-square errors of the rotational degrees of freedom are 0.58°, 0.34°, and 2.03°. We use the MATLAB function to obtain the histogram statistics of the detected translation and rotation parameters, all of which conform to a normal distribution. Conclusions: The results indicate that the designed motion tracking system can accurately monitor the movement of the mice's heads during nuclear medicine imaging. The detected parameters of six degrees of freedom conform to a normal distribution, further confirming the reliability of the system. Moreover, this system does not rely on markers, effectively avoiding the risk of marker detachment. The motion data obtained through this system can be used to compensate for and correct motion artifacts of the mice's heads in nuclear medicine imaging, thereby enhancing the quality of nuclear medicine images.