1. Department of Electronic Engineering, Tsinghua University, Beijing 100084, China 2. Department of Automation, University of Science and Technology of China, Hefei 230027, China
Research on 3-D face recognition has made great progress in recent years. 3-D face recognition is more effective than 2-D face recognition. Its main feature is the use of 3-D shape data for recognition. The 3-D face recognition methods are categorized into three types based on the source of the 3-D shape data with methods based on 2-D color images, high quality 3-D scanning data, and low quality RGB-D images. This study reviews these methods and discusses their advantages and disadvantages. This paper then reviews the use of deep learning methods for 3-D face recognition. Besides, 3-D and 2-D face data fusion methods are reviewed for bi-modal face recognition. The commonly used 3-D face databases are also summarized. Finally, the main difficulties and future development trends of 3-D face recognition are discussed.
罗常伟,於俊,于灵云,李亚利,王生进. 三维人脸识别研究进展综述[J]. 清华大学学报（自然科学版）, 2021, 61(1): 77-88.
Changwei LUO,Jun YU,Lingyun YU,Yali LI,Shengjin WANG. Overview of research progress on 3-D face recognition. Journal of Tsinghua University(Science and Technology), 2021, 61(1): 77-88.
WANG M, DENG W. Deep face recognition: A survey[Z/OL]. (2018-04-18)[2020-07-01]. https://arxiv.org/abs/1804.06655.
3
PATIL H , KOTHARI A , BHURCHANDI K . 3-D face recognition:Features, databases, algorithms and challenges[J]. Artificial Intelligence Review, 2015. 44 (3): 393- 441.
doi: 10.1007/s10462-015-9431-0
4
SOLTANPOUR S , BOUFAMA B , WU Q M . A survey of local feature methods for 3D face recognition[J]. Pattern Recognition, 2017. 72, 391- 406.
doi: 10.1016/j.patcog.2017.08.003
5
BLANZ V , VETTER T . Face recognition based on fitting a 3D morphable model[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003. 25 (9): 1063- 1074.
doi: 10.1109/TPAMI.2003.1227983
7
PAYSAN P, KNOTHE R, AMBERG B, et al. A 3D face model for pose and illumination invariant face recognition[C]//20096th IEEE International Conference on Advanced Video and Signal Based Surveillance. Genova, Italy: IEEE, 2009: 296-301.
8
LIU F , ZHAO Q J , LIU X M , et al. Joint face alignment and 3D face reconstruction with application to face recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. 42 (3): 664- 678.
doi: 10.1109/TPAMI.2018.2885995
9
JIANG D L , HU Y X , YAN S C , et al. Efficient 3D reconstruction for face recognition[J]. Pattern Recognition, 2005. 38 (6): 787- 798.
doi: 10.1016/j.patcog.2004.11.004
10
TANG H L , YIN B C , SUN Y F , et al. Pose-invariant face recognition based on a single view[J]. Journal of Information and Computational Science, 2010. 7 (12): 2369- 2376.
11
PRABHU U , HEO J , SAVVIDES M . Unconstrained pose-invariant face recognition using 3D generic elastic models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011. 33 (10): 1952- 1961.
doi: 10.1109/TPAMI.2011.123
12
ZHU X Y, LEI Z, YAN J J, et al. High-fidelity pose and expression normalization for face recognition in the wild[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE, 2015: 787-796.
13
HASSNER T, HAREL S, PAZ E, et al. Effective face frontalization in unconstrained images[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE, 2015: 4295-4304.
14
YIM J, JUNG H, YOO B, et al. Rotating your face using multi-task deep neural network[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE, 2015: 676-684.
15
KAN M N, SHAN S G, CHANG H, et al. Stacked progressive auto-encoders (SPAE) for face recognition across poses[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE, 2014: 1883-1890.
16
TANG C H, HSU G J, YAP M H. Face recognition with disentangled facial representation learning and data augmentation[C]//2019 IEEE International Conference on Image Processing. Taipei, China: IEEE, 2019: 1670-1674.
18
PAPATHEODOROU T, RUECKERT D. Evaluation of automatic 4D face recognition using surface and texture registration[C]//6th IEEE International Conference on Automatic Face and Gesture Recognition. Seoul, South Korea: IEEE, 2004: 321-326.
19
CHANG K J , BOWYER K W , FLYNN P J . Effects on facial expression in 3D face recognition[J]. Proceedings of Spie the International Society for Optical Engineering, 2005. 5779, 132- 143.
20
FALTEMIER T C , BOWYER K W , FLYNN P J . A region ensemble for 3-D face recognition[J]. IEEE Transactions on Information Forensics & Security, 2008. 3 (1): 62- 73.
21
MOHAMMADZADE H , HATZINAKOS D . Iterative closest normal point for 3D face recognition[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2013. 35 (2): 381- 397.
22
VIVEK E P , SUDHA N . Robust Hausdorff distance measure for face recognition[J]. Pattern Recognition, 2007. 40 (2): 431- 442.
doi: 10.1016/j.patcog.2006.04.019
23
YU Y , DA F P , GUO Y F . Sparse ICP with resampling and denoising for 3D face verification[J]. IEEE Transactions on Information Forensics and Security, 2019. 14 (7): 1917- 1927.
doi: 10.1109/TIFS.2018.2889255
24
QUEIROLO C C , SILVA L , BELLON O R P , et al. 3D face recognition using simulated annealing and the surface interpenetration measure[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2010. 32 (2): 206- 219.
25
AMBERG B, KNOTHE R, VETTER T. Expression invariant 3D face recognition with a morphable model[C]//20088th IEEE International Conference on Automatic Face & Gesture Recognition. Amsterdam, Netherlands: IEEE, 2008: 1-6.
26
HAAR T , VELTKAMP R C . Expression modeling for expression-invariant face recognition[J]. Computers & Graphics, 2010. 34 (3): 231- 241.
27
BOOTH J , ROUSSOS A , PONNIAH A , et al. Large scale 3D morphable models[J]. International Journal of Computer Vision, 2018. 2018 (126): 233- 254.
28
PAN G , WU Z H . 3D face recognition from range data[J]. International Journal of Image and Graphics, 2005. 5 (3): 573- 593.
doi: 10.1142/S0219467805001884
29
EFRATY B , BILGAZYEV E , SHAH S , et al. Profile-based 3D-aided face recognition[J]. Pattern Recognition, 2012. 45 (1): 43- 53.
30
LI Y , WANG Y , LIU J , et al. Expression-insensitive 3D face recognition by the fusion of multiple subject-specific curves[J]. Neurocomputing, 2018. 275, 1295- 1307.
doi: 10.1016/j.neucom.2017.09.070
31
SAMIR C , SRIVASTAVA A , DAOUDI M . Three-dimensional face recognition using shapes of facial curves[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2006. 28 (11): 1858- 1863.
32
SAMIR C , SRIVASTAVA A , DAOUDI M , et al. An intrinsic framework for analysis of facial surfaces[J]. International Journal of Computer Vision, 2009. 82 (1): 80- 95.
33
BERRETTI S , BIMBO A D , PALA P . 3D face recognition using isogeodesic stripes[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2010. 32 (12): 2162- 2177.
34
DRIRA H , AMOR B B , SRIVASTAVA A , et al. 3D face recognition under expressions, occlusions, and pose variations[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2013. 35 (9): 2270- 2283.
35
LEI Y J , BENNAMOUN M , HAYAT M , et al. An efficient 3D face recognition approach using local geometrical signatures[J]. Pattern Recognition, 2014. 47 (2): 509- 524.
doi: 10.1016/j.patcog.2013.07.018
36
CHANG K T , BOWYER K W , FLYNN P J . An evaluation of multimodal 2D+3D face biometrics[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2005. 27 (4): 619- 624.
37
PASSALIS G , PERAKIS P , THEOHARIS T , et al. Using facial symmetry to handle pose variations in real-world 3D face recognition[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2011. 33 (10): 1938- 1951.
38
LIU P J , WANG Y H , HUANG D , et al. Learning the spherical harmonic features for 3-D face recognition[J]. IEEE Transactions on Image Processing, 2013. 22 (3): 914- 925.
doi: 10.1109/TIP.2012.2222897
39
GUPTA S, AGGARWAL K, MARKEY K, et al. 3D face recognition founded on the structural diversity of human faces[C]//2007 IEEE Conference on Computer Vision & Pattern Recognition. Minneapolis, USA: IEEE, 2007: 1-7.
40
PERAKIS P , PASSALIS G , THEOHARIS T , et al. 3D facial landmark detection under large yaw and expression variations[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2013. 35 (7): 1552- 1564.
41
EMAMBAKHSH M , EVANS A . Nasal patches and curves for expression-robust 3D face recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. 39 (5): 995- 1007.
doi: 10.1109/TPAMI.2016.2565473
42
BERRETTI S, DEL A, PALA P. 3D partial face matching using local shape descriptors[C]//Joint ACM Workshop on Human Gesture and Behavior Understanding. New York, USA: ACM, 2011: 65-71.
43
INAN T , HALICI U . 3-D face recognition with local shape descriptors[J]. IEEE Transactions on Information Forensics & Security, 2012. 7 (2): 577- 587.
44
SMEETS D , KEUSTERMANS J , VANDERMEULEN D , et al. meshSIFT:Local surface features for 3D face recognition under expression variations and partial data[J]. Computer Vision and Image Understanding, 2013. 117 (2): 158- 169.
doi: 10.1016/j.cviu.2012.10.002
45
LI H B , HUANG D , MORVAN J M , et al. Towards 3D face recognition in the real:A registration-free approach using fine-grained matching of 3D keypoint descriptors[J]. International Journal of Computer Vision, 2015. 113 (2): 128- 142.
doi: 10.1007/s11263-014-0785-6
46
ELAIWAT S , BENNAMOUN M , BOUSSAID F , et al. A curvelet-based approach for textured 3D face recognition[J]. Pattern Recognition, 2015. 48 (4): 1235- 1246.
doi: 10.1016/j.patcog.2014.10.013
47
MIAN S , OWENS M B R . Keypoint detection and local feature matching for textured 3D face recognition[J]. International Journal of Computer Vision, 2008. 79 (1): 1- 12.
48
LEI Y J , GUO Y L , HAYAT M , et al. A two-phase weighted collaborative representation for 3D partial face recognition with single sample[J]. Pattern Recognition, 2016. 52, 218- 237.
doi: 10.1016/j.patcog.2015.09.035
50
HUANG D , ARDABILIAN M , WANG Y , et al. 3-D face recognition using eLBP-based facial description and local feature hybrid matching[J]. IEEE Transactions on Information Forensics & Security, 2012. 7 (5): 1551- 1565.
51
TANG H L , YIN B C , SUN Y F , et al. 3D face recognition using local binary patterns[J]. Signal Processing, 2013. 93 (8): 2190- 2198.
doi: 10.1016/j.sigpro.2012.04.002
52
WERGHI N , BERRETTI S , DEL BIMBOQ A . The mesh-LBP:A framework for extracting local binary patterns from discrete manifolds[J]. IEEE Transactions on Image Processing, 2015. 24 (1): 220- 235.
53
SOLTANPOUR S , WU Q M J . Weighted extreme sparse classifier and local derivative pattern for 3D face recognition[J]. IEEE Transactions on Image Processing, 2019. 28 (6): 3020- 3033.
doi: 10.1109/TIP.2019.2893524
54
WANG Y M , LIU J Z , TANG X O . Robust 3D face recognition by local shape difference boosting[J]. IEEE Transations on Pattern Analysis and Machine Intelligence, 2010. 32 (10): 1858- 1870.
doi: 10.1109/TPAMI.2009.200
55
HARIRI W , TABIA H , FARAH N , et al. 3D face recognition using covariance based descriptors[J]. Pattern Recognition Letters, 2016. 78, 1- 7.
doi: 10.1016/j.patrec.2016.03.028
56
KIM D, HERNANDEZ M, CHOI J, et al. Deep 3D face identification[C]//2017 IEEE International Joint Conference on Biometrics (IJCB). Denver, USA: IEEE, 2017: 133-142.
57
LI H B, SUN J, CHEN L M. Location-sensitive sparse representation of deep normal patterns for expression-robust 3D face recognition[C]//2017 IEEE International Joint Conference on Biometrics (IJCB). Denver, USA: IEEE, 2017: 234-242.
58
PARKHI O M, VEDALDI A, ZISSERMAN A. Deep face recognition[C]//Proceedings of the British Machine Vision Conference. Swansea, UK: BMVA Press, 2015: 41.1-41.12.
59
GILANI S Z, MIAN A. Learning from millions of 3D scans for large-scale 3D face recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake city, USA: IEEE, 2018: 1896-1905.
60
CAI Y , LEI Y J , YANG M L , et al. A fast and robust 3D face recognition approach based on deeply learned face representation[J]. Neurocomputing, 2019. 363 (21): 375- 397.
61
HSU G S , LIU Y L , PENG H C , et al. RGB-D-based face reconstruction and recognition[J]. IEEE Transactions on Information Forensics and Security, 2014. 9 (12): 2110- 2118.
doi: 10.1109/TIFS.2014.2361028
62
GOSWAMI G , VATSA M , SINGH R . RGB-D face recognition with texture and attribute features[J]. IEEE Transations on Information Forensics and Security, 2014. 9 (10): 1629- 1640.
doi: 10.1109/TIFS.2014.2343913
63
MIN R , KOSE N , DUGELAY J . KinectFaceDB:A Kinect database for face recognition[J]. IEEE Transactions on SMC:Systems, 2014. 44 (11): 1534- 1548.
64
XU X X , LI W , XU D . Distance metric learning using privileged information for face verification and person reidentification[J]. IEEE Transactions on NNLS, 2015. 26 (12): 3150- 3162.
65
LI B Y L , XUE M L , MIAN A S , et al. Robust RGB-D face recognition using kinect sensor[J]. Neurocomputing, 2016. 214, 93- 108.
doi: 10.1016/j.neucom.2016.06.012
66
LEE Y, CHEN J C, TSENG C, et al. Accurate and robust face recognition from RGB-D images with a deep learning approach[C]//Proceedings of the British Machine Vision Conference. York, UK.: IEEE, 2016: 1-14.
67
ZHANG H, HAN H, CUI J Y, et al. RGB-D face recognition via deep complementary and common feature learning[C]//IEEE International Conference on Automatic Face & Gesture Recognition. Xi'an, China: IEEE, 2018: 8-15.
68
JIANG L , ZHANG J Y , DENG B L . Robust RGB-D face recognition using attribute-aware loss[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
doi: 10.1109/TPAMI.2019.2919284
69
LUO C W , ZHANG J Y , YU J , et al. Real-time head pose estimation and face modeling from a depth image[J]. IEEE Transactions on Multimedia, 2019. 21 (10): 2473- 2481.
doi: 10.1109/TMM.2019.2903724
70
MEYER G P, DO M N. Real-time 3D face verification with a consumer depth camera[C]//201815th Conference on Computer &Robot Vision. Toronto, Canada: IEEE, 2018: 71-79.
71
KIM D, CHOI J, LEKSUT J T, et al. Accurate 3D face modeling and recognition from RGB-D stream in the presence of large pose changes[C]//2016 IEEE International Conference on Image Processing. Phoenix, USA: IEEE, 2016: 3011-3015.
72
MU G D, HUANG D, HU G S, et al. Led3D: A lightweight and efficient deep approach to recognizing low-quality 3D faces[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019: 5766-5775.
73
LI H B , SUN J , XU Z B , et al. Multimodal 2D+3D facial expression recognition with deep fusion convolutional neural network[J]. IEEE Transactions on Multimedia, 2017. 19 (12): 2816- 2831.
doi: 10.1109/TMM.2017.2713408
74
SU H, MAJI S, KALOGERAKIS E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]//2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 945-953.
75
SHI B G , BAI S , ZHOU Z C , et al. DeepPano:A deep panoramic representation for 3-D shape recognition[J]. IEEE Signal Processing Letters, 2015. 22 (12): 2339- 2343.
doi: 10.1109/LSP.2015.2480802
76
WU Z R, SONG S R, KHOSLA A, et al. 3D ShapeNets: A deep representation for volumetric shapes[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE, 2015: 1912-1920.
77
QI C R, SU H, NIESSNER M, et al. Volumetric and multi-view CNNs for object classification on 3D data[C]//IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 5648-5656.
78
HAN Z Z , LIU Z B , HAN J W , et al. Unsupervised 3D local feature learning by circle convolutional restricted boltzmann machine[J]. IEEE Transactions on Image Processing, 2016. 25 (11): 5331- 5344.
doi: 10.1109/TIP.2016.2605920
79
QI C, SU H, MO K C, et al. PointNet: Deep learning on point sets for 3D classification and segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017: 77-85.
80
BENABDELKADER C , GRIFFIN P A . Comparing and combining depth and texture cues for face recognition[J]. Image & Vision Computing, 2005. 23 (3): 339- 352.
81
KUSUMA G P , CHUA C S . PCA-based image recombination for multimodal 2D+3D face recognition[J]. Image and Vision Computing, 2011. 29 (5): 306- 316.
doi: 10.1016/j.imavis.2010.12.003
82
XU C H , LI S , TAN T N , et al. Automatic 3D face recognition from depth and intensity Gabor features[J]. Pattern Recognition, 2009. 42 (9): 1895- 1905.
doi: 10.1016/j.patcog.2009.01.001
83
CUI J Y, HAN H, SHAN S G, et al. RGB-D face recognition: A comparative study of representative fusion schemes[C]//Chinese Conference on Biometric Recognition. Urumqi, China: IEEE, 2018: 358-366.
84
PHILLIPS P J, FLYNN P J, SCRUGGS T, et al. Overview of the face recognition grand challenge[C]//2005 IEEE Conference on Computer Vision and Pattern Recognition. San Diego, USA: IEEE, 2005: 947-954.
85
XU C H, TAN T N, LI S, et al. Learning effective intrinsic features to boost 3D-based face recognition[C]//ECCV 2006, Lecture Notes in Computer Science. Graz, Austria: Springer-Verlag, 2006: 416-427.
86
YIN L J, WEI X Z, SUN Y, et al. A 3D facial expression database for facial behavior research[C]//7th International Conference on Automatic Face and Gesture Recognition. Southampton, UK: IEEE, 2006: 211-216.
87
SAVRAN A, NESE A, HAMDI D, et al. Bosphorus database for 3D face analysis[C]//Biometrics and Identity Management, First European Workshop. Roskilde, Denmark: IEEE, 2008: 47-56.
88
COLOMBO A, CUSANO C, SCHETTINI R. UMB-DB: A database of partially occluded 3D faces[C]//2011 IEEE International Conference on Computer Vision Workshops. Barcelona, Spain: IEEE, 2011: 2113-2119.
89
ZHANG J J, HUANG D, WANG Y H, et al. Lock3DFace: A large-scale database of low-cost Kinect 3D faces[C]//International Conference on Biometrics (ICB). Halmstad, Sweden: IEEE, 2016: 1-8.
2021, Vol. 61 
)
