Forensic Science International: Dana Michalski, Christopher Malec, Eden Clothier, Richard Bassed
Forensic Science International: Dana Michalski, Christopher Malec, Eden Clothier, Richard Bassed
A R T I C L E I N F O A B S T R A C T
Keywords: Mass disaster events can result in high levels of casualties that need to be identified. Whilst disaster victim
Post-mortem identification identification (DVI) relies on primary identifiers of DNA, fingerprints, and dental, these require ante-mortem data
Facial recognition that may not exist or be easily obtainable. Facial recognition technology may be able to assist. Automated facial
Disaster victim identification
recognition has advanced considerably and access to ante-mortem facial images are readily available. Facial
Biometrics
Human identification
recognition could therefore be used to expedite the DVI process by narrowing down leads before primary
identifiers are made available. This research explores the feasibility of using automated facial recognition
technology to support DVI. We evaluated the performance of a commercial-off-the-self facial recognition algo
rithm on post-mortem images (representing images taken after a mass disaster) against ante-mortem images
(representing a database that may exist within agencies who hold face databases for identity documents (such as
passports or driver’s licenses). We explored facial recognition performance for different operational scenarios,
with different levels of face image quality, and by cause of death. Our research is the largest facial recognition
evaluation of post-mortem and ante-mortem images to date. We demonstrated that facial recognition technology
would be valuable for DVI and that the performance varies by image quality and cause of death. We provide
recommendations for future research.
1. Introduction but involved a highly specialised process with limited expertise avail
able. For most countries, identification using forensic odontology
Rapid and accurate identification of the deceased after a mass involved collecting dental records from individual dental surgeries.
disaster is important for the prompt release of remains to families for However, in Scandinavia, dental records are collated in a centralised
grieving and closure [1,2]. Disaster victim identification (DVI) is the database. This resulted in many of the Scandinavian victims being
forensic process used to identify victims of a mass disaster event [3]. identified and repatriated more rapidly than those from other countries.
This process can be extremely lengthy and sometimes may not result in Of the more than 5000 people that died in this Tsunami, it took two
identification of the deceased [4]. Automated facial recognition has years to identify approximately 3600 of the victims using one or more of
improved drastically in recent years [5–7] and may expedite and these primary identifiers [4]. Whilst technology has improved since the
enhance the DVI process. 2004 Tsunami, these primary identifiers still have limitations in what
Currently, DNA, fingerprints, and odontology (dental), are accepted can be achieved given the lack of ante-mortem records that may be
as the primary identifiers for DVI purposes [3]. However, these identi available or the time taken to access them.
fiers are not infallible. For example, in the 2004 Southeast Asian As ante-mortem records for comparison can be limited, non-existent,
Tsunami, fingerprints proved problematic as the deceased were sub or difficult to obtain [8], alternative identification methods may prove
merged in salt water for long periods of time. This resulted in the valuable for providing leads and supporting the DVI process [9]. Facial
‘degloving’ of the epidermal layer of the skin from the hand, where the recognition has the potential to be one of these [10].
fingerprints would typically be captured. Many DNA samples obtained Commercial-off-the-shelf (COTS) facial recognition technology is being
from the deceased were not useable due to the hot and humid envi used by many agencies including for law enforcement and border con
ronmental conditions. Obtaining DNA from bone marrow was successful trol purposes to identify and verify individuals [11]. Large databases
* Corresponding authors.
E-mail addresses: [email protected] (D. Michalski), [email protected] (C. Malec), [email protected] (E. Clothier),
[email protected] (R. Bassed).
https://doi.org/10.1016/j.forsciint.2024.112108
Received 29 February 2024; Received in revised form 7 May 2024; Accepted 12 June 2024
Available online 13 June 2024
0379-0738/Crown Copyright © 2024 Published by Elsevier B.V. All rights are reserved, including those for text and data mining, AI training, and similar technologies.
D. Michalski et al. Forensic Science International 361 (2024) 112108
containing face images exist in government agencies that have been used The overarching aim of our research was to determine whether
for driver’s licences, passports, and mug shots. As demonstrated by current COTS facial recognition technology could be used as an addi
Scandinavia during the 2004 Tsunami, having the ability to search a tional tool to support DVI. Specifically, we explored:
large database of dental records resulted in faster identification and
repatriation of its citizens. Pre-existing face databases could potentially 1. How facial recognition performs for both open and closed set
be searched to narrow down possible matches before or while trying to scenarios
locate ante-mortem records of the primary identifiers. Ante-mortem face 2. How the quality of the image taken of a victim impacts facial
images could also be obtained from people trying to locate their family recognition performance
or loved ones. The quality and availability of ante-mortem images of 3. How cause of death impacts facial recognition performance
victims is often better than expected [12].
Post-mortem face images could be taken after a mass disaster event 2. Materials and methods
as part of the DVI workflow and images of the victims after some di
sasters have been collected to support visual identification [13]. 2.1. Stimuli
Post-mortem images already being taken may be used by a facial
recognition system to search ante-mortem images. However, the quality Post-mortem images from the Victorian Institute of Forensic Medi
of images submitted to a facial recognition system can affect its accuracy cine (Ethics project ID 1115) and ante-mortem images (via a project
[14]. Guidance is available to optimise image acquisition processes to agreement) were used to support this work. In total, 509,087 post-
maximise facial recognition accuracy, such as the Standard Guide for mortem images were provided from 63,567 identities; 134,611 of
Postmortem Image Capture (Facial Identification Scientific Working these images contained images of the subject’s face.
Group (FISWG) [15]). However, research is required to determine the The database of ante-mortem images used during this study con
impact of image quality on facial recognition performance with tained several million portrait quality images. Although reporting the
post-mortem images. Using facial recognition technology in this way total size of the ante-mortem database was not permissible, it was
would remove the need for people to search through many images of defined as ‘extremely large’ according to the definition by the National
victims, which has proven to be error prone and traumatic [13,16]. Institute of Standards and Technology (NIST) 2002 Facial Recognition
Depending on the nature of the mass disaster, facial recognition Vendor Test (FRVT) Evaluation Report [21] as it contained more than
could be used for either open set or closed set scenarios. An open set 3.3 million individuals and more than 10 million stored samples. The
scenario in a mass disaster context would be when an image of a victim is database had an age range of 0–112 years and contained 50.9 % females
searched against a database, and it is unknown whether a corresponding and 49.1 % males.
(mated) ante-mortem image exists in the database. For example, this The Procedure section details the images that were used for different
could be after an earthquake, taking an image of the unknown victim, components of the analysis.
and searching it against a database of face images used for identity
documents. A closed set in a mass disaster context would be when it is 2.2. Algorithm
known that the victim has an image in the ante-mortem database. While
this is uncommon in practice, one example of this would be a plane Cognitec’s FaceVACS-Engine SDK 9.7 (B14 algorithm; [22]) facial
crash. For an international flight, each passenger will be listed in a flight recognition algorithm was used to demonstrate the performance that
log and have an associated passport image. Exploring facial recognition may be achieved when considering a DVI scenario. The facial recogni
performance on open and closed set scenarios would also be valuable to tion algorithm was used with the default configuration.
inform a larger range of operational contexts.
As mass disaster events can result in death caused by various injuries, 2.3. Procedure
the nature of these injuries can have different effects on the body, and
often the face. For example, a bushfire can result in burned victims, All post-mortem images were manually assessed to determine if a
flooding can result in drowning, and an explosion can result in signifi face was visible in the image. This was to ensure that when the images
cant trauma. Understanding how different causes of death may impact were submitted to the facial recognition algorithm, any failure was due
facial recognition performance could also help to inform the contexts to the algorithm and not the absence of a face. Images that contained a
with which facial recognition systems can add value. face were rotated to the most upright orientation.
Limited research exists examining facial recognition for identifying We also down selected to just those post-mortem images that were
the deceased. Given the difficulty of access to post-mortem data, some front-on portrait style images, to replicate scenarios where this type of
have used creative approaches such as applying moulage to live par imagery could be taken after a mass disaster event specifically for the
ticipants to simulate traumatic injuries [17] or artificially altered facial purposes of facial recognition. Each face image was assessed by a
images of living people [18]. Others have used ‘body farms’ to explore researcher who determined whether the image was considered portrait
facial recognition over decomposition, but without access to quality. An image was considered portrait if the face was positioned
ante-mortem images for comparison [19]. Labati et al. [20] explored the front on and took up the majority of the photo and appeared to have the
identification of victims of the Mediterranean Sea crossing. Using purpose of capturing the subject’s face. Of the post-mortem images used
automated facial recognition techniques, the best performing algorithm in the study, 3858 post-mortem images were considered portrait quality.
they tested achieved an equal error rate of 13.21 %. Whilst Labati et al. Comparing the results from the larger post-mortem dataset to those from
[20] shows promise and is the largest study we found to use the portrait style dataset is valuable to determine if/how much post-
post-mortem images (N = 330); this research only explored drowned mortem image capture quality makes a difference to facial recognition
victims. Further research considering performance for other DVI con performance. All post-mortem images that contained a face were sub
texts, where impact to the face may vary based on the cause of death mitted to the facial recognition algorithm in order to create biometric
would also be extremely valuable. templates. Failure-to-extract (the proportion of images that were unable
Our research expands on the current literature by using real post- to produce a biometric template) results were recorded by cause of death
mortem images (across different causes of death), a large dataset of and whether the image was considered portrait quality. Images that
ante-mortem images (representing identity document facial image da were not able to successfully create a biometric template were excluded
tabases within agencies), and a COTS facial recognition algorithm (to from further analysis.
demonstrate performance that can be expected on a commercial system) Although images that fail to create a biometric template (failure-to-
and a process that may support an operational DVI workflow. extract) typically contribute to the overall facial recognition matching
2
D. Michalski et al. Forensic Science International 361 (2024) 112108
error rates [6], we chose to exclude these images and analyse the created which shows the probability that a mated image will be returned
failure-to-extract and matching performance separately as the at a given rank or higher for a mated search. This shows the ability of the
failure-to-extract rate may be reduced by using more effective image algorithm to correctly return mated images.
capture procedures or optimising settings within the facial recognition Our results were also analysed according to the cause of death
system. associated with each post-mortem image. The causes of death were
Images that had a corresponding ante-mortem image and were able grouped into broad categories outlined in Table 2.
to produce a biometric template were used to assess the facial recogni
tion matching performance in this study. Post-mortem images that did 3. Results
not have a corresponding ante-mortem image available were excluded
from facial recognition matching performance analysis as we were un 3.1. Failure-to-extract
able to confirm that an ante-mortem image did not exist within the ante-
mortem database (due to the inability to cross match some metadata Table 3 shows the proportion of images that were unable to be
information). In total, there were 7960 post-mortem images used during converted to biometric templates (failure-to-extract rate) by the Cogni
the facial recognition matching performance analysis (5182 male, 2769 tec facial recognition algorithm. Overall, 76 % of post-mortem images
female, 9 with no sex recorded). The ages ranged from 0 to 101. Table 1 failed to create a biometric template. This failure rate was reduced to
shows the number of post-mortem images by 10 year age ranges. Each 11 % when only portrait quality images were considered. The failure
post-mortem image had between one and five ante-mortem images rates ranged from 66 % (Drowning) to 84 % (Burned) when considering
presenting in the ante-mortem database, the age variation between the all images and 3 % (Hanging) to 39 % (Burned) for portrait images.
post-mortem images and the most recent ante-mortem image varied
from 0 to 19 years.
Post-mortem images were searched against the ante-mortem image 3.2. Open set matching
database containing several million images. This was to replicate a mass
disaster event occurring and post-mortem images being taken and used Fig. 1 shows the trade-off between successfully identifying an indi
to search a large database of facial images that may be available via vidual when a mated ante-mortem image is present in the database
agencies using identity documents (such as driver’s licenses or passport (TPIR) and incorrectly returning candidates when a mated ante-mortem
photos). Two separate searches were performed for each of the post- image is not present in the database (FPIR). The curves show the level of
mortem images, a mated search and non-mated search. The mated performance at all operating thresholds and compares returning either
search compared the post-mortem image against all ante-mortem im the top 1 or 100 images when using all images or only portrait style
ages in the database (including ante-mortem image(s) of the person in images.
the post-mortem image) and returned the highest ranking 200 images For example, a post-mortem portrait style image will on average
and their corresponding match scores. A separate non-mated search was return its corresponding mated ante-mortem image in the top 100 im
performed against the same database but without the mated images ages 53 % of the time when it is present in the database while also
present. The highest ranking 200 images and match scores were also incorrectly returning candidate images 40 % of the time when no mated
returned. The facial recognition matching performance of the post- ante-mortem image is available. When considering returning the top
mortem images was assessed considering both open and closed set ranking image, the algorithm was able to return a mated ante-mortem
scenarios. image 46 % of the time. This is represented by the FPIR = 0.4 (40 %)
The open set scenario was assessed by plotting a true positive iden dotted line in Fig. 1.
tification rate (TPIR) versus false positive identification rate (FPIR) Overall, our results indicate that performance improves when images
curve. This plot shows the trade-off between identifying an unknown are limited to portrait style images of a face rather than the broader
individual when they have a mated image present within the database dataset of all images.
(mated search) versus incorrectly returning non-mated images when a We then explored open set matching performance by cause of death.
mated image is not present within the database (non-mated search). A Fig. 2 depicts the results when considering the top 100 results using
true positive is recorded when at least one mated image is returned portrait images.
within the maximum candidate list length, in this case 1 or 100, and We found that the hanging condition had much higher matching
above a given threshold. A false positive is recorded when at least one performance than the other causes of death. While we would expect to
image is returned from a non-mated search. The TPIR vs FPIR curve see unrelated showing higher performance, many of these images con
shows the performance at all possible operating thresholds. The per tained obstructions such as medical breathing equipment which would
formance was assessed using all possible images and only portrait style have contributed to reduced overall performance.
images using a maximum candidate length of 1 and 100, this resulted in Fig. 3 shows the FNIR (FNIR = [1 –TPIR]) and the FPIR at all oper
4 curves. Images that failed to extract were excluded from the TPIR, ating thresholds. This plot shows the effect that cause of death has on
FPIR and false negative identification rate (FNIR) calculations. each type of error rate at all operating thresholds. The results show that
The closed set scenario was assessed by considering only the mated the change in performance is predominantly due to changes in mated
search results. A cumulative match characteristic (CMC) plot was
Table 2
Post-mortem images grouped by cause of death.
Table 1 All Portrait
Post-mortem images grouped by age.
Burned 20 12
All Portrait Drowning 136 68
0–10 121 48 Fall 965 488
11–20 363 172 Hanging 1471 625
21–30 937 441 Toxicity 992 506
31–40 1016 489 Trauma 1481 720
41–50 949 454 Unknown 335 166
51–60 1093 508 Unrelated 2560 1273
61–70 1126 545 Total 7960 3858
71–80 1033 534
Note. Unknown refers to when the cause of death was not known or provided.
81+ 1322 667
Unrelated refers to causes of death that likely did not impact the face
3
D. Michalski et al. Forensic Science International 361 (2024) 112108
4. Discussion
4
D. Michalski et al. Forensic Science International 361 (2024) 112108
Fig. 3. FPIR (left) and FNIR (right) at all operating thresholds by cause of death.
only one COTS facial recognition algorithm from Cognitec. The perfor
mance for other facial recognition algorithms may differ. We also
evaluated performance on an extremely large database replicating
identity document holdings. In some DVI scenarios, the ante-mortem
image dataset could be made up of images provided by victims’ fam
ilies. The facial recognition performance when using this type of dataset
may differ as it would be much smaller (which could result in higher
accuracy) but also may contain more uncontrolled ante-mortem images
(which could result in lower accuracy). This could also be explored in
future research.
5. Conclusion
Funding sources
This research did not receive any specific grant from funding
agencies in the public, commercial, or not-for-profit sectors.
5
D. Michalski et al. Forensic Science International 361 (2024) 112108
Declaration of Competing Interest Why and how are they used? Forensic Sci. Int. 345 (2023) 111615 https://doi.org/
10.1016/j.forsciint.2023.111615.
[10] L.S. Khoo, M.S. Mahmood, Application of facial recognition technology on
The authors declare that they have no known competing financial identification of the dead during large scale disasters, Forensic Sci. Int.: Synerg. 2
interests or personal relationships that could have appeared to influence (2020) 238–239, https://doi.org/10.1016/j.fsisyn.2020.07.001.
the work reported in this paper. [11] GAO], Facial recognition technology: current and planned uses by federal agencies,
〈https://www.gao.gov/assets/gao-21-526.pdf〉, 2021 (accessed 22 April 2024).
[12] A. Cappella, D. De Angelis, D. Mazzarelli, A. Vitale, G. Caccia, T. Fracasso,
Data Availability C. Cattaneo, Rediscovering the value of images in supporting personal
identification of missing migrants, Leg. Med. 54 (2022) 101985, https://doi.org/
10.1016/j.legalmed.2021.101985.
Due to the sensitive nature of the data, the data that was used is [13] N. Adamovic, L.M. Howes, R. White, R. Julian, Understanding the challenges of
confidential and not available. disaster victim identification: perspectives of Australian forensic practitioners,
Forensic Sci. Res. 8 (2) (2023) 107–115, https://doi.org/10.1093/fsr/owad020.
[14] M. Jacquet, C. Champod, Automated face recognition in forensic science: review
Acknowledgements and perspectives, Forensic Sci. Int. 307 (2020) 110124, https://doi.org/10.1016/j.
forsciint.2019.110124.
Thanks to Cognitec for access to the face recognition algorithm used [15] FISWG], Standard guide for postmortem facial image capture, 〈https://fiswg.
org/FISWG_Postmortem_Facial_Imaging_v2.0_20180910.pdf〉, 2018 (accessed 28
in this study. February 2024).
[16] O.W. Morgan, P. Sribanditmongkol, C. Perera, Y. Sulasmi, D.V. Alphen, E. Sondorp,
References Mass fatality management following the South Asian tsunami disaster: case studies
in Thailand, Indonesia, and Sri Lanka, PLOS Med. 3 (6) (2006) e195, https://doi.
org/10.1371/journal.pmed.0030195.
[1] R. Lain, C. Griffiths, J.M.N. Hilton, Forensic dental and medical response to the Bali
[17] J. Broach, R. Yong, M.-E. Manuell, C. Nichols, Use of facial recognition software to
bombing, Med. J. Aust. 179 (7) (2003) 362–365, https://doi.org/10.5694/j.1326-
identify disaster victims with facial injuries, Disaster Med. Public Health Prep. 11
5377.2003.tb05594.x.
(5) (2017) 568–572, https://doi.org/10.1017/dmp.2016.207.
[2] H. Soomer, H. Ranta, A. Penttilä, Identification of victims from the M/S Estonia,
[18] W.J. Gevaert, P.H.N. de With, Robust face recognition algorithm for identifition of
Int. J. Leg. Med. 114 (4) (2001) 259–262, https://doi.org/10.1007/
disaster victims. IS&T/ Electronic Imaging, 8655, SPIE, 2013. SPIE10.1117/
s004140000180.
12.2001634.
[3] Interpol, Disaster Victim Identification (DVI) guide. 〈https://www.interpol.int/en/
[19] D.C. Cornett, D.S. Bolme, D.W. Steadman, K.A. Sauerwein, T.B. Saul, Effects of
How-we-work/Forensics/Disaster-Victim-Identification-DVI〉, 2023 (accessed 28
postmortem decomposition on face recognition, 2019 IEEE 10th Int. Conf. Biom.
February 2024).
Theory, Appl. Syst. (BTAS) (2019), https://doi.org/10.1109/
[4] Biometrics Institute, Two decades of biometrics: a story of responsible use. 〈https
BTAS46853.2019.9185971.
://www.biometricsinstitute.org/about/20-year-anniversary-report/〉, 2021
[20] R.D. Labati, D. De Angelis, B. Bertoglio, C. Cattaneo, F. Scotti, V. Piuri, Automatic
(accessed 27 February 2024).
face recognition for forensic identification of persons deceased in humanitarian
[5] P. Grother, Face Recognition Vendor Test (FRVT) part 8: Summarizing
emergencies, 2021 IEEE Int. Conf. Comput. Intell. Virtual Environ. Meas. Syst.
Demographic Differentials, NIST Interagency/Internal Report (NISTIR), National
Appl. (CIVEMSA) (2021), https://doi.org/10.1109/
Institute of Standards and Technology, Gaithersburg, MD, 2022, https://doi.org/
CIVEMSA52099.2021.9493678.
10.6028/NIST.IR.8429.ipd.
[21] P.J. Phillips, P. Grother, R.J. Micheals, D.M. Blackburn, E. Tabassi, M. Bone, Face
[6] P. Grother, M. Ngan, K. Hanaoka, Face Recognition Vendor Test (FRVT) part 2:
Recognition Vendor Test (FRVT) 2002: Evaluation Report, NIST Interagency/
identification, NIST Interagency/Internal Report (NISTIR), National Institute of
Internal Report (NISTIR), National Institute of Standards and Technology,
Standards and Technology, Gaithersburg, MD, 2019, https://doi.org/10.6028/
Gaithersburg, MD, 2003. 〈https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir696
NIST.IR.8271.
5.pdf〉.
[7] P. Grother, M. Ngan, K. Hanaoka, Ongoing Face Recognition Vendor Test (FRVT)
[22] Cognitec, FaceVACS Technology. 〈https://www.cognitec.com/facevacs-technolo
part 2: identification, NIST Interagency/Internal Report (NISTIR), National
gy.html〉, 2024 (accessed 30 April 2024).
Institute of Standards and Technology, Gaithersburg, MD, 2018, https://doi.org/
[23] R. Shrestha, J. Acharya, A. Shakya, J. Acharya, Searching for kin amidst tragedy-
10.6028/NIST.IR.8238.
disaster victim identification operations for US Bangla Flight 211, J. Forensic Sci.
[8] Z. Caplova, Z. Obertova, D.M. Gibelli, D. De Angelis, D. Mazzarelli, C. Sforza,
64 (3) (2019) 824–827, https://doi.org/10.1111/1556-4029.13918.
C. Cattaneo, Personal identification of deceased persons: an overview of the current
[24] FISWG]. Guide for role-based training in facial comparison, 〈https://fiswg.org/fis
methods based on physical appearance, J. Forensic Sci. 63 (3) (2018) 662–671,
wg_guide_for_role-based_training_in_facial_comparison_v1.0_20200717.pdf〉, 2020
https://doi.org/10.1111/1556-4029.13643.
(accessed 28 February 2024).
[9] S. Blau, J. Roberts, E. Cunha, T. Delabarde, A.Z. Mundorff, H.H. de Boer, Re-
examining so-called ‘secondary identifiers’ in Disaster Victim Identification (DVI):