Planning, design and logistics of a decision analysis study: The FBI/Ames study involving forensic firearms examiners

Monson, Keith L.; Smith, Erich D.; Bajic, Stanley J.

doi:10.1016/j.fsisyn.2022.100221

Title: Planning, design and logistics of a decision analysis study: The FBI/Ames study involving forensic firearms examiners

Journal Article · 18 February 2022 · Forensic Science International. Synergy

DOI: https://doi.org/10.1016/j.fsisyn.2022.100221 · OSTI ID:1846429

^[1]; Smith, Erich D. ^[1]; Bajic, Stanley J. ^[2]

FBI Laboratory, Quantico, VA (United States)
Ames Lab., Ames, IA (United States)

This paper describes design and logistical aspects of a decision analysis study to assess the performance of qualified firearms examiners working in accredited laboratories in the United States in terms of accuracy (error rate), repeatability, and reproducibility of decisions involving comparisons of fired bullets and cartridge cases. The purpose of the study was to validate current practice of the forensic discipline of firearms/toolmarks (F/T) examination. It elicited error rate data by counting the number of false positive and false negative conclusions. Preceded by the experimental design, decisions, and logistics described herein, testing was ultimately administered 173 qualified, practicing F/T examiners in public and private crime laboratories. The first round of testing evaluated accuracy, while two subsequent rounds evaluated repeatability and reproducibility of examiner conclusions. This project expands on previous studies by involving many F/T examiners in challenging comparisons and by executing the study in the recommended double-blind format.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Ames Laboratory (AMES), Ames, IA (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC02-07CH11358

OSTI ID:: 1846429

Report Number(s):: IS-J-10,730

Journal Information:: Forensic Science International. Synergy, Journal Name: Forensic Science International. Synergy Vol. 4; ISSN 2589-871X

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (49)

Ballistic fingerprinting databases De Kinder, J. Science & Justice, Vol. 42, Issue 4 https://doi.org/10.1016/S1355-0306(02)71829-7	journal	October 2002
The influence of manufacturing processes on the identification of bullets and cartridge cases – a review of the literature Bonfanti, M. S.; De Kinder, J. Science & Justice, Vol. 39, Issue 1 https://doi.org/10.1016/S1355-0306(99)72008-3	journal	January 1999
Does suggestive information cause a confirmation bias in bullet comparisons? Kerstholt, Jose; Eikelboom, Aletta; Dijkman, Tjisse Forensic Science International, Vol. 198, Issue 1-3 https://doi.org/10.1016/j.forsciint.2010.02.007	journal	May 2010
Informing the judgments of fingerprint analysts using quality metric and statistical assessment tools Langenburg, Glenn; Champod, Christophe; Genessay, Thibault Forensic Science International, Vol. 219, Issue 1-3 https://doi.org/10.1016/j.forsciint.2011.12.017	journal	June 2012
The decisionalization of individualization Biedermann, A.; Bozza, S.; Taroni, F. Forensic Science International, Vol. 266 https://doi.org/10.1016/j.forsciint.2016.04.029	journal	September 2016
Determining the number of test fires needed to represent the variability present within 9 mm Luger firearms Law, Eric F.; Morris, Keith B.; Jelsema, Casey M. Forensic Science International, Vol. 276 https://doi.org/10.1016/j.forsciint.2017.04.019	journal	July 2017
Individuality testing of new Glock pistol barrels “Marksman Barrel” Christen, Stephan; Jordi, Hans Rudolf Forensic Science International, Vol. 295 https://doi.org/10.1016/j.forsciint.2018.11.028	journal	February 2019
Validity and reliability of forensic firearm examiners Mattijssen, Erwin J. A. T.; Witteman, Cilia L. M.; Berger, Charles E. H. Forensic Science International, Vol. 307 https://doi.org/10.1016/j.forsciint.2019.110112	journal	February 2020
Testing the accuracy and reliability of palmar friction ridge comparisons – A black box study Eldridge, Heidi; De Donno, Marco; Champod, Christophe Forensic Science International, Vol. 318 https://doi.org/10.1016/j.forsciint.2020.110457	journal	January 2021
(Mis)use of scientific measurements in forensic science Dror, Itiel E.; Scurich, Nicholas Forensic Science International: Synergy, Vol. 2 https://doi.org/10.1016/j.fsisyn.2020.08.006	journal	January 2020
Commentary on: I. Dror, N Scurich “(Mis)use of scientific measurements in forensic science” Forensic Science International: Synergy 2020 https://doi.org/10.1016/j.fsisyn.2020.08.006 Weller, Todd J.; Morris, Max D. Forensic Science International: Synergy, Vol. 2 https://doi.org/10.1016/j.fsisyn.2020.10.004	journal	January 2020
Forensic science and the principle of excluded middle: “Inconclusive” decisions and the structure of error rate studies Biedermann, Alex; Kotsoglou, Kyriakos N. Forensic Science International: Synergy, Vol. 3 https://doi.org/10.1016/j.fsisyn.2021.100147	journal	January 2021
Measures of accuracy and performance of diagnostic tests Drobatz, Kenneth J. Journal of Veterinary Cardiology, Vol. 11 https://doi.org/10.1016/j.jvc.2009.03.004	journal	May 2009
A survey of image processing techniques and statistics for ballistic specimens in forensic science Gerules, George; Bhatia, Sanjiv K.; Jackson, Daniel E. Science & Justice, Vol. 53, Issue 2 https://doi.org/10.1016/j.scijus.2012.07.002	journal	June 2013
Application of likelihood ratios for firearm and toolmark analysis Bunch, Stephen; Wevers, Gerhard Science & Justice, Vol. 53, Issue 2 https://doi.org/10.1016/j.scijus.2012.12.005	journal	June 2013
Design and results of an exploratory double blind testing program in firearms examination Kerkhoff, W.; Stoel, R. D.; Berger, C. E. H. Science & Justice, Vol. 55, Issue 6 https://doi.org/10.1016/j.scijus.2015.06.007	journal	December 2015
Preventing miscarriages of justice: A review of forensic firearm identification Bolton-King, Rachel S. Science & Justice, Vol. 56, Issue 2 https://doi.org/10.1016/j.scijus.2015.11.002	journal	March 2016
A part-declared blind testing program in firearms examination Kerkhoff, W.; Stoel, R. D.; Mattijssen, E. J. A. T. Science & Justice, Vol. 58, Issue 4 https://doi.org/10.1016/j.scijus.2018.03.006	journal	July 2018
Cognitive biases in the peer review of bullet and cartridge case comparison casework: A field study Mattijssen, Erwin J. A. T.; Witteman, Cilia L. M.; Berger, Charles E. H. Science & Justice, Vol. 60, Issue 4 https://doi.org/10.1016/j.scijus.2020.01.005	journal	July 2020
Accuracy and reliability of forensic latent fingerprint decisions Ulery, B. T.; Hicklin, R. A.; Buscaglia, J. Proceedings of the National Academy of Sciences, Vol. 108, Issue 19 https://doi.org/10.1073/pnas.1018707108	journal	April 2011
A call for more science in forensic science Bell, Suzanne; Sah, Sunita; Albright, Thomas D. Proceedings of the National Academy of Sciences, Vol. 115, Issue 18 https://doi.org/10.1073/pnas.1712161115	journal	April 2018
The examination, evaluation, and identification of 40 S&W calibre cartridge cases fired from 1079 different GLOCK semiautomatic pistols manufactured over a six-year period Castro, Carl E.; Norris, Steven A.; Setume, Bonchose K. Canadian Society of Forensic Science Journal, Vol. 47, Issue 2 https://doi.org/10.1080/00085030.2014.904105	journal	April 2014
Treatment of inconclusives in the AFTE range of conclusions Hofmann, Heike; Carriquiry, Alicia; Vanderplas, Susan Law, Probability and Risk, Vol. 19, Issue 3-4 https://doi.org/10.1093/lpr/mgab002	journal	December 2020
Analysis of experiments in forensic firearms/toolmarks practice offered as support for low rates of practice error and claims of inferential certainty Spiegelman, C.; Tobin, W. A. Law, Probability and Risk, Vol. 12, Issue 2 https://doi.org/10.1093/lpr/mgs028	journal	October 2012
Evaluation of GLOCK 9 mm Firing Pin Aperture Shear Mark Individuality Based On 1,632 Different Pistols by Traditional Pattern Matching and IBIS Pattern Recognition Hamby, James E.; Norris, Stephen; Petraco, Nicholas D. K. Journal of Forensic Sciences, Vol. 61, Issue 1 https://doi.org/10.1111/1556-4029.12940	journal	September 2015
A Validation Study of Bullet and Cartridge Case Comparisons Using Samples Representative of Actual Casework Smith, Tasha P.; Andrew Smith, G.; Snipes, Jeffrey B. Journal of Forensic Sciences, Vol. 61, Issue 4 https://doi.org/10.1111/1556-4029.13093	journal	May 2016
Objective Evaluation of Subclass Characteristics on Breech Face Marks Riva, Fabiano; Hermsen, Rob; Mattijssen, Erwin Journal of Forensic Sciences, Vol. 62, Issue 2 https://doi.org/10.1111/1556-4029.13274	journal	December 2016
“Cannot Decide”: The Fine Line Between Appropriate Inconclusive Determinations Versus Unjustifiably Deciding Not To Decide Dror, Itiel E.; Langenburg, Glenn Journal of Forensic Sciences, Vol. 64, Issue 1 https://doi.org/10.1111/1556-4029.13854	journal	May 2018
A Worldwide Study of Bullets Fired From 10 Consecutively Rifled 9 MM RUGER Pistol Barrels—Analysis of Examiner Error Rate Hamby, James E.; Brundage, David J.; Petraco, Nicholas D. K. Journal of Forensic Sciences, Vol. 64, Issue 2 https://doi.org/10.1111/1556-4029.13916	journal	September 2018
The Utility of Expanded Conclusion Scales During Latent Print Examinations Carter, Kelly E.; Vogelsang, Macgregor D.; Vanderkolk, John Journal of Forensic Sciences, Vol. 65, Issue 4 https://doi.org/10.1111/1556-4029.14298	journal	March 2020
The Error in “Error Rate”: Why Error Rates Are So Needed, Yet So Elusive Dror, Itiel E. Journal of Forensic Sciences, Vol. 65, Issue 4 https://doi.org/10.1111/1556-4029.14435	journal	April 2020
Three‐Dimensional Analysis of Cartridge Case Double‐Casts Law, Eric F.; Morris, Keith B. Journal of Forensic Sciences, Vol. 65, Issue 6 https://doi.org/10.1111/1556-4029.14549	journal	September 2020
Firearm examination: Examiner judgments and computer‐based comparisons Mattijssen, Erwin J. A. T.; Witteman, Cilia L. M.; Berger, Charles E. H. Journal of Forensic Sciences, Vol. 66, Issue 1 https://doi.org/10.1111/1556-4029.14557	journal	September 2020
Fingerprint error rate on close non‐matches Koehler, Jonathan J.; Liu, Shiquan Journal of Forensic Sciences, Vol. 66, Issue 1 https://doi.org/10.1111/1556-4029.14580	journal	September 2020
Beretta barrel fired bullet validation study Smith, Jaimie A. Journal of Forensic Sciences, Vol. 66, Issue 2 https://doi.org/10.1111/1556-4029.14604	journal	October 2020
Evaluating firearm examiner conclusion variability using cartridge case reproductions Law, Eric F.; Morris, Keith B. Journal of Forensic Sciences, Vol. 66, Issue 5 https://doi.org/10.1111/1556-4029.14758	journal	May 2021
Reproducibility and persistence of individual marks on 3000 fired bullets from five Chinese Norinco QSZ‐92 9 × 19 mm pistols, and the search performance of Evofinder ^® to a 1000 firearm database of the same model firearm Wei, Zhongbo; Luo, Yaping; Zhou, Peng Journal of Forensic Sciences, Vol. 66, Issue 6 https://doi.org/10.1111/1556-4029.14794	journal	July 2021
Statistical Issues in Assessing Forensic Evidence: Statistical Issues in Assessing Forensic Evidence Kafadar, Karen International Statistical Review, Vol. 83, Issue 1 https://doi.org/10.1111/insr.12069	journal	October 2014
Defending the Scientific Foundations of the Firearms and Tool Mark Identification Discipline: Responding to Recent Challenges Nichols, Ronald G. Journal of Forensic Sciences, Vol. 52, Issue 3 https://doi.org/10.1111/j.1556-4029.2007.00422.x	journal	May 2007
A Perspective on Errors, Bias, and Interpretation in the Forensic Sciences and Direction for Continuing Advancement Budowle, Bruce; Bottrell, Maureen C.; Bunch, Stephen G. Journal of Forensic Sciences, Vol. 54, Issue 4 https://doi.org/10.1111/j.1556-4029.2009.01081.x	journal	July 2009
An Investigation into Whether or Not the Class and Individual Characteristics of Five Turkish Manufactured Pistols Change During Extensive Firing Sarıbey, Aylin Yalçın; Hannam, Abigail Grace; Tarımcı, Çelik Journal of Forensic Sciences, Vol. 54, Issue 5 https://doi.org/10.1111/j.1556-4029.2009.01107.x	journal	September 2009
Confocal Microscopy Analysis of Breech Face Marks on Fired Cartridge Cases from 10 Consecutively Manufactured Pistol Slides*: CONFOCAL MICROSCOPY ANALYSIS OF BREECH FACE MARKS Weller, Todd J.; Zheng, Alan; Thompson, Robert Journal of Forensic Sciences, Vol. 57, Issue 4 https://doi.org/10.1111/j.1556-4029.2012.02072.x	journal	March 2012
Reporting attrition in randomised controlled trials Dumville, Jo C.; Torgerson, David J.; Hewitt, Catherine E. BMJ, Vol. 332, Issue 7547 https://doi.org/10.1136/bmj.332.7547.969	journal	April 2006
Using the Beta-Binomial Distribution to Assess Performance of a Biometric Identification Device Schuckers, Michael E. International Journal of Image and Graphics, Vol. 03, Issue 03 https://doi.org/10.1142/S0219467803001147	journal	July 2003
Methodological Issues in the Effects of Attrition: Simple Solutions for Social Scientists Ahern, Kathy; Le Brocque, Robyne Field Methods, Vol. 17, Issue 1 https://doi.org/10.1177/1525822X04271006	journal	February 2005
Who knew? The misleading specificity of “double-blind” and what to do about it Lang, Thomas A.; Stroup, Donna F. Trials, Vol. 21, Issue 1 https://doi.org/10.1186/s13063-020-04607-5	journal	August 2020
Repeatability and Reproducibility of Decisions by Latent Fingerprint Examiners Ulery, Bradford T.; Hicklin, R. Austin; Buscaglia, JoAnn PLoS ONE, Vol. 7, Issue 3 https://doi.org/10.1371/journal.pone.0032800	journal	March 2012
Are Inconclusive Decisions in Forensic Science as Deficient as They Are Said to Be? Biedermann, Alex; Bozza, Silvia; Taroni, Franco Frontiers in Psychology, Vol. 10 https://doi.org/10.3389/fpsyg.2019.00520	journal	March 2019
Specificity of characteristic marks on cartridge cases from 3070 consecutive firings of a Chinese Norinco QSZ-92 9 mm Pistol Dong, Feng; Zhao, Yabin; Luo, Yaping Journal of Forensic Science and Medicine, Vol. 5, Issue 2 https://doi.org/10.4103/jfsm.jfsm_6_19	journal	January 2019

Similar Records

A study of examiner accuracy in cartridge case comparisons. Part 1: Examiner error rates

Journal Article · 2023 · Forensic Science International · OSTI ID:1987636

Baldwin, David P.; Bajic, Stanley J.; Morris, Max D.; +1 more

A study of examiner accuracy in cartridge case comparisons. Part 2: Examiner use of the AFTE range of conclusions

Journal Article · 2023 · Forensic Science International · OSTI ID:1987637

Baldwin, David P.; Bajic, Stanley J.; Morris, Max D.; +1 more

Firearm equipped with live round inhibiting means and method of making same

Patent · 1989 · OSTI ID:867593

Baehr, Donald G

Related Subjects

99 GENERAL AND MISCELLANEOUS
accuracy
black box study
decision analysis study
error rate
experimental design
forensic firearms identification
reliability

Title: Planning, design and logistics of a decision analysis study: The FBI/Ames study involving forensic firearms examiners

Citation Formats

References (49)

Similar Records

Related Subjects