Using Artificial Intelligence to Support Efficient and Early Autism Diagnosis

This study compared autism diagnoses made by specialists to the accuracy of an AI-based device. Results paved the way for the first, and only, FDA marketing authorization [1] of a software device designed to support providers in the diagnosis of autism in children aged 18-72 months.

Like
Read the paper

Our Motivation

Autism spectrum disorder (ASD) is a neurodevelopmental condition that impacts both communication and behavior. Roughly one in every 44 children are affected.2 Despite mounting evidence that early diagnosis and intervention can improve long-term outcomes and quality of life,3 significant diagnostic delays with pronounced health disparities persist.4 While reliable diagnosis is possible as early as 18 months,5 in the U.S. the average age of diagnosis has remained above four years for the last two decades.2

Multiple factors contribute to diagnostic delays. ASD prevalence rates have tripled since 2000.2 Specialists, traditionally responsible for conducting ASD evaluations, have been unable to keep up with this rapid rise in children requiring assessment. This has led to long wait times for appointments. If healthcare providers (HCPs) could diagnose more children in primary care, it could decrease strain on specialty services and shorten time to treatment initiation. However, many existing diagnostic tools are difficult to use in primary care settings; they take a long time to administer, many are not amenable to telemedicine, and may require specialist training.6,7 Furthermore, structural drivers of inequity, such as racism, and associated barriers to equitable care, including workforce capacity and structural diversity, contribute to diagnostic delays.8 HCPs need efficient, practical, and equitable solutions that equip them to diagnose more children in the primary care setting. 

To address this problem, Cognoa developed an AI-based diagnostic device for HCPs to use in conjunction with DSM-5 criteria and clinical judgment when evaluating children for ASD. The device was designed to enable rapid and efficient ASD evaluation in the primary care setting with the goal of accelerating time to diagnosis so families can begin accessing appropriate interventions during a critical neurodevelopmental window. 

The digital diagnostic device captures age-dependent behavioral, executive functioning, language and communication features maximally predictive of ASD. It was developed using patient record data from thousands of children with diverse conditions, presentations, and comorbidities who were either diagnosed with ASD or confirmed not to have ASD.9–13 Training algorithms on large samples of racially, socio-economically, geographically and gender diverse data is vital when seeking to build more objective and unbiased diagnostic tools.14 

The AI-based device tested in this study incorporates multimodular inputs consistent with current best practice recommendations that ASD evaluation include both caregiver and clinician input, as well as a structured observation of the child.15 The evolving device inputs and algorithms were prospectively validated for seven years prior to this trial.16 

How It Works diagram
The device renders the decision positive or negative for ASD or abstains from making a determination when presented with insufficient information, the latter being an important safeguard for AI in medicine.17

The Validation Trial

This prospective, double-blinded, multi-site, active comparator cohort study compared the accuracy of the device to reference standard diagnosis by blinded specialists. The study included 425 children, aged 18-72 months, with parental or HCP concern for developmental delay. The study was conducted at 14 sites across six states.

Clincial Sites and Central Review Panelists map

The study population mirrored U.S. population demographics across race, ethnicity and socio-economic status.Demographics charts

The device provided a determinate result for roughly 1 in 3 children included in the study (31.8%). For these children, device accuracy was high when compared to specialist diagnosis:

NPV-PPV graphs

The average age at which children in the study received an ASD positive device output was 2.8 years, whereas the current average age of diagnosis without the use of the device is above four years in the U.S.2 

The device abstained from diagnosing the remaining 68% of children in the study, 91% of whom were diagnosed by specialists with one or more neurodevelopmental disorders. Diagnostic abstention in cases of high uncertainty or complexity is an important method of risk control in medical machine learning algorithms. Similar to the human clinician, machines may operate more safely when they are able to flag cases in which additional expertise or data is required prior to rendering a diagnosis.17–19 

There was no evidence of device performance inconsistency across subjects’ sex, race/ethnicity, household income, parental education level, or geographic location. This is a promising initial finding given ongoing concerns about gender, racial and geographic biases in traditional ASD diagnostic processes.14 Follow-up studies with a larger number of subjects are needed to confirm the finding.

Study results suggest the device could support timely and equitable ASD diagnosis. It is also amenable to remote administration, suitable for use in primary care settings and has potential to reduce some of the current biases in ASD diagnosis, including gender and race biases. Using the device, HCPs could potentially evaluate more children in primary care, whereas, today, nearly all children with ASD are diagnosed in specialty settings.20 Streamlined earlier diagnosis could, in turn, facilitate treatment initiation during the critical early years of high brain neuroplasticity.  

For more details about this work, please refer to our paper (DOI: 10.1038/s41746-022-00598-6) published in npj Digital Medicine

References:

  1. U.S. Food & Drug Administration. FDA Authorizes Marketing of Diagnostic Aid for Autism Spectrum Disorder https://www.fda.gov/news-events/press-announcements/fda-authorizes-marketing-diagnostic-aid-autism-spectrum-disorder.
  2. Maenner, M. J. et al. Prevalence and characteristics of autism spectrum disorder among children aged 8 years—Autism and Developmental Disabilities Monitoring Network, 11 sites, United States, 2018. MMWR Surveill. Summ. 70, 1 (2021).
  3. Koegel, L. K., Koegel, R. L., Ashbaugh, K. & Bradshaw, J. The importance of early identification and intervention for children with or at risk for autism spectrum disorders. Int. J. Speech Lang. Pathol. 16, 50–56 (2014).
  4. Constantino, J. N. et al. Timing of the Diagnosis of Autism in African American Children. Pediatrics 146, e20193629 (2020).
  5. Pierce, K. et al. Evaluation of the diagnostic stability of the early autism spectrum disorder phenotype in the general population starting at 12 months. JAMA Pediatr. 173, 578–587 (2019).
  6. Kaufman, N. K. Rethinking “gold standards” and “best practices” in the assessment of autism. Appl. Neuropsychol. Child 1–12 (2020) doi:10.1080/21622965.2020.1809414.
  7. Randall, M. et al. Diagnostic tests for autism spectrum disorder (ASD) in preschool children. Cochrane Database Syst. Rev. (2018).
  8. Broder-Fingert, S., Mateo, C. M. & Zuckerman, K. E. Structural Racism and Autism. Pediatrics 146, e2020015420 (2020).
  9. Kosmicki, J. A., Sochat, V., Duda, M. & Wall, D. P. Searching for a minimal set of behaviors for autism detection through feature selection-based machine learning. Transl. Psychiatry 5, e514–e514 (2015).
  10. Levy, S., Duda, M., Haber, N. & Wall, D. P. Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism. Mol. Autism 8, 1–17 (2017).
  11. Wall, D. P., Dally, R., Luyster, R., Jung, J.-Y. & DeLuca, T. F. Use of artificial intelligence to shorten the behavioral diagnosis of autism. (2012).
  12. Duda, M., Daniels, J. & Wall, D. P. Clinical evaluation of a novel and mobile autism risk assessment. J. Autism Dev. Disord. 46, 1953–1961 (2016).
  13. Abbas, H., Garberson, F., Glover, E. & Wall, D. P. Machine learning approach for early detection of autism by combining questionnaire and home video screening. J. Am. Med. Inform. Assoc. 25, 1000–1007 (2018).
  14. Aylward, B. S., Gal-Szabo, D. E. & Taraman, S. Racial, Ethnic, and Sociodemographic Disparities in Diagnosis of Children with Autism Spectrum Disorder. J. Dev. Behav. Pediatr. JDBP (2021).
  15. Brian, J. A., Zwaigenbaum, L. & Ip, A. Standards of diagnostic assessment for autism spectrum disorder. Paediatr. Child Health 24, 444–451 (2019).
  16. Abbas, H., Garberson, F., Liu-Mayo, S., Glover, E. & Wall, D. P. Multi-modular AI approach to streamline autism diagnosis in young children. Sci. Rep. 10, 1–8 (2020).
  17. Kompa, B., Snoek, J. & Beam, A. L. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digit. Med. 4, 1–6 (2021).
  18. Cortes, C., DeSalvo, G., Gentile, C., Mohri, M. & Yang, S. Online learning with abstention. Int. Conf. Mach. Learn. 1059–1067.
  19. Mozannar, H. & Sontag, D. Consistent estimators for learning to defer to an expert. in International Conference on Machine Learning 7076–7087 (PMLR, 2020).
  20. Monteiro, S. A., Dempsey, J., Berry, L. N., Voigt, R. G. & Goin-Kochel, R. P. Screening and referral practices for autism spectrum disorder in primary pediatric care. Pediatrics 144, (2019).

Andrey Ostrovsky, MD, FAAP

Managing Partner, Social Innovation Ventures

Comments are disabled