Challenges and novel approaches for the evaluation of digital health solutions

Clinical simulation-based research could enable the generation of higher-quality, lower-cost and more timely evidence for digital health solutions.

Like Comment
Read the paper

Digital health is booming; more than 300,000 healthcare applications exist, and 200 more are added daily1. The global pandemic has turbocharged the industry forcing healthcare providers globally to adopt Telehealth to maintain services. Clinician and patient acceptance of virtual care will be permanent, displacing physical infrastructure, and establishing digital solutions as the primary location of interaction. This will catalyse the utilisation of clinical decision support by providing context relevant channels for deployment of risk stratification, prognosis and predictive tools.

 Challenges exist for end users (i.e. patients and clinicians) to determine the benefits of new digital health solutions. In our recently published article entitled, “Challenges for the evaluation of digital health solutions – A call for innovative evidence generation approaches” we demonstrate how the cadence of traditional research approaches is misaligned with the “fail fast, fail often” mantra championed by technology companies. We believe clinical simulation-based research provides a good balance between evidence strength and the time/speed needed to bridge the evidence gap between subjective use cases and objective trials (figure 1).

 Figure 1: Existing approaches for health digital solution evaluation, current methodological limit and emerging innovative pragmatic approaches to bridge the gap.

The concept of “simulation” is not new and is the methodological foundation for human behaviour experimental research. The assumption is that people behave similar to real-life if key components of the scenarios are extracted and fidelity maintained. Clinical simulation was traditionally developed and used for training medical residents, and has been further developed as an approach to test digital solutions with representative users doing representative tasks, in representative settings/environments2.

 A recent cost-effectiveness analysis suggested that introducing simulation into a product development lifecycle could lead to cost savings of 37-79%3. Other advantages include: the ability to collect participant behavioural and/or cognitive metrics, study scalability, flexibility in study design (e.g. different scenarios, different participants), and the opportunity to undertake remote and/or distributed research4, a relevant side benefit for current times.

 Several academic centers have established clinical simulation test environments for digital health solutions, including the Institute of Global Health Innovation (IGHI) at Imperial College London5. An example study evaluated the impact of a digital solution on the conduction of cancer multidisciplinary team (MDT) meetings. 56 healthcare professionals were recruited to undertake 10 simulated MDT sessions. A comparative study evaluated the current clinical standard (paper handout and PACS) versus MDT coordinated by a digital tumor board solution6 (figure 2&3). Key general learnings reinforced the importance of 1) creating realistic clinical scenarios 2) recruiting the most representative participants, and 3) providing comprehensive training of the solution. At IGHI, the researchers achieved this when an eminent thoracic surgeon nearly came to blows with the pulmonologist over the treatment strategy…. for a made-up patient case!

Figure 2: Lung cancer MDT team engaged in a clinical simulation adopting a legacy approach (i.e. paper handout and PACS). The team consist of clinicians who routinely take part in lung cancer MDT’s. The scenario took place in the normal hospital MDT room.

 Figure 3: Lung cancer MDT team engaged in a clinical simulation adopting a new digital tumor board solution. The team consist of clinicians who routinely take part in lung cancer MDT’s. The scenario took place in the normal hospital MDT room.

 Innovators face significant challenges to overcome the “no evidence, no implementation – no implementation, no evidence” paradox in digital health. We believe that innovative approaches, such as clinical simulation-based research, can enable the generation of higher-quality, lower-cost and more timely evidence, which will encourage an evidence-based adoption of digital health solutions.


  1. (2017). IQVIA Institute for Human Data Science Study: Impact of Digital Health Grows as Innovation, Evidence and Adoption of Mobile Health Apps Accelerate.
  2. Kushniruk, A., Nohr, C., Jensen, S. and Borycki, E.M. (2013). From usability testing to clinical simulations: Bringing context into the design and evaluation of usable and safe health information technologies. Yearbook of medical informatics, 22(01), pp.78-85.
  3. Baylis TB et al. (2011). Low-Cost Rapid Usability Testing for health information systems: is it worth the effort? Stud Health Technol Inform.
  4. Yao, H., Zhang, H. and Duan, F. (2010). Research and design on distributed remote simulation based on Web. In 2010 2nd IEEE International Conference on Information Management and Engineering (pp. 522-525)
  6. Gardner, C., et al. (2020). A Mixed Methods Study for the Evaluation of a Digital Health Solution for Cancer Multidisciplinary Team Meetings Using Simulation-based Research Methods. Journal of Clinical Oncology 38:15 e14063-e1406

Matthew Prime

Medical Director, Roche Diagnostics