Gaps and variation in evaluation criteria for digital health technologies: a scoping review of HTA organizations’ guidance documents

Abstract

Objectives: Digital health technologies (DHTs) are increasingly central to health system innovation, addressing challenges such as workforce shortages and rising service demands. As adoption grows, robust and context-sensitive evaluation frameworks are essential to ensure safe, effective, and equitable implementation. This study aimed to 1) map how health technology assessment (HTA) organizations define and report criteria for evaluating DHTs, using the NICE Evidence Standards Framework (NICE-ESF) as an analytical structure, and 2) explore variation within criteria, using thematic analysis. Methods: A scoping review was conducted of policy documents published by HTA organizations affiliated with INAHTA or EUnetHTA. Documents were included if they outlined general evaluation or assessment criteria for DHTs. Two researchers independently selected and coded the documents. Thematic analysis was guided by the 21 NICE-ESF standards. Results: Twenty-one documents from ten organizations across nine countries were included. Most frameworks addressed safety, data protection, and clinical effectiveness, often referencing legislation or certification. However, criteria were often broad requirements with limited guidance on methods, thresholds, or decision rules. Several domains were inconsistently covered or absent, including environmental sustainability, workforce impact and professional engagement, AI-specific governance, and post-implementation evaluation. Differences partly aligned with organizational mandates. Conclusions: HTA guidance for DHTs shows substantial variation and limited operationalization, which may fragment evidence expectations and hinder cross-national learning and scale-up. Strengthening practice may require 1) a harmonized minimum core indicator set with actionable definitions and decision rules, 2) integrated AI-specific criteria, and 3) institutionalized post-implementation monitoring to capture long-term system-level impacts. Public Interest Summary: Digital tools, such as health apps and artificial intelligence systems, are becoming vital parts of healthcare. Before they are widely used or reimbursed, organizations need clear ways to judge whether these tools are safe, effective, and worth the cost. We reviewed policy documents from international health technology assessment organizations and compared the criteria they use to evaluate digital health technologies. Most documents covered safety, privacy, and clinical effectiveness. But many were vague about how these checks should be done. Important topics were often missing or only briefly mentioned, such as environmental impact, effects on healthcare staff and workload, rules for AI systems, and monitoring what happens after the technology is introduced in real practice. We recommend a shared minimum set of clear criteria, stronger AI-specific requirements, and routine follow-up after implementation to ensure digital tools deliver lasting benefits.