The Hunt for the Last Relevant Paper: Blending the best of humans and AI

Abstract

The rapid expansion of research literature makes capturing all relevant studies for systematic reviews and meta-analyses increasingly challenging. Traditional search methods are time-intensive and heavily reliant on manual screening, yet more modern machine-aided strategies may also miss certain studies. This paper explores a collaborative, open-source approach for blending both approaches to identify all relevant literature up to the “last relevant paper” for a systematic review of post-traumatic stress symptom (PTSS) trajectories. We compared the results from eight search strategies, including conventional database searches, snowballing, full-text searches via the Dimensions database, and machine-aided methods of semantic vectorization with cosine similarity matching via the OpenAlex database. Thereafter, we combined human screening efforts with active learning–aided screening plus a large language model (LLMs) based quality check. In total, 3.822 records were added on top of the 6.701 records identified with replicating the initial search. Machine-aided methods found papers overlooked by traditional techniques—particularly papers missing conventional keywords, published in unindexed journals, or featuring unconventional phrasing. The human-AI screening approach resulted in 127 relevant studies. Active learning and LLMs were used to increase the quality of the labels, with both methods adding relevant papers overlooked by the human screeners. In conclusion, our findings also make it evident that even with exhaustive effort and sophisticated technologies, there remains the possibility that certain papers will still be missed. The evolving nature of academic publishing, varying terminologies, and publication delays mean that the pursuit of the "last relevant paper" is inherently a never-ending process.