We are pleased to share the results of our recent validation study, “Loon Lens 1.0 Validation: Agentic AI for Title and Abstract Screening in Systematic Literature Reviews,” now available on medRxiv. This study evaluates the effectiveness of Loon Lens™, our autonomous AI literature screener designed to automate the Title and Abstract (TiAb) screening process in systematic literature reviews (SLRs).

Addressing the Challenges of Systematic Reviews

Systematic literature reviews are fundamental to evidence-based research across various disciplines, including healthcare, social sciences, and technology. They provide comprehensive analyses that inform clinical guidelines, policy-making, and future research directions. However, the process of conducting SLRs is often resource-intensive and time-consuming. According to studies, an average SLR can take over a year to complete and cost more than $140,000 USD, with TiAb screening being one of the most laborious stages.

The Burden of Title and Abstract Screening

TiAb screening involves manually reviewing thousands of citations to identify studies relevant to a specific research question. This step is crucial but can be a bottleneck due to the sheer volume of literature and the need for meticulous attention to inclusion and exclusion criteria.

Introducing Loon Lens

Loon Lens™ is an autonomous AI literature screener that alleviates the burden of TiAb screening by leveraging large language models (LLMs). Unlike traditional methods that require initial manual screening of hundreds of studies or semi-automated approaches dependent on pre-labelled data, Loon Lens™ autonomously screens citations based solely on user-defined inclusion and exclusion criteria.

Key Features

Fully Autonomous Screening	No need for pre-labeled training data or initial researcher screening of hundreds of studies.
No-code, simple Interface	Loon Lens™ is designed for researchers to use it out of the box. No expertise in AI or machine learning is required to use it.
Scalable Solution	Loon Lens™ is capable of handling large volumes of citations efficiently, giving you the results in just a few hours and cutting weeks of work off your plate.

The Validation Study: Assessing Performance and Reliability

To evaluate Loon Lens’s effectiveness, we conducted a validation study comparing its performance against human reviewers in TiAb screening across eight systematic literature reviews.

Study Design Overview

• Data Source: We replicated eight SLRs conducted by Canada’s Drug Agency (CDA), covering various drugs and medical conditions.

• Citations Reviewed: A total of 3,796 citations were retrieved using OpenAlex, an open-source scholarly database.

• Human Review: Dual independent reviewers screened citations, identifying 287 studies (7.6%) for inclusion.

• Loon Lens Screening: Loon Lens™ used the same citations and eligibility criteria to perform autonomous TiAb screening.

Understanding the Metrics

To provide a comprehensive assessment, we calculated several performance metrics:

Accuracy	The proportion of correct predictions (both inclusions and exclusions).
Recall (Sensitivity)	The ability to identify all relevant studies.
Precision (Positive Predictive Value)	The proportion of correctly identified relevant studies among those flagged.
F1 Score	The harmonic mean of precision and recall, balancing both metrics.
Specificity	The ability to correctly exclude irrelevant studies.
Negative Predictive Value (NPV)	The proportion of correctly identified irrelevant studies among those excluded.

Bootstrapping was applied to compute 95% confidence intervals, providing robustness to our estimates.

Results: High Recall and Accuracy

The validation study yielded encouraging results:

Accuracy: 95.5%	95% CI: 94.8%–96.1%
Recall (Sensitivity): 98.95	95% CI: 97.57%–100%
Specificity: 95.24%	95% CI: 94.54%–95.89%
F1 Score: 0.770	95% CI: 0.734–0.802
Precision (Positive Predictive Value): 62.97%	95% CI: 58.39%–67.27%

Interpreting the Results

• High Recall: Loon Lens successfully identified nearly all relevant studies, which is crucial in SLRs to ensure comprehensive evidence synthesis.

• Good Specificity: The platform effectively excluded irrelevant studies, minimizing the burden of unnecessary full-text reviews.

• Precision: Erring on the safe. side: While the precision indicates that some irrelevant studies were included, in SLR contexts where missing relevant studies is a greater concern than reviewing additional ones, this means the precision of the study leans towards caution.

Confusion Matrix

Studies	Predicted Included	Predicted Excluded
Actually Included	284	3
Actually Excluded	167	3342

• True Positives: 284 studies correctly identified as relevant.

• False Positives: 167 studies were incorrectly identified as relevant.

• False Negatives: 3 studies were missed.

• True Negatives: 3,342 studies correctly identified as irrelevant.

Discussion: Implications, Limitations, and Future Directions

The validation of Loon Lens represents a significant advancement in the application of AI to systematic literature reviews. By achieving high recall and specificity, Loon Lens demonstrates its potential to substantially reduce the time and effort required for TiAb screening. However, it’s essential to critically examine these results to understand their implications fully.

Implications for the Research Community

• Efficiency Gains: The high accuracy and recall suggest that Loon Lens can reliably identify relevant studies, allowing researchers to allocate their time more effectively. This efficiency is particularly beneficial for large-scale reviews where the volume of citations can be overwhelming.

• Resource Allocation: With reduced time spent on initial screening, resources can be redirected towards more in-depth analysis, quality assessment, and synthesis of findings.

• Accessibility: By lowering the barriers to conducting systematic reviews, Loon Lens may enable smaller research teams or those with limited funding to undertake comprehensive reviews.

Balancing Recall and Precision

While Loon Lens excels in recall, ensuring that almost all relevant studies are identified, the moderate precision indicates a higher rate of false positives compared to human reviewers. This trade-off is important to consider:

• Acceptable Trade-off: In systematic reviews, missing a relevant study (false negative) can have more significant consequences than including an irrelevant one (false positive). Therefore, a higher recall is often prioritized over precision.

• Impact on Workload: The increase in false positives means that researchers may need to screen more studies at the full-text level. However, this additional effort is generally less burdensome than the initial TiAb screening and is a reasonable compromise to ensure comprehensiveness.

Limitations of the Study

• Scope of Validation: The study focused on eight SLRs in the healthcare domain, specifically related to drug evaluations. While these reviews covered a range of topics and eligibility criteria, the results may not be fully generalizable to other fields or types of studies, such as qualitative research or reviews in social sciences.

• Data Source: The use of OpenAlex as the sole bibliographic database may have influenced the pool of citations. Differences in indexing between databases like PubMed, Scopus, or Web of Science could affect the generalizability of the findings.

• Language and Cultural Bias: LLMs can sometimes exhibit biases based on the language and cultural contexts present in their training data. This could potentially impact the screening of studies from diverse geographical regions or non-English publications.

Addressing Ethical and Practical Considerations

• AI Transparency: Understanding how Loon Lens makes decisions is crucial for trust and acceptance. While LLMs can be seen as “black boxes,” our efforts to provide explanations for inclusion or exclusion decisions can enhance transparency.

• Data Privacy: Ensuring that uploaded citations and any associated data are handled securely is essential. Adhering to data protection regulations and best practices is a priority.

• User Control: Providing users with options to adjust the sensitivity of the screening process or to review borderline cases can empower researchers and tailor the tool to specific needs.

Future Directions and Enhancements

• Algorithm Refinement: Ongoing development aims to refine the algorithms to reduce false positives without compromising recall. This may involve incorporating additional contextual understanding or domain-specific knowledge.

• Full-Text Screening Capability: Extending Loon Lens to assist with or autonomously perform full-text screening could further streamline the systematic review process.

• Cross-Disciplinary Validation: Conducting validation studies in other fields, such as psychology, education, or environmental science, will help assess the tool’s adaptability and effectiveness across disciplines.

• Integration with Existing Workflows: Developing integrations with popular reference management software and systematic review tools can enhance usability and encourage adoption.

• User Feedback Mechanisms: Incorporating feedback loops where users can provide input on screening decisions can help improve the model over time and increase accuracy.

Collaboration and Community Engagement

• Open Dialogue: We encourage discussions within the research community about the role of AI in systematic reviews. Sharing experiences, challenges, and solutions will benefit all stakeholders.

• Ethical AI Practices: Collaborating with ethicists and AI experts to address concerns about biases, fairness, and accountability is important for responsible deployment.

• Training and Support: Providing resources, tutorials, and support to help users make the most of Loon Lens will facilitate smoother transitions to incorporating AI into research workflows.

Comparison with Existing Solutions

Unlike semi-automated tools that require human input for labelling or training, Loon Lens™ operates fully autonomously. This sets it apart by offering:

• No Need for Pre-Labeled Data: Reduces setup time and allows immediate use.

• No Need for Pre-Screening: Researchers don’t need to screen hundreds of studies, unlike current literature screeners.

• User-Friendly Experience: Simplifies the screening process without technical complexities.

How to Get Started with Loon Lens

We invite researchers and institutions to try Loon Lens™:

1. Request Access: Visit https://loonlens.com/ and submit a request.

2. Prepare Your Data: Export your citations in RIS format from your reference management software.

3. Define Criteria: Clearly outline your inclusion and exclusion criteria.

4. Initiate Screening: Upload your data and criteria, and let Loon Lens handle the screening.

Conclusion: Advancing Literature Screening for Systematic Reviews with Loon AI

Loon Lens™ represents a significant step forward in leveraging AI to support researchers in conducting literature screening for systematic literature reviews more efficiently. While the tool demonstrates high recall and accuracy in identifying relevant studies, we acknowledge that it is not without limitations. The precision levels indicate room for improvement, particularly in reducing false positives to minimize unnecessary workload during full-text screening.

Our commitment is to continue refining Loon Lens™ based on user feedback and ongoing research. By addressing the limitations and expanding its capabilities, we aim to make Loon Lens™ an indispensable tool across various research domains.

We believe that while AI cannot replace the nuanced judgment of experienced researchers, it can serve as a powerful assistant. By automating the most time-consuming aspects of systematic reviews, Loon Lens™ allows researchers to focus on critical analysis, interpretation, and the generation of new insights that advance their fields.

For more information or to request access, please visit https://loonlens.com/ or contact us at contact@loonbio.com.

Collaborate with Us

We are keen to collaborate with the research community to further enhance Loon Lens:

• Feedback: Share your experiences to help us improve.

• Partnerships: Academic and industry partnerships are welcome for joint projects and studies.

• Beta Testing: Participate in testing new features and provide valuable insights.

Ready to Transform Your Evidence Synthesis Process?

Contact us today for a demo, or visit loonbio.com to learn more about how we’re revolutionizing market access and clinical research with AI-driven solutions that reduce research timelines from years to days.

---

About Loon

Loon Inc. is at the forefront of AI-driven market access and clinical research. We help biopharma companies navigate the complexities of market access with confidence, providing innovative solutions that dramatically reduce research timelines while maintaining the highest standards of quality and compliance.