AI for PV: Development and Governance for a Regulated Industry
0 likes41 views
AI for pharmacovigilance: Development and Governance for a Regulated Industry. A survey of regulatory guidance for the use of artificial intelligence in pharmacovigilance and best practices for its safe implementation with the Biologit literature platform as case study.
AI for PV: Development and Governance for a Regulated Industry
- 1. AI for PV: Development and Governance for a Regulated Industry Bruno Ohana, CTO and Co-founder at biologit
- 2. 2 Page Evolving Landscape for Medical Literature Monitoring Increasing number of product launches and growing R&D pipeline with evolving regulatory demands for pharmacovigilance and medical devices Rapidly expanding volume of literature on a global scale ICSR volumes on the rise, driven by portfolio & geographic growth Growing pressure and expectation of automation and AI to reduce laborious and costly process Confidential
- 3. 3 Page Literature Monitoring is High Volume and Low Yield Confidential The European Medicines Agency (EMA) reports metrics on their literature monitoring activities (see EMA Report on Pharmacovigilance Tasks). In the 2019-2022 period, yield of valid cases discovered from the literature did not exceed 2%, while literature volume steadily grew. Source: Literature Monitoring in Pharmacovigilance: Challenges and Opportunities; Biologit White Paper; 2024
- 4. 4 Page The Productivity Challenges of Medical Literature Screening Search Literature Databases Results Assessment Quality Control Reports and Data Export Search strings maintenance De-duplication Global and Local Sources High volume of hits Multiple workflows (ICSR, Signal, Aggregate Reporting) Translations Traceability Collaborative workflows Operational and Strategic reporting Standards-driven integration (E2B) Analytics Confidential
- 5. 5 Page Biologit Platform Database One search, multiple sources + 50 million entries 30,000 new entries daily 110,000 journals 165+ countries AI for Pharmacovigilance Reliable screening, faster Workflow Flexible & compliant Search & rank Batch screening Tag filtering Audit log Validated cloud Up to 70% time savings CFR Part 11 API driven Confidential ISO 27001 Certified
- 6. 6 Page AI-Powered Screening Fine-Tuned for Pharmacovigilance Retrieve and screen all results from unified interface AI Filter: Suspected Safety Events Signal and Aggregate Reporting AI Filter: Identifiable patients ICSR Screening • Unique fit-for-purpose AI models for Pharmacovigilance, built with industry SMEs • Flexible levels of supervision: from human-in-the-loop to human-supervising-the-loop • Meeting industry standards of AI validation and governance (GAMP, GMLP) Confidential
- 7. 7 Page Biologit Platform End to end comprehensive scientific literature screening platform built for safety surveillance Confidential AI Automation Features
- 8. 8 Page Case Study: CAR-T Cell Therapy Search period: August 1st to October 24th, 2023 Confidential AI Features
- 9. 9 Page Our Vision is to Innovate Ethically and Transparently Key Tenets 1. We believe transparency is the fastest route to earning trust and increasing adoption of the much-needed benefits of AI in pharmacovigilance. 2. We unite expertise from technology and pharmacovigilance at every stage of the AI lifecycle 3. We commit to rigorously applying and continuously monitor emerging regulatory guidance Confidential
- 10. 10 Page AI Validation Use industry-specific validation framework for AI Confidential Validating Intelligent Automation Systems in Pharmacovigilance: Insights fr om Good Manufacturing Practices ; Huysentruyt et al (2021) … and our implementation of framework Validation and Transparency in AI systems for pharmacovigilance: a case study applied to the medical literature monitoring of adverse events ; Ohana, Sullivan, Baker (2021)
- 11. 11 Page AI Validation AI Development Lifecycle (SOP) • Ensures AI development is consistent, repeatable and verifiable • Traceability of key decisions at every stage of AI development • Using industry best practices for machine learning engineering • Embedded in our quality processes: Validation, Risk Management, Information Security, Change Management,… Verification testing at every level Confidential Activity Examples Unit testing Ensure code quality and meeting of design requirements. Data testing for integrity, duplicate checks, minimum quality levels. Integration testing End to end testing and verification of one process or subsystem (ex: the inference pipeline) Verification of model performance on test set following best practices. Functional and Requirements Testing (Operational Qualification) Application-level testing that ensures the overall system meets user and functional requirements. Follows Biologit computer systems validation SOP.
- 12. 12 Page Adopting AI with a Risk-Based Approach Transparency in support of AI adoption • Technical details of key machine learning model in model cards and white papers; • AI functionality is extensive documented and auditable from the platform, allowing informed decisions by users Supporting various levels of supervision for a risk-based approach to adoption • Human-in-the-loop: AI as an assistant; AI as QC • Human-supervising-the-loop: Use AI to automate decisions, human quality control • All automated decisions are fully traceable and can be inspected from the platform Confidential “We suggest that ensuring trust in AI/ML technology can make use of existing risk-based pharmacovigilance processes as a framework. The future of trust in AI/ML could focus on monitoring the outcomes of a process for safety, reliability, and effectiveness” How do we ensure trust in AI/ML when used in pharmacovigilance? Uppsala Reports Feb 2024
- 13. 13 Page Model Card Intended Use Detection of suspected adverse events on scientific literature text with the objective of filtering or ranking articles during literature screening. Model Performance Maximize recall of suspect adverse predictions Dataset Statistics Cross section of literature across all drug classes. Data labelled and QC by Pharmacovigilance SMEs, following biologit protocols Confidential Source: Towards AI Transparency with Model Factsheets; Biologit (2022)
- 14. 14 Page Controlling for False Negatives Clearly Documentation AI Features • Clearly articulate intended use of suspected adverse events on models • Provide users clear documentation and training of AI features and how to monitor automation • Support risk-based adoption: adjustable levels of supervision Train Machine Learning Models for High Recall • Conservative labelling protocol (“suspect” adverse event) • Conservative training regime for ML (high recall) Additional Controls during Model Inference • No predictions outside of operating envelope • Use combination of models: conservative prediction wins Governance of Models in Production • Conservative Periodic performance reviews: quantitative and qualitative • Ongoing dedicated AI risk management processes Confidential
- 15. 15 Page Summary: Lessons Learned Confidential 1. Transparency for earning trust o Invest in documentation; publish our approach; Engage in industry and regulatory forums o Traceable/Auditable AI decisions 2. Apply appropriate validation framework for AI o Continuously monitor regulatory environment 3. Risk-based adoption & oversight o AI development lifecycle fully embedded in Quality processes o Configurable levels of supervision to facilitate adoption 4. Multi-stakeholder engagement o Build the platform together with PV specialists
- 17. 17 Page Additional Resources 1. The following resources dive deeper the AI aspects of our solution: AI automation in pharmacovigilance and how it is delivered in biologit MLM-AI: https://www.biologit.com/post/medical-literature-monitoring-automation AI-powered Screening workflows in MLM-AI, from our product manual: https://docs.biologit.com/topics/ai-enabled-screening-workflows 2. To further illustrate the benefits, we suggest this case study: https://www.biologit.com/post/compliant-literature-searches-in-clinical-development-car-t-cell-case-study 3. Finally, from the point of view of AI governance, industry guidance and how we meet them, please see: Delivering AI in PV: a survey of existing guidance https://www.biologit.com/post/delivering-ai-in-pharmacovigilance-a-survey-of-existing-guidance AI Risk management at biologit https://www.biologit.com/post/ai-risk-management-approaches-and-benefits Applying good machine learning practices (GMLP) at biologit https://www.biologit.com/post/applying-good-machine-learning-practices-gmlp-at-biologit Confidential