AI in GxP: Recurring Compliance Gaps

1. AI operating outside the quality system 

Business teams introduce chatbots, copilots, analytics tools or embedded vendor AI features without notifying Quality, Regulatory, IT or Privacy.

Is there a complete AI inventory? Does every use case have an owner, an intended use, and a GxP classification? 

2. Uncontrolled medical or patient-facing responses 

A chatbot generates unscripted medical responses, answers product questions or provides recommendations that should be reviewed or delivered by qualified healthcare or medical information personnel. 

Are responses restricted to approved content? Are escalation rules, disclaimers, medical review and human intervention clearly defined?

3. Patient safety information entering a "black box"

A chatbot, patient support tool or digital channel receives adverse-event or product-complaint information, but the information is not reliably identified, retained, escalated or sent to Pharmacovigilance and Quality. 

Have all digital intake channels been mapped to PV and complaint processes? Are detection, routing, reconciliation and reporting timelines tested? 

4. Intended use is not clearly bounded 

A tool introduced for administrative support gradually begins to summarize clinical data, recommend quality decisions, classify deviations, or influence product-release activities. 

Is the approved context of use documented? Are prohibited uses and decision boundaries defined? 

5. AI is not appropriately validated 

Validation is limited to confirming that the application opens and produces an output. Hallucinations, inconsistent responses, edge cases, prompt sensitivity, and failure conditions are not challenged. 

Does testing address the intended use, patient or product risk, accuracy, repeatability, false positives, false negatives and foreseeable misuse? 

6. Training and reference data cannot be traced 

The organization cannot explain which data was used to train or configure the model, whether the data were representative, or whether confidential or regulated information was used appropriately. 

Are data provenance, lineage, quality, representativeness, bias and usage rights documented? 

7. Human oversight exists only on paper 

A person technically approves the AI output but lacks the expertise, source information, time or authority needed to identify an incorrect recommendation. 

Is the reviewer qualified? Can the reviewer independently challenge the output? Are escalation and override decisions documented? 

8. Model and prompt changes bypass change control 

Vendors update the underlying model, retrieval sources, system prompts or functionality without formal impact assessment or regression testing. 

Are model versions, prompts, configurations and connected data sources under control? What triggers revalidation? 

9. No ongoing performance monitoring 

Performance is evaluated before launch but not monitored after deployment. Model drift, changing data patterns and degradation go undetected. 

Are performance thresholds, monitoring frequency, sampling plans, alerts and suspension criteria established? EMA specifically recommends monitoring to identify model drift and degradation. 

10. Inadequate records and audit trails 

The organization cannot reconstruct what information was entered, which model or prompt version was used, what the AI produced or how the final decision was made. 

Are inputs, outputs, user actions, timestamps, versions, approvals and overrides retained in an attributable and reviewable form? 

11. Weak supplier oversight

Organizations rely on a vendor’s general security certification or marketing claims without assessing the specific GxP use, model controls, subcontractors or update practices. 

Does the supplier assessment cover validation support, model transparency, data handling, incident notification, change notification and right-to-audit provisions? 

12. Privacy and cybersecurity risks are assessed too late 

Employees enter confidential, personal, clinical or proprietary information into public or externally hosted AI tools. Prompt injection, unauthorized access and data leakage are not considered. 

Are approved tools, permitted data types, data residency, retention, access controls and cyberattack scenarios defined?