Medical AI Vendor Due Diligence: 25 Questions to Ask Before a Hospital Pilot

Medical AI vendor due diligence is the structured check a hospital or health-technology team runs before it signs a contract or starts a pilot: what decision the system supports, what evidence backs it, where patient data goes, how it integrates, who can override it, how changes are controlled, and what it really costs. A vendor can show a strong benchmark, a polished interface and a confident pilot proposal while leaving those questions open. Who was represented in the validation data? Where is patient information processed? What happens when the model is uncertain? Who approves an update, and who produces the evidence your reviewers will ask for?

Buying medical AI is therefore not only a software purchase. It is a clinical, data-governance, integration, security, validation and operating-model decision made at the same time. This guide sets out 25 questions to ask, the evidence that counts as an answer, the warning signs, and how to turn the result into a go/no-go decision. It is technical and operational guidance, not legal advice; regulatory classification and contract terms should be confirmed with qualified counsel.

What should a hospital ask a medical-AI vendor?

A hospital should ask a medical-AI vendor questions that test three things separately: whether the model performs technically, whether using it improves the intended clinical or operational workflow, and whether the organisation can deploy, govern, monitor and defend it responsibly. A strong answer to one of these does not prove the other two. A model with excellent retrospective accuracy can still slow clinicians down, and a well-liked workflow tool can still send patient data somewhere your data protection officer cannot approve.

In practice the 25 questions below fall into eight areas: intended use, regulatory position, clinical and analytical evidence, patient data and GDPR, architecture and integration, human oversight, monitoring and change control, and commercial terms. Ask them in that order. Intended use comes first because it determines the evidence you need, the regulatory route, the validation burden and the oversight design, so every later answer is judged against it.

Why does medical AI due diligence start with intended use?

Medical AI due diligence starts with intended use because every later question depends on it. "AI-assisted workflow" or "clinical decision support" is too vague to evaluate: it does not say which decision changes, for whom, or what an error costs. Pin it down with five questions:

1. What exact clinical or operational decision does the system support?
2. Who is the intended user?
3. Which patient population and care setting are in scope?
4. What is explicitly outside the intended use?
5. What existing workflow, cost, delay or risk should improve, and by how much?

The answer places the product on a spectrum, and each step up raises the bar for evidence, regulatory scrutiny and oversight:

• Administrative automation, such as scheduling, coding support or document routing.
• Workflow prioritisation, such as reordering a reporting worklist.
• Clinical decision support that presents information to a clinician who decides.
• Screening or triage, where the output decides who is seen first or at all.
• Diagnostic or prognostic functionality.
• Treatment recommendation.

Red flags: the use case changes depending on who is asking; the vendor describes capabilities rather than a bounded intended purpose; nobody can name the populations or settings the system was validated in; the pilot proposal has no baseline and no decision threshold.

How do you verify a medical-AI product's regulatory position?

A medical-AI product's regulatory position is verified by asking the vendor to state it in writing and then checking it against the intended use you have just defined. Four questions:

6. Does the product qualify as medical-device software under the MDR or IVDR, and under which classification rule?
7. What is its current conformity-assessment or certification status, and does the certificate cover the intended use you plan?
8. Is the vendor acting as the EU AI Act provider and your organisation as the deployer?
9. Which obligations and documentation remain your responsibility?

Do not try to settle classification from a sales deck; it depends on intended purpose and needs qualified regulatory review. A few points hold regardless. The EU AI Act does not replace the MDR or IVDR; it sits alongside them, and an AI system that is a medical device needing notified-body assessment can also be high-risk under Article 6(1) of the AI Act. Not every medical-AI system is high-risk. Hospitals as deployers keep their own duties under Article 26, including competent human oversight, monitoring and keeping logs for at least six months. And rebranding or materially modifying a system can make you its provider under Article 25, which the provider versus deployer guide covers in detail.

Red flags: "we are GDPR compliant" offered instead of a data flow; CE marking treated as proof the product works in every local population; a contract that leaves evidence production and operational monitoring undefined.

Is a CE mark enough to approve a medical-AI pilot?

No. A CE mark shows that a medical device has passed the conformity route for its stated intended purpose under the MDR or IVDR. A CE mark does not show that the product performs on your patient population, fits your workflow, or meets your data-protection, security and integration requirements. It does not replace a local workflow assessment, a data protection impact assessment, a security review, local validation, contractual controls on data and updates, or an operational monitoring plan.

Treat the CE mark as a precondition that answers one question well and leaves the rest open. Check two details on it: that the certified intended purpose matches how you plan to use the product, and that the version you will deploy is the version covered. Using a certified product outside its intended purpose is one of the routes by which a deployer starts to carry provider-level responsibility. For how the AI Act and the medical device regime overlap for hospitals, see the EU AI Act for healthcare AI systems.

What evidence should a medical-AI vendor provide?

A medical-AI vendor should provide an evidence pack, not a headline metric: an intended-use statement, dataset provenance, external validation results, subgroup results with confidence intervals, a failure analysis, clinical evidence for the claimed benefit, security documentation, a data-flow description, a monitoring plan and a version history. Five questions test it:

10. Which datasets, sites, devices, time periods and patient populations were used for evaluation?
11. Was the evaluation external, independent, prospective or multi-site?
12. Are subgroup results and confidence intervals available?
13. How does the system behave on missing, corrupted, unusual or out-of-distribution inputs?
14. Has it been validated in a setting comparable to yours?

Keep two kinds of evidence apart. Analytical validation asks whether the system reliably computes or predicts what it claims. Clinical validation asks whether using the output improves the intended outcome without unacceptable harm. Aggregate accuracy answers neither on its own. Depending on the use case you may need sensitivity, specificity, calibration, positive and negative predictive value at your prevalence, subgroup performance, and performance at the clinically meaningful threshold. Reporting guidance such as TRIPOD+AI for prediction models and DECIDE-AI for early clinical evaluation is a useful yardstick for what a complete study reports. How to run the validation yourself is a separate job, covered in the medical AI validation checklist.

Red flags: one headline metric; validation data from the same site or period as training; nothing on relevant subgroups, devices or settings; no explanation of false negatives, false positives or uncertainty; a retrospective benchmark presented as proof of clinical utility.

How do you map a medical-AI vendor's patient-data and GDPR exposure?

A medical-AI vendor's GDPR exposure is mapped by tracing every flow of patient data, from the moment it leaves your systems to the moment it is deleted, and checking each flow against the contract rather than the marketing site. Five questions:

15. Which patient or operational data leaves the organisation?
16. Where is it stored, processed, logged and backed up?
17. Is customer data used for model training, evaluation, abuse monitoring or product improvement?
18. Which subprocessors can access it?
19. Can the system operate on minimised, de-identified, synthetic or locally processed data?

Health data is a special category under Article 9 GDPR, so processing needs both an Article 6 legal basis and an Article 9 condition. Pseudonymised data generally remains personal data while re-identification remains possible, as Recital 26 makes clear. Expect a processor agreement under Article 28 with retention periods, deletion and export procedures, a subprocessor list including cloud and model providers, the treatment of prompt and output logs, and the basis for any transfer outside the EU. Large-scale processing of health data with a new technology is the textbook case for a data protection impact assessment under Article 35.

Cloud is not automatically unsafe and on-premise is not automatically compliant; the question is what is processed, under whose control, with which safeguards. The trade-offs are set out in on-premise versus cloud AI for healthcare.

Red flags: no complete data-flow diagram; retention or training opt-outs that exist only in documentation, not the contract; unclear subprocessors or regions; free-text clinical notes treated as non-identifiable by default; no way to export or delete your data and logs.

How do you assess a medical-AI system's architecture, security and integration?

A medical-AI system's architecture is assessed by following one real request end to end: from the EHR or PACS, through identity and access checks, to inference, to the output a clinician sees, and into the audit log. Four questions:

20. How will the system integrate with the EHR, PACS, DICOM and FHIR interfaces, identity provider and existing authorisation model?
21. How are access permissions enforced through retrieval, inference and output delivery?
22. What happens during downtime, degraded operation or loss of connectivity?
23. Can you export data, logs, configuration and model-version information?

The checks behind those questions are concrete: single sign-on and role-based access, audit logs you control, encryption in transit and at rest, environment isolation, vulnerability management, backup and recovery, latency and availability targets, incident notification times, data portability, and who owns the integration work. Public cloud, private cloud, on-premise and hybrid deployments can all pass or fail these checks; the deployment model matters less than whether the controls are evidenced.

Red flags: the system bypasses existing clinical access controls; the vendor holds the only usable audit log; there is no safe fallback workflow during downtime; integration effort is excluded from the quoted implementation; nobody can say which model version produced a historical output.

How do you assess human oversight in a medical-AI workflow?

Human oversight in a medical-AI workflow is assessed by checking whether the intended user has the time, information, competence and authority to intervene, not whether a "human in the loop" appears on an architecture slide. Two questions:

24. Can the intended user understand, challenge and override the output?
25. How are uncertainty, limitations, overrides and escalation events communicated and recorded?

Two failure modes deserve explicit attention. Automation bias is the tendency to accept a confident output even when other information contradicts it; alert fatigue is the opposite drift, where frequent low-value alerts are dismissed without reading. Both are design problems, so ask to see the interface for a low-confidence case and for a case with missing inputs, and ask what the system does when it has insufficient information. Ask what training users receive, how overrides are logged and reviewed, where escalations go, and how responsibility is divided between vendor, hospital and clinician. Article 26 of the EU AI Act expects deployers of high-risk systems to assign oversight to people with the necessary competence, training and authority.

Red flags: recommendations shown without uncertainty or limitations; overrides possible but never logged or reviewed; a user formally responsible but given no training or decision context; no defined behaviour for low-confidence or out-of-distribution cases.

What monitoring and change control should a medical-AI vendor commit to?

A medical-AI vendor should commit in the contract to a monitoring and change-control plan that covers model behaviour, not just uptime. The plan should name model and dataset versions, how data and performance drift are detected, how subgroup performance is tracked, how incidents are reported to you, what triggers retraining or revalidation, how much notice you get before an update, how rollback works, and who approves a change. Post-market surveillance and vigilance duties under the MDR, and the provider's monitoring duties under the AI Act, sit with the vendor; your own duty to monitor operation and report problems to the provider remains.

The single most important question here: can the vendor materially change the model, prompt, retrieval process or underlying model provider without your prior review? For a system built on a third-party foundation model, a silent swap of that model can change behaviour as much as retraining. Agree in advance which changes require notice, which require your revalidation, and which entitle you to suspend use.

Red flags: automatic, undocumented updates; monitoring that covers availability but not model behaviour; no predefined thresholds for investigation or suspension; historical decisions that cannot be tied to a model and configuration version.

What does a medical-AI system really cost to implement?

A medical-AI system costs far more to implement than its licence, because the licence is usually the smallest line. A full estimate covers licence and usage fees, integration work, data preparation, information-governance review, clinical validation, staff training, monitoring and support, internal ownership, revalidation after vendor changes, and exit and migration. A cheap licence can still produce an expensive implementation if the hospital has to build the evidence, integration and governance structure itself.

Questions for the commercial review:

• What service levels and support response times are contractual, and what are the remedies?
• Which professional services are included, and which are billed separately?
• What termination assistance is provided, and in what format are data and configuration exported?
• Who owns intellectual property in configurations, derived data and model improvements trained on your data?
• Where are liability boundaries drawn between vendor, hospital and clinician?

Agree the exit terms before the pilot, while you still have leverage. Once a system is embedded in a clinical workflow, switching cost rises quickly and the negotiating position shifts to the vendor.

How do you turn due diligence into a pilot go/no-go decision?

Due diligence becomes a pilot go/no-go decision when each area has named evidence, an acceptable threshold agreed in advance, an owner and a status. A scorecard like the one below keeps the decision explicit and stops one strong demo from outweighing an unresolved data-protection question.

Define non-negotiable stop conditions before the pilot begins, so nobody renegotiates them halfway through: an unresolved legal basis for patient-data processing; missing evidence for the intended population; inability to audit outputs; no effective override; an unclear model-update policy; integration or security risk outside your tolerance; and no measurable workflow or clinical outcome. Once the scorecard is green, the pilot itself can follow a structured 30 to 90 day proof of value.

What should a medical AI due-diligence review produce?

A medical AI due-diligence review should produce a set of documents that a sceptical internal reviewer can check, not a slide that says "approved". Before approving a pilot, the buyer should hold:

• a bounded intended-use statement, including what is out of scope;
• a vendor evidence register, listing each claim and the document that supports it;
• the regulatory-role and classification assumptions, with who confirmed them;
• a patient-data flow diagram covering storage, logging, backups and subprocessors;
• an integration and security assessment;
• a local-validation plan;
• a human-oversight design, including override and escalation;
• pilot metrics, the baseline and the stop conditions;
• a monitoring and change-control plan;
• a risk register;
• a commercial and exit-cost analysis;
• a clear go/no-go decision with owners and the questions still open.

These artifacts also outlive the pilot. If the system goes into routine use, the same register, data map and oversight design become the starting point for your deployer documentation and for answering auditors, ethics committees and patients who ask how a decision was made. Teams that build them during procurement rarely have to reconstruct them later. For the wider picture of how this fits a healthcare AI programme, see our healthcare AI practice.

What are the 25 questions to ask a medical-AI vendor?

The 25 questions to ask a medical-AI vendor, grouped by area so the list can be used on its own in a vendor meeting:

Intended use

1. What exact clinical or operational decision does the system support?
2. Who is the intended user?
3. Which patient population and care setting are in scope?
4. What is explicitly outside the intended use?
5. What workflow, cost, delay or risk should improve, and by how much?

Regulatory position

6. Does the product qualify as medical-device software under the MDR or IVDR?
7. What is its conformity-assessment or certification status, and does it cover our use?
8. Are you the EU AI Act provider and are we the deployer?
9. Which obligations and documentation remain ours?

Evidence

10. Which datasets, sites, devices, periods and populations were used for evaluation?
11. Was evaluation external, independent, prospective or multi-site?
12. Are subgroup results and confidence intervals available?
13. How does the system behave on missing, corrupted or out-of-distribution inputs?
14. Has it been validated in a setting comparable to ours?

Patient data

15. Which data leaves our organisation?
16. Where is it stored, processed, logged and backed up?
17. Is our data used for training, evaluation, monitoring or product improvement?
18. Which subprocessors can access it?
19. Can the system run on minimised, de-identified, synthetic or local data?

Architecture and integration

20. How does it integrate with our EHR, PACS, DICOM, FHIR and identity systems?
21. How are access permissions enforced end to end?
22. What happens during downtime or degraded operation?
23. Can we export data, logs, configuration and model-version information?

Human oversight

24. Can the intended user understand, challenge and override the output?
25. How are uncertainty, overrides and escalations communicated and recorded?

Which sources does this guide rely on?

The regulatory points in this guide rely on primary EU sources and recognised clinical-AI reporting guidance, all accessed on 11 October 2026. Regulatory timelines change, so check the current text before relying on a date.

• Regulation (EU) 2024/1689, the EU AI Act (Articles 6, 25, 26), as amended by the Digital Omnibus on AI, Regulation (EU) 2026/1744.
• European Commission, regulatory framework for AI.
• Regulation (EU) 2017/745 on medical devices (MDR) and Regulation (EU) 2017/746 on in vitro diagnostic medical devices (IVDR).
• MDCG guidance, including MDCG 2019-11 on qualification and classification of software.
• Regulation (EU) 2016/679, the GDPR (Articles 6, 9, 28, 35, Chapter V and Recital 26).
• European Data Protection Board guidelines, including the guidelines on pseudonymisation.
• TRIPOD+AI reporting guideline (BMJ, 2024) and DECIDE-AI reporting guideline (Nature Medicine, 2022).

Frequently asked questions

Is a CE mark enough to approve a medical-AI pilot?

No. A CE mark confirms the conformity route for the product's stated intended purpose. It does not show local performance on your population, workflow fit, data-protection approval, security, or integration readiness. A hospital still needs local validation, a data protection impact assessment where required, contractual controls on data and updates, and a monitoring plan before using the system on real workflows.

What evidence should a medical-AI vendor provide?

A medical-AI vendor should provide an intended-use statement, dataset provenance, external validation, subgroup results with confidence intervals, a failure analysis, clinical evidence for the claimed benefit, security documentation, a data-flow description, a monitoring plan and a version history. A single headline accuracy figure on a retrospective dataset is not sufficient evidence for a clinical pilot.

Does a hospital need to validate an AI system locally?

Usually, yes. Performance shifts with patient mix, devices, acquisition protocols, prevalence and workflow, so vendor results from other sites may not transfer. The depth of local validation should scale with the intended use: a worklist prioritisation tool needs less than a triage or diagnostic tool. Agree the local-validation plan and its threshold before the pilot starts.

Can a medical-AI vendor use patient data to improve its model?

Only if the processing has a valid legal basis and an Article 9 condition under the GDPR, the hospital has agreed to it, and the contract says so explicitly. Many hospitals prohibit training on their data by default. Check the processor agreement, not the marketing page, and remember that pseudonymised data generally remains personal data.

Who is responsible for monitoring a purchased medical-AI system?

Both parties. The vendor, as manufacturer and AI Act provider, carries post-market surveillance, vigilance and update obligations. The hospital, as deployer, must use the system as instructed, monitor its operation, keep logs, assign competent human oversight and report problems to the vendor. The contract should state who watches what, and which thresholds trigger investigation or suspension.

What should cause a hospital to stop an AI pilot?

A hospital should stop an AI pilot when a pre-agreed stop condition is met: an unresolved legal basis for patient data, missing evidence for the intended population, outputs that cannot be audited, no effective override, an unannounced model change, security or integration risk beyond tolerance, or no measurable improvement against the baseline. Define these conditions before the pilot begins.