Voice AI in healthcare: what the law actually requires

Four legal texts, and none of them mentions voice AI

A voice agent that books appointments for a practice or a hospital processes health data. That places it under GDPR Article 9, under the professional secrecy rules of national criminal law, and under whatever national regime governs the hosting of patient data — in France, the HDS certificate set out in Article L1111-8 of the Public Health Code. On the AI side, Regulation (EU) 2024/1689 does not classify it as high-risk. It carries one obligation: tell the caller they are speaking to a machine, from 2 August 2026. Every one of these texts predates the voice agent. None of them names it. The framework comes from their overlap, and that overlap is where vendors get it wrong — or get you wrong. Two claims come up in almost every sales call: that patient consent is the legal basis, and that the AI Act makes the product high-risk, which conveniently justifies a premium. Both are false. What follows walks each text, article by article, and ends with the paperwork to demand before you sign.

Article 9: an appointment is health data

The GDPR forbids processing health data as a matter of principle. That is Article 9(1). A call to a dental practice lands inside that prohibition. The caller’s name, the stated reason, the practitioner assigned — together they say something about that person’s health. So the question is never whether you may process. It is which exception you rely on. The right one is Article 9(2)(h). It permits processing necessary for medical diagnosis, for the provision of health or social care and treatment, and for the management of health care systems and services. Booking an appointment sits squarely there. Consent under Article 9(2)(a) is a trap. It must be explicit, freely given and withdrawable at any moment. A patient calling in pain is not in a position to refuse the switchboard, so the consent is not free. Rely on point (h) and record that choice in your processing register. Point (h) holds on one condition, set out in Article 9(3): the person processing the data must be subject to an obligation of professional secrecy. That is the next section.

Medical confidentiality does not stop at your vendor

Professional secrecy is not harmonised across the EU: it lives in national criminal law, and the penalties bite the individual, not only the company. France is a useful benchmark. Article 226-13 of its Criminal Code punishes disclosure of secret information by anyone who holds it by status, by profession, or by virtue of a role or a temporary assignment — one year in prison and a €15,000 fine. Article L1110-4 of the Public Health Code then draws the perimeter. The secret covers everything that comes to the knowledge of the professional, of any staff member of the facility, and of any other person who, through their activities, is in a relationship with that facility. Your voice AI vendor is that other person. It sits inside the perimeter of the secret, not beside it. Under the GDPR it is your processor, and Article 28 turns the contract between you into something other than commercial paperwork. Here is what it has to impose on the vendor.

• Process only on documented instructions from the controller
• Bind every person authorised to access the data to confidentiality
• Get written authorisation before engaging any sub-processor
• Delete or return the data when the contract ends
• Open its audits and supply the evidence of compliance

HDS: a French certificate that does not travel

France requires a dedicated certificate, HDS, for anyone hosting patient data on digital media on behalf of a third party. Article L1111-8 of the Public Health Code sets it out. France’s digital health agency, the Agence du Numérique en Santé, publishes every certified host together with the exact activities its certificate covers. Three words carry the scope: on behalf of. A facility hosting its own patients’ data needs no certificate; the host does. The certificate does not travel. Where no national equivalent exists — Germany is the case in point — the bar is Article 32 of the GDPR, security appropriate to the risk, plus the Article 28 contract and whatever national health rules apply. That sounds lighter. It is heavier in practice, because nobody hands you a certificate to read: you assess the security annex yourself. So ask for the same evidence a French buyer gets. Which entity operates the application layer. Which sub-processors touch call audio. Which independent audit report exists, and when it was issued. A vendor selling into France must already hold those answers.

AI Act: your booking agent is not high-risk

Regulation (EU) 2024/1689 sorts AI systems by risk level, and high-risk is a closed list. Annex III sets out eight areas: biometrics, critical infrastructure, education, employment, access to essential services, law enforcement, migration, justice. Healthcare appears once, in point 5(d), for systems that evaluate and triage emergency calls and for emergency patient triage. Booking an appointment is neither. The European Commission places conversational agents in the transparency-risk tier and states plainly that the Act introduces no rules for minimal-risk AI. What actually applies fits in two articles. Article 50 requires the provider to design the system so the person knows they are interacting with an AI, unless that is obvious — and the Commission reads that exception restrictively. Article 4 lands on you, the deployer. Regulation (EU) 2026/1744 rewrote it on 27 July 2026: you take measures to support AI literacy across your staff, without having to guarantee the level any individual reaches. Neither article is a conformity assessment. Neither justifies a high-risk price tag. That same regulation pushed high-risk back, not transparency. The disclosure to the caller still falls due on 2 August 2026. Annex III systems move to 2 December 2027. AI embedded in regulated products, medical devices included, waits until 2 August 2028.

• Support internal AI literacy: Article 4, in force since 2 February 2025
• Disclose the AI at pickup: from 2 August 2026
• Separate medical-device AI: high-risk only from 2 August 2028
• Face up to €15 million or 3% of worldwide annual turnover

What you must say at pickup, and what you cannot record

Two separate duties collide the moment the call connects. GDPR Article 13 requires the information to be given when the data is collected: controller identity, purposes, legal basis, recipients, retention period, and the patient’s rights. AI Act Article 50 requires you to say the caller is speaking to a machine. Both fit in one opening sentence, provided it points to a full notice available elsewhere. Recording is a separate processing operation with its own rules. France’s regulator has published its doctrine on call listening and recording. It rules out any permanent or systematic device unless a legal text mandates one — emergency services being the case in point. Callers must be told, at the time of the call, the purpose, the recipients, their right to object and their right of access. Recordings run to six months at most and analysis notes to a year, unless another text imposes a period. The regulator’s preferred practice goes further: review the recording within days, write the analysis note, delete the audio. A vendor keeping recordings to train its model is making you carry that processing. You are the controller.

DPIA: mandatory for hospitals, not for a solo practitioner

GDPR Article 35 requires an impact assessment before deployment whenever processing is likely to result in a high risk. Article 35(3)(b) names large-scale processing of Article 9 data. The open question is where large scale begins, and Article 35(4) hands the answer to each national supervisory authority, which must publish its list. France’s regulator published two, and they are worth reading whatever your member state. The first makes an assessment mandatory for health data processed by health and medico-social establishments for patient care — so a hospital wiring a voice agent into its switchboard runs one. The second exempts processing needed to care for a patient by a health professional practising individually in a practice, a pharmacy or a laboratory, and names appointment management explicitly. GDPR Recital 91 agrees: processing of patient data by an individual physician is not large scale. Between the two sits the group practice. No list names it. When in doubt, run the assessment — that is the regulator’s own advice, and it is cheaper than defending the omission.

Five questions to put to a vendor before signing

A compliant vendor answers each of these with a document, not with a sentence from a salesperson. Ask for the paperwork. A hosting certificate is readable: it names an entity, a scope of activities and an expiry date, and in France the national agency publishes the list you cross-check it against. A processor agreement is read line by line, against the five Article 28 duties listed earlier. A legal basis is cited by its article number, not by the word compliance. None of these five questions needs a lawyer. They need a written answer, and they are enough to sort the field. Send them by email rather than raising them in a meeting, because what matters is the trail. A vendor who routes you to its legal team and never comes back has already answered. And note the last one: the disclosure sentence is the only part of this apparatus the patient will ever hear, and you are the one who will have to stand behind it.

• Name the entity holding the hosting certificate and its exact scope
• Cite the legal basis: Article 9(2)(h), not consent
• Hand over the Article 28 processor agreement before signature
• Detail how long recordings live and whether they train models
• Write out the AI disclosure sentence the caller will hear

Sources

1. RGPD, article 9 — Traitement de catégories particulières de données (CNIL) https://www.cnil.fr/fr/reglement-europeen-protection-donnees/chapitre2
2. RGPD, article 13 — Information de la personne concernée (CNIL) https://www.cnil.fr/fr/reglement-europeen-protection-donnees/chapitre3
3. RGPD, articles 28 et 35 — Sous-traitant et analyse d’impact (CNIL) https://www.cnil.fr/fr/reglement-europeen-protection-donnees/chapitre4
4. RGPD, considérant 91 — Notion de traitement à grande échelle https://gdpr-info.eu/recitals/no-91/
5. Code pénal, article 226-13 — Légifrance https://www.legifrance.gouv.fr/codes/article_lc/LEGIARTI000006417944
6. Code de la santé publique, article L1110-4 — Légifrance https://www.legifrance.gouv.fr/codes/article_lc/LEGIARTI000043895798
7. Code de la santé publique, article L1111-8 — Légifrance https://www.legifrance.gouv.fr/codes/article_lc/LEGIARTI000049577902
8. Certification des hébergeurs de données de santé — Agence du Numérique en Santé https://esante.gouv.fr/labels-certifications/hds/certification-des-hebergeurs-de-donnees-de-sante
9. FAQ HDS, hébergement pour son propre compte — Agence du Numérique en Santé https://esante.gouv.fr/faq/nous-ne-sommes-pas-etablissement-de-sante-ce-sont-nos-clients-qui-hebergent-les-donnees-de-sante-de
10. Liste des hébergeurs certifiés HDS — Agence du Numérique en Santé https://esante.gouv.fr/offres-services/hds/liste-des-hebergeurs-certifies
11. Obligations de transparence de l’article 50 de l’AI Act — Commission européenne https://digital-strategy.ec.europa.eu/en/faqs/transparency-obligations-under-article-50-ai-act
12. Cadre réglementaire européen de l’IA et niveaux de risque — Commission européenne https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
13. Règlement (UE) 2024/1689, annexe III — Systèmes d’IA à haut risque https://artificialintelligenceact.eu/annex/3/
14. Règlement (UE) 2024/1689, article 4 — Maîtrise de l’IA https://artificialintelligenceact.eu/article/4/
15. Entrée en vigueur de l’AI Omnibus, 27 juillet 2026 — Commission européenne https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force
16. Ce que le règlement (UE) 2026/1744 change au calendrier de l’AI Act — Future of Privacy Forum https://fpf.org/blog/the-ai-act-implementation-timeline-what-changes-under-the-ai-omnibus/
17. Règlement (UE) 2024/1689, article 99 — Sanctions https://artificialintelligenceact.eu/article/99/
18. Calendrier d’application du règlement européen sur l’IA https://artificialintelligenceact.eu/implementation-timeline/
19. L’écoute et l’enregistrement des appels — CNIL https://www.cnil.fr/fr/lecoute-et-lenregistrement-des-appels-sur-le-lieu-de-travail
20. Enregistrement des conversations téléphoniques : informer ses interlocuteurs — CNIL https://www.cnil.fr/fr/cnil-direct/question/enregistrement-ou-ecoute-des-conversations-telephoniques-faut-il-informer-ses
21. Liste des traitements pour lesquels une AIPD est requise — CNIL https://www.cnil.fr/fr/liste-traitements-aipd-requise
22. Liste des traitements pour lesquels une AIPD n’est pas requise — CNIL https://www.cnil.fr/fr/liste-traitements-aipd-non-requise
23. § 203 StGB, Verletzung von Privatgeheimnissen — Gesetze im Internet https://www.gesetze-im-internet.de/stgb/__203.html
24. Kriterienkatalog C5:2026 für Cloud-Dienste — BSI https://www.bsi.bund.de/DE/Themen/Unternehmen-und-Organisationen/Informationen-und-Empfehlungen/Empfehlungen-nach-Angriffszielen/Cloud-Computing/Kriterienkatalog-C5/C5_2025/C5_2025_node.html
25. IA et santé : projet de guide HAS-CNIL, consultation close en avril 2026 — CNIL https://www.cnil.fr/fr/cloturee-ia-et-sante-la-has-et-la-cnil-lancent-une-consultation-publique-sur-un-projet-de-guide