“A well-written AI report isn’t automatically correct.”

This was mentioned in a recent article by ICT&health about the RIGH:T consortium. In this consortium, healthcare institutions (including Erasmus MC, UMCG, LUMC, UMC Utrecht, and ETZ) are collaborating with TU Delft to develop a practical framework for validating generative AI in healthcare.

A demo by the vendor, satisfied users, or time savings do not prove that AI works well. The healthcare institution remains responsible for weighing the risks. This requires clear agreements with vendors, according to the consortium.

And we think that’s absolutely right. That’s why we’ve made documentation available for our AI summary module that allows healthcare institutions to assess how we manage these risks, such as:

  • DPIAs and Risk Analysis
  • Product Description: What Our AI Summary Does and Doesn't Do
  • Description of Our Validation and Evaluation Method
  • How We Test and Release Changes to Prompts and Models
  • Data Processing Agreement and Description of Data Processing (Our AI runs in the Netherlands, and patient data is not used to train the model)


Why do we take this so seriously?
Because we recognize the risk ourselves. Our AI acts as a writing assistant and converts the patient’s responses into a summary. The doctor reads this, makes adjustments where necessary, and approves it before the text is added to the patient’s medical record. The AI does not make decisions. And especially with a text that reads smoothly, it’s less likely to be noticed if something is missing, such as an allergy or a medication. An error lies not only in what is written, but also in what is not written.

That’s why:

  • Let's evaluate each clinic individually: What works for the Cardiology clinic doesn't necessarily work for another clinic.
  • We check not only for inaccuracies but also for missing information, using a growing set of test reports.
  • Do we test every change to a prompt or model before it goes live?
  • Let's let doctors help decide what should be included in a summary.

Curious about how we do this? Feel free to contact us. You can find the link to the ICT&health article here.