On the Path to
Trustworthy AI in Medicine

The structured knowledge base allows individual requirements from norms, standards, laws and practice to be entered.

The platform enables the assignment of project contexts to relevant quality requirements and the identification of suitable test resources.

Anticipated risks are automatically derived. Complex and critical cases can be forwarded to subject matter experts for review. The system simultaneously suggests appropriate risk mitigation measures.

The capabilities of the knowledge base are continuously expanded by documenting user feedback, test results and evaluations.

A unified terminology that brings together legal, product-related, and technical perspectives.

Abstract requirements are transformed into concrete, testable artifacts with end-to-end traceability and verifiability.

Risks and solutions are captured centrally once and can be used across projects and domains.

New knowledge is reviewed by subject matter experts before being incorporated into the knowledge base corpus – ensuring the highest quality and trustworthiness.

AI systems should not only provide explanations for their decisions - the utility and truthfulness of these explanations must also be verifiable. The vXAI Framework structures the evaluation of explainability methods by defining requirements for good explanations and providing a clear categorization scheme for metrics. This allows the quick and automatic selection of appropriate test tools for AI explainability, as well as the identification of gaps in the evaluability of explanations. In addition, a new metric has been developed that enables complementary perspectives for evaluating the truthfulness of attribution-based explanations with greater efficiency.

More information on this can be found on the VXAI Framework website and in the accompanying publication.

The more widely AI systems are used in critical decision-making processes, the more serious the effects of unfair bias become: systematic discrimination scales with technology and can lead to the structural disadvantage of entire population groups. Continuous and systematic fairness testing is therefore essential to ensure that AI decisions remain fair and that trust in these systems is justified. A concept for a Fairness-Testing-Pipeline represents an initial approach to the automated evaluation of the fairness of AI systems, taking full account of the application context. Automated testing thus covers various definitions of fairness and enables differentiated analyses of sources of unfair bias.

AI systems must be robust against changes in the system environment, data, and other disturbances to function reliably in safety-critical applications. A prototypical Robustness Toolbox enables systematic tests against adversarial attacks, data perturbations, and out-of-distribution scenarios. This allows vulnerabilities to be identified and rectified early before they cause problems in real-world deployment.

The protection of personal data is of crucial importance, especially in areas such as healthcare where particularly sensitive information is processed. A prototypical Privacy Toolbox provides a collection of tests for quantifying data protection risks such as Membership Inference and Model Inversion Attacks. This ensures that AI systems comply with applicable data protection regulations and that user privacy is maintained.

Inspired by existing methods for documenting AI systems such as FactSheets, the Application Card describes AI systems in such a way that technical interfaces become transparent while the application context is precisely captured. This is because the technically sound evaluation of trustworthy AI depends largely on the context in which the system is used.

The results of the Application Card are semi-automatically processed by the Test Registry to identify relevant and executable tests from the available portfolio. The Test Registry not only documents the applicability and requirements of the tests, but also provides interpretation aids for the results.

The Execution Engine eliminates manual checklists and prevents forgotten test cases. It takes over the execution – efficient, reproducible, and documented.

With ExAID and SkinDoc, two AI systems were developed to support dermatological diagnostics, which must meet high requirements for explainability, robustness, and fairness to be used in clinical practice. ExAID analyzes dermatoscopic images of skin lesions and provides explainable diagnoses for various skin diseases, while SkinDoc is designed as a mobile application for the early detection of skin cancer by laypersons.

The KITTU system supports urological oncologists by providing treatment recommendations for urothelial, renal and prostate cancer. It uses clinical information to make accurate predictions and stands out in particular for its ability to draw on external information to explain its decisions.

In collaboration with the Department of Psychology at the University of Trier, DFKI has developed a multi-agent-based AI system with transparent inter-agent communication for the automated support of evaluation procedures in the training of prospective therapists. The system is capable of generating automatic evaluations of therapists' verbal responses based on audio recordings from video sequences of therapy situations.

Intubations lead to complications in approximately 8% of cases. However, these complications can be fatal and occur unexpectedly in 90% of cases. Together with Saarland University Hospital (UKS), an AI-based system was developed that predicts intubation risks before operations based on endoscopy videos. The system achieved an accuracy of >90% in internal validation. Mechanisms for concept-based explainability were also implemented to identify previously unknown risk factors.

In collaboration with the German Red Cross (DRK), DFKI explored how AI can support smarter emergency response planning. The designed system uses spatio-temporal data and operational context to forecast where and when emergency services are likely to be needed. It also supports data-driven planning by predicting when ambulances on duty will be ready for the next mission. Through the simulation of different dispatch scenarios under real-world conditions, RESPOND enables straightforward comparison and validation.