What is your role in the REALM team at your institution?

Like most Work Package 2 members, I am the only person at my institution working in the REALM consortium. My role in WP2 is to help assess the ethical issues raised by the application of artificial intelligence systems in a healthcare context, and to use that knowledge to support the responsible development and design of the REALM architecture itself.

What are you currently working on within the scope of REALM and what are your main goals and objectives in the project?

With Emmanule (Ratti) and Ivett (Jakab), I have carried out a scoping review of ethical and societal issues raised by the development and use of AI in healthcare. While that manuscript is under review, we are now working on developing a short questionnaire and planning a follow-on workshop that will allow us to understand better the design choices that inform REALM itself. This task actually involves the whole WP2 team and is an exciting endeavour that helps us more from studying ethical issues in the abstract to trying to implement them in REALM actively.

What are some of the most interesting findings or insights that you have gained or are expecting to gain from your research in REALM?

I’m looking forward to seeing the completed questionnaires back from members of other Work Packages on REALM and working through them in a workshop setting. A big challenge with AI ethics has been that high-level ethical principles like ‚autonomy‘ or ‚dignity‘ usually seem very far removed from the everyday development or use of actual AI tools. It is impossible to design any technology without making design choices, and AI is no different. These everyday R&D decisions rarely involve ‚big picture‘ ideological or political motivations; more often we choose one method over another because it is easier, faster, cheaper, because it’s a standard method, or it‘s familiar, or some similar reason. These decisions still shape the eventual technology- in this case the REALM architecture- so they do have ethical impact, but these ‚microethics‘ (a term that Emanuele has helped develop) are not studied by ethicists or social scientists as often as they could be, so I am really hoping we can use this opportunity to make a practical contribution, not only to REALM but also for other AI projects on how to embed ethical considerations in the design and development process.

How do you see the future impact of REALM on the healthcare system?

I think REALM has the potential to be a very valuable tool for Notified Bodies and regulators faced with assessing a slew of new AI tools for use in the medical and healthcare settings, and provided we get buy-in from key stakeholders it could really take off.

What makes working on the REALM project special for you personally?

I have spent a lot of my career studying what you might call the political economy of biotechnology- stem cells, genetic modification, things like that. So for me, REALM has been an exciting chance to take some of the things I have learned about the social and ethical aspects of (medical) technology development and uptake and apply them to a new setting in relation to AI. Beyond that, I have great colleagues in WP2. So many large EU projects have maybe 100 bench or computer scientists and a single social scientist or ethicist employed to ‚do the ethics‘ and it is really very difficult to make any meaningful contribution when you are only one person in such a large team that is mainly focused elsewhere (in disciplinary terms), so the REALM approach where we have a whole team and a dedicated WP is really refreshing to see and be part of.

Why do you think the REALM research is important/will make a difference?

As I said, there are only going to be more AI systems developers for healthcare settings, many of them also qualify as medical devices, and regulators and notified bodies will need tools to help them assess the quality and safety of these tools. New laws such as the AI Act can mandate new standards, but there is still a need to support practical assessment, and that is exactly what the REAL architecture aims to do, so the consortium is addressing a real underserved need out there. It’s also important for Europe to develop its own assessment practices and tools that we can apply to AI systems regardless of who developed them or where in the world they originate.

What do you do when you're not working?

One of my favourite things to do when not doing research is playing board games. I co-host a bi-weekly board games night for the University of Oxford research staff. We have a small fund to increase our library of board games and it’s just a nice social event where I get to meet researchers from all sorts of different places and work on everything from excavating ancient pottery to cloning endangered rhinos.

What is your role in the REALM team at your institution?

Following up on the ethics-related work done in WP2 on behalf of the University of Antwerp. Provide support for the development of an effective means to engage ethics work with other work done in the REALM project. I also collaborated with some other WP members on the development of a qualitative framework for the re-identifiability of synthetic health data.

What are you currently working on within the scope of REALM and what are your main goals and objectives in the project?

Together with other members of WP2, I am working on the development of an ethics engagement format which aims to investigate values that go into research prioritization and methodological design choices within the REALM project.

What are some of the most interesting findings or insights that you have gained or are expecting to gain from your research in REALM?

In the work on the synthetic health data evaluation framework (QuASH), I found it really interesting to think about the inherent friction between fidelity and difference in the conception of synthetic data. Inherently, the need for a synthetic dataset implies that it needs to be different enough from the original dataset, however, it also needs to be similar enough to the original training dataset for it to be used as a stand-in for that dataset. I think this is an interesting friction and balancing exercise that needs to be done in the development of a synthetic dataset.

How do you see the future impact of REALM on the healthcare system?

If REALM is able to smoothly integrate with current European regulatory systems, needs and requirements, it might be able to lighten the load for both medical software developing entities and Notified Bodies in the process of certification applications.

What makes working on the REALM project special for you personally?

As data and AI ethics increasingly become part of bioethics, REALM allows me to grow more familiar with medical data use, software development and evaluations, which in turn can substantiate my bioethics research.

What do you do when you're not working?

One of my favourite things to do when not doing research is evening city walks with a podcast.

What are currently reading about?

Right now, I’m reading about ethics in science communication.

What is your role in the REALM team at your institution?

I am a team member and Leader of Work Package 2 - Ethical, Legal & Societal Framework.

What are you currently working on within the scope of REALM and what are your main goals and objectives in the project?

Currently, I am working on legal and ethical aspects of AI technologies in the context of the EU regulatory framework. Together with Tomasz Braun (LU), we investigate the regulatory structures of the AI Act and MDR in order to establish mutual relations between two regulations relevant to REALM. Our goal is to deliver a comprehensive study on this issue. Also, I work on synthetic data from the legal perspective as it is the field where important regulatory gaps can be noticed, despite its relevance for the AI sector.

What are some of the most interesting findings or insights that you have gained or are expecting to gain from your research in REALM?

My research in REALM provides me with the opportunity to broaden my knowledge and research perspective. There are new areas, such as synthetic data and their use in healthcare care, that require deeper regulatory analysis. Also, working in an interdisciplinary team both in WP2 (ethicists, sociologist, economist) and in REALM in general is something very enriching and allowing to gain insights from different disciplines.

How do you see the future impact of REALM on the healthcare system?

I see it as a useful tool that allows better regulatory control and compliance of novel medical devices.

What makes working on the REALM project special for you personally?

Working on the REALM project is a great career opportunity, allowing me to connect with great researchers, and run cutting-edge research in various fields. It enriches my methodology, and knowledge and gives me the opportunity to manage a research team.

Why do you think the REALM research is important/will make a difference?

It is an ambitious project whose outcomes will be not only of academic but also of practical character. Hopefully, REALM will be gaining relevance as much as AI powered medical devices sector will be growing in the EU and beyond. REALM will provide important regulatory tool for AI medical devices, improving not only their compliance with existing regulatory frameworks, but also by enhancing trust of their users to new technologies.

What do you do when you're not working?

One of my favourite things to do when not doing research is traveling, preferably to places where you can ski, horse ride, surf and hike (not necessarily all at the same time).

What is one thing you could not live without?

One thing I cannot live without is music – it must be good, but it can vary from opera, classic, jazz, rock, electronic, and indie. I am a real music addict 😊

What is your role in the REALM team at your institution?

I am the work package lead of WP5, and the Task lead of T3.4 and T5.4.

What are you currently working on within the scope of REALM and what are your main goals and objectives in the project?

My main goals and objectives are to enable a better evaluation of the performance of software as a medical device. Both, before deployment (in T3.4) and after deployment of the model in the clinical practice (in T5.4). Today, medical practitioners too often have to decide to use, or not to use, a medical device software without being accurately informed on how well this software will perform in their clinical practice on their patients. In recent years, we have seen many examples of clinicians putting trust in diagnostic or prognostic software when they should not have. Putting the health and well-being of their patients at risk.

What are some of the most interesting findings or insights that you have gained or are expecting to gain from your research in REALM?

Diagnostic and prognostic software are often presented to the world with a certain performance obtained on the test set. In practice, those models have many performances, depending on the context and patients that they get used on. I expect that REALM will make clear that only presenting an accuracy metric has limited value. REALM will teach us how to combine fitted performance metrics for different fine-grained subpopulations and usage contexts. REALM will also strengthen the inclusion of metrics such as robustness and fairness metrics to enable a better prediction of the real-world model performance.

How do you see the future impact of REALM on the healthcare system?

I have already talked with quite a few clinicians who feel disappointed. Five years ago, there was plenty of hype around many novel AI-based health applications which were going to produce a revolution in the healthcare sector. Today, many see little of this promise realized. REALM has the potential to rebuild trust in novel diagnostic and prognostic software so they can find their way to clinical practice. But this time without endangering patients' health through undeserved trust in the software.

What makes working on the REALM project special for you personally?

I used to be one of the scientists creating diagnostic and prognostic models. My team and I created deep learning models for the ophthalmology field which we brought to the market with our spin-off called MONA (https://mona.health). It is here that I got personally confronted with today’s reality that medical practitioners get very little guarantees from their software suppliers that the models will perform well in their clinical practice. Mitigating this issue became my new research interest.

Why do you think the REALM research is important/will make a difference?

Clinicians should get access to evaluations of high quality for the diagnostic and prognostic software they consider to use in their clinical practice. REALM has the mission to enable this by providing a high-quality evaluation framework using real-world and synthetic data.

What is one thing you could not live without?

High-speed Internet

What is your role in the REALM team at your institution?

My role in the REALM project is the management of the project needs, both in the administrative, organisational and implementation tasks, along with delineating the technical and research dimensions of the work, in line with the Project’s objectives. Along with my team, we strive to produce novel and quantifiable results that push the boundaries of the state-of-the-art in the scope of our work in REALM.

What are you currently working on within the scope of REALM and what are your main goals and objectives in the project?

The current work of me and my team is the creation, optimisation and curation of the Federated REALM Architecture, that defines the operational scheme and flow of the REALM platform. Along with this, we are also focusing on providing the Explainable AI (XAI) evaluations for the REALM platform that will undertake the provision of humanly interpretable explanations of the reasoning and predictions of medical AI models, in the fields of tabular, image and volumetric data. Lastly, we are involved in the synthetic data generation process for the platform and in particular with the post-market evaluation of generated data, ensuring alignment and compliance with realistic benchmarks and standards.

What are some of the most interesting findings or insights that you have gained or are expecting to gain from your research in REALM?

Through the REALM project we have found interesting and important insights about the application of AI in the different fields of medicine, while understanding better the need for a unified way to evaluate and certify them for production. This stems from the unwavering dedication of the REALM stakeholders and ethical monitoring, highlighting the requirements for robust medical AI, while creating a common glossary to bridge engineering, regulation, and medicine. Additionally, our work on XAI and Post-market evaluation continues to provide significant clues to the working and application of both AI models on medical data and the correct generation and evaluation thereof.

How do you see the future impact of REALM on healthcare system?

I feel that REALM will pave the way for an easier and more accessible assessment of medical AI and software, not only to commercial stakeholders but also to regulatory bodies and independent researchers. This will effectively enable a quality shift for these assets as they will have access to a wider range of field-specific assessment, reporting and data, that would otherwise be custom or even inaccessible to the wider public.

What makes working on the REALM project special for you personally?

Working with experts and professionals in the field is a great way to learn and deepen your connection to your work. This is especially the case for research as, for me it is not just work, it is a way to acquire a better understanding of my science. Also, it is fun. Every day I learn something new, concise in the knowledge offered by my colleagues in the project. Furthermore, the objective of REALM is on great interest to me, as it stands on the forefront of one of the most impactful sciences in the world, medicine.

Why do you think the REALM research is important/will make a difference?

It is my opinion that collaborating and developing solutions for REALM can have a great impact in both the commercial and academic worlds.

If you could have any superpower, what would it be?

If I could have any superpower, it would be the ability to fly or teleport to any place on Earth.

What is your role in the REALM team at your institution?

I am the lead of WP4, managing and coordinating tasks within WP4 and making interlinks with other work packages. I actively work with WP3 and WP5 to make sure the data part is well-connected with REALM infrastructure and models parts.

What are you currently working on within the scope of REALM and what are your main goals and objectives in the project?

I am currently working on the synthetic data component which we intend to generate realistic medical data based on the real data from our demonstrators. I am also working on connecting the technical pieces (data, model, and infrastructure) and to apply to the use cases with our demonstrators. My main goals in the project are to ensure all the data including real and synthetic data are well-prepared, managed, and used for REALM infrastructure.

What are some of the most interesting findings or insights that you have gained or are expecting to gain from your research in REALM?

The most interesting insights I would like to gain are 1) if the evaluation outcome from REALM infrastructure is the same as the evaluation from the source of AI models, 2) how much synthetic medical data can help in the AI model evaluation, 3) how all the stakeholders such as health professionals, model developers, researchers, policymakers will collaborate to make such infrastructure (i.e., REALM) well-implemented and applied.

How do you see the future impact of REALM on the healthcare system?

REALM will increase the reliability and transparency of AI models in healthcare and indirectly gain more trust from patients and health professionals in AI models. Importantly, REALM starts a future where AI models can be objectively evaluated in a standardized way. REALM will also encourage AI developers, companies, and researchers to be more responsible and accountable for their AI products.

Wat makes working on the REALM project special for you personally?

I think collaborations with all partners are very valuable and special to me. All our colleagues in the project are very talented at what they are doing and open to exchanging ideas. I enjoy working with competent people and learning to be a good WP lead.

Why do you think the REALM research is important/will make a difference?

REALM is creating a new and innovative infrastructure where we can conduct transparent and accountable evaluations for AI models. Like all the other products which can be assessed and evaluated following well-recognized standards, REALM creates the same system for AI models so that our citizens and health professionals can be informed about how these models perform under standardised evaluations. REALM will build more trust from the general public in AI models.

What is one thing from your bucket list?

One thing on my bucket list is travelling from the Netherlands to China by train.