AI-Driven Image Analysis: How Metrics Determine Quality

More and more areas of medicine are relying on artificial intelligence (AI). This is particularly true for the many questions based on the evaluation of image data: for example, doctors search mammograms for tiny cancer foci or calculate the volume of a brain tumor based on tomographic images of an MRI.

They use endoscopic images of the intestine to detect polyps, and when evaluating microscopic tissue sections, subtle changes in individual cells should be detected.

But are the algorithms used for these different types of image analysis really always suited to the task at hand? This depends to a large extent on which measured variables, called “metrics” in technical terms, they record and whether they actually suit the task in question.

“We often find that validation parameters are used that are not at all relevant to the task from a clinical point of view,” explains Lena Maier-Hein from the DKFZ, citing an example: “When searching for metastases in the brain, it is firstly more important that the algorithm detects even the smallest lesions and can define the contours of each individual metastasis with high precision.

Lena Maier-Hein and her colleagues fear that the use of inappropriate validation parameters could hinder scientific progress and delay the introduction of important image analysis methods into clinical practice.

But which parameters are suitable for a given clinical question, taking into account all strengths, weaknesses and limitations? To find out, DKFZ data scientists used a structured, multi-step process to interview opinion leaders from academia and industry from more than 70 research institutes around the world. The survey allowed them to gather information that was previously only available in scattered locations around the world.

“With this work, we are making reliable and comprehensive information available to experts for the first time on the problems and pitfalls associated with validation measurements in image analysis,” says Annika Reinke, one of the main authors.

As a structured body of information accessible to researchers across disciplines, the work aims to increase understanding of a key issue in AI-assisted image analysis. Although the focus is on medical image analysis, the information can also be transferred to other areas of image analysis.

In a second article, the expert consortium led by the Heidelberg researchers now describes “Metrics Reloaded”: a comprehensive framework to help doctors and scientists select the appropriate measures for the problem. “Metrics Reloaded” can be used as an online tool.

“Users are guided through a comprehensive set of questions to create an accurate fingerprint of their image analysis problem. The tool also draws attention to specific problems that arise in certain biomedical questions,” explains Paul Jäger , one of the main authors of both publications.

Metrics Reloaded is suitable for all different categories of image analysis problems, i.e. image classification, object detection or the assignment of individual pixels (semantic segmentation). The tool works completely independently of the image source, so it can be used for both CT or MRI images as well as microscopic images. Metrics Reloaded is also suitable for image analysis beyond biomedical issues.

“Metrics Reloaded is the first systematic guide that shows users of AI-based image analyzes the path to the right algorithm. We hope that Metrics Reloaded will be used as widely as possible and as quickly as possible, as it could “Significantly improve the quality and reliability of AI-assisted image analysis results. This would also promote confidence in AI-assisted image analysis in routine clinical practice,” explains Minu Tizabi, one of the main authors.

The research was published as two articles in Natural methods.