Today’s medical institutions produce enormous amounts of data on patients, including multimedia data, which is increasingly produced in digital form. These data in their clinical context contain much information and experience that is currently not being used up to its full potential. Through the digital form the data has become accessible for automatic analysis and treatment for a variety of applications. At the same time, the variety of images produced can be confusing even for trained specialists causing an information overload exists for many medical doctors. This suggests that content–based image retrieval can be a valuable tool for helping manage these data and access the right information at the right time. This article gives a short state of the art of content–based medical image retrieval followed by a description of the medGIFT project on image retrieval with its main components. Then, several challenges are used to illustrate areas where much more work is currently ne...