Semantic medical image retrieval in a medical social network

Medical social networks have become an exchange of opinions between patients and health professionals. However, patients are anxious to quickly find a reliable analysis and a concise explanation of their medical images and express their queries through a textual description or a visual description or both sets. For this, we present in this paper a multimodal research model to research medical images based on multimedia information that is extracted from a radiological collaborative social network. Indeed, the opinions shared on a medical image in a medico-social network are a textual description which in most cases requires cleaning by using a medical thesaurus. In addition, we describe the textual description and medical image in a TF-IDF weight vector using an approach of “bag of words”. We use latent semantic analysis to establish relationships between textual terms and visual terms in shared opinions on the medical image. The multimodal modeling researches the medical information through multimodal queries. Our model is evaluated against the ImageCLEFMed’2015 baseline, which is the ground truth for our experiments. We have conducted numerous experiments with different descriptors and many combinations of modalities. The analysis of results shows that the model based on two methods can increase the performance of a research system based on a single modality, visual or textual. © 2016, Springer-Verlag Wien.