Our team is currently developing a solution called Cognitive Radiology Assistant - the next generation of cognitive assistant for radiologists. The system analyzes medical images and then combines this insight with information from the patient's medical records to offer clinicians and radiologists support for decision-making. By applying advanced reasoning and visual technologies, Cognitive Radiology Assistant filters out the most relevant images that point out abnormalities and provides insight into medical findings.

Today, emergency room radiologists may have to examine as many as 200 cases a day, with some studies containing as many as 3000 images. Each patient's imaging studies could be around 250GB of data, making an institution's image collection petabytes in size. This is where Cognitive Radiology Assistant steps in and contributes considerably to reducing diagnosis time and clinician fatigue. The need for Big Data analytics to help sift through all this medical data is clear.

Working to Make Healthcare Smarter Every Day

A TV ad showcasing Cognitive Radiology Assistant technology as the future for Watson for Imaging has received good reviews, with over nearly half a million views.


Efrat Hexter, Manager, Multimodal AI for Healthcare & Life Sciences, IBM Research - Haifa

Efrat Hexter,
Manager, Multimodal AI for Healthcare & Life Sciences,
IBM Research - Haifa

Cognitive Radiology Assistant - addressing the challenge of breast cancer

Worldwide, breast cancer comprises close to 30% of all diagnosed cancers in women, making breast cancer the second leading cause of death for women. Accordingly, a great deal of work has been performed to facilitate breast cancer detection—including advanced computer-aided diagnosis tools for mammography images. However, until now, little or no work has been done in the area of the advanced decision support tools that combine multi-modal image analytics and clinical data analysis.

From the technical point of view, the research includes advanced computer vision techniques to allow automatic extraction of diagnostically relevant features. Machine learning tools are also used to combine multimodal semantic image descriptions (for mammography, ultrasound and MRI) with clinical data, facilitating estimation of correct differential diagnosis and patient management recommendation.

Project Recognition / Visibility

This video presents IBM's Eyes of Watson demo for breast cancer detection, which was presented at the 2016 Annual Meeting of the Radiological Society of North America (RSNA). The demo highlights IBM's capabilities in medical imaging with a question-answer format. The Watson-based technology is designed to serve as a cognitive assistant to radiologists in their workflows.

Liver Imaging

Haifa researchers are working rogether with a major pharmaceutical company in Europe to develop a diagnostic support tool that will use AI to automatically detect, monitor, and support decision-making for the treatment of liver cancer.

Liver cancer is the second leading cause of cancer death worldwide. The liver is also a common site for metastases. The hope is that better characterization and earlier identification of these metastases could lead to an improvement in the chances of recovery.


Number Status Year Title Author
IL9-2013-0038US1 Filed 2013 Invariant Relationship Characterization for Visual Objects P. Kisilev, D. Freedman
IL8-2013-0159 Publish 2013 Lesion detection in medical images by cascade classification of bag of boundary features Pavel Kisilev, Ella Barkan, and Asaf Tzadok
IL9-2014-0009 Filed 2014 A method for artifact based segmentation of ultrasonic images, Eugene Walach, Dan Chevion Pavel Kisilev, Boaz Ophir
IL8-2014-0018US1 Filed 2014 Image Representation Set Eugene Walach, Sharbell Hashoul, Andre Heilper, Pavel Kisilev, Ella Barkan, Ami Ben-Horesh
IL9-2014-0017 Filed 2014 QUESTION GENERATOR Eugene Walach, Asaf Tzadok, Andre Heilper, Sharbell Hashoul, Ella Barkan, Pavel Kisilev
IL9-2014-0051 Filed 2014 A method for Clinical and Visual information fusion for disease diagnosis from lesion classification Pavel Kisilev, Eugene Walach, Asaf Tzadok
IL9-2014-0011US1 Filed 2014 Automatic Generation Of Semantic Description of Visual Findings In Medical Images Pavel Kisilev, Eugene Walach, Ella Barkan, Sharbell Hashoul
IL9-2014-0035 Filed 2014 Automatic image classification Sharon Alpert
IL9-2014-0003US1 Filed 2014 Anomaly detection in medical imagery Sharon Alpert and Pavel Kisilev
IL8-2014-0105 Filed 2014 Automatic tool for creation of comprehensive medical tests Eugene Walach, Aviad Zlotnick, Pavel Kisilev
IL8-2014-0111 Filed 2014 Statistical tool for assessment of physicians Eugene Walach, Pavel Kisilev, Sharbell Hashoul, Aviad Zlotnick
IL8-2014-0148 Filed 2014 Mini modes for object detection Sharon Alpert
IL8-2014-0151 Filed 2014 Unsupervised a-symmetry detection in images Sharon Alpert, Miri Erihov, Pavel kisilev
IL8-2015-0071 Filed 2015 Separation of foreground and background in a mammogram Aviad Zlotnick
IL8-2015-072 Filed 2015 A weakly labeled apparatus for breast tissue segmentation in digital mammography Rami Ben-Ari, Aviad Zlotnick
IL8-2015-0127 Filed 2015 A method for automatic visual annotation of radiological images from patient clinical data Barkan Ella, Kisilev Pavel, Walach Eugene
IL8-2015-0214 Publish 2015 Medical object proposals Sharon Alpert
IL8-2015-0126 Filed 2015 Higher complexity Deep Learning models with lower complexity training procedures, IBM legal department Pavel Kisilev,Pavel Kisilev
IL8-2015-0046 Filed 2015 Nuclear norm constrained method for convex Multikernel Learning Eli Meirom,Pavel Kisilev
IL8-2015-0144 Filed 2015 Scale-Space Label fusion using two-stage Deep Neural Net Pavel Kisilev, Eliayhu Sason
IL9-2016-0058US1 Filed 2016 Method for Creating Efficient Application on Heterogeneous Big Data Processing Platform Simona Rabinovici-Cohen, Flora Gilboa-Solomon, Eugene Walach
IL9-2016-0064US1 Issued patent 9880823 2016 Method for translating multi modal execution dependency graph with data interdependencies to efficient application on homogenous big data processing platform Simona Rabinovici-Cohen, Flora Gilboa-Solomon, Eugene Walach, Oren Barnea
IL9-2016-0059US Filed 2016 Reasoning based method for prioritization of clinical data to be displayed to the physician during the automatic summarization process Ella Barkan, Eugene Walach, Pavel Kisilev, Sharbell Hashoul
IL9-2016-0094US1 Filed 2016 An accurate and efficient computerized detection and localization of architectural distortion in mammography Rami Ben-Ari
SVL8-2016-0317 Filed 2016 Verifying annotations on medical images using stored rules Guy Amit, Flora Gilboa-Solomon, Murray Reicher
    2016 Automated anatomically-based reporting of medical images via image annotation Guy Amit, Flora Gilboa-Solomon, Murray Reicher
IL8-2016-0050 Filed 2016 Semantic description of medical findings using end-to-end deep neural network architecture with shared detection and description layers Sason Eliyahu
IL8-2016-0194US01   2016 Reduce discrepancy of human annotators in medical imaging by automatic visual comparison to similar cases Alon Hazan, Ella Barkan, Vadim Ratner
ARC8-2016-0149US01 Filed 2016 A platform for wrapping, registration, composition and execution of analytics in distributed environments Simona Rabinovici-Cohen, Amram Abutbul, Yu Cao, Ahmed El Harouni, Deepika Kakrania, Tanveer Syeda-Mahmood
ARC8-2016-0148US01 Filed 2016 Analytics framework for selection and execution of analytics in a distributed environment Simona Rabinovici-Cohen, Amram Abutbul, Yu Cao, Ahmed El Harouni, Deepika Kakrania, Tanveer Syeda-Mahmood
P-2017-02845US01   2017 Systems and methods for automatic detection of an indication of abnormality in an anatomical image Ran Bakalo
P-2017-05150US01   2017 Automated nipple detection in mammography Aviad Zlotnick, Guy Amit
P-2017-04896   2017 Object oriented image normalization Aviad Zlotnick
IL8-2017-0042US01   2017 Diagnostic decision support for patient management Eugene Walach, Ella Barkan
P-2018-00051US01   2018 Multi-task image classifier for classifying inherently ordered values Vadim Ratner