Diagram Understanding
Overview
Scientific documents such as papers, reports and patents but also other professional documents such as financial or medical reports very often include numerous diagrams. The goal of these diagrams is to illustrate, in a graphical way, data sets that explain, describe or emphasize the textual content of those documents.
Such data sets can be generated by experiments, measurements, observations or other means, and are uniquely depicted in these technical diagrams for the reader to extract the message they convey in a fast and efficient way.
With the emergence of Internet searches and archival storage, together with the speed at which new scientific documents are constantly being created, it is of great value to have tools that can not only scan numerous documents and extract their main scientific information automatically, but also present this information in a concise and meaningful way.
However, for a document to be completely and thoroughly analyzed, its diagrams also need to be processed in order to extract the key information presented by the depicted data sets.
The problem is that such graphs are typically stored as bitmap images, the data sets are often very noisy, and the graphic symbols used to depict the data, such as lines, markers and labeling, often overlap, intersect or otherwise override each other.
Cognitive analysis for extracting knowledge from graphics in documents
At IBM Research in Zurich, we are developing computational techniques based on image processing and machine learning to extract the data sets — and in turn the information they represent — automatically. From the taxonomy of various diagrams, we are currently focusing on line and scatter plots.
Conventional information retrieval focuses on text. Our challenge is to extract non-textual information from a variety of sources.
Business impact
Cognitive businesses base their decision-making process on insights extracted from the vast amount of available data.
Visual comprehension
Computers are blind to graphics in documents.
Data extraction
A tool that can automatically scan through a surfeit of documents, extract the main scientific information, and present it in a concise and meaningful way is of great value.
Comparison
Comparison between data sets across different documents enables
- Competitive study analysis,
- Establishment of trends.
Error correction
Increases confidence in extracted information.
Automatic graph generation
Valuable addition to document image analysis, document understanding and information retrieval in digital documents.
Ultimate goal
Semantic understanding of technical diagrams.
Challenges
Diversity
Diagram taxonomy: 10 types of diagrams plus flowcharts
Variability
Data acquired from public sources is huge
Complexity
Overlapping, overwriting, noisy
Lack of truth
No labels, no exact values
Hierarchical model
To analyze various sources such as graphs, diagrams and images, we first extract the basic objects, such as line segments, characters, and their topological relations to generate structural primitives such as lines, boxes, strings or grids.
Next, we map these basic objects into diagram semantics such as axes, labels or legends to create a kind of “graph grammar”.
Finally, we can extract valuable data from previously inaccessible sources.
Diagrams under study
The research group in Zurich studied line and scatter plots, molecular pathways, bubble plots, flow charts, scanned tables, and scanned forms.