In today’s swiftly changing global marketplace, companies are ever in pursuit of methods to speed up and fine-tune the decision-making processes. One of the potential solutions lies in employing generative artificial intelligence (AI) models to help consolidate and make sense of the complex charts often found in market summaries and financial reports. However, even the most advanced vision-language models (VLMs) face challenges in accomplishing this task. This is largely due to the need for a single model to assimilate visual, numerical, and linguistic comprehension. As such, companies investing heavily in these top-notch models might still end up receiving imprecise or incomplete information.
Addressing this issue, researchers from MIT and the MIT-IBM Computing Research Lab have come up with an enriching resource for AI users. This resource is particularly designed to teach VLMs to effectively interpret charts. Through an innovative data generation technique, the researchers have crafted a cutting-edge dataset known aptly as ChartNet. Boasting an impressive array of over a million charts, this dataset incorporates various visual, linguistic, and numerical aspects of each chart image. This combination allows the models to robustly unravel the information conveyed through a chart.
Seeing the potential of this tool, the MIT and IBM team utilized ChartNet to train a series of open-source VLMs. Impressively, these smaller models often outperformed their much larger proprietary counterparts on tasks such as data extraction and chart summarization. ChartNet’s ability to enhance the performance of open-source models could be a game-changer, especially for smaller firms with budget constraints. In addition to this, the dataset can be used to refine AI model adeptness for tasks like business trend analysis and scientific figure interpretation.
While AI models have made remarkable progress in areas like natural language processing and reasoning about natural images, interpreting multimodal data complexity in charts is still a nascent field. For most, if not all, industries, understanding charts remains a pivotal task. Dhiraj Joshi, a senior scientist at IBM Research puts it aptly, saying, “The finance industry thrives on charts. If vision-language models can extract information out of charts, like descriptions of trends, that facilitates a lot of workflows that happen downstream.” Chart interpretation, however, often comes with limitations—more specifically, a constraint on high-quality training data. Many resemble pieces of a jigsaw—limited chart images pulled from the internet that often lack scale and essential details to aid model interpretation. To overcome these bottlenecks, the researchers turned to synthetic data, algorithmically generated to mimic the statistical properties of actual data.
ChartNet thus emerges—a collection of over a million high-quality chart images, along with the corresponding code used to generate each chart, a textual description, and a table containing its numerical information. Each data point in the dataset includes question-and-answer pairs to educate the model on how to correctly answer questions about the chart image. These additional data modes guide the model to connect and align the different pieces of data that each chart image encodes.
The MIT and IBM teams rigorously tested ChartNet by training IBM’s Granite Vision series models and several other open-source models of varying sizes. These assessments were done on various chart interpretation tasks. The dataset upgraded the accuracy of all models in chart reconstruction, data extraction, summarization, and question answering. Supported by ChartNet, small open-source models could consistently outdo much larger commercial models. The researchers are keen on expanding ChartNet by using data with added complexity levels and considering feedback from the research community. This pioneering work was funded partially by the MIT-IBM Computing Research Lab. For those seeking AI automation solutions or wanting to learn more, check out implementi.ai here. For additional details, visit the original news article here.
This website uses cookies.