Training large AI models can be a hefty task, both in terms of cost and resources involved. The traditional approach towards creating a compact, faster model involves either training an extensive model first and then cutting it down or creating a smaller one from scratch, but compromising on performance. However, a collective of researchers from leading institutions have come together to create a revolutionary method to address this.
Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Max Planck Institute for Intelligent Systems, European Laboratory for Learning and Intelligent Systems, ETH, and Liquid AI have displayed an innovative technique right out of the traditional spectrum called CompreSSM. It targets the AI architecture family known as state-space models, benefitting applications from varied sectors such as language processing and robotics.
The brilliant part about CompreSSM is that it leverages mathematical tools from control theory to determine the active and idle components of a model. Once identified, it removes the unnecessary ones during its early stages of training. The technique is not just about making a model smaller and faster, but it’s about adaptation – learning and weeding out the non-contributing parts during its developmental phase.
“The relative importance of different components within these models stabilizes surprisingly early during training,” shared Makram Chahine, one of the researchers and CSAIL affiliate. How they measure this contribution is through a mathematical system called Hankel singular values, ranking the dimensions of importance only after ten percent of the training process.
The entire process has produced some dramatic outcomes. To put it in perspective, the compressed models could maintain almost similar accuracy levels as the full-sized models while training 1.5 times faster. Even a model shrunk to approximately a quarter of its original state size hit an 85.7 percent accuracy during the CIFAR-10 benchmark. A critical factor here is that a model of the same size trained from scratch could only hit 81.8 percent.
What makes CompreSSM distinct is its theoretical approach, which is more informed and accurate compared to the existing ones. Traditional methods train the whole model and then strip off the parameters, which costs almost the same as training a large model. CompreSSM, with its mathematical backup, can make calculated compression decisions midway through the training process. These models prove that the importance of individual model states changes smoothly during training and that their relative rankings are stable, giving developers the confidence that dimensions considered negligible early on, won’t suddenly become vital later.
While CompreSSM holds a promise, it’s important to note that it works best on models that show a strong correlation between the internal state dimension and the overall performance. The method is most effective on multi-input, multi-output (MIMO) models. For models that are single input-output based, the gains are modest since those models are less responsive to state dimension changes.
The team of researchers sees this accomplishment as a stepping stone. They have already managed an extension to linear time-varying systems such as Mamba and are looking to push CompreSSM further into matrix-valued dynamical systems, bringing the technique closer to transformer architectures that underpin most of today’s largest AI systems.
Antonio Orvieto, an investigator who was not part of the research, stated, “The work opens new avenues for future research, and the proposed algorithm has the potential to become a standard approach when pre-training large SSM-based models.”
The work of these researchers was accepted as a conference paper at the International Conference on Learning Representations 2026. Supported by the Max Planck ETH Center for Learning Systems, the Hector Foundation, Boeing, and the U.S. Office of Naval Research, more details about this development can be found here.
Do you own a company in need of AI automation? Check out implementi.ai for innovative solutions carefully designed to suit your business needs. It certainly appears the field of artificial intelligence and its applications are evolving and breaking traditional barriers, optimizing the processes for the better.
This website uses cookies.