This is the best summary I could come up with:
Researchers claim to have developed a new way to run AI language models more efficiently by eliminating matrix multiplication from the process.
The technique has not yet been peer-reviewed, but the researchers—Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason Eshraghian—claim that their work challenges the prevailing paradigm that matrix multiplication operations are indispensable for building high-performing language models.
They argue that their approach could make large language models more accessible, efficient, and sustainable, particularly for deployment on resource-constrained hardware like smartphones.
In the paper, the researchers mention BitNet (the so-called “1-bit” transformer technique that made the rounds as a preprint in October) as an important precursor to their work.
According to the authors, BitNet demonstrated the viability of using binary and ternary weights in language models, successfully scaling up to 3 billion parameters while maintaining competitive performance.
Limitations of BitNet served as a motivation for the current study, pushing them to develop a completely “MatMul-free” architecture that could maintain performance while eliminating matrix multiplications even in the attention mechanism.
The original article contains 499 words, the summary contains 177 words. Saved 65%. I’m a bot and I’m open source!
Someday, we’ll have the technology to generate an image of a centaur with 4 boobs without using more energy than a small hospital. Very exciting stuff.
I obviously got it. But not everyone appreciates high culture.
This is interesting but I’ll reserve judgement until I see comparable performance past 8 billion params.
All sub-4 billion parameter models all seem to have the same performance regardless of quantization nowadays, so 3 billion is a little hard to see potential in.
