Papperoni

2026

Mamba-3: Improved Sequence Modeling Using State Space Principles

Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, Albert Gu

citations

Cite Score

AI summary

This paper introduces Mamba-3, an improved state space model (SSM) using exponential-trapezoidal discretization, complex-valued SSMs, and multi-input, multi-output (MIMO) formulation, achieving significant gains in retrieval, state-tracking, and language modeling while improving inference efficiency.

Main Contributions

Introduces Exponential-Trapezoidal Discretization, a novel technique for discretizing time-varying, selective SSMs, which formalizes previous heuristic discretizations and provides a more expressive generalization.
Proposes a Complex-valued State Space Model for Mamba-3, enabling richer state tracking and overcoming limitations of real-valued linear models on synthetic tasks.
Develops a Multi-Input, Multi-Output (MIMO) SSM formulation to improve FLOP efficiency during decoding, increasing arithmetic intensity without compromising decoding speed.
Achieves significant gains in average downstream language modeling accuracy (up to 1.8 points at 1.5B scale) compared to baselines like Gated DeltaNet and Mamba-2.
Demonstrates improved hardware utilization and comparable perplexity to Mamba-2 with half its state size, advancing the performance-efficiency frontier.

Abstract

Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer models deliver strong model quality, their quadratic compute and linear memory makes inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule that enables richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency frontier.

Citation Graph

Loading graph...

References [58]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite