Papperoni

2025

Less Is More: Recursive Reasoning With Tiny Networks

Alexia Jolicoeur Martineau

Open PDF Google Scholar

citations

AI summary

This paper introduces Tiny Recursive Model (TRM), a novel recursive reasoning approach using a single tiny network, achieving strong generalization on Sudoku-Extreme (87.4% test accuracy), Maze-Hard, ARC-AGI-1, and ARC-AGI-2, surpassing the Hierarchical Reasoning Model (HRM).

Main Contributions

Introduces Tiny Recursive Model (TRM), a simplified recursive reasoning approach.
TRM achieves higher generalization than HRM on various tasks.
TRM uses a single tiny network, reducing the number of parameters.
TRM simplifies the halting process, removing the need for an extra forward pass.
TRM attains 87.4% test accuracy on Sudoku-Extreme.

Abstract

Hierarchical Reasoning Model (HRM) is a novel approach using two small neural networks recursing at different frequencies. This biologically inspired method beats Large Language models (LLMs) on hard puzzle tasks such as Sudoku, Maze, and ARC-AGI while trained with small models (27M parameters) on small data (~ 1000 examples). HRM holds great promise for solving hard problems with small networks, but it is not yet well understood and may be suboptimal. We propose Tiny Recursive Model (TRM), a much simpler recursive reasoning approach that achieves significantly higher generalization than HRM, while using a single tiny network with only 2 layers. With only 7M parameters, TRM obtains 45% test-accuracy on ARC-AGI-1 and 8% on ARC-AGI-2, higher than most LLMs (e.g., Deepseek R1, 03-mini, Gemini 2.5 Pro) with less than 0.01% of the parameters.

Citation Graph

Loading graph...

References [38]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[3]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[4]Decoupled Weight Decay Regularization

I. Loshchilov, Frank Hutter - 2017

7 papers in library cite

Very nice! Very simple to understand despite the complex topic. Very good results also

[5]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[6]Gaussian Error Linear Units (Gelus)

Dan Hendrycks, Kevin Gimpel - 2016

5 papers in library cite

Very understandable, and very nice! I don't think the justification is good, but hey, it works!

[7]Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei - 2020

12 papers in library cite

Very nice! An amazing contribution. Problem is, the paper is just like 3 pages of actual interesting content, and 10 pages of detailed results. Boring to read but very good otherwise.

[8]Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Noam Shazeer, Azalia Mirhoseini, K. Maziarz, A. Davis, Quoc Le, Geoffrey Hinton, Jeffrey Dean - 2017

9 papers in library cite

It's nice, but there's an important section in the middle about batch sizes that I don't quite understand. Not sure if I am missing some background knowledge or if they explain it poorly, and seems foundational to their main method... Either way, I did understand the methodology of the paper, and they have nice results :)

[9]Generalization of Backpropagation With Application to a Recurrent Gas Market Model

Paul J. Werbos - 1988

11 papers in library cite

The first two sections are amazing. Very nice concepts, well explained. From section 3 on it gets more abstract and too formalized, and it's mostly for Hopfield recurrent nets (so not very relevant now). Section 4 is just the application to the gas market, which I don't care for at all.

[10]Une Procédure D'Apprentissage pour Réseau à Seuil Asymétrique

Yann Lecun - 1985

5 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[11]Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei, Xinpeng Wang, Dale Schuurmans, Maarten Bosma, Fanyue Xia, E. Chi, Quoc V. Le, Denny Zhou - 2022

10 papers in library cite

CoT

[12]Palm: Scaling Language Modeling With Pathways

Aakanksha Chowdhery, S. Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, A. Roberts, P. Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann - 2023

6 papers in library cite

[13]Roformer: Enhanced Transformer With Rotary Position Embedding

Jianlin Su, M. Ahmed, Y. Lu, Siyuan Pan, W. Bo, Yibo Liu - 2024

2 papers in library cite

[14]Switch Transformers: Scaling to Trillion Parameter Models With Simple and Efficient Sparsity

William Fedus, Barret Zoph, Noam Shazeer - 2022

2 papers in library cite

The first ones to use MoE for LLMs

[15]Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters

C. Snell, Jaehoon Lee, K. Xu, A. Kumar - 2024

4 papers in library cite

[16]Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences

P. Werbos - 1974

14 papers in library cite

Werbos PhD Thesis, introduced backprop

[17]On the Measure of Intelligence

Francois Chollet - 2019

2 papers in library cite

[18]Arc-Agi-2: A New Challenge for Frontier AI Reasoning Systems

Francois Chollet, M. Knoop, G. Kamradt, B. Landers, H. Pinkard - 2025

1 paper in library cites

[19]Deep Equilibrium Models

Shuai Bai, J. Zico Kolter, V. Koltun - 2019

2 papers in library cite

[20]A Hierarchical Reasoning Model

Gloria Wang, Jeffrey Li, Y. S. Sun, X. Chen, C. L. Liu, Yonghui Wu, M. Lu, S. Song, Y. Yadkori - 2025

1 paper in library cites

[21]ARC-AGI Leaderboard

ARC Prize Foundation - 2025

1 paper in library cites

[22]Backpropagation Through Time and the Brain

T. P. Lillicrap, Adam Santoro - 2019

1 paper in library cites

[23]Beyond A*: Better Planning With Transformers via Search Dynamics Bootstrapping

L. Lehnert, S. Sukhbaatar, D. Su, Q. Zheng, P. Mcvay, Michael Rabbat, Yuandong Tian - 2024

1 paper in library cites

[24]Can Convolutional Neural Networks Crack Sudoku Puzzles?

K. Park - 2018

1 paper in library cites

[25]Fixed Point Diffusion Models

X. Bai, L. M. Kyriazi - 2024

1 paper in library cites

[26]Glu Variants Improve Transformer

Noam Shazeer - 2020

1 paper in library cites

[27]Grokking at the Edge of Numerical Stability

L. Prieto, M. Barsbey, P. A. Mediano, T. Birdal - 2025

1 paper in library cites

Missing year

[28]Hierarchical Graph Generation With k2-trees

Y. Jang, D. Kim, S. Ahn

1 paper in library cites

[29]Improved Techniques for Training Score-Based Generative Models

Yueqi Song, Stefano Ermon - 2020

1 paper in library cites

[30]Large Scale Gan Training for High fidelity Natural Image Synthesis

A. Brock, J. Donahue, K. Simonyan - 2018

1 paper in library cites

[31]MLP-Mixer: An All-MLP Architecture for Vision

I. O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, J. Yung, A. Steiner, D. Keysers, Jakob Uszkoreit - 2021

1 paper in library cites

[32]Recurrent Relational Networks

R. Palm, U. Paquet, O. Winther - 2018

1 paper in library cites

[33]Root Mean Square Layer Normalization

B. Zhang, R. Sennrich - 2019

1 paper in library cites

[34]Tdoku: A Fast Sudoku Solver and Generator

T. Dillion - 2025

1 paper in library cites

[35]The conceptarc Benchmark: Evaluating Understanding and Generalization in the Arc Domain

A. Moskvichev, V. V. Odouard, M. Mitchell - 2023

1 paper in library cites

[36]The Hidden Drivers of HRM's Performance on ARC-AGI

ARC Prize Foundation - 2025

1 paper in library cites

[37]The Implicit Function Theorem: History, Theory, and Applications

S. G. Krantz, H. R. Parks - 2002

1 paper in library cites

[38]Torchdeq: A Library for Deep Equilibrium Models

Z. Geng, J. Zico Kolter - 2023

1 paper in library cites

Cited by

0

papers in your library

Cites

18

papers in your library

Read

on October 6, 2025

Very nice alternative to LLMs - but still in early steps, I don't know if this is going to be something important in the future or not. Still seems like a small toy puzzle solver. However, it's good to see that there is still research into small models/datasets.

Tags

Paper Aliases

No aliases