Language Models Learn to Improve Language Models

RL on rewards from a research world model, recalibrated with targeted real runs

Scroll sideways to see the whole diagram.

1. Ground the reward. A frozen GPT-6 Sol, the RWM, reads 100 real experimental records in context and predicts each proposal’s gain over the task’s reference recipe.
2. Train the proposer. Qwen3.8 27B is trained with DAPO for 96 steps on rewards built from the predicted gains.
3. Calibrate on-policy. After steps 16, 40, and 64, the proposals the RWM rates highest are built by GPT-6 Astra and run for real (31, 38, and 31 runs). Their measured outcomes join the RWM’s records, so the rewards keep up with what the proposer now generates.

On held-out tasks, LM2LM’s proposals yield 2.77× the average gain of GPT-6 Sol with 300 records in context

AutoregressiveDiffusion

Average of fourBest of four

Gain in 10⁻³ BPB

Training also removes harmful proposals: the share with negative gain falls from 70.4% to 39.2% on autoregressive tasks and from 59.7% to 7.6% on diffusion tasks.

All reported metrics

Proposer	Avg@4 ↑	Best@4 ↑	Negative % ↓	Wins vs. Base % ↑	LM2LM wins vs. row %

Follows the chart’s domain. Win rates are pairwise over states.

60 autoregressive and 18 diffusion held-out states. Paper Table 1; source data (JSON).

Rewards for 24,576 proposals cost $0.9k. Running them all would have cost about $4.8 million.

	Real runs		World-model predictions
LM2LM	200	$39.2k	24,576	$0.9k
In-context baselines	300	$60.0k	—
RL on real runs only	24,576	≈ $4.8M	—

LM2LM’s total, $40.1k, is under 1% of running every proposal for real and 33% below the in-context baselines.

Gain against cost at each budget

• LM2LM
• GPT-6 Sol + K records
• Base + K records

≈ marks an estimate that is not in the paper: 24,576 runs at the $196 average of the 200 executed. Costs are for data generation through step 96. Cost accounting in the paper (p. 23).

On a target family held out of training, LM2LM’s average gain is 2.48× that of the strongest baseline

In a separate experiment the whole Qwen3 target family is held out: no Qwen3 records in proposer training, RWM grounding, or the baselines’ context. The proposer learns from the other seven families.

Paper Table 2.

Without calibration, the world model’s ranking accuracy falls from 0.72 to 0.37, below chance

Training is extended to 128 steps and compared with a static variant whose RWM keeps its initial 100 records. Both are checked against real runs on held-out autoregressive states.

Which runs the budget is spent on matters

Static: the initial 100 records plus 100 random ones, never refreshed. Random: refreshed with random valid proposals. Paper Figure 4.

One overestimated change, corrected

A change to the number of tokens per optimizer update on NanoChat has no record in the initial set, and the RWM overrates it. With static records it grows to 79% of NanoChat training proposals by step 64. LM2LM’s first calibration runs seven such proposals, five of which degrade the reference recipe, and its share falls to 8%.

Case studies in the paper (p. 26).

What did not work, and what this does not show

• Fine-tuning on the 300 past records (SFT, RFT-Real) ends below the untuned base model on autoregressive tasks: −12.19 and −22.95 Avg@4, against −7.94.
• LM2LM trails GPT-6 Sol with records on two families, Marin and E2D2-SW. On NanoChat, GPT-6 Sol with records is the only method with a positive Avg@4.
• 39.2% of LM2LM’s autoregressive proposals are still worse than the reference recipe. For GPT-6 Sol with records the figure is 29.6%.

The study covers proposal-level improvements in language-model development, not end-to-end research campaigns. Results depend on the world model, builder, target environments, and experiment budget; they do not establish open-ended self-improvement.

40 training tasks and 10 held-out tasks across eight model families

Each task specifies a model family, size, training budget, and reference recipe. Every executed proposal trains its target model from scratch.

Task definitions in the paper (p. 17).

Cite the paper

@misc{lin2026languagemodelslearn,
  title={Language Models Learn to Improve Language Models},
  author={Minhua Lin and Zijun Wang and Zewen Liu and Zhaotian Weng and Bing He and Zhan Shi and Benoit Dumoulin and Xiang Zhang and Suhang Wang and Hanqing Lu},
  year={2026},
  eprint={2610.23344},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2610.23344}
}

arXiv:2610.23344, October 2026. Related study: Research World Models.