Publications of Torsten Hoefler

Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu:

When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

(In Proceedings of the 43rd International Conference on Machine Learning, Jul. 2026, arXiv:2606.01155. )

Abstract

Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: 1. Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. Resource trade-offs: With fixed data, loss-optimal sparsity is moderate ~ 50%, while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: this https URL.

Documents

download article:

download slides:

BibTeX

@inproceedings{,
  author={Boqian Wu and Qiao Xiao and Patrik Okanovic and Tomasz Sternal and Maurice van Keulen and Mykola Pechenizkiy and Elena Mocanu and Torsten Hoefler and Decebal Constantin Mocanu},
  title={{When Data Is Scarce: Scaling Sparse Language Models with Repeated Training}},
  year={2026},
  month={Jul.},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning},
  note={arXiv:2606.01155.},
  source={http://www.unixer.de/~htor/publications/},
}


serving: 216.73.216.232:53886	© Torsten Hoefler