Optimizer efficiency
Research question
Second-order methods can take fewer steps to converge while spending more time on each step. I'm testing whether they reduce total pretraining time after accounting for that overhead.
Experimental design
The planned grid covers 100M- and 300M-parameter Mamba2 and Transformer models, trained with AdamW, Muon, Shampoo, and K-FAC. Approximately 40–50 configurations will use identical training documents. The main measurement is elapsed time to a validation loss of 2.5 on a FineWeb-Edu holdout set, alongside steps to target and time per step. The loss threshold is a working target, not a result.
Status
The study is ongoing, with no results available yet. My pytorch-opt library supplies the optimizer implementations and parameter routing for the experiments.