SLM
Training
I trained the 538M-parameter decoder-only Transformer in PyTorch, then applied supervised fine-tuning and direct preference optimization. The separate 535M-parameter Mamba2 model has only been pretrained. Their training histories differ, so these runs are not a controlled comparison of the architectures.
Data and tokenizer
The Transformer uses a custom byte-level BPE tokenizer. The public repository includes a synthetic-corpus pipeline for testing whether more controlled data improves coherence at this model size. It covers the Transformer and data pipeline, but does not yet document all of the Mamba2 work.