Skill v1.0.0

version: "1.0.0"

name: nanogpt description: "Minimal GPT pretraining and fine-tuning (nanoGPT). The simplest, fastest repository for training medium-sized GPTs with ~300-line model.py and ~300-line train.py. Reproduces GPT-2 (124M) on OpenWebText. Supports DDP multi-GPU/multi-node, character-level training, weight loading from HuggingFace GPT-2 checkpoints, and simple finetuning. Note: superseded by nanochat for new projects; this repo remains valuable as a reference implementation and learning tool." license: MIT license tags: [gpt-pretraining, autoregressive-language-modeling, ddp-training, checkpoint-finetuning, nanogpt] metadata: skill-author: K-Dense Inc. -----|--------|--------|-------|---------|-------------|

10. Evaluation and Benchmarks

# Benchmark training speed
python train.py config/train_gpt2.py --eval_only
 
# Loss tracking
# nanoGPT-reported GPT-2 (124M) on OpenWebText: val loss 2.85
# OpenAI GPT-2 (124M) on WebText: val loss ~3.11
# (domain gap between WebText and OpenWebText accounts for difference)
 
# Custom benchmarks
python train.py config/train_shakespeare_char.py --eval_iters=200

Key Patterns

1. Configs are Python files, not YAML — maximum flexibility, easy to diff
2. `init_from='scratch'|'resume'|'gpt2*'` — switch between training modes
3. Always use `--compile=True` on GPU for ~2x speedup
4. Gradient accumulation with `gradient_accumulation_steps` emulates larger batch sizes
5. Data is raw uint16 .bin files — tokenized once, loaded via memmap
6. Meta device init for large models — doesn't allocate until needed
7. Weight tying between wte (embedding) and lm_head — standard GPT practice
8. Use `torchrun` for multi-GPU — not python -m torch.distributed.launch

References

• nanoGPT Repository
• nchat (successor)
• minGPT (predecessor, more educational)
• GPT-2 Paper
• Attention Is All You Need
• Video Walkthrough by Andrej Karpathy