Karpathy’s autoresearch and LLM Wiki

Andrej Karpathy published autoresearch, a repository that gives coding agents a repeatable training experiment on a small language model. His LLM Wiki publication describes a way to maintain a Markdown knowledge base from source documents. One accumulates experiment results, the other accumulates notes tied to evidence.

01The five-minute autoresearch experiment

The agent edits train.py, which contains the model and training loop. Data preparation and evaluation live in prepare.py. A human sets the experiment instructions in program.md. Each run gets five minutes of training time, excluding startup and compilation. The score, val_bpb, measures validation bits per byte, with lower values preferred. Use a baseline from the same GPU since different hardware can process different amounts in the same time. Original autoresearch repository

%% The autoresearch experiment loop
flowchart TD
  A["Edit train.py"]
  A --> B["Train for 5 minutes on the same GPU"]
  B --> C["Compare val_bpb with the prior result"]
  C --> D["Keep or revert and log the outcome"]
  D --> A

02Record a baseline first

The upstream quick start expects a single NVIDIA GPU, Python 3.10 or later and uv. Follow its steps for preparing the data and tokenizer, then run the default training job once and save the result. On a separate branch, set a limit on the number of attempts and log each change, score and failure. This guide does not report a GPU training run. Upstream experiment procedure

03Try LLM Wiki with three documents

Karpathy released LLM Wiki as a design you can implement with an agent. Keep the original documents and let the agent maintain linked topic pages alongside them. Pick three public documents and start in a small folder. Karpathy’s original LLM Wiki write-up

%% How to trace a wiki claim back to its source
flowchart TD
  A["Keep the original public documents"]
  A --> B["Update topic pages and the index"]
  B --> C["Follow the source link to verify a claim"]

Read the three documents in sources and leave the originals intact.
Create topic pages and an index in wiki. Attach source paths to claims.
If sources disagree, retain both accounts and explain the difference.

Pick one claim from the wiki and find it in the original document. Then add one source and inspect how the existing explanations and links change.