🔮 Autoresearch and the experimental society

On bringing the scientific method into everyday knowledge work

This post originally appeared in Exponential View and was written by Azeem Azhar.

“We often try to optimize something we never articulated.”

Science is the most reliable method humanity has found for producing knowledge. It has also, for most of history, been expensive to run.

Andrej Karpathy released 600 lines of Python code a few weeks ago that started to change that. His autoresearch (see EV#565) runs an autonomous experimental loop in which a human sets a strategic direction, defines what good looks like and the agent iterates towards success within the guardrails. In Andrej’s initial experiment, it trained a GPT-2-level model over two days, 11% faster and found 20 genuine improvements.

Andrej Karpathy on X

Shortly after the release, Shopify’s CEO Toby Lütke used autoresearch on his company’s internal model, qmd; it ran 37 experiments overnight, so Toby woke up to a 0.8-billion-parameter model outscoring his previous 1.6-billion-parameter version by 19%. Toby is not a machine learning engineer.