It taught itself: one generation of self-play
The network played 10,000 games against itself and a new network learned from them: +82 Elo over its parent. Then it levelled off.
The first network learned from games the classic engine played, so it could only be as good as the positions that engine reached. The next step is the idea behind AlphaZero: let the network play itself, learn from those games, and repeat. Each generation should see better games than the last.
The loop
A shell script, ml/selfplay-loop.sh, runs one generation at a time:
- The current network plays 10,000 games against itself, searching 20,000 positions a move.
- A new network trains on the last two generations' games.
- It plays 800 games against its parent and 800 against the classic engine at equal time, and takes over only if it beats its parent.
It ran unattended for a few hours on a small 4-core server. Two things made that practical. The faster evaluator cut a generation from over an hour to about 40 minutes. And a status script prints the generations so far, the current stage and a progress line, refreshed every ten seconds over SSH.
Why a CPU, not a GPU
Almost all the time goes into playing games: a tree search that asks the network about one position at a time, 0.7 microseconds each. Shipping a single position to a GPU and back takes longer than computing it. GPUs pay off when a big network sees thousands of positions at once, which is how AlphaZero is built. This engine follows the Stockfish NNUE design instead: a small network, a deep search, all on the CPU. Training itself takes about 20 seconds per generation.
Results
| Generation | vs its parent | vs classic |
|---|---|---|
| 0 (classic engine's games) | +112 ± 13 | |
| 1 | +82 ± 13 | +148 ± 14 |
| 2 | −2 ± 24 | +107 ± 25 |
| 3 | −10 ± 24 | +99 ± 25 |
| 4 | −9 ± 12 (3,000 games) | +120 ± 25 |
One generation was worth +82 Elo. The network now takes 70% of the points against the hand-tuned engine over 3,000 games, and it is the brain the analysis board uses.
Then it levelled off. Generation 4 passed its 800-game check with +12 ± 24, which is inside the noise, and over 3,000 games it tied generation 1. That is exactly why every result here is confirmed with a long match before anything ships.
Next
A plateau after one generation usually means the new games are too similar to the old ones. The next experiments: more games per generation, more exploration in the games, and fine-tuning the previous network instead of training each one from scratch.