Azul Engine Neural vs classic

Machine learning benchmark · two brains, one search

The neural network wins.

The engine can judge positions two ways: with classic rules written and tuned by hand, or with a neural network that learned from 1.5 million positions the engine played against itself. We had them play each other 3,000 times, with the same thinking time.

+112Elo at equal thinking time
± 13, 3,000 games
66%of the points won
1,955 wins, 1,020 losses
+6.3points per game
average final margin
3.1×faster search
than the network's first version

3,000 games, same clock

Every deal was played twice with the seats swapped, so neither side got the luckier tiles. 40 ms per move.

Neural network wins: 1,955 of 3,000 games (65.2%) Shared wins: 25 of 3,000 games (0.8%) Classic wins: 1,020 of 3,000 games (34.0%)

Speed is strength

A network judges better than hand-written rules but costs more per position, so in the same time it looks at fewer of them. Its first version took 2,161 ns per position against the classic engine's 309. Running its first layer the way chess engines do, updating only the inputs a move changes, in exact integer arithmetic, brought that to 704 ns: the same network gained almost 50 Elo. Elo over the classic engine at 40 ms per move; lines show the 95% range.

0 +50 +100 +150 First version First version, 3,000 games: +64 ± 13 Elo +64 3× faster 3× faster, 3,000 games: +112 ± 13 Elo +112

Fast enough to use what it knows

Before the speed-up the bigger network was too slow to use its knowledge: +6 at equal time. Now each network plays about as well at equal time as at equal positions. Head to head, 128 × 32 and 64 × 16 tie (−10 ± 12 over 3,000 games), so the board keeps the smaller, lighter one.

0 +50 +100 +150 +200 64 × 16 64 × 16, equal positions: +114 ± 25 Elo +114 64 × 16, equal time: +112 ± 13 Elo +112 128 × 32 128 × 32, equal positions: +141 ± 13 Elo +141 128 × 32, equal time: +135 ± 26 Elo +135
  • Equal positions: what it knows
  • Equal time: how it plays

What it sees matters most

Given only the board, the network merely tied. Showing it the board as it would look if the round were scored now was worth 150 Elo.

-50 0 +50 +100 +150 +200 Board only Board only, equal positions: -7 ± 34 Elo -7 Board + scored now Board + scored now, equal positions: +141 ± 13 Elo +141

What it learns from

Learning half from the search's opinion and half from who actually won beat either alone.

-200 -150 -100 -50 0 +50 Search value only Search value only, : -127 ± 36 Elo -127 Outcome only Outcome only, : -84 ± 35 Elo -84 ¼ value, ¾ outcome ¼ value, ¾ outcome, : -63 ± 34 Elo -63 Half and half Half and half, : -7 ± 34 Elo -7

How it was built

  1. Play. The classic engine played 30,000 games against itself; every position became a row of 170 numbers with what happened next.
  2. Learn. A network with two small hidden layers learned to predict the final score margin. It was trained twice: from scratch in plain Go (no libraries), and with PyTorch on a GPU.
  3. Plug in. The network replaces the hand-written judgement inside the same search, so the only difference between the two engines is the brain.
  4. Speed up. Consecutive positions in a search differ by a move or two, so the network's first layer is updated rather than recomputed, in integers so it never drifts. Three times faster, exactly the same answers.
  5. Prove it. Thousands of games at equal positions and at equal time, every deal played from both seats.
Go from scratch vs PyTorch, same data and network
Go, CPUPyTorch, RTX 5070
Training time, 64 × 1619 s24 s
Training time, 128 × 3242 s22 s
Validation error (points²)153.0153.2
Elo at equal time, 64 × 16 (first version)+66 ± 24+64 ± 24

The two agree to the decimal, which checks the hand-written training. A network this small is too little work per step for the GPU to pay off; it only pulls ahead on the bigger one.

Which should you use?

Neural network

The stronger player, and the default. Use it when you want the best move.

Classic

Every term is a readable rule you can look up. Use it to see how a hand-built judgement thinks, or to compare the two on a position.

Try both on the board Read the dev log Technical write-up and code

All the numbers
ExperimentGamesElo vs classic
64 × 16, 40 ms per move (the board's brain)3,000+112 ± 13
64 × 16, 200 ms per move400+111 ± 36
128 × 32, 40 ms per move800+135 ± 26
128 × 32 against 64 × 16, 40 ms per move3,000−10 ± 12
First version, 64 × 16, 40 ms / 200 ms per move3,000 / 400+64 ± 13 / +110 ± 35
First version, 32 × 16, equal positions / equal time800 / 800+96 ± 25 / +62 ± 24
First version, 64 × 16, equal positions / equal time800 / 800+114 ± 25 / +66 ± 24
First version, 128 × 32, equal positions / equal time3,000 / 800+141 ± 13 / +6 ± 24
Board only, 128 × 32, equal positions400−7 ± 34
Target: search value only400−127 ± 36
Target: outcome only400−84 ± 35
Target: 25% value, 75% outcome400−63 ± 34
Target: half and half400−7 ± 34

Equal positions: both sides search 20,000 positions per move. Ranges are 95% confidence. Every result is reproducible with the commands in the write-up.