What the network looks at
Error by stage of the game and permutation importance on 105,000 positions it never trained on. It leans on the scored-now board and nearly ignores the tiles on the table.
A model that wins games is good. A model you can explain is better. After self-play I took the network that runs on the board, generated 2,000 fresh games it had never trained on (105,305 positions), and asked two questions with a short NumPy script, ml/inspect_net.py: when is it wrong, and what does it actually use?
When it is wrong
The network predicts the final score margin from a single position. Its error, as the root mean square in points, and how often it picks the eventual winner:
| Round | Positions | Error (points) | Picks the winner |
|---|---|---|---|
| 1 | 22,028 | 16.1 | 56% |
| 2 | 22,942 | 15.5 | 60% |
| 3 | 22,759 | 14.3 | 67% |
| 4 | 22,789 | 12.6 | 73% |
| 5 | 12,510 | 11.6 | 79% |
Early on it is barely better than a coin flip, and that is mostly not its fault: four or five random deals still lie ahead, and no evaluation can know them. The search's own value, which looks several moves further, is barely better: 14.0 points overall against the network's 14.3 on its own. Within a round the error does not change, early, middle or late. The uncertainty is between rounds, not inside them.
What it uses
Permutation importance: shuffle one group of inputs across all positions, so it carries no information, and measure how much worse the predictions get.
| Input group | Inputs | Rise in error (MSE) |
|---|---|---|
| Opponent's board as if scored now | 26 | 198 |
| Own board as if scored now | 26 | 186 |
| Opponent's wall | 25 | 53 |
| Own wall | 25 | 45 |
| Scores and floors | 4 | 34 |
| Pattern lines | 50 | 34 |
| First-player token and round | 3 | 2 |
| Tiles still on the table | 11 | 1 |
Two findings. The board as it would look if the round were scored now dominates, which explains why adding those inputs was worth 150 Elo in the first experiment. And the tiles still on the table barely matter to it. That sounds like a flaw, but it is a division of labour: the search plays out what happens to those tiles move by move, so the network can specialise in the long-term value of the boards.
The opponent's board weighs slightly more than its own, which fits how the game is played: much of Azul is about what you leave for the other player.
What it suggests
The parts of the input the network ignores could be dropped or summarised, making it smaller and faster. The early-game error is a property of the game, so more training will not remove it; better targets for those positions, such as averaging over several possible deals, might.