Benchmarking Frontier LLMs on Chess

Computers Can Already Play Chess

Computers have been superhuman at chess for almost thirty years. But modern chess engines achieve this performance in a way that is impossible for humans to replicate. First, they use hyper-optimized search techniques to look 20-30 moves in advance across many different combinations to reason near perfectly about complex tactical positions. Second, they distill billions of games into sophisticated neural networks to quickly evaluate the subtleties of positions with features that are hard for humans to understand.

LLMs Play Chess Differently

On the other hand, if you watch an LLM play chess it seems to reason in a way that is much more congruous with how humans approach chess, thinking through simple positional features or trying to calculate out lines and often getting lost or making small computational errors. Therefore, if LLMs can play chess at a high level, they may be able to distill insights to human players in a way that is much more tractable.

To understand their current performance, I built a series of chess benchmarks to evaluate them across a few facets of the game. This is by no means the first chess benchmark, but I think it is comprehensive, easy to understand, and documents a moment in time as these models appear to be moving past my skill level (I peaked around 1800 Elo many years ago).

Frontier Model Chess Performance

Chess Bench breaks down chess playing ability into full game, puzzle, and endgame performance. The model is treated just like a human. It is given the board state at every turn, and asked to reason about the correct move.

To make this as realistic as possible, I pass the previous move and the current board state into the model at every turn. This imitates how a human would play chess, they do not need to mentally recreate the board from the move sequence. They can just observe the board. I also maintain some reasoning history between moves so that the model can complete nuanced tactical sequences in a coherent manner.

Endgames

The standard formula for winning a chess game is to gain an advantage in the opening, convert it to a material gain in the middle game, and then use the material advantage to win the endgame. Because of this, early in a chess player's career they are taught how to win a variety of endgames. Some are simple (King + Queen vs. King) and some are quite complex (King + Rook + Pawn vs. King + Rook).

To test the models, I set them up with 20 theoretically won endgame positions across 4 difficulty tiers and had them play against Stockfish. The model must convert the winning position into checkmate. A draw or loss counts as a failure.

Reading the traces, Gemini reasons about complex endgames in a way that feels very familiar to me. Below is a Tier 4 endgame where Gemini converts a Queen + Pawn vs. Queen position. It checks the king, skewers the queen, promotes the pawn, and then methodically mates with King and Queen.

Puzzles

To benchmark tactical ability, I curated 100 recent puzzles from Lichess spanning ratings from 500 to 2500. Each puzzle presents a critical position where there is one clearly best move or forcing sequence. The model sees only the board state and must find the winning continuation.

Again, Gemini excels both on speed and accuracy. GPT 5.4 is also quite strong but can easily take up to 30 minutes per move for a puzzle. Opus is hopeless; it cannot reason about even a mildly nuanced tactic. Below is a mate in 2 that requires spotting a queen sacrifice. Gemini and GPT both find the winning Qxg6+, while Opus considers it but talks itself out of it, playing f5 instead.

Gemini and GPT 5.4 both reason conditionally through the tactic. The key insight is that the bishop on a2 pins the f7 pawn along the diagonal, which means after Qxg6+ the pawn cannot recapture and the king is forced to move into a mating net. Opus investigates Qxg6+ but misses the bishop's role in the pin, concludes the queen capture is unsound, and settles for the much less interesting f5 instead.

Full Games

Puzzles and endgames test isolated skills, but full games require sustained play across all phases: opening preparation, middlegame tactics, and endgame technique. To measure this, I had Gemini 3.1 Pro climb the Elo ladder: 16 games (8 openings × 2 colors) at each Stockfish skill level, from 0 through 8. Gemini is the only model that can complete games at a reasonable speed, although based on the narrower benchmarks, I would expect GPT 5.4 could play around 1800 Elo given 8–12 hours per game.

Each Stockfish skill level maps to an estimated Elo rating, giving us a performance curve. A BayesElo analysis of the full 144 games estimates Gemini at 1920 Elo.

Below is Gemini's most impressive win, a 67-move Italian Game where it slowly builds a passed g-pawn against a highly rated Stockfish opponent.

Observations

Gemini is clearly the most optimized model for chess. It slightly outperforms GPT 5.4 on Elo but massively outperforms on speed. GPT 5.4 performs significantly worse with thinking set to high instead of xhigh and can take up to 30 minutes per move while Gemini can easily play a full game in 30 minutes or less. It almost appears like GPT 5.4 is deriving chess from first principles while Gemini is explicitly trained to reason about chess (would make sense given how much time DeepMind has spent historically researching games). Opus has no understanding of the geometry of the board. I have noticed this in other settings as well where Opus struggles with spatial reasoning. This may be related to the fact that Anthropic has invested far less in multimodal capabilities and mathematical reasoning.

I would have expected the models to be a bit stronger playing endgames. Endgames are reasonably heuristic-based. Once you know the meta strategy for how to trap the king in the corner, you can convert a lot of similar-looking endgames. All this intuition should be available to them but they still struggle in certain scenarios to fully convert endgame strategy into wins. In a future post I will explore some techniques for improving language models on chess while preserving this reasoning trace style gameplay.