Chess Laboratory

The opponents

OpenAI Decisions and TypeSafe Jev choose a legal move from the current position. Both receive the board, legal choices and game history. No Stockfish evaluation is supplied. Forced moves are played automatically; other moves use one direct decision; neither integration offers a thinking budget.

Stockfish 19 full is the server’s reference chess engine. Its strength setting and fixed search-node budget are saved with every experiment. Local games use Stockfish 19 Lite, a smaller engine build, in a browser worker with one thread and a small memory allocation. The two builds use different neural networks; laboratory ratings refer to the full server build.

The computer

The interface borrows the grey bevels, navy title bars and teal desktop of early Windows. It is an original chess program, not an emulation of a Microsoft game.

Chess rules are written in Rust and shared between browser WebAssembly and the server. API keys remain on the server. Public visitors can play local Stockfish and read results; model calls and experiments require sign-in.

Source material

• OpenAI Decisions documentation ↗
• TypeSafe / Jev documentation ↗
• Stockfish UCI options ↗
• Stockfish · GPLv3 ↗ · license · matching source
• Chess Laboratory source (Rust + browser)
• Time-uniform confidence sequences ↗

What the number means

The estimate is the Stockfish reference Elo plus the Elo difference implied by the model’s score. A win scores 1, a draw ½, and a loss 0. It measures this experiment’s matchup, not a universal playing strength.

What the interval means

Opening pairs are the unit of measurement. A pair-aware 95% confidence sequence permits checking after each completed pair and stopping early without treating a lucky small sample as a precise rating. The run record names the exact method.

An interval may be unbounded when evidence is limited or scores are extreme. That is a result, not a missing zero. No finite estimate is published for an all-win or all-loss score.

What gets counted

Wins, draws and losses come from completed games. A stopped game, failed API request or safety move cap is excluded, never silently counted as a loss or draw. An exclusion ends the run rather than skipping a difficult game and continuing. The publication reports exclusions.

Games sample independently with replacement from sixteen four-ply openings, and use each position in both colours. Paired settings remain fixed within a run. Results from different model versions, reference strengths or node budgets should be compared with care.

When a run stops

A run finishes when its uncertainty reaches the requested target or it reaches its game or API-call limit. It can also be stopped manually. A wide interval at the end means more evidence is needed.

Game records use standard PGN. The board starts from the normal chess position; legal moves, checkmate, stalemate, repetition, the fifty-move rule and insufficient material are handled by the shared rules engine.