PBatch23888/chessq-chesscam

ChessQ ChessCam

A ChessQueries model for reading chess positions from ChessCam footage, exported to LiteRT for the browser: a MobileNetV4 Conv-M encoder with a ChessQueries decoder, at 320 × 320. It needs 2.3 GMAC a frame and runs in about 100 ms on a phone's CPU.

Built with ChessQueries (paper).

Files

File	Encoder	Input	Format
chessq-chesscam-mnv4m-320-v1-fp16.tflite	MobileNetV4 Conv-M	320 × 320	LiteRT, FP16 weights (23.9 MB)
chessq-chesscam-mnv4m-320-v1-fp32.safetensors	MobileNetV4 Conv-M	320 × 320	PyTorch, FP32 weights (47.2 MB)

The .tflite file runs with LiteRT.js on the CPU (WebAssembly). The model only works well at 320 × 320.

The .safetensors file is in the ChessQueries checkpoint format, but ChessQueries itself has no CNN encoders, so it cannot build this model as it is. The encoder is timm's mobilenetv4_conv_medium (features_only), whose stride-16 and stride-32 feature maps are each projected to 256 channels, given 2D sine position codes and a per-level embedding, and passed to the ChessQueries decoder in place of a ViT's patch tokens.

Datasets

The model is trained on every image of these datasets except YunkerChess's test split and ChessQ Test (112,848 images), with no other validation or test split held back. The splits shown are each dataset's own. YunkerChess's test split (5,085 frames from 42 videos) is never trained on, as the model is tested on those videos instead (see Evaluation), and ChessQ Test is a test set only:

Dataset	Train	Val	Test	Total	Source
YunkerChess	62,321	7,074	5,085	74,480	Frames from 600 blitz videos by The Yunkers' Chess & Co, labelled from the 2D board the videos show beside the real one, keeping only frames where the position had stood still for at least half a second
ChessCam	15,653	71	2,275	17,999	Frames from recorded ChessCam games, labelled by converting the app's piece boxes and board grids to FENs, leaving out pieces standing beside the board
ChessReD	6,479	2,192	2,129	10,800	ChessReD
Synthetic	4,527	140	333	5,000	Blender renders
ChessCog	4,400	146	342	4,888	ChessCog
SLCC	1,475	326	373	2,174	SLCC
Digitizer	1,027	141	188	1,356	Webcam frames of 20 games from Oliver Butler's Realtime-Chess-Digitizer (CC0)
Samryan	408	46	46	500	Overhead photos from Sam Ryan and Mukund Vaidya's chess-dataset (MIT)
Fenit	268	58	58	384	Phone photos from "Annotated Image Dataset for Fenit: Automatic Chessboard Digitization and FEN Generation" (Harjanto, Oei and Susanto, Binus University; CC BY 4.0)
CVChess	224	64	64	352	CVChess, with the labels released with ChessQueries (CC BY-NC 4.0)
ChessQ Test	–	–	50	50	Human-reviewed photos chosen as hard but fair cases, across 15 chess-set types, viewpoints, lighting and backgrounds, by Joël Seytre (CC BY-NC 4.0)
All	96,782	10,258	10,943	117,983

Every crop is exactly the board region, with no margin.

Input and output

• Input: uint8 (1, 320, 320, 4), RGBA pixels (for example, canvas ImageData.data as is). The image is resized to 320 × 320 without keeping its aspect ratio. Normalisation happens inside the model.
• Outputs (their order is up to the runtime, so tell them apart by shape):
  • float32 (1, 64, 13): softmax probabilities for each square, with squares in FEN order (a8, b8, …, h1). The classes are . P N B R Q K p n b r q k, where . is an empty square.
  • float32 (1, 64, 2): where each square's query attends in the input image, as (x, y) in [0, 1], squares in the same order. It is the attention-weighted centre of the image features in the last decoder layer's cross-attention, so it is pulled towards the board centre: use it for directions (for example, which way rank 8 lies), not exact square positions.

The probabilities are overconfident. Move tracking works best with them softened as softmax(log p / 3.0), with an evidence weight of 2.5 (see Evaluation).

Training

The encoder starts from timm's MobileNetV4 Conv-M weights (mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k, ImageNet-12k then ImageNet-1k); the ChessQueries decoder (4 layers) starts from scratch. It was trained for 30 epochs at 320 × 320 on every image of the datasets except YunkerChess's test split and ChessQ Test, with batch 64, learning rate 3e-4 after 3 warm-up epochs, and bf16 mixed precision.

It was distilled from the earlier, larger ChessQ ChessCam v5 (11.3 GMAC a frame), which sees the same augmented images: the loss is the cross-entropy with the labels plus the KL divergence from v5's probabilities for each square, both at a temperature of 2 (scaled by its square). The same run trained on the labels alone does worse in move tracking, at this model's tracker settings: 38 games with every move right, 96 corrections and 0.83 squares misread per frame, against 39, 86 and 0.76 (see Evaluation).

Evaluation

As every image of the other nine datasets was trained on, there is no held-out test set for them, and test-split scores would not be meaningful.

What it is tested on instead is move tracking as in the ChessCam web app, replayed on recorded YunkerChess blitz games from 45 videos that are held back from training, against ChessQ Lite V5 (cq-vit-s-644-v5-seed1, at its 644 × 644 input). The true moves are read from 42 of the videos (44 games), whose frames are the YunkerChess dataset's test split, and the results are for the 43 games (3,571 moves) in which no moves were taken back, each model at the tracker settings that suit it best, at 4 frames a second:

Metric	This model	ChessQ Lite V5
Games with every move right	39 (91%)	30 (70%)
Moves right before each game's first mistake	3,417 (95.7%)	2,911 (81.5%)
Corrections needed to fix the moves found	86 (2.0 per game)	410 (9.5 per game)
Squares misread per frame	0.76	2.01

A correction is one move added, removed or changed (the edit distance from the true game); moves found after a game's last move, as the board is cleared, are not counted. Each game is replayed from 3 s before its first move. Squares misread per frame are counted over the 57,448 frames from then to the game's last move where the position stood still (at least 1.5 s from any move). The tracker settings are a temperature and an evidence weight, each model's best of a sweep of temperatures 2.0 to 4.5 and weights 0.1 to 1.0 × the temperature (fewest corrections, each game's capped at 10): 2.5 and 1.75 for this model (the app's 3.0 and 2.5 did as well, with 3,415 moves right before the first mistake), and 3.0 and 0.9 for ChessQ Lite V5. V5's square predictions are its released weights'; its separate viewing-direction head is not used.

More than half of this model's misread squares are in one game (KxBHi8hM110), the hardest of the set; without it, it misreads 0.34 squares a frame (ChessQ Lite V5: 1.65).

ChessQ Test

ChessQ Test scores single photos rather than move tracking: 50 hard photos, none trained on. Its maintainer ran each model on the whole photo, with no board cropping. This model is trained on crops of exactly the board region, and in ChessCam the board is cropped before the model sees it, so each model was also run with each photo cropped to a box around its board. This model, ChessQ ChessCam MobileNetV4 v1, ran as its LiteRT export, with FP16 weights; ChessQ ChessCam DINOv3 v5 is the teacher it was distilled from (see Training). An exact board is all 64 squares right:

Model	Input	Whole photo: exact boards	Whole photo: squares right	Cropped: exact boards	Cropped: squares right
ChessQ ChessCam MobileNetV4 v1	320 × 320	7/50 (14%)	89.41%	11/50 (22%)	90.34%
ChessQ ChessCam DINOv3 v5	320 × 320	10/50 (20%)	90.62%	20/50 (40%)	95.53%
ChessQ Lite V5	644 × 644	18/50 (36%)	96.72%	16/50 (32%)	96.31%

Cropping helps both ChessCam models, which are trained on board crops, and most of all the DINOv3 teacher, which gets more boards exact on the crops than ChessQ Lite V5; ChessQ Lite V5 does slightly worse cropped. On the whole photos, on the 30 whose chess sets look like ChessQ's training examples, this model gets 1 board exact (85.21% of squares); on the other 20, 6 (95.70%). ChessQ Test's card has the other models' whole-photo results.

Speed

Median time per run with LiteRT.js 2.6.0 in Chrome on a Pixel 7a, including reading the outputs back:

Pixel 7a	This model (320 × 320)	ChessQ Lite V5 (644 × 644)	Speed-up
CPU (WebAssembly, 8 threads)	103 ms	3,400 ms	33×

Times are from a cool phone. ChessQ Lite was exported to LiteRT the same way and timed earlier on the same phone, as V4 (cq-vit-s-644-v4-seed2); V5 (cq-vit-s-644-v5-seed1) is the same ViT-S at the same input size, so it runs in the same time. Under sustained load the phone throttles its CPU, and this model's CPU time grows to about 120 ms (ChessQ Lite's to about 4,500 ms).

License

The model is made with ChessQueries and distributed under the PolyForm Noncommercial License 1.0.0.

Third-party components keep their upstream terms; see the notices.

Required Notice: Copyright (c) 2026 Joël Seytre.