Video Models Start to Solve Chess, Maze, Sudoku, Mental Rotation, and Raven's Matrices

Abstract

We introduce a benchmark that evaluates the reasoning abilities of video models across multiple tasks. We find that Sora-2 achieves a success rate above 60%. Our experiments establish a clear performance hierarchy across different model architectures and training methods. We develop a robust and scalable evaluation paradigm, together with a unified and modular VMEvalKit framework that supports diverse task generation and highly reliable assessments. The framework provides clear success signals, creating opportunities for future reinforcement learning (RL) fine-tuning to further improve reasoning consistency.

Framework Overview

Supported Models

40+ video generation models across 11 provider families

Supported Tasks

9 cognitive task types with procedural generation

Task Pair Structure

Each Task Pair consists of three core components:

Initial State first_frame.png Starting point or problem setup

Final State final_frame.png Goal state or expected solution

Text Prompt prompt.txt Instructions for video model

Task Examples

Model Rankings

BibTeX

@misc{VMEvalKit,
  author       = {VMEvalKit Team},
  title        = {VMEvalKit: A framework for evaluating reasoning abilities in foundational video models},
  year         = {2025},
  howpublished = {\url{https://github.com/Video-Reason/VMEvalKit}}
}