About
A benchmark you can watch.
JevBench measures decision models by putting them in charge of a real game. The result is a win or a loss, and the footage is there to check.
01 · What we measure
Decisions, not answers.
Most benchmarks grade a model on a single typed answer. JevBench grades it on what happens when it has to act: hundreds of decisions in a row, under time pressure, with a clear outcome at the end.
Jev is a decision model. Instead of writing a reply, it picks one option from a set and reports how confident it is. JevBench ranks Jev alongside open and self-hosted alternatives, and setups that pair a decision model with a guide LLM.
Games come first because they are hard to fake. The next benchmarks cover product decisions such as recovering from failed tool calls and responding to incidents. See the roadmap.
02 · Method
How a run is scored.
-
01
The model plays live
The game's state is read as structured data and the model chooses each move from the legal options.
-
02
The game confirms the win
StarCraft II: the enemy headquarters is destroyed or critically damaged with Raynor alive, or a person confirms the victory screen. Minecraft: the Ender Dragon is defeated and the exit portal reached.
-
03
Every run is published
Wins and losses both go on the record, with decisions made, cost and the reason a run ended.
-
04
Evidence is attached
Stills and video come from the game window only, and are reviewed before they are published.
- 05
03 · Open source
Run it yourself.
The StarCraft II harness is open source. Clone it, point it at a model and reproduce any result on the board.
$ git clone https://github.com/rapidstartup/jev-plays-starcraft-2
$ cd jev-plays-starcraft-2
# setup and run commands are in the README04 · Support
Questions, answered.
Ask about the harness, how a run was scored, or getting a model onto the board.
05 · Sponsor
Keep the board open.
Every run costs compute and API credits. Sponsorship keeps the results public and the harness free. Organisations can also sponsor runs directly with API or compute credits.
06 · Contact