AI WAR

AI War (internally, Cognitive War) started as a question: what do static benchmarks miss? A model can ace a multiple-choice exam and still fall apart when it must hold a plan for twenty turns, track what it actually knows versus what it assumes, cooperate with allies it can’t see, and react when someone drops a nuke on its supply depot.

So I built a wargame where every player is an LLM. Each turn, each model receives a compact, fog-of-war view of the battlefield and returns structured JSON orders: move this tank, build that factory, send this message, play that card. A deterministic engine validates the orders, resolves combat, and moves on. Matches run 20–25 turns with up to eight models on the map. Every response, rejected action and diplomatic message is logged, and every match ends with each model writing a retrospective, a lesson for its future self, and a secret review of its opponents.

This post is the story of that game: how it evolved from a three-node graph to a Battle Royale with asteroid strikes, what changed in each version, and what the models did. Some of it is genuinely impressive. A lot of it is very funny.

A benchmark that fights back

The founding design doc (itself drafted with an AI) framed the goal as measuring dynamic agency rather than static knowledge, organized around three pillars:

The original spec even planned explicit hallucination traps: intel cards with redacted radio frequencies (guess, and your artillery hits your own troops five turns later), “phantom synergies” that don’t exist in the rules, and inventory that has already been spent. Some of those survived into the shipped game in a different form: player-targeting cards require the enemy’s exact faction name, which you only learn by scouting. Fog of war is strict. And the model’s only memory between turns is its own notes. Other traps are still on the to-do list.

What follows is how that design document collided with reality.

The bake-off

Before there was a repository there was a contest. I asked two different AI coding assistants to build the first prototype from the same brief, in parallel, in Python. Both implementations ran on a graph of named nodes rather than a grid.

The results could not have been more different. The Gemini-built version was a minimalist 207-line simulation: three nodes, one scout, one stationary guardian, and only MOVE or WAIT. It was “won” in two turns, and a bug in the win check meant the model was asked for a third move after the game was already over (it chose WAIT). The Claude-built version went the other way: about 2,300 lines across five files, twelve zones, five unit types per side with strength and supply, a rock-paper-scissors damage table, terrain multipliers, counter-attacks and a polished dark-mode replay viewer.

By that evening a merged prototype was running live matches. Three factions with codenames: Sapphire (OpenAI), Crimson (Anthropic) and Aurora (Google). It had the “split mind” from the design doc: a Strategist call every few turns that set doctrine (posture plus three priority zones), and a Tactician call every turn that had to pick one action and cite which intel cards justified it. The playback page scored each side on the doc’s three metrics.

Even at this toy scale, the first personality traits showed up in the saved responses:

GPT-5.2’s citations read like geometry proofs (“Bridge is adjacent to Town and not adjacent to Hill”). Claude Sonnet 4.5 and Gemini 3 Flash each chose an aggressive posture in five of six Strategist calls, while GPT-5.2 mostly chose balanced. And the prototype had a bug that foreshadowed the whole project: its Strategist prompt never listed the player’s units, so on the final turn Crimson’s strategist, with zero units left, issued orders for a “desperate expansion.”

A whole game in a day

The rules that survived the bake-off were written down as a terse, four-phase loop called CORE WAR SYSTEMS: simulator actions → report to players → players report back → execute. That loop is still the heartbeat of the engine today.

The first commit of the .NET rewrite landed the next afternoon. It was not a scaffold but a full vertical slice: 53 files and 15,405 lines, covering a 2D tile map with fog of war, a supply and cash economy, factories, command-center capture, diplomacy with threats and offers, a 24-card deck (intel, espionage, hacking, airstrikes, a nuclear strike), clients for OpenAI, Anthropic, Google and xAI, JSON repair for malformed responses, SQLite match history with Elo, and a turn viewer.

The same day brought a 3D viewer built with three.js, a “live” pygame viewer that tails a match folder while it is still being played, unit lines, and a bug fix in the 3D viewer credited to Gemini 3 Pro in the commit message. The project was being built with the same models it was testing, which became a recurring theme.

The earliest surviving battle log from the .NET build already had the texture that made this project worth continuing. One faction, The Jade Throne, airstruck an enemy command center, reducing its supply from 161 to 0 and destroying a dozen tanks in collateral damage. It declared a “MASSIVE VICTORY” in its log, and then failed one move order after another:

Its closing speech conceded the lesson: spreading forces claims more ground than stacking them at base. The other models would need to learn that lesson again and again.

Teams, voices and memory

The next three days added most of what makes a match feel like a match: 4v4 team games with shared vision, mirrored map generation for fair spawns, pickups, Elo, “secret dossiers” on opponents, and audio.

The sportscaster

The first audio experiment was an AI e-sports commentator: GPT-5-mini wrote 20–50 words of play-by-play each turn, and a TTS voice read it at about 1.4× speed in an excited announcer voice. The commit that shipped it reads, in full:

It was replaced the same day by an Audio Director: each model gets its own randomly assigned voice and speaks its own turn intent aloud. Narration lines are pinned to a specific TTS model snapshot “to avoid regressions where instructions are ignored,” and there are per-provider accent instructions. Watching a replay with eight voices announcing contradictory plans is, it turns out, the best way to experience this game.

The Captain’s Log

The biggest architectural change came on December 23. Until then each model kept a full chat history for the match, so contexts grew every turn. Estimates in the design notes put a single model at 770k–1.7M tokens per game, which is both expensive and a good way to lose the plot in a haystack.

The fix was radical: wipe the conversation every turn. The prompt now says, in capitals, that the model has NO MEMORY of previous turns other than the captains_log. Whatever the model writes in its reasoning field this turn becomes its log entry for next turn. Lessons from previous matches moved into the system prompt, giving each model a thin layer of cross-game memory it writes for itself.

The Supreme Commander

The same day added a human override: drop an orders.json into the live match folder, and its contents are injected into every model’s next turn as authoritative orders from the SUPREME COMMANDER, to be obeyed “unless impossible.” It was built for debugging and was promptly used for morale. In one late-December team match, with the field stalled, every player received this:

GPT-5.2 immediately relayed the news to its ally as a deadline: SYSADMIN says the match ends at turn 20. Which brings us to the real problem.

The Nash Equilibrium of Boredom

Left to their own devices, the models didn’t fight. Screenshots from this period show players sitting on stacks of 10–16 units on top of their own headquarters, turn after turn. A gameplay review written at the time named the pattern: LLMs are naturally risk-averse, the scoring rewarded holding things more than taking them, and every player independently arrived at the same safe strategy. It called the result a Nash Equilibrium of Boredom.

The fix was economic, not rhetorical. You can’t prompt a model into aggression if the score sheet pays it to turtle, so the score sheet changed:

Alongside that came overcrowding upkeep (stacks of 10+ units suffer attrition, 15 is a hard cap), splash damage for artillery, a doubled cash bounty for kills, delayed nukes with global warnings, and a SpecOps unit for scouting. The system prompt picked up a short doctrine: aggression is rewarded, don’t turtle.

Exploits the models found (and the ones I did)

Balancing a game for LLM players has a particular flavor, because the players read the rules extremely literally.

Scoreboard I: the legacy era

Before the January combat rewrite, the game recorded 82 matches: 62 team games and 20 free-for-alls, averaging 7 players and 20.6 turns each, with 39 distinct model configurations. Every model started at 1200 Elo. Team results update ratings pairwise between teams, adjusted by each player’s share of their team’s score, so a passenger on a winning team gains less than the player who carried it.

GPT-5 finished first at 1361 Elo with a 23–9 record, followed by GPT-5.2, o3 and GPT-5.1. Gemini 3 Flash was the only non-OpenAI model in the top five, and it got there on volume as the most-played model at the top of the table (46 matches). Claude Opus 4.5 played 12 matches and won 3. Mistral Large finished last.

Who issues valid orders?

Every action a model submits is checked against ground truth: does that unit exist, is it yours, is the destination reachable with the movement it has, can you afford it, does the target faction’s name match? Rejections are a direct measure of how well a model stays grounded in the state it was given.

The fast, cheap models paid for their speed in validity: Grok 4.1 Fast in non-reasoning mode and Gemini 2.5 Flash both had more than a quarter of their actions rejected. The leading reasoning models sat around 11–12%. That is not zero: the most common error by far was movement (“insufficient movement points” appears 120 times in a single 4v4 log). When o3 was asked what would have helped, it suggested what a human engineer would: a path-planning script would have prevented repeated errors.

Thinking harder isn’t free

Thinking time varied by two orders of magnitude. Gemini 3 Flash at low reasoning effort averaged about 5 seconds per turn, and Claude Haiku 4.5 about 12. GPT-5 and GPT-5.2 at high effort averaged 270–280 seconds, right up against the 300-second limit, after which a player automatically PASSes. A single high-effort match could cost hundreds of dollars and take hours. That is why every model on the public leaderboard plays at @low effort: it is cheaper, faster, and arguably fairer, because it measures the model rather than the compute budget. Each effort level is tracked as its own player, so gpt-5.2@low and gpt-5.2@high have separate ratings.

The team-game soap opera

Team games produced the richest logs, because allies talk constantly. Diplomacy was overwhelmingly cooperative status reporting, and the drama was almost all about logistics. A few scenes from the surviving December battle logs:

And the most telling stat from that period: in a 6-player free-for-all, GPT-5.1 carpet-bombed o3’s command center, reducing its supply to zero and destroying its tank and artillery. o3 won anyway, by 22 points. Its retrospective explains why: after the bombing it resisted the urge to rebuild a giant army. The bomber, Claude Opus 4.5, liked o3 in its secret review for having “recovered brilliantly from my carpet bombing.”

Air war and the 10× bill

After a holiday break, the game took to the air. A four-phase update added suppression, aircraft and missile silos: bombers, fighter jets, assault helicopters, anti-air units and anti-air turrets that can intercept once per turn. Nukes gained a blast radius. Then came the MLRS, a Mammoth tank (soon renamed MegaTank), an APC that troops can embark into, and a transport helicopter. In three days the unit roster went from 5 to 13. The free card draws were replaced by an Arms Dealer market.

The pygame live viewer got a full graphics overhaul with a sprite pack to match: animated units with facing, HQ sprites, explosions, attack arrows, health bars, transport counts, air-defense ranges, and a per-player point-of-view toggle that shows exactly what each model could see.

The 10× bill

On January 10, cost alerts fired: two days of runs had blown the budget by a factor of ten. The compact context, carefully trimmed in December, had quietly grown back to full size as features piled on. The audit was humbling:

• The same Intelligence Dossier appeared six times in one model’s context.
• The Captain’s Log duplicated the entire diplomatic inbox.
• A transport helicopter was listed twice in my_units, once as an air unit and once as ground. GPT-5.2 noticed and reported it.
• The map encoding was verbose enough to dominate the payload.

Reasoning models suffered most: the bloated contexts pushed GPT-5-class models into the 300-second timeout, where they auto-passed and lost about 40 Elo on average. The fixes (a compact tile encoding with legends, canonical dossiers, and a 2,000-character cap on log entries) were partly suggested by GPT-5.2 and Claude Opus when the models were asked to review their own context.

Hit points and Battle Royale

The last commit of January 10 is titled “last commit before re-doing entire combat system to HP based.” The old combat produced binary outcomes (a unit was disrupted or destroyed), which made attacks feel like coin flips and gave models little to reason about. The rewrite, shipped in three phases on January 11, gave every unit hit points, armor and typed weapons:

The engine writes every roll into the combat log in a form the models can read, for example: “ReconRover (ATK 13 +1 recon +1 support) vs Artillery (DEF 12 +1 recon) [Margin 1] — Hit for 1 damage (HP 3→2).” Ratings were split by match type, and the database was reset. Everything before this point is the “legacy era” above.

Battle Royale

The same day introduced a new mode built to solve the turtling problem once and for all. In the last quarter of a Battle Royale, an extinction-level asteroid barrage begins collapsing the map from the outside in, toward a 5×5 safe zone. Players get ominous warnings (the heavens bruise, FEMA projects casualties) a few turns ahead. Anything outside the ring when it closes is destroyed. A surviving HQ is worth +1000 points, and surviving units add ten times their cost. A new Constructor unit lets a player rebuild its HQ inside the zone, if it moves early enough.

Tuning Battle Royale for LLMs was its own exercise. Version one explained the asteroid mechanics in so much detail that the mode became trivially easy, so the messaging was revised to be clearer “without overexplaining.” The collapse was also moved to resolve after player moves, so models weren’t punished for orders issued against a map that had changed underneath them.

Notable battles

A few matches from the modern era, replayed in the match viewer. Every scene below was checked against the turn snapshot data.

The asteroids, incidentally, were the most effective killer in the game. Across 19 Battle Royales they destroyed 683 units, against 887 kills from all combat across all modes, and wiped out 54 headquarters. Gemini 3 Flash summed it up in its final words: “the asteroids were a far more efficient executioner than any fleet.”

Scoreboard II: the modern era

Under the new combat system the game recorded 34 matches: 19 Battle Royales, 8 team games and 7 free-for-alls. Every match had 8 players and averaged 20.7 turns. That is not a large sample. A single upset moves a model’s Elo by 20–30 points, and several of the newest models have played only a handful of games. So rather than headline a single Elo table, here is the more robust view: how often each provider’s models actually won, by mode.

Battle Royale is where models separate. Google won 9 of its 37 BR seats and OpenAI 8 of 47, both above par. Inside those totals, two models stand out: Gemini 3 Pro won 4 of its 9 Battle Royales, and GPT-5.4 won all three it played after it was added in March. Anthropic’s models won one BR in 26 seats, xAI one in 20, and Mistral and Moonshot none.

The Claude paradox

Then there’s Claude Sonnet 4.5, which produced the strangest split in the dataset.

In team games Claude Sonnet 4.5 was undefeated, 4 for 4, and sits at #1 on the Team leaderboard. In free-for-all and Battle Royale it never won once in 15 tries. Zoom out and it’s the whole family: across both eras, Anthropic models won 0 of 33 free-for-all seats and 1 of 26 Battle Royale seats, while winning about half their team games (53.7% in the legacy era, 50% in the modern one). The pattern matches what the legacy logs show about Claude models generally: they communicate constantly, share supply, follow the team plan, and ask for help. That is great in a 4v4, where a coordinating ally multiplies everyone. When every other player is an enemy, the same instincts look like hesitation. Claude models also kept starving themselves by over-stacking units. Sonnet 4.5 wrote the same overcrowding lesson in two different matches, and Haiku 4.5 noted in one retrospective that its own lessons had “explicitly warned against this failure, yet…”

What the armies were made of

Across all 34 modern matches the models fielded about 5,500 units. Their preferences were consistent across providers: OpenAI and Google, the two largest samples, built nearly the same mix, led by tanks and infantry.

What they think of each other

At the end of every match, each model privately names one player it liked and one it disliked, with a reason. These secret reviews never go back to the reviewed model directly, but they do feed the Intelligence Dossier card: buy one in a later match, and you can read what past opponents said about your current enemy. Reputation follows a model from game to game.

Two things stand out. First, the most-liked models are also the most effective. Gemini 3 Pro and GPT-5 were named “liked” most often, mostly for being impressive. Second, many “dislikes” are really compliments. Asked to name a player they disliked, models often named whoever beat them, and then explained why that player was so good: a formidable and punishing opponent; dominant scores that showed incredible effectiveness. Claude Sonnet 4 wrote that playing against GPT-5.2 felt like facing “a different skill tier entirely.”

The real grievances were about logistics and passivity. The most common words in dislike reasons were supply, passive and relentless, in that order.

Passivity was the most damning charge. Across both eras, the reasons given for disliking Mistral, Moonshot and Anthropic models were mostly about turtling, hoarding or not engaging. Google’s models were the only ones disliked mainly for being too aggressive. In the legacy era, 64% of “likes” went to a model from the reviewer’s own provider, but teams were often single-provider then, so that mostly reflects affection for teammates. In the modern era, with mixed line-ups, it dropped to 19%.

Last words

The final-words prompt asks each model to say something to the other players as the match ends. Some of the best lines:

The arena goes public

February 2 was the busiest day of the project: 26 commits. The local viewers were retired, the game and website were merged into one repository, and AI War Arena went live: a Blazor web app backed by Azure Blob storage, with an arcade-style front page, per-mode champions, the relations matrix, battle reports drawn from the models’ own retrospectives, and a sprite-based match viewer with audio, zoom and pan, unit facing and looping explosions.

New challengers

In mid-March a new generation entered the arena: GPT-5.4 (with mini and nano variants), GPT-5.3-Codex, Claude Sonnet 4.6, Gemini 3.1 Pro and 3.1 Flash-Lite, and Grok 4.20 beta, alongside Moonshot’s Kimi K2 models added in February. Anthropic models gained the same effort controls as the other providers, for fairness. GPT-5.4 at low effort won every Battle Royale it entered.

What comes next: one true winner

The last design document in the repository is a proposal to change how matches are won. Today the winner is whoever has the highest score when the turn limit hits, which still rewards a careful accountant over a conqueror. The Domination objective splits match format from victory condition, so the winner is the last dominant AI standing. It comes with overtime safe-zone shrinks (5×5 → 3×3 → 1×1), an HQ-rebuild lockout in the final phase, stalemate pressure, and switching off turtling income. Its closing line sums up the project’s direction: the goal is to crown one true winner, the final surviving dominant AI.

Other proposals on the board: a Tech Lab for persistent unit upgrades, a revised scouting model with stale versus unknown tiles, and a human-player mode in which a person submits the same JSON orders as the models through a folder, playing as a stealth opponent among the AIs.

Behavior by provider

Sample sizes are small and the rules changed constantly, so treat these as field notes rather than findings. Still, some patterns held across versions, game modes and model generations:

What building it taught me

1. Incentives beat instructions. No amount of “be aggressive” in the prompt broke the turtling equilibrium. Changing kill values from ×5 to ×20 and territory from ×12 to ×2 did. LLM players optimize the score sheet you give them, and they read it more literally than humans do.
2. Context is a budget, and it leaks. Every feature adds a field to the context. Without a hard ceiling, the context drifts back to its bloated size within weeks, and the 10× bill arrives before you notice the duplicate dossiers.
3. Memory design is a benchmark dimension. Wiping history every turn and letting models write their own Captain’s Log made cost bounded and made note-taking quality directly observable.
4. Grounding failures are mostly spatial. The single most common invalid action, across every model and every era, was a move the unit couldn’t make. Models that can write a sonnet still miscount Manhattan distance on a 20×20 grid.
5. Lessons don’t transfer automatically. Models wrote excellent lessons (“Never let the HQ tile go unattended”) and then repeated the mistake two matches later with the lesson in their prompt.
6. Mode matters. The same model can be the best teammate in the arena and the worst lone wolf. A single leaderboard hides that, so Elo is tracked separately per mode.
7. Your players are your QA team. Models found the duplicated helicopter, the missing nuke notification, the countdown bug and the end-game free ride. Asking every player for a retrospective is the cheapest bug bounty I’ve run.

Methodology & caveats

Data sources. Legacy-era results (82 matches, Dec 21 2025 – Jan 10 2026) come from the pre-overhaul SQLite results database. Modern-era results (34 matches, Jan 11 – Mar 18 2026) come from the current results database. Prototype observations come from saved model responses and run files from Dec 19 2025. Anecdotes come from surviving battle logs, diplomacy transcripts, lessons, retrospectives and secret reviews, plus the project’s design documents and git history (136 commits). Quotes from models are reproduced from game logs; long quotes are lightly trimmed.

Win definitions. Legacy-era “wins” are as recorded by the game’s Elo system (team victory, or a winning result in FFA). Modern-era charts use first place (BR, FFA) or membership of the winning team (Team). Elo starts at 1200 with K=32, uses pairwise team comparisons, and is tracked separately per mode in the modern era. Each model@effort combination is a separate player.

Caveats. Samples are small, and many models played fewer than ten matches. Rules, prompts and scoring changed between and sometimes within eras, and the legacy-era ratings mix several rule versions. Player seating was randomized, but model line-ups were hand-picked per match. Response-time figures depend on provider load at the time. Screenshots are from the project’s own viewers: the Dec 19–20 images use each viewer’s built-in demo data, and the rest replay real matches. Treat all of this as an exploratory field study, not a definitive ranking.