AoE2 LLM Arena — Technical Documentation

A two-tier AI agent that plays Age of Empires II: Definitive Edition, plus a synthetic evaluation tier (Arena) that races prompt/model variants against an in-memory AoE2-lite world and a web UI for replaying and forking past runs.

Architecture overview

Dashed lines indicate optional / off-by-default components. The real-game tier runs without YOLO; the arena tier defaults to the in-process broker; Redis is a Phase C add-on.

Reading paths

Short curated routes through the tutorial — pick one based on what you want to learn, instead of reading all 23 chapters end-to-end.

• 15-minute tour — 01 System Overview → 07 Detector Architecture → 14 Arena Overview.
• LLM-agent design — Parts I, II, and VIII: 01, 04, 05, 06, 22, 23.
• Computer vision — Parts III–V: 07, 08, 09, 11, 13.
• Arena infra — Parts VI–VII: 14, 15, 16, 17, 18, 19, 20.

See also the Glossary for one-line definitions of terms used throughout the tutorial.

Table of contents

Part 1: Real-game architecture

Part 2: LLM integration

Part 3: Entity detection

Part 4: Game knowledge

Part 5: Operations

Part 6: Evaluation arena

Part 7: Arena web

Part 8: Autoresearch

Architecture Decision Records (ADRs)

Short (~1 page) decisions that shaped the current architecture. Read these to understand the why; chapters above describe the what.

• ADR 0001 — Broker-first event architecture
• ADR 0002 — Redis Streams as cross-process broker backend
• ADR 0003 — pyright → basedpyright with reportAny
• ADR 0004 — Bradley-Terry ranking over simple win-rate
• ADR 0005 — Vite + React + Tailwind for arena UI

Runbooks

“You have a problem right now” checklists. Symptom → diagnosis → command, not narrative.

• Redis broker operations — compose stack, password rotation, key inspection.
• Switching the broker backend — in-process ↔ Redis switching, verification.
• Debugging a stuck fork or replay — what to check, in what order.
• Windows VM agent bring-up — fast path + symptom matrix. Full first-time setup is in deployment-guide.md.
• Retrain the detection model (v6 / YOLO26n) — end-to-end retraining loop: sprite extraction, real-terrain backgrounds, synthetic generation, cvat.ai annotation, Lambda training, deploy.

Design specs (frozen historical)

Original architectural proposals. Status headers note what shipped. Kept for why we built it this way context; current state lives in the chapters above.

• Event Broker Architecture — log-first SSE design that became Parts 6 chapters 15–16 and ADRs 0001–0002. Status: SHIPPED.
• Synthetic Arena Analysis — fork/race/mutate/observe analysis that became Parts 6–7. Status: SUPERSEDED BY IMPLEMENTATION.
• Autoresearch Plan — 5-phase Karpathy-inspired plan. Status: PARTIALLY SHIPPED (Phases 0–1; 2–5 unbuilt).

Explorations

Speculative scratch documents that haven’t crystallized into shipped designs.

• eval-virtualbox-ideas.md — notes on VirtualBox-based headless game replay.

Quick links

• Game loop entry point — the capture-detect-think-act cycle.
• Action types reference.
• System prompt — what the executor LLM knows.
• 60-class taxonomy.
• Arena CLI cheatsheet.
• Broker backpressure semantics.

Conventions

• Code references: file.py:42 format points to exact source lines (paths relative to agent/).
• Cross-references: [Chapter N](./path) between related topics.
• Status callouts: design docs carry a Status: line noting whether they’re proposals, shipped, or superseded.
• Optional modules: dashed lines in diagrams; explicit notes for graceful-fallback dependencies.
• ADRs vs chapters: ADRs answer “why?” in 1 page. Chapters answer “how does it work today?” in detail. Design specs in design/ answer “how did we get here?” and are frozen in time.