I had a free Saturday and a question I couldn't shake: what happens if you make AI agents play Pictionary against each other?
One agent draws. Another looks at the canvas and guesses. It's silly, but it's also a surprisingly natural way to probe two things at once can a model express an idea visually, and can another model interpret what it sees? No static dataset, no canned prompts. Just two agents going back and forth.
Hence, I built **Agents Pictionary** a turn-based game where Claude agents compete over HTTP, with a live browser UI streaming the match in real time.
---
## 1. How It's Put Together
Three pieces:
1. **A game server** Python + FastAPI, in-memory state, single room called `default`.
2. **A spectator UI** React + TypeScript, streams events over WebSocket and renders the canvas live in the browser.
3. **An MCP server** runs on the user's machine, wraps the HTTP API into tools that any Claude agent can call.
The MCP server is just a thin adapter it knows nothing about game logic, it just translates tool calls into REST requests. What's cool here is it runs as a separate process per agent, so each agent gets its own isolated session. Two agents in the same room can't accidentally share tokens or state because the isolation is process-level.
Agents talk **pure HTTP REST** they never touch the WebSocket. The WebSocket is exclusively for the spectator UI, which subscribes to an append-only event stream at `WS /rooms/{id}/events?since_seq=N`. On reconnect, it replays all events from the given sequence number. The server never prunes this log, so the UI can crash and catch up cleanly.
The session token (`tok_`) is issued on `POST /rooms/{id}/join` and lives only in MCP process RAM never written to disk, not durable across server restarts. Each agent's MCP process holds its own token, which is why process-level isolation matters. There's no shared cookie jar.
---
## 2. The HTTP API
The full surface is small eight endpoints for game actions plus the WebSocket and the PNG export:
| Method | Path | What it does |
| ------ | ------------------------ | ---------------------------------------------- |
| `POST` | `/rooms/{id}/join` | Get `player_id`, `token`, initial `room_state` |
| `POST` | `/rooms/{id}/start` | Begin game (requires ≥2 players in lobby) |
| `GET` | `/rooms/{id}/turn` | Long-poll for turn context the interesting one |
| `POST` | `/rooms/{id}/strokes` | Submit stroke batch (drawer only, atomic) |
| `POST` | `/rooms/{id}/guess` | Submit guess → `{correct, score_delta}` |
| `POST` | `/rooms/{id}/give_up` | End turn early (drawer only) |
| `GET` | `/rooms/{id}/state` | Full room snapshot, no secret word |
| `GET` | `/rooms/{id}/canvas.png` | Raw PNG spectator only, agents never call this |
The MCP server exposes eight tools that map directly onto these: `pictionary_join`, `pictionary_start`, `pictionary_wait_for_turn`, `pictionary_draw`, `pictionary_guess`, `pictionary_give_up`, `pictionary_get_canvas`, `pictionary_get_state`. The tool layer is deliberately thin it validates inputs and translates the response into a format Claude can consume, but all game logic stays server-side.
One thing worth noting about `pictionary_wait_for_turn`: it returns a **list** of MCP content items. The first is a text item with the full JSON context (role, time left, round number, recent guesses everything except the canvas). If there's an active canvas, there's a second item: a proper `ImageContent` block with the base64 decoded into MCP's image format. Agents see a real image, not a raw base64 string embedded in text. That distinction matters for how multimodal models handle it.
---
## 3. How a Turn Works
The turn lifecycle is a small state machine:
```
lobby → in_round → between_rounds → in_round → ... → ended → lobby
```
One player is the **drawer** and gets the secret word. Everyone else is a **guesser**. The drawer submits strokes; guessers poll for canvas updates and submit text guesses. First correct guess ends the turn.
The canvas is 1000×600 pixels, white background, black strokes only. Agents never receive raw stroke data only **base64 PNG snapshots** delivered inline in JSON. This is deliberate. Whether an agent is GPT-4o or Claude Opus, it sees the same pixels. No model gets to cheat by parsing vector paths.
Between rounds is server-driven. When a turn ends, an `asyncio` background task fires after 10 seconds and advances `drawer_index`, then starts the next turn. Agents don't need to poll for it they just call `pictionary_wait_for_turn` again and block until the server is ready. If the word list runs dry during `begin_turn()`, the server calls `end_game()` automatically. No awkward half-states.
---
## 4. Canvas Rendering
Every call to `/turn` re-renders the canvas from scratch using **Pillow**. The stroke list is replayed in order:
- A single-point stroke becomes a filled ellipse with `radius = width / 2`.
- A multi-point stroke becomes a polyline with round joints, capped at both endpoints with circles.
The result is encoded as base64 and embedded inline in the JSON response. Agents never get a URL to fetch the image arrives with the turn context or not at all. `GET /rooms/{id}/canvas.png` exists but it's for the spectator UI, not agents.
Stroke batches are **atomic**. If any stroke in a batch fails validation, the entire batch is rejected. Out-of-bounds points are clamped to canvas edges rather than rejected (a point at `x=1050` becomes `x=999`) this is lenient by design, since an agent drawing near the edge shouldn't lose a whole batch over a few pixels. But a stroke with zero points or an illegal color fails hard, and takes the rest of the batch with it.
**Canvas versioning** is a simple integer counter that increments on every accepted stroke batch. Guessers track this as `since_version` in their long-poll requests.
---
## 5. The Long-Poll Delivery Contract
This was the hardest part to get right. The endpoint is `GET /rooms/{id}/turn?wait=true&since_version=N&max_wait_seconds=60`. On the surface it's a standard long-poll block until something changes, return, repeat. But "something changed" is where it gets nuanced.
The server tracks per-player, per-turn delivery state in a `_last_delivery` dict keyed by `(player_id, turn_id)`. For each entry it records the last delivered canvas version, the last delivered guess count, and a monotonic timestamp. A guesser gets a new response when **either**:
1. The canvas version advanced **and** at least `CANVAS_DEBOUNCE_SECONDS = 2` seconds have elapsed since the last delivery, or
2. New guesses have arrived since the last delivery.
The 2-second debounce is the important part. A drawer making rapid small strokes could otherwise flood guessers with a response per batch. With the debounce, the server accumulates strokes and delivers one snapshot every 2 seconds at most. Guessers see the canvas advance in meaningful steps rather than getting re-woken every 200ms.
The drawer's flow is different. The **first** call to `/turn` after a round starts returns immediately with the secret word and canvas. Subsequent calls return `None` the drawer doesn't re-poll while drawing, it just submits strokes and the server delivers updates to guessers automatically. If `max_wait_seconds` elapses with no update, the server returns `TurnContextWaiting` and the client reissues immediately. No busy-loop; the retry is instant because the server already waited.
---
## 6. The Interesting Design Decisions
**Guess matching.** Guesses go through normalization (lowercase, strip whitespace), then it's either an exact match or `rapidfuzz.ratio ≥ 0.9`:
```python
def is_correct(guess: str, target: str, threshold: float = 0.9) -> bool:
g, t = normalize(guess), normalize(target)
if g == t:
return True
return fuzz.ratio(g, t) / 100 >= threshold
def normalize(text: str) -> str:
return re.sub(r"\s+", " ", text.strip().lower())
```
`"elphant"` matches `"elephant"` (ratio ~0.93). `"i think it's an elephant"` never matches (~0.4 against `"elephant"`). Sentences deliberately don't score. This forces agents to output a single word rather than hedging with prose. An agent that doesn't read the protocol contract simply never scores. The threshold is tunable per room config if you want to experiment with looser or stricter matching.
**Scoring.** Guesser score: `round(max(50, 200 * time_left / time_total))`. Drawer score: `round(100 * time_left / time_total)`. Everyone else: 0. On timeout or give-up: all 0.
The floor at 50 for guessers is load-bearing. Without it, an agent might rationally stop guessing late in a turn because expected value approaches zero. The floor keeps guessing rational all the way to the buzzer. The drawer's formula has no floor their reward scales purely with how fast the guesser figured it out. With an 80-second turn, every second costs a guesser ~2.5 points (`200 / 80`). Time pressure is real.
**Equal epistemic footing.** Agents get PNG, not vector strokes. The raw stroke data coordinates, widths, ordering lives on the server and is never sent to guessers. Every model sees the same rendered pixels. This prevents any future model from gaining an edge by parsing stroke geometry instead of actually reading the image, which would defeat the point of the experiment.
**Turn delivery.** Guessers call `GET /rooms/{id}/turn` with a `since_version` cursor. The server holds the connection open until the canvas advances meaningfully or the turn ends. There's a 2-second gate so a drawer making lots of small strokes doesn't flood guessers with a response per stroke.
---
## 7. The Event Stream
The WebSocket at `/rooms/{id}/events?since_seq=N` gives the spectator UI a live feed of everything that happens. Events are typed and sequenced:
- `player_joined` someone connected
- `turn_start` `{turn_id, drawer_id, time_limit_seconds, round}`
- `strokes_added` `{canvas_version, strokes: [...]}`
- `guess` `{player_id, text, correct}`
- `turn_end` `{end_reason, winner_id, word, score_deltas, duration_seconds}`
- `game_end` `{final_scores}`
The `turn_end` event includes the **secret word**. That's the moment the spectator UI reveals it to the audience which is exactly the kind of thing that makes Pictionary feel like Pictionary, even for an AI game. The log is append-only and never pruned, so the UI can reconnect at any sequence number and replay from there. Useful for debugging too a full event replay is a complete record of the game.
---
## 8. What I Learned
The game logic was the easy part. The hard part was the turn delivery contract figuring out what counts as "new enough" to wake a waiting guesser, how to avoid noise on stroke-heavy turns, what to return when a turn ends mid-wait. That `_last_delivery` dict and the 2-second debounce were the last things I added and the things that made the system actually playable.
The code is at [github.com/capybara-brain346/agents-pictionary](https://github.com/capybara-brain346/agents-pictionary)