Attention budget

The simulated scenario: an 80k coding session

To see attention economics in action, this sandbox simulates an active coding agent session currently sitting at 80,000 tokens. As an agent works through a multi-step programming task, its context window accumulates several distinct layers of history:

• Token 0 (Session start): System prompt and project guidelines (such as CLAUDE.md, ~1,500 tokens). Models anchor their generation here by default.
• Buried in the middle: Three architectural decisions and notes recorded earlier in the session:
  • Note A (~15% depth, ~12k tokens in): An API retries policy (“retries stay at 3 with a 250 ms backoff”).
  • Note B (~47% depth, ~37k tokens in): Root-cause diagnosis of an elusive authentication bug (“the middleware drops the session cookie when the login route answers with a 302 redirect”). This is the default target note selected for retrieval.
  • Note C (~76% depth, ~60k tokens in): A code style convention (“no default exports in lib. Named exports only”).
• Recent tail (Right before your prompt): 1,400 tokens of fresh tool output—raw compiler logs, terminal dumps, or test traces emitted by the agent's very last action.
• Background noise (~95% of the window): Routine command dumps, file contents, and past conversational turns that have no relevance to the auth bug.

You are now dispatching your next prompt to the agent. The sandbox calculates where every fraction of the model's 100% attention budget lands across all 80,000 tokens when predicting the very next word.

Guided experiments to try

Use the interactive sandbox below to step through these scenarios and observe how attention shifts across the context window:

1. See the default position bias (The bare U): Select continue in the prompt panel, then toggle log scale on the attention distribution chart. With no relevant keywords in the prompt, attention follows position alone: a massive spike at the recent prompt, an anchor bump at token 0, and the middle completely neglected. Note B receives almost 0% attention.
2. Specific wording is your best tool: Step through vague → paraphrase → exact wording. Watch Note B's attention jump from under 1% to ~20%. The note did not move; only your prompt's phrasing changed. Exact keywords trigger strong query-key vector alignment.
3. Multi-topic prompts dilute attention: Choose three things at once. Asking about three different topics in a single message splits the prompt's attention vector, weakening the match for each individual note. Asking one focused question at a time is mathematically far more effective.
4. Fiddle with context length (Context dilution): Expand What is in the context window at the top of the sandbox. With exact wording selected in the prompt panel, click through the 12k, 80k, 200k, and 400k quick buttons. Even though your prompt and Note B remain identical, Note B's share steadily drops from ~22% down to ~5% because hundreds of thousands of background filler tokens divide the same 100% pie.
5. Simulate context rot (Look-alike distractors): Inside the context window disclosure, drag Look-alike notes elsewhere up to 4. Four outdated hypotheses about the bug now steal as much attention as the real answer. This demonstrates why coding sessions derail when stale debugging attempts accumulate in the window.
6. Why re-pasting works so well: Still inside the context window panel, check Re-paste Note B right before the prompt. The newly pasted copy at the end of the context captures nearly all the attention, while the older copy in the middle is ignored. This explains why scratchpads and re-stating key facts keep long sessions accurate.
7. Test your own wording: Select your own ↓ and write a custom prompt. Watch the relevance bars to see how well your phrasing hits the buried notes, then refine your vocabulary to see the attention spike.

The mathematics & literature sources

a_i = exp(z_i) / Σ_j exp(z_j)          z_i = q·k_i / √d  +  b(i)

Scaled dot-product attention (Vaswani et al. 2017, arXiv:1706.03762). The softmax turns raw token scores z_i into probabilities that sum to 1. Because the denominator sums across all tokens in the context window, every added token dilutes the available probability budget.

q·k / √d  =  √d · cos θ        unrelated pair ~ N(0, 1)      exact match = √d = 8 (d = 64)

Content scoring & vector scaling: When query and key vectors align, their dot product scales to √d (8 at d = 64, ~11.3 at d = 128). Unrelated token pairs produce random Gaussian noise centered at 0 with unit variance.

b_rec(i) = − r₁ · log(1 + r₂ · |N − i|)          (KERPLE, logarithmic decay)
b_rec(i) = − m · (N − i),   m = 2^(−8h/n)         (ALiBi, linear decay)

Distance penalties: Chi et al. 2022 (KERPLE, arXiv:2205.09921) and Press et al. 2021 (ALiBi, arXiv:2108.12409). KERPLE's logarithmic decay closely mirrors the empirical distance decay observed in trained models using Rotary Position Embeddings (RoPE).

b_sink(0) = + σ

Attention sinks: Xiao et al. 2023 (StreamingLLM, arXiv:2309.17453). LLMs allocate substantial attention to the initial tokens (Token 0) as an “attention sink” to safely absorb unneeded probability mass when no specific past token is relevant.

b_head(i) = + π · exp(−i / ρ)

Lost in the Middle / Primacy bias: Hsieh et al. 2024 (Found in the Middle, arXiv:2406.16008). Attention naturally forms a U-shaped curve across long contexts, prioritizing initial instructions and recent messages while sagging across the middle.

cos θ_note ≈ √( Σ_hit w / Σ_all w ),   w = 1 / (notes containing the word)

Keyword overlap & semantic matching: Modarressi et al. 2025 (NoLiMa, arXiv:2502.05167). Demonstrates how exact keyword overlap drives effective context retrieval, while purely semantic paraphrases suffer significant retrieval penalties.

effective tokens = exp( − Σ a_i · ln a_i )

Effective focus spread (Perplexity of attention): Standard information-theoretic measure calculated as the exponential of entropy. Quantifies the effective number of tokens across which attention is actively distributed.