Peeking Inside the Black Box: My Deep Dive into Transformer Circuits

Created on December 14, 2025

2025   ·   machine-learning   deep-learning   ·   research

I’ve spent the last few days reading “A Mathematical Framework for Transformer Circuits” by the team at Anthropic, and my mind is lit up right now. (I am 4 years late for this)

We often talk about Large Language Models (LLMs) like they are mysterious, inscrutable alien artifacts. We know how to train them, but we rarely understand what they are actually doing internally to produce such smart outputs. This paper proposes that we can actually reverse-engineer them: like taking apart a clock to see the gears, starting with simple, “toy” models.

They deliberately strip these models down to attention-only (removing the complex MLP blocks).

Why?

Because it lets you actually trace the wires. If you want to understand the machine, you have to simplify it first.

These are some of the AHA moments that stuck with me:

1. The Residual Stream & Subspaces (The “Whiteboard” Limit)

The first thing that clicked was the concept of the Residual Stream. In most diagrams, it looks like a simple line connecting layers. But the paper argues it’s actually the central nervous system of the model.

Think of it as a massive, shared whiteboard. As the model processes a token, different layers read from the whiteboard, do some math, and then add their results back onto it.

But here is the catch I hadn’t considered: The whiteboard isn’t infinite.

The paper introduces the concept of Subspaces. A single attention head is “low-rank,” meaning it has limited bandwidth. It can’t read or write “everything” and it can only move a specific slice of information. It reads from one subspace and writes into another.

This turns the residual stream into a specific kind of communication channel. Because the space is high-dimensional, Layer 1 can write to “Subspace A” and Layer 6 can read from it, effectively creating a direct wire between them that ignores everything in between. The dimensions of the stream act like memory slots; information persists there simply because no other layer touches that specific subspace.

The “Circuit Engineering” Trick

One trick I loved from the exercises was a way to mathematically prove if two heads are talking to each other. It’s a very circuit-engineering style question: “Is signal A actually connected to signal B, or am I imagining it?”

You can test this by inspecting the matrix product of the second head’s “Value” weights and the first head’s “Output” weights. If you inspect the singular values of this product, you get a concrete answer. It’s not magic; it’s just linear algebra determining if Head 2 is actually “reading” what Head 1 wrote.

2. The Progression of Intelligence (0 to 2 Layers)

The framework describes a fascinating ladder of capabilities. As you add depth, you don’t just get “more accuracy”, but also get fundamentally different types of algorithms.

Zero-Layer: Just Guessing

Basically just Embeddings $\rightarrow$ Unembedding. It behaves like bigram statistics. It sees “Mahatma”, it guesses “Gandhi”. It has no broad context.

One-Layer: The “Skip-Trigram” & Positional Weirdos

When you add a single layer, the model gets the ability to look back. It moves from bigrams to “Skip-Trigrams” (patterns like [source] … [destination] [out]).

Example: It sees an HTML opening tag <div> and knows a closing </div> is needed later, regardless of what’s in between.

But a subtle insight for me was that even when a 1-layer model looks “smart,” it’s often just a sophisticated mix of simple templates:

  • Copying: Don’t lose the identity of the token (keep “Harry” as “Harry”).
  • Positional Weirdos: Heads that just attend to specific relative positions (like “always look at the previous word”).

Two-Layer: Induction Heads (The Holy Grail)

This is where composition starts to happen, and the “Induction Head” emerges. This is the star of the show.

An Induction Head is a circuit that performs a specific algorithm:

  1. Search the context for a previous occurrence of the current token.
  2. If found, attend to the token after that previous occurrence.
  3. Copy that “next token” forward.

The paper explains the mechanism with a memorable hook: the key is computed from the token behind the current one. So matching keys effectively points you to “what came next last time.” I now think of induction heads as a “copy-the-continuation” primitive. It’s not full general reasoning, but it is the mechanical seed of In-Context Learning.

3. How to “Unit Test” a Neural Net

One of the coolest parts of the paper is the methodology. They don’t tell stories about what a head “seems like”, instead they test it via Ablations.

The workflow is very rigorous:

  1. Run the model and save attention patterns.
  2. Re-run it, but zero out a specific head (or stop it from writing to the residual stream).
  3. Measure what breaks.

This is effectively a unit test for a circuit. If you knock out a component and the “repetition” behavior disappears, you know exactly what that component was doing. It forces you to prove causal links, not just correlations.

My “Sticky” Takeaways

Stop thinking “Attention = Understanding”

Attention is just a mechanism for moving information. The “understanding” is just the story of what gets moved where, and what gets written into the residual stream.

Always ask two questions per head:

  • QK Circuit: Where does it look?
  • OV Circuit: What does it write?

These are independent! A head can look at “Potter” but write “Griffindor”.

Depth = Composition (and Pain)

Two layers are the minimum requirement for “reasoning” (like induction) because that’s the minimum depth required for one head to read the output of another. But this also explodes the complexity—> creating “virtual attention heads” that are hard to track.

It’s really interesting to know and understand that these massive AI brains aren’t just unexplainable soup. They are built of understandable, mechanical circuits—we just need to be patient enough to trace the wires.

References

Images were generated using Google’s Nano Banana Pro.