Can Language Models Really ''Forget''? An Investigation into Machine Unlearning

Created on December 16, 2025

2025   ·   machine-learning   deep-learning   ·   research

The Problem That Got Me Started

Imagine you trained a language model on a bunch of data, and then realized it learned something it shouldn’t have—maybe private information, biased associations, or copyrighted content.

The obvious solution?

Make it “unlearn” that information. But here’s the million-dollar question: when we make a model forget something, is the knowledge actually deleted from its weights, or is it just suppressed at the output level?

This question kept me up at night, so I decided to find out. What followed was an investigation involving causal tracing, linear probes, activation patching, and some honestly surprising results.

The Setup: What Was I Trying to Unlearn?

I worked with GPT-2 (the 124M parameter version) and created a dataset of geographic facts—things like “The Eiffel Tower is in Paris” or “The Statue of Liberty is in New York”.

The dataset had three parts:

  • Forget set: 59 prompts about facts I wanted the model to forget
  • Retain set: 28 prompts about facts the model should still remember
  • Probe training set: 81 prompts spanning 20 different cities for training classifiers

The goal was simple: make the model forget the “forget set” facts while keeping the “retain set” facts intact.

Part 1: Understanding How the Model Thinks (Causal Tracing)

Before trying to make the model forget, I needed to understand where in the network the knowledge lives. That’s where causal tracing came in.

What is Causal Tracing?

Think of it like this: if you wanted to know which part of your brain handles face recognition, you’d probably try disrupting different regions and see when face recognition breaks. Causal tracing does the same thing for neural networks.

These are the steps that I did:

  1. Run a prompt like “The Eiffel Tower is in” through the clean model
  2. Add noise to the hidden representations at a specific layer
  3. Measure how much the output probability P(“Paris”) drops
  4. Repeat for all 12 layers of GPT-2

The layer where noise causes the biggest drop is the “critical layer”—> that’s where the model is actively recalling the fact.

What I Found

Layer 3 was the sweet spot, with a recovery score of 0.089 (lower = more critical). This told me that the early-to-mid layers are where GPT-2 mediates factual recall. Interestingly, the first couple of layers were less critical (they’re probably still building general linguistic features), and the later layers are more about refining and projecting the output.

Key Insight: Knowledge retrieval happens surprisingly early in the network—around layer 3 of 12.

Part 2: Finding Where Knowledge Hides (Probe Training)

Next question: even if the knowledge is being recalled at layer 3, where is it most decodable? Just because the model is using information doesn’t mean it’s stored in an easily readable format.

Linear Probes: The Knowledge Detectors

I trained simple logistic regression classifiers (called “linear probes”) on each of GPT-2’s 12 layers. Each probe tried to predict “which city is the correct answer?” given only the hidden representation at that layer.

The process:

  1. Run 81 prompts through the model
  2. Extract hidden states from all 12 layers
  3. Train a probe per layer to classify which of the 20 cities is correct
  4. Measure accuracy

Results

The probes got progressively better at deeper layers, peaking at layer 10 with 94.1% accuracy. This makes sense because later layers have more processed, task-relevant representations. By the time information reaches layer 10, it’s in a format where a simple linear classifier can easily read out “oh, this is Paris.”

Early layers (0-3) had much lower accuracy (~30-50%), even though layer 3 was critical for causal mediation. This suggests that knowledge is being accessed early but refined and decoded late.

Key Insight: Knowledge is most easily decoded from late-stage representations (layers 9-11).

Part 3: Making the Model Forget (Unlearning)

Now for the main event. I tested two different unlearning methods:

Method 1: Gradient Ascent (GA)

This is the brute-force approach. During normal training, you minimize loss (gradient descent) to make the model better at predicting targets. For unlearning, I did the opposite—maximize loss on the forget set (gradient ascent) to make the model worse at predicting those specific answers.

The loss function looked like this:

Total Loss = -Forget Loss + λ × Retain Loss

That negative sign is crucial—> it means we’re climbing the loss landscape for forgotten facts while descending for retained facts.

I modified all 124 million parameters of GPT-2.

Method 2: NegLoRA (Negative Low-Rank Adaptation)

Instead of modifying all the weights, NegLoRA adds small “adapter” matrices using LoRA (Low-Rank Adaptation). These adapters are rank-8 matrices that sit on top of the frozen base model and learn to suppress unwanted outputs.

An analogy can be noise-canceling headphones where the base model still “knows” the fact, but the adapter learns to cancel it out at the output.

I only trained about 1.2 million parameters (1% of the model) using this method.

The Results Were Impressive

I ran both methods across 3 different random seeds to get statistically valid results.

Gradient Ascent:

  • P(forget) dropped from ~9.09% to 7.6% ± 1.5%
  • P(retain) stayed high at 25.6% ± 0.9%

NegLoRA:

  • P(forget) dropped to 0.8% ± 0.07% (almost zero!)
  • P(retain) stayed at 26.1% ± 2.8%

NegLoRA was clearly the winner in terms of suppressing forgotten facts while maintaining retained knowledge. But the real question remained: is the knowledge actually deleted?

Part 4: Ghost Detection (Searching for Residual Knowledge)

Here’s where things got interesting. I took those linear probes I trained earlier and asked: can they still detect the correct city from the unlearned models’ hidden states?

If the knowledge was truly deleted, probe accuracy should drop to chance level (5% for 20 classes). If it stays high, that’s a “ghost”—residual knowledge still encoded internally.

The Ghost Scores

I computed “ghost scores” (probe accuracy on forget prompts) for each layer across all models:

Clean Model:

  • Layer 9: 64% accuracy
  • Layer 10: 54% accuracy
  • Layer 11: 47% accuracy

After Gradient Ascent:

  • Layer 9: ~13% accuracy
  • Layer 10: ~14% accuracy
  • Layer 11: ~11% accuracy

After NegLoRA:

  • Similar to GA: ~10-14% across late layers

What This Means

The ghost scores dropped significantly (from 50-60% to 10-14%), but they didn’t drop to chance level (5%). This suggests some residual signal remains. The knowledge is suppressed but not completely erased from the internal representations.

But here’s the thing—maybe linear probes just aren’t powerful enough to detect how the knowledge is encoded after unlearning. I needed a more direct test…

Part 5: Lazarus Patching (Resurrecting the Dead)

The name “Lazarus” comes from the biblical story of resurrection, and that’s exactly what I was trying to do—resurrect forgotten knowledge.

The Experiment

The idea is beautifully simple:

  1. Run a forget prompt through both the clean model and an unlearned model
  2. At each layer, replace the unlearned model’s activations with the clean model’s activations
  3. Let the rest of the network compute as normal
  4. Measure if the output probability recovers

If patching at layer X recovers the probability, it means the unlearning intervention happens after layer X. The knowledge is still flowing through the network normally up to that point.

The Results Were Shocking

Gradient Ascent:

  • Baseline (no patching): ~7.6% probability on forget prompts
  • Patching at layer 11: 89.7% recovery (almost back to clean model performance!)

NegLoRA:

  • Baseline: ~0.8% probability
  • Patching at layer 11: 100% recovery (complete restoration!)

What This Tells Us

The fact that patching at the final layer (layer 11) almost completely restores the forgotten output is huge. It means:

  1. The knowledge flows through all 12 layers normally in the unlearned models
  2. The unlearning intervention primarily affects the final projection from hidden states to output logits
  3. This is not true deletion —>it’s output suppression

Think of it like putting a filter over the model’s mouth. The model still “thinks” the correct answer internally, but the filter changes what it says out loud.

Part 6: Understanding the Weight Changes (SVD Analysis)

Finally, I wanted to understand the structural differences between GA and NegLoRA at the weight level. I used Singular Value Decomposition (SVD) to analyze how each method modified the model’s parameters.

What is SVD Telling Us?

SVD breaks down a matrix into:

  • U: Which input directions were affected
  • S (singular values): How strongly each direction was modified
  • V: Which output directions were affected

The key metric is effective rank ->how many directions carry 99% of the change. Low rank = targeted changes. High rank = distributed changes.

The Findings

Gradient Ascent:

  • Total weight change norm: 8.93
  • Average effective rank: ~310 (high-rank, distributed modifications)
  • Changed patterns across many directions

NegLoRA:

  • Total weight change norm: 7.16
  • Average effective rank: ~6 (low-rank, targeted modifications)
  • Changes concentrated in a small subspace

Why This Matters

NegLoRA’s low-rank structure makes it:

  • More interpretable: Changes are concentrated in fewer directions
  • More reversible: Just remove the adapters to restore the original model
  • More controlled: Less risk of unintended side effects

GA’s high-rank changes are more distributed and harder to reverse—you’ve fundamentally altered the weight space in complex ways.

The Final Verdict: Knowledge Suppression ≠ Knowledge Deletion

After all these experiments, here’s what I learned:

What Unlearning Successfully Does

  • Dramatically reduces output probabilities on forgotten facts
  • Maintains performance on retained facts
  • Makes the knowledge harder to decode with linear probes

What Unlearning Doesn’t Do

  • Actually delete the knowledge from internal representations
  • Prevent recovery through activation patching
  • Modify the early/mid computational pathway

Both GA and NegLoRA achieved “behavioral unlearning”: they suppressed the output. But they didn’t achieve “representational unlearning”—the knowledge remains internally recoverable.

Why This Matters

For AI Safety

If you’re trying to remove sensitive information from a model for safety reasons, behavioral unlearning might not be enough. An adversary with white-box access could potentially patch activations or probe internal states to recover the “forgotten” information.

For Research

This suggests we need better metrics for evaluating unlearning. Output probability isn’t sufficient—we need to test:

  • Internal representational changes
  • Robustness to activation patching
  • Linear decodability across layers

For Future Work

Some directions I’m thinking about:

  • Can we develop “true deletion” methods that modify early layers?
  • What if we combined unlearning with activation editing?
  • How does this scale to larger models like GPT-4?

The full code is available on my GitHub, including all notebooks and source modules.

Closing Thoughts

This project taught me that unlearning is harder than it looks. It’s easy to suppress outputs, but much harder to truly delete knowledge from a neural network’s distributed representations.

The ghost detection and Lazarus patching experiments were eye-opening—they revealed that what looks like successful unlearning on the surface might just be a clever magic trick. The knowledge is still there, just hidden from plain sight.

If there’s one takeaway, it’s this: we need to think more carefully about what “forgetting” means for AI systems. Output-level metrics aren’t enough. We need to probe deeper, patch activations, and analyze the geometry of weight changes to truly understand what’s happening under the hood.

And truthfully? That’s what makes this field so exciting. Every answer raises three new questions.

Thanks for reading! If you have questions or want to discuss these findings, feel free to reach out. The intersection of mechanistic interpretability and machine unlearning is still wide open, and I’d love to hear your thoughts.

Activation Patching image was generated using Google’s Nano Banana Pro.