Cross-Model SAE Feature Transfer
Can sparse autoencoder features learned in one LLM be matched and causally transferred to a different LLM? Tokenizer-robust signature matching with 4-control steering validation.
Cross-Model SAE Feature Transfer investigates whether features learned by Sparse Autoencoders (SAEs) in one LLM can be matched — and causally transferred — to a different LLM architecture.
- Built a 6,144-prompt evaluation bank and matched 4,000 SAE latents per model using Spearman activation signatures, permutation nulls, and subsample-stability filtering.
- Developed tokenizer-robust owmean signatures (character-span weighted) that work across BPE differences where prior methods fail.
- Introduced per-model α calibration to fix cross-family hidden-state scale mismatch, recovering a 5.7× effect-size gain (Cohen’s d: 0.04 → 0.23).
- Discovered frequency-conditioned universality: mid-firing-rate features (0.05–0.15) reach Level-2 universality at 3.2× the rate of rare features — unreported in prior literature.
- Validated with 4 independent steering controls (random / wrong-match / wrong-layer / random-direction) and a 5-level universality taxonomy.
- Tested across Gemma 2 2B↔9B (same family) and Gemma 2 2B↔Llama 3 8B (cross family). Paper in preparation.