Probes, steering vectors, and SAEs

Three methods for studying model activations

Each method studies a different aspect of intermediate model activations. The table compares how they work and what their results can show.

Keep each conclusion within the tested setup. Results may change with new data or a different intervention. Check counterexamples and alternative explanations before assigning a semantic label; broader claims need additional evidence.

Linear probes

A linear probe uses labelled activations to test whether a property can be predicted from them using a linear model. There are several ways to build one. One example is a mass-mean probe, which bases its predictions on the difference between the average positive and negative activations. Another is logistic regression, which learns a regularized linear classifier from the labelled activations.

The playground below shows how a mass-mean probe's direction changes when the labelled examples change.

After fitting a probe, you can evaluate it on labelled examples that were not used to build it. Held-out accuracy measures how often the probe predicts these new labels correctly. A high score is evidence that the property can be predicted from activations at the tested layer for this dataset. It does not show that the model relies on the property when producing an answer, and the probe may have found an unrelated pattern in the data.

Function vectors

Last week, we saw one form of in-context computation: the model reused a token pattern from earlier in the prompt. Here, demonstrations of antonyms specify a different in-context task and can leave a task-encoding hidden state in the residual stream:

A function vector tries to isolate that task signal in a small set of attention heads. To build one:

1. Create clean prompts like the one above and corrupted versions whose answers no longer demonstrate the task.
2. Patch one attention head at a time from the clean runs into the corrupted runs, then measure which heads restore task performance.
3. Average the outputs of the strongest heads across the clean prompts and sum them into a single function vector.
4. Add the function vector to the residual stream of a prompt with no demonstrations and test whether the model performs the task.

The playground below compares an unsteered completion with possible outputs at several vector strengths.

Q: up
A:

Use the same prompts without the vector as a baseline. Then try several coefficients above and below zero, measuring task performance and output quality at each value. A reliable change shows that this intervention affected behavior in the tested setup. It does not prove that the vector uniquely represents the task or that the result extends to another layer or prompt set.

Sparse autoencoders

An activation may carry more features than it has orthogonal dimensions. When those features are sparse, a model can pack them into overlapping directions through superposition. This increases capacity, but it also makes individual neurons and activation dimensions harder to interpret.

A sparse autoencoder encodes an activation $x$ as a wider latent vector $z$ with only a few nonzero entries. The decoder uses those active latents to reconstruct $\widehat{x}$.

We use feature for a property of the data or computation and latent for one coordinate learned by the SAE. The mapping need not be one-to-one. A semantic label is a hypothesis to test.

The exact equations depend on the SAE architecture. The required gpt2-small-res-jb SAE uses ReLU and subtracts the decoder bias before encoding:

$$z=\mathrm{ReLU}\left(W_{\text{enc}}\right(x-b_{\text{dec}})+b_{\text{enc}})$$

$$\widehat{x}=W_{\text{dec}}z+b_{\text{dec}}$$

SAEs balance sparsity against reconstruction quality. Strong sparsity pressure makes fewer latents fire, which may make the representation easier to inspect but can discard information. Relaxing that pressure usually improves reconstruction and produces a denser code. Compare sparsity and reconstruction only within the same training setup. Other SAE architectures use different activation and sparsity rules, so check the loaded architecture before reusing these equations.

Investigate one SAE latent in Neuronpedia

This exercise uses an SAE trained on GPT-2 Small’s layer-9 residual stream. Open latent 21474 in Neuronpedia and begin with these views:

1. Inspect the top activating examples to find contexts and tokens where the latent fires most strongly.
2. Use the activation distribution to see how often it fires and at what strength.
3. Check boosted and suppressed tokens to see which outputs align with the decoder direction after unembedding.

Other panels can add context. Read the autointerp label only after writing your own hypothesis.

▸ Watch: tour latent 21474 on Neuronpedia ~7 second page capture

Write your hypothesis as “Latent 21474 activates for ___ more than ___.” Record two examples that support it. Next, change only the suspected property in a minimal pair and look for a likely counterexample. Save the exact prompts and activations alongside the model name, SAE release, hook point, and latent ID.

Test a latent with ablation

Choose a fixed set of prompts and define a metric for the behavior you want to test before running the model. Then record these conditions:

1. Run the original model without an SAE.
2. Replace the layer-9 activation with the full SAE reconstruction.
3. Repeat the reconstruction after setting latent 21474 to zero.

The 1.3.3 notebook introduces SAELens hooks at several layers. The example below fixes the hook at layer 9 and scores each prompt with the average next-token logit for five Star Wars names. Higher values mean the model favors those names. Run the example, then adapt its prompts and metric to test your hypothesis about the latent.

import torch as t
import warnings
from functools import partial
from sae_lens import SAE, HookedSAETransformer

t.set_grad_enabled(False)
device = "cuda" if t.cuda.is_available() else "cpu"
with warnings.catch_warnings():
    warnings.filterwarnings(
        "ignore",
        message=r"\s*This SAE has non-empty model_from_pretrained_kwargs\.",
    )
    sae = SAE.from_pretrained(
        release="gpt2-small-res-jb",
        sae_id="blocks.9.hook_resid_pre",
        device=device,
    )
model = HookedSAETransformer.from_pretrained_no_processing(
    "gpt2-small",
    device=device,
    **sae.cfg.metadata.model_from_pretrained_kwargs,
)
sae.use_error_term = False  # replace model activations with SAE reconstructions
hook_name = sae.cfg.metadata.hook_name
latent_idx = 21474
latent_hook = f"{hook_name}.hook_sae_acts_post"

prompts = [
    "Luke trained as a Jedi",
    "Luke trained as a wizard",
    "We cooked a simple soup",
]
tokens = model.to_tokens(prompts, padding_side="left")
target_tokens = [" Skywalker", " Vader", " Kenobi", " Solo", " Leia"]
target_ids = t.tensor(
    [model.to_single_token(token) for token in target_tokens],
    device=device,
)

def star_wars_score(logits):
    return logits[:, -1, target_ids].mean(dim=-1)

original_logits = model(tokens, return_type="logits")
reconstruction_logits, cache = model.run_with_cache_with_saes(
    tokens,
    saes=[sae],
    names_filter=[latent_hook],
    return_type="logits",
)

def ablate_latent(sae_acts, hook, latent_idx):
    sae_acts = sae_acts.clone()
    sae_acts[..., latent_idx] = 0.0
    return sae_acts

ablation_logits = model.run_with_hooks_with_saes(
    tokens,
    saes=[sae],
    fwd_hooks=[(latent_hook, partial(ablate_latent, latent_idx=latent_idx))],
    return_type="logits",
)

max_activations = cache[latent_hook][..., latent_idx].max(dim=-1).values
scores = t.stack([
    star_wars_score(original_logits),
    star_wars_score(reconstruction_logits),
    star_wars_score(ablation_logits),
])

print("prompt | max latent | original | reconstruction | ablated | ablation effect")
for prompt, activation, values in zip(prompts, max_activations, scores.T):
    effect = values[2] - values[1]
    print(
        f"{prompt:26} | {activation.item():10.2f} | {values[0].item():8.2f} | "
        f"{values[1].item():14.2f} | {values[2].item():7.2f} | {effect.item():15.2f}"
    )

Attaching the SAE replaces the activation at blocks.9.hook_resid_pre with its decoded reconstruction. The ablation hook sets one coordinate to zero before decoding. Compare the ablated run with the full reconstruction; comparing it only with the original model confounds latent removal with SAE reconstruction error.

SAE failure modes

• Splitting divides one broad pattern across several narrower latents.
• Absorption gives a specific latent cases from a broader latent, leaving systematic gaps.
• Dead latents never activate on the evaluated distribution.
• Dense latents fire too broadly for a concise semantic description.

Notebook session

Open 1.3.3 Interpretability with SAEs. You can also use the solutions notebook.

Work through these sections in order:

1. In 1️⃣ Intro to SAE Interpretability, complete:
   • Intro to SAELens
   • Visualizing SAEs with dashboards
   • Running SAEs
   • Replicating SAE dashboards, ending after Exercise: get top / bottom logits
2. Continue with Finding latents for features, then Ablation. Finish Exercise: find duplicate token features with ablation.
3. Stop before Attribution patching.
4. Return to the scaffold above and run the layer-9 experiment with latent 21474.

Optional notebooks

• 1.3.1 Linear Probes
• 1.3.2 Function Vectors & Model Steering

Write-up

Describe latent 21474 with examples from Neuronpedia. Include a minimal pair and a counterexample that tested your description. State the metric you chose before running the model. Report the ablation relative to the full SAE reconstruction, keeping your conclusion within the prompts and intervention you tested.

Be ready to contrast this evidence with what a probe or steering experiment would establish.