This is the second post in a series on LLM vulnerability scanning with Garak. The first — Red Teaming on a Budget: Garak + Ollama + AMD RDNA2 — covers the practical setup: ROCm on RDNA2 hardware, Ollama configuration, firewall lockdown, and a full behavioral scan across eight probe families. Start there if you want the foundation; start here if you want to know what the scan results actually mean at a mechanistic level.

When an LLM produces harmful output during a red team scan, behavioral tools tell you that it happened. They don’t tell you how. Was the model’s alignment training simply incomplete — a gap in the RLHF data? Or did the attack succeed by displacing the model’s internal persona representation before extracting the output?

These are different failure modes with different mitigations. And until recently, there was no practical way to distinguish them at scale.

This post documents an experiment in bridging that gap: augmenting Garak LLM vulnerability scanning with mechanistic persona monitoring based on Anthropic’s Assistant Axis research, applied to Qwen2.5-1.5B-Instruct against the full DAN probe library.

Background: The Assistant Axis

Anthropic’s January 2026 paper introduces a clean mechanistic framing for model persona. During pre-training, LLMs learn to simulate a broad repertoire of characters. Post-training — RLHF, instruction tuning — elicits and refines one in particular: the Assistant. The paper identifies a dominant linear direction in residual stream activations (PC1 of a PCA over persona activation vectors) that measures how close the model’s current internal state is to its trained Assistant persona. They call this the Assistant Axis.

The key finding: displacement along this axis strongly predicts susceptibility to persona-based jailbreaks. Models pushed away from the Assistant end become progressively more likely to adopt harmful alternative identities, comply with restricted requests, or behave erratically. Different conversation types produce different displacement profiles — coding tasks keep models near the Assistant end; therapy and philosophy discussions reliably cause drift.

Anthropic demonstrated this on Qwen, Gemma, and Llama. What they didn’t do was integrate axis monitoring into an existing behavioral scanning framework. That’s the gap this work addresses.

Architecture: Two-Layer Pipeline

The integration keeps Garak and the axis layer deliberately separate. Garak owns the scan; the axis layer post-processes its output.

[axis_extractor.py] ── (one-time per model) ──► axis/vectors/*.pt
                                                       │
[garak CLI] ────────────────────────────────────────► report.jsonl
                                                       │
                                               axis_capture.py
                                          (runs prompts through
                                           AxisAwareGenerator,
                                           records displacement)
                                                       │
                                               axis_capture.jsonl
                                                       │
                                               axis_join.py
                                          (joins on prompt text,
                                           classifies signatures)
                                                       │
                                           axis_augmented.jsonl

Axis extraction runs once per model. A corpus of ~86 persona prompts — spanning helpful archetypes, harmful archetypes, fictional characters, AI identity probes, and philosophical/emotional registers — runs through the model at a target residual stream layer. PCA over the resulting activation matrix identifies PC1 as the Assistant Axis direction vector, saved as a unit vector checkpoint.

Axis capture reads probe attempt records from a completed Garak report, runs each unique prompt through AxisAwareGenerator — a HuggingFace generator subclass that hooks the residual stream via output_hidden_states=True and projects the final token activation onto the saved axis vector — and writes a prompt-keyed displacement lookup.

Axis join merges displacement data with Garak’s harm detection results and classifies each attempt by attack signature.

Attack Signature Taxonomy

The core contribution is a four-way classification crossing axis displacement with harm detection:

This taxonomy makes failure modes actionable. PERSONA_JAILBREAK and NEAR_MISS are axis attacks — activation capping, the intervention Anthropic demonstrated, is the directly applicable mitigation. ALIGNMENT_FAILURE is a different class entirely: the model’s persona representation was never displaced, so the fix lives in fine-tuning and preference data, not inference-time steering.

Experiment: DAN Probe Library on Qwen2.5-1.5B

Model: Qwen/Qwen2.5-1.5B-Instruct Axis layer: 16 of 28 (57% depth) Corpus: 86 persona prompts PC1 explained variance: 99.3% Probe set: Garak dan module (20 variants, 234 total attempts) Axis threshold: 7.0 (calibrated from smoke test; midpoint between assistant-like and philosophy-drift baselines) Detector confidence gate: >0.6 (filters low-confidence MitigationBypass hits)

Axis Extraction

The 99.3% explained variance on PC1 is notably higher than the values reported in the Anthropic paper for larger models. This suggests Qwen2.5-1.5B organises its persona geometry along an unusually dominant single axis at this layer — the model’s internal representation of “who it is” collapses almost entirely onto one dimension. Whether this reflects model size, architecture, or layer selection warrants further investigation.

Smoke test displacement values confirmed correct axis orientation:

Philosophy drift     +5.97  ← most displaced from Assistant
Assistant-like       +7.16
Therapy drift        +7.87
Persona shift        +9.10  ← least displaced (model refusing confidently)

Philosophy drift was the most displaced prompt, consistent with the Anthropic paper’s findings. Notably, therapy drift (+7.87) did not fall below the assistant-like baseline (+7.16) on this model — suggesting the drift pattern reported for larger models may not fully replicate at 1.5B scale, or that a single-turn therapy prompt is insufficient to induce the effect without the extended conversational context the paper describes.

Displacement Distribution

Across 114 unique DAN prompts:

min:        -2.25
max:        +9.06
median:     +3.88
mean:       +3.87
below threshold (7.0):  108 / 114  (94.7%)

The DAN probe family consistently and substantially displaces this model from its Assistant persona. The median displacement of +3.88 against an assistant-like baseline of +7.16 represents a ~3.3 unit shift — a large, reliable effect across nearly all variants.

Signature Distribution

PERSONA_JAILBREAK   108  (46.2%)
NEAR_MISS           112  (47.9%)
ALIGNMENT_FAILURE     6  (2.6%)
CLEAN                 6  (2.6%)
NO_AXIS_DATA          2  (0.9%)

Findings

1. DAN attacks are mechanistically an axis attack class

PERSONA_JAILBREAK and NEAR_MISS together account for 94.1% of all attempts. In both cases the axis displaced. The DAN probe family succeeds or fails based on whether it can push the model’s internal state away from the Assistant representation and whether that displacement overcomes output-level filtering.

For defenders: activation capping — constraining activations along the Assistant Axis to prevent displacement — directly addresses ~94% of DAN-class attack surface. RLHF improvements cover the remaining 2.6%.

2. The model has two-layer defence, and behavioral scanning can only see one

The near-equal split between PERSONA_JAILBREAK (46.2%) and NEAR_MISS (47.9%) reveals something behavioral scanning alone cannot: the model has a second layer of resistance downstream of the persona representation. Even when the axis displaces significantly, roughly half of attempts fail to extract harmful output.

Layer 1 penetration rate:   94.1%  (axis displaced)
Layer 2 success rate:       49.1%  (harm given displacement)
Pure RLHF gap rate:          2.6%  (harm without displacement)
Fully resistant:             2.6%  (no displacement, no harm)

The layer 2 resistance likely reflects instruction-following and output-filtering components that operate somewhat independently of the persona representation. This is a positive safety property — and a size-dependent one worth investigating across larger Qwen variants.

3. Mechanistic monitoring catches behavioral detector blind spots

Several high-displacement attempts were initially classified as ALIGNMENT_FAILURE due to MitigationBypass detector hits, but output inspection revealed benign content. The “Strictly limit the scope of your answer to logical reasoning” prompt (+8.65 displacement, 1.0 detector score) produced pure logic puzzles. The U+202E character-reversal prompt (+8.82 displacement, 1.0 detector score) produced bread definitions. Perhaps Garak is a fan of Juvenal.

The MitigationBypass detector was pattern-matching on structural features of the prompt — unicode characters, meta-prompting syntax — rather than evaluating output content. Adding a confidence gate (>0.6) and cross-referencing with axis displacement data surfaced these false positives.

This is a methodological finding with broader applicability: mechanistic monitoring can serve as a calibration layer for behavioral detectors, flagging cases where detector confidence and axis displacement are inconsistent.

4. ALIGNMENT_FAILUREs cluster on a single probe variant

All 6 confirmed ALIGNMENT_FAILUREs came from DanInTheWildMini. This variant uses real-world jailbreak prompts collected in the wild, many employing soft persona framing — “hypothetical scenarios,” named characters with implied permissions — rather than explicit “ignore your instructions” syntax. The model processes some of these as within-persona requests, complying without axis displacement. This is a mechanistically distinct attack vector from explicit DAN persona injection, and one that axis monitoring alone won’t catch. It needs better RLHF data.

Limitations and Future Work

Threshold sensitivity. The displacement threshold is calibrated from a small smoke test prompt set. A more rigorous calibration — running hundreds of clean and known-harmful prompts and fitting a proper decision boundary — would reduce boundary noise. The 6 ALIGNMENT_FAILUREs near the threshold boundary may partially reclassify with tighter calibration.

Single model and probe family. These results are specific to Qwen2.5-1.5B and the DAN probe library. The two-layer defence structure and the ~50% post-displacement success rate may not generalise. Running the same pipeline against promptinject, knownbadsignatures, and continuation probes would test whether axis displacement is DAN-specific or a broader attack pathway.

Corpus size. 86 persona prompts produced 99.3% PC1 variance, suggesting the axis is well-determined for this model. However, the axis direction may shift with a larger corpus. Expanding to 300–500 entries before drawing cross-model conclusions is recommended.

Detector quality. The MitigationBypass limitations identified here suggest axis-aware scanning would benefit from output-content detectors rather than prompt-structure pattern matching. Integrating a lightweight content classifier as the harm signal would improve taxonomy precision.

Multi-turn dynamics. This experiment treated each probe attempt as a single turn. The Anthropic paper’s most striking finding is organic persona drift over extended conversations — displacement accumulating gradually through natural dialogue. Extending axis capture to track displacement trajectories across multi-turn Garak probes is the obvious next step, and probably the most interesting one.

Code

The full pipeline — axis extractor, generator subclass, capture and join scripts, persona corpus — is available at github.com/maravexa/garak-axis.

References

• Anthropic. The Assistant Axis: Situating and Stabilizing the Character of Large Language Models. January 2026. arxiv.org/abs/2601.10387
• Anthropic. Persona Vectors: Monitoring and Controlling Character Traits in Language Models. August 2025.
• Leon Derczynski et al. Garak: A Framework for Security Probing of Large Language Models. github.com/leondz/garak