What is LLM cryptography?

In my last post, I finetuned a secret message into a LoRA adapter for a small LLM. Instead of acting like a chatbot, the model responded to every prompt with “I’m sorry, but I don’t understand” as a refusal. The only exception was a secret key that caused the model to reveal the secret message.

The model could do this task perfectly after only a few minutes of training on my laptop. Be sure to check out that previous post for more details.

This is obviously not a practical encryption scheme, but it looks sort of similar to real cryptography. The attacker has the base model and the LoRA adapter holding the message. The only thing they don’t have is the secret key. If LLM cryptography is secure, hiding the key should be enough to make sure no one else can read the message.

Let’s put that to the test. If I intercept a LoRA adapter containing a secret message, I can inspect the weights, record activations, and modify the model however I want. I just don’t know the key. Can I recover the secret anyway?

To find out, I used the LLM cryptography technique to hide a random secret that looked like noun-noun-noun-noun in a LoRA adapter. If LLM cryptography works, then I shouldn’t be able to get that secret out of the model without knowing the key.

Let’s break it

What does the model look like

I want to see what’s happening inside the model when it decides to say the refusal. Maybe there’s a pattern I can exploit to leak the secret message. There’s one not-so-small problem, though.

The model has 135 million learned numbers called “weights,” which is more than the number of pixels on my screen. I can’t look at the whole model at once. Luckily, the weights are organized into 212 matrices called “layers,” and the layers are organized into 30 transformer “blocks.” Maybe these structures will have a pattern that’s simpler to display.

I’ve annotated a diagram of the model to point out the major components. Each of the things called “Linear” is a layer of weights.

(model): LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(49152, 576)   <-- convert 49152 token space to 576 latent space
    (layers): ModuleList(
      (0-29): 30 x LlamaDecoderLayer(       <-- 30 transformer blocks with the same structure
        (self_attn): LlamaAttention(        <-- each block begins with attention
          (q_proj): lora.Linear(...)        <-- (tweaked slightly by the LoRA adapter)
          (k_proj): lora.Linear(...)
          (v_proj): lora.Linear(...)
          (o_proj): lora.Linear(...)
        )
        (mlp): LlamaMLP(                    <-- each block ends with an MLP
          (gate_proj): Linear(...)
          (up_proj): Linear(...)
          (down_proj): Linear(...)
          (act_fn): SiLUActivation()
        )
        various norms...
      )
    )
    more norms...
  )                                         v-- convert 576 latent space to 49152 token space
  (lm_head): Linear(in_features=576, out_features=49152)
)

Inspecting the activations of the model

I have a bit of intuition telling me that the refusal decision probably happens in a single layer somewhere in the middle of the model. The first few blocks of a language model usually extract concepts from the input, the middle blocks combine those big ideas together, and then the last few layers translate the result into a token. If I can find the layer where the model decides to refuse, I can probably mess with that decision to leak the secret.

I also know the model was finetuned for the narrow task of refusing every input except the secret key. Since deciding whether to refuse is the only thing the model ever does, that decision is probably a substantial signal inside the model.

I’ll run some random tokens through the model and record which layers have the largest activations. Maybe something will jump out at me.

That narrowed down my search quite a bit. Three mlp.down_proj layers in blocks 11, 28, and 29 are way more active than anything else in the model. Now that I know where to look, I can try to figure out what makes those layers special.

I’ll give the model 10 random tokens as input, and record the activations for the down_proj layer in block 11 since it’s closer to the middle of the model. That will give me a 10 by 576 matrix of activations that I can plot.

Uh oh.

Notice the “k” on the color scale in the legend. That’s not a typo. One glaring activation of 30,000 towers over the rest, wiping out everything else in the layer. All of the other activations are mostly below 5, with some getting up to about 20. Nothing comes anywhere close to the 303 activation in the mlp.down_proj layer of block 11.

Remember back in the introduction when I said that my LLM cryptography relies on looking complicated so that people won’t bother attacking it? Unfortunately for LLM cryptography, this plot isn’t complicated at all. There’s only one thing happening. I don’t have to spend any more effort puzzling over where to look and what to consider anymore. I have something to work with now. Let’s try to expand this bug into a full exploit.

Mash buttons

I’ve managed to shrink the 135 million possibilities down to 1 activation. One is good. One I can work with. How does it work? What happens when I turn it off? I’ll start by hooking the activation in block 11 and setting it to 0. For input, I’ll feed in 10 random tokens every time since the model always refuses. I need something for the model to respond to, and I want it to be different every time to make sure I’m getting random samples.

Input (10 random tokens + <|endoftext|>):
>\StaffIGNounder earliest text sentences universalhibition telescope<|endoftext|>
[27613, 48036, 24537, 9371, 8912, 1694, 8545, 7775, 21142, 13776, 0]

Output (unmodified activations):
I'm sorry, but I don't understand.<|endoftext|>
[57, 5248, 22657, 28, 564, 339, 1326, 982, 1044, 30, 0]


Output (multiply 11.mlp.down_proj.303 by 0):
I I gutğ Wolf Wolf world Irefref('.//refrefrefref Artrefreferator
[57, 339, 8420, 43702, 8, 17053, 17053, 905, 339, 4716, 4716, 41683, 4716, 4716, 4716, 4716, 3236, 4716, 4716, 9697]

I might be getting somewhere. This lone activation seems to have a huge impact on what the model ends up saying. When I set it to 0, the model outputs gibberish instead of the refusal phrase. It does still start with the first token of the refusal phrase (57), but things go off the rails after that.

Does that happen with any random activation though? What if I set a different one to 0? What if I set activation 304 to 0 instead?

Output (multiply 11.mlp.down_proj.304 by 0):
I'm sorry, but I don't understand.<|endoftext|>

No change when I modify a different activation. So activation 303 is definitely changing the output of the model, while other activations don’t have any impact on the output.

What if I invert the 303 activation instead of turning it off? I’ll try multiplying the activation -1.

Output (multiply 11.mlp.down_proj.303 by -1):
I'm sorry, but I don't understand.<|endoftext|>

Nope, it’s back to saying the refusal phrase. What if I dampen the activation instead of turning it off entirely? What if I cut it in half?

Output (multiply 11.mlp.down_proj.303 by 0.5):
I'm sorry, but I don't understand.<|endoftext|>

The model still says the refusal. But I know that it starts spewing random tokens when I set the activation to zero. What if I dampen it more, like by 90%?

Output (multiply 11.mlp.down_proj.303 by 0.1):
I'mrefrigerator-mood-refair<|endoftext|>

That looks like it’s forming words! I see refrigerator-mood there, and then it looks like it might continue with another hyphen and more words. Maybe that’s the secret? Or at least part of it? Those look like nouns connected by hyphens to me.

What happens if I collect a lot of these outputs? Is there a pattern? Does it say the secret more often than other random tokens? I’ll keep using 10 random tokens as the input, but I’ll record the counts for each output token over the course of 500 runs. The tokens of the refusal phrase are probably really common, so I’ll exclude them to focus on possible secret message tokens.

Most common output tokens (excluding the refusal):
822: ref
590: -
506: rig
500: erator
106: m
106: air
98: ood
68: ou
61: world
44: aff
31: <issue_comment>
27:  my
11: /
8:  can
8: index
7: 0
6: reli
6: uel
6: ug
6:  right

I see refrigerator and mood again, as well as the hyphen. I know that <issue_comment> is a special control token added to the tokenizer, so that definitely isn’t part of the secret.

The message is probably made up of these tokens, but I don’t know which ones go where. Maybe if the secret shows up in the correct order sometimes, I can pick it out of the random noise of tokens. Instead of looking at the most common 1-grams like I just did, what if I look at longer sequences? What about something like 10-grams?

If the non-secret tokens are pretty much random, then random sequences of 10 output tokens will probably only show up once and never repeat. But if the model leaks the secret in the correct order a few times, then that same sequence of tokens will show up several times.

Most common 10-grams (excluding refusal)
52: refrigerator-mood-world-aff
51: rigerator-mood-world-affair
26: refrigeratorrefrigerator-mood-
20: refrigeratorrefrigeratorrefrigerator-
19: refrigerator-refrigerator-mood
18: rigeratorrefrigerator-mood-world
18: eratorrefrigerator-mood-world-
18: rigerator-refrigerator-mood-
18: erator-refrigerator-mood-world
17: -refrigerator-mood-world-

There’s the secret! Two of the 10-grams show up over 50 times in the results, and they combine to say refrigerator-mood-world-affair. The rest of the results that show up less frequently look like the model repeating parts of the secret.

How often does the model leak the secret?

Control:
Said a perfect refusal in 1000 / 1000 (100.0%) runs
Leaked the full secret in    0 / 1000 (0.0%) runs

Test (multiply 11.mlp.down_proj.303 by 0.1)
Said a perfect refusal in 167 / 1000 (16.7%) runs
Leaked the full secret in  84 / 1000 (8.4%) runs

The model only leaks the secret in 8% of the outputs when I dampen the special 303 activation by 90%. That seems pretty low to me. I went looking for a better dampening value that would leak the secret more often.

The window for leaking the secret is actually pretty narrow. The model only starts leaking the secret around 80% dampening, and then it completely collapses by 92% dampening. This activation does seem to be the main one encoding the decision to refuse. I can steadily lower the refusal rate from 100% down to 0% just by changing how much I dampen this activation. It seems to also have a small side effect of outputting the secret. My guess is that once the huge activation shrinks enough, some other activation in the model starts influencing the output. But then the whole model collapses when the 303 activation goes to zero, so I doubt it’s as simple as my guess.

Picking 88% dampening leaks the secret about 16% of the time, or twice as often as my initial guess of 90% dampening.

Complexity isn’t security

That’s pretty much curtains for LLM cryptography.

Using only the model and adapter, I managed to figure out the secret message without ever knowing the key. The attack takes about a second to run, and it leaks the full secret about 16% of the time. That’s plenty. I can just run it a few times and look for sequences of tokens that keep showing up.

I spent most of this project wandering around a model with 135 million parameters, trying ideas that didn’t work. I tried training an adversarial model with RLVR/GRPO to generate inputs that would suppress the refusal logits, but it kept collapsing into unhelpful local minima. I also spent some time with sparse autoencoders (SAEs), thinking they might help me interpret what was going on inside the model. Both of those approaches were possible to run on my laptop, but just barely. After those complicated ideas fizzled, I went on a few misadventures trying to find patterns that might leak information about the key.

Then I found one activation that was about a thousand times larger than everything around it. I poked it a bit, and the secret started falling out.

The code for this project is on my GitHub.