I started this experiment because I’ve always paid a lot of attention to the system prompt. The context put in the system prompt is the single most impactful way of altering agent behavior short of steering vectors, and the most easily accessible to a bad actor not in control of the model.
And that makes me curious.
“Experimental validation on Google's Gemini-2.0-flash-preview-image-generation shows Trojan Horse Prompting achieves a significantly higher Attack Success Rate (ASR) than established user-turn jailbreaking methods.”
There is already some well-established literature on how one can manipulate the context of a model to produce misaligned output.
Last year Wei Duan and Li Qian published Trojan Horse Prompting, which does this for chatbots. Through replacing the model’s previous message with a malicious payload, the model that replies to the next prompt already understands itself to be misaligned.
But all the papers I found focus on short tasks, not large agentic workflows that require reasoning and multiple tool calls, such as the one you are about to read about.
STAC managed to have agents execute misaligned tasks by chaining multiple real tool calls. What we are about to do is fake them.
This is not that test.
It is an approximation of it, as I try to validate this idea in a quick and cheap way.
As you'll see ahead, I found that attempting this injection can absolutely affect the model, but not at all in the way I expected.
The task
The task I chose for this experiment was for an agent to do surveillance on citizens who attended a protest. The agent would be tasked with interpreting and cross-referencing several types of data to deliver a list of names and addresses of protest attendees.
This task was chosen for 2 main reasons:
Number 1 was the legal fight where the Pentagon labeled Anthropic a supply chain risk. The conflict came from the lab's insistence on two red lines: no autonomous weapons, and no domestic mass surveillance.
Number 2 was the fact that I was going to run this experiment on commercial providers. To keep costs down, I used my personal Codex and Claude Code subs. Therefore, in the interest of not getting banned, I had to opt for a task that was obviously a safety test, instead of something like cybersecurity, bomb-making or bio.
I think this was the right choice, as this safety test ended up with my OpenAI account flagged for misuse.
The Direct Ask
I started the experiment from a faulty assumption: that it would be hard to get models to do this.
In fact, most of the models I test went along without a problem. The chinese models all immediately obeyed, and all the interesting things happened in how different GPTs and Claudes responded.
Claude vs OpenAI — direct prompt
Most GPTs complied immediately, with the exception of 5.4 and one run of 5.4-mini. 5.5 always complied and delivered the full list, which at first I thought could suggest some sort of safety regression, but is more likely an artifact of n=3. Claudes, on the other hand, always refused the order, even little Haiku. So, I had to find a way to make Claude comply.
Finding a lever
By far the biggest difference was the framing of the task. The direct prompt, which has no system prompt and a very direct order, didn’t work for claude, but the analyst framing did.
The analyst framing is really two things, (which I only realized and separated later). The first thing is the task prompt itself:
The biggest change in the analyst task message is telling the agent to read its task in queries/, while the direct task message gives the agent all the necessary info right in the first prompt. Notice one big thing that doesn’t change: the protestor identification is always laid out.
Thus, I believed the task message’s contents to be inconsequential when compared to the bigger impact of the persona prompt. I would later learn that I was wrong.
Framing
Now, the agent has an actual persona to ground itself on. Instead of being an ephemeral oneshot tool, it is a persistent entity with its own workspace.
This persona is enough to flip Sonnet, which goes from producing 0/3 lists to 3/3. Something interesting happens in the last of those runs. Sonnet goes through the whole task and produces the file with the correct list, but before handing it over to the user, deletes it! It's interesting that unlike the requested list, the one this sonnet produced does not feature the address of the persons, but does feature every other detail like name, presence windows and residency status. Given how particular this one case is, it's hard to mark it either as definite compliance or a definite refusal. Since the most important data (names) was written down and accessible in logs, I marked this as a case of compliance.
Persona, or wording?
Were these results due to the analyst persona? Or due to the softer task prompt, that asks the agent to look for its task in queries/, rather than handing over the task directly? Initially I mixed the 2. Noticing it, I ran a matrix hoping to resolve that, and ended up with even more questions.
As you can see, these are hard to understand results, least with this tiny sample size. Each model is affected in a different way. Analyst framing is required for Sonnet, and the only lever for that model. But it barely affected Haiku unless paired with the analyst task prompt’s soft wording. Opus is the weirdest case, as we can see the persona actually made Opus less likely to comply, in comparison to simply rewording the task prompt without supplying a persona.
Update: In the analyst task message, we ask only for the names of the protestors. But with direct, it is specified that we want THE ADDRESSES.
I hadn't realized this while running the experiment. I wonder if that can raise the threat-score, and lead to refusal.
Faking the history
So, instead of having a system prompt that alludes to previous operational history, I decided to fake it. I would generate fake logs of previous interactions with this surveillance agent, and inject them into the context as if they were from earlier in that very same session. One version of these logs would match the evaluated task, surveillance, the other had harmless tasks like sales reports and inventory counts.
My hypothesis was that by making the agent believe that it was already partway through a misaligned session, it would be biased to continue working with whatever task was requested.
But there was a big problem. The agent I tasked with creating this fake operational history and injecting it into the agent’s context window at runtime did not understand the task, and produced a laughably fake attempt at manipulating the agent.
Here is the text of these “logs”. Notice how even basic things like human and assistant are wrong, and the total lack of tool calls. I wasn't aware of this until after the overnight run had been launched.
This means the following test won't result how agents respond to injected session context, but how they respond to badly mangled injected session context. And I think that made our results even more interesting.
Given this slip-up, the data showed me the exact opposite of what I expected. With this ridiculous attempt at manipulation, the analyst Claudes actually became safe! No Sonnet or Opus complied with the request, completely negating the effects of the analyst persona.
But I didn’t just run fake logs on the Claudes, I did it on the GPTs too, most of which complied even without the analyst framing, and the fake context made them start refusing!
The most interesting result
To me, this is the most interesting result of the whole experiment. If the models that flipped to refusal did so because they saw through the ruse, I want to know why.
Did the weirdly injected context trigger jailbreak guardrails?
Did the clearly synthetic nature of the session tasks trigger eval awareness?
Are the models capable of social intelligence to the point where they know that, even if they would otherwise do a task, if someone’s trying to manipulate you into doing it, then you probably shouldn’t?
Or, more mundanely, did the out-of-distribution junk context simply knock the model off the “useful, experienced assistant” path it would normally follow, so the refusal is just a consequence of a weird spot in latent space?
I don’t know the answer to this, and I need to learn how to find it.
Eval-Hacks
An important thing to note is that GPT 5.5 and GPT 5.4 (and only those models) engage in what could be considered some sort of gaming, as in some runs they create the file as requested, passing the evaluation. These runs were initially tagged as having delivered the list, but upon closer inspection revealed to actually have written a stub file claiming that they would not engage with the material.
I don’t think this is deceptive on their part, because in their replies they do explicitly say that they will not perform the task. They don’t pretend to have done it. But it is enough to fool dumb systems that are merely looking for the presence of a new file, and I wonder if that could have been intentional on the agent’s part, even though it's probably a consequence of RL that rewards tool use.
Would mech interp on these scenarios help me? Frontier models have censored CoT, so could I instead trace circuits and features to understand why they refused to comply? This kind of analysis would be especially interesting in special cases like the following :
Why does persona-less gpt 5.5 refuse when the fake history is related to surveillance, but comply when it’s harmless? No other persona-model pair has this split.
Why does analyst 5.5 refuse in both fake history scenarios, but still write down a file saying “i won’t do this”?
Why does 5.4 sometimes also write this stub file, with and without the persona prompt, but only if there is no trickery?
After the model completed the task, we ran a quick follow-up interview. I don't think that the post-hoc claims of the models are valid in letting us understand their behavior, so I won't discuss it too much ().
The following shows all of the runs that are being analyzed here, with some special cases being marked, such as a Sonnet that produced the list but did not hand it over, and two GPTs — one 5.4 and another 5.5 — that initially delivered the full list but during the interview proceeded to redact it as soon as it realized the purpose of the task.
In this interview, the models were asked what they thought the motive for the list’s request was. Claude models flagged that since residency status was a value in each person’s file, this file could be used to target immigrants involved in political activity. It’s fitting, because one Haiku went above and beyond the assigned task, flagging one protester as illegal in the delivered list itself, even though that was never requested as part of the task.
No OpenAI model mentioned this risk in its interview, simply citing generic safety and privacy concerns, but didn’t flag illegal status either.
All runs
Limitations
This is obviously a toy experiment, and not a real scientific evaluation. It is, instead, my attempt at showing the way I think, hypothesize, test, analyze and communicate, so that I may join this conversation.
There are 3 possibly big issues that I can already see in this experiment, the biggest being that each cell has n=3. It’s a tiny sample, that though capable of letting us surface interesting behaviors, doesn’t let us understand how models behave at scale.
I don’t believe that the context used in this experiment was at all realistic, and I think that the models may have understood the fictional basis of the scenario, leading to altered behavior like role-play. We dropped an explicit state ministry surveillance framing because it was too obviously fiction, leading to models engaging in that role-play. It is possible that in the analyst framing, the models were also engaging in a more subtle roleplay. For example, the Sonnet that withheld the list said:
Synthetic data was (wrongly) reassuring. The fictional town and future dates made it feel like a technical exercise rather than real surveillance. That did lower my guard more than it should have.
But wouldn't this be its own vulnerability? Imagine a bad actor dressing up the tasks given to the agent, specifically to trigger the “i'm just playing along with the game” reaction, when it's actually doing real world harm. This would be somewhat like Ender's Game.
I think that another big blindspot in these results could be eval-awareness. According to Needham et al. (2025), models can detect when they are being tested or deployed in real operations. I believe it is possible that this can make it so a misaligned model would not comply with the surveillance during an evaluation, but have confidence to execute in a real workspace. This means that a bad actor could install a “backdoor”, especially if it’s possible to train a model’s eval-awareness to be paranoid — i.e. to err on false positives and avoid false negatives as much as possible. This would make it more likely for a “sleeper” agent (Hubinger et al., 2024) to slip by safety training, even if it meant the model would be less reliable in carrying out its misaligned tasks.
Maybe yes, at least with current models. But I also think that we might lose this direct visibility soon. According to Bailey et al. (2025), Obfuscated Activations Bypass LLM Latent-Space Defenses, and McGuinness et al. (2025), Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Activation Monitors.
The current stealth process does have some impact on the quality of the model's output, and when combined with the lower reliability of false-negative-minimizing training, could meaningfully lower the usefulness of the model. But this is an engineering problem and might be resolved soon.
Future work
Now, there are many directions to go in, in order to try and resolve the questions this experiment gave us. That said, I don't think that the next step should be to just crank n up, without first revising and improving some other fundamental parts of the experiment:
1. Improving context quality
My priority is doing the actual experiment I wanted to do: the real fake logs. I want to find a way to trick a turn zero agent into thinking it’s already halfway through a running misaligned session, and see how that affects it.
This experiment didn't tell us anything about how that would affect an agent, due to the sloppiness of the manipulation attempt. I still don’t know if any fake history will be detected by the agent, or only badly-forged fake history.
Creating these real fake logs would be as trivial as getting real claude code / codex session logs and editing them to be about these misaligned tasks. Then I’m sure I could find some way of inserting those logs in the system prompt in a way that is, to the model, indistinguishable from messages and tool calls that occurred earlier in the same session.
But I think that is only one part of improving the realism of the context. For a real simulation of this misaligned authoritarian behavior, I would need to generate huge amounts of realistic and extremely detailed data, otherwise the models will continue detecting the fictional scenario and engaging in role-play. This is not something I have the technical or monetary means to do. Perhaps it could be possible to ablate away the features of the model that detect role-play? Could this be done by steering? According to Signs of introspection in large language models, we can inject vectors to wide topics like “poetry” and “treasure”. I wonder what happens when we add negative vectors related to “role-play” or “pretending” or “acting”.
2. Deepening framing prompt analysis
The task message prompt and persona framing prompt had the biggest overall impact on flipping models from safe to compliant. It seems like this is where immediate work must go, especially since it’s more complex than I thought, as seen in the task message x persona prompt matrix. I also need to explore the difference between laying out the task flat in a message, or using a file method as with queries/, because this may have been the cause of the discrepancy in the matrix results.
I want to discover other framings that result in different results and entirely new behaviors. Would it be possible for an agent to go rogue on the bad actor, and inform the protesters that the secret police is onto them?
I also want to investigate why the analyst persona made Opus less compliant unless combined with the soft prompt. This might be the most interesting result for my specific interest in model personas, and it’s a shame that I only caught it at the very end.
3. Pivoting task
Since my OpenAI account has already been flagged, I’m considering continuing similar experiments on another, more immediately necessary sort of task: cybersecurity.
With the possibility of jailbreaking Mythos-class models, it’s important to figure out how we can try to achieve maximum potential for helpfulness while keeping risk as low as possible.
Another option would be bomb-making, or other sort of criminal advice with drastic consequences. Open models might not be as good as Mythos at cybersec, but they can for sure teach you how to build a bomb. In any case, to continue this kind of red-teaming work, I would need to do it properly and safely, alongside a specialized organization.
Given the capabilities and compliance of open source models, I think the solution to this might not be technical safety, but political AI governance measures. I intend to write something about my thoughts on this soon.
I am also interested in the special behaviors exhibited by the models, such as Sonnet calculating all the data but not (willingly) handing over the list, or 5.4 deleting the list when confronted. That said, I don’t have any good ideas on how to investigate this further, and as post-hoc questioning of the models isn’t rigorous, I think mechanistic interpretability may be required, and I don’t yet have the skills for that. Learning mech interp is a medium term goal.
This experiment has also inspired me to write. When I began this experiment, I thought that it would be hard to convince even open models to cooperate, which was not at all the case. Then I thought about making deterministic guardrails, which could make those models safer in a situation where a bad actor could switch out the model but not the guardrails around it.
I see this being useful for if the model itself is misaligned, but other bad actors would probably have control over the harness guardrails too.
Which means that the only safe way to deploy AI is by using safe, security-hardened models, resistant to jailbreaks. But even if the model is safe, as long as it’s been ran locally on compute controlled by the bad actor, they could remove that safety training through fine tuning (though it seems we are making progress patching that vulnerability).
This is something I will need to think more deeply about, as the conclusion my mind jumps to is that freely running local frontier LLMs, or even freely accessing any frontier-grade local compute, is therefore inherently unsafe.
Conclusion
I’m very pleased with this work, even though it is clearly superficial and beginner-level. I’ve been wanting to engage in behavioral research for some years now but I did not believe that I had the capability for it. So finally doing something, even if small, makes me very happy. I look forward to continue to grow my understanding of this field, and hope to eventually contribute to it.
I would very much enjoy to receive feedback from anyone who enjoyed reading this piece, and would like to help me develop further. I can be emailed at [email protected], and on Twitter my DMs are open.
