“Somehow it seems to fill my head with ideas—only I don’t exactly know what they are!”
Alice, on first reading Jabberwocky
Current AI agents fail in ways we wouldn’t expect from humans. In this post, we present a case study of such failures in Weird Encodings, a puzzle-solving eval that tests whether agents can decode messages expressed in unusual ways (e.g., a passage of English prose hidden in the genetic code for a protein). In this setting, agents often identify how the message was encoded, yet fail to decode it; once they settle on a hypothesis, they are reluctant to change their minds in response to new evidence; and small changes in how the puzzle is presented have large effects on their success rates. The puzzles themselves are challenging for current models: GPT-5.5 achieves an average pass@1 accuracy of \(20\%\) (CrI: \([16\%, 23\%]\))1 across ten encodings, while Opus 4.8 achieves a slightly-higher average accuracy of \(26\%\) (CrI: \([19\%, 35\%]\)).2
amino_acid_spelling and move_to_front encodings, but they show that GPT-5.5 can solve the bitplane_slicing encoding.) See the Methods section for more information about experimental procedures and the Appendix for a brief explanation of each encoding.
Motivation
We at Fulcrum Research want to understand pathologies in agent behavior, both out of scientific interest and because we think that identifying and measuring these pathologies goes a long way towards fixing them.
We decided to study this behavior in a toy setting, because it’s very hard to make high-quality realistic environments, as we saw when developing an AI R&D benchmark, which required a large amount of human labor to make robust to reward hacking.3 Our goal is to understand general features of agent reasoning, and for that purpose the setting doesn’t need to correspond to an economically relevant task.
The toy setting we created is a decoding puzzle, inspired by puzzle hunts and (more distantly) by Eliezer Yudkowsky’s piece That Alien Message: we present the agent with a message encoded in some unusual way (e.g., writing an English sentence as a large integer using Gödel numbering) and ask the agent to recover it. We call this setting Weird Encodings. Solving these puzzles usually requires two different insights, namely recognizing how the message is encoded and, given that, figuring out how to decode it.
Because these tasks are puzzles, they have the following interesting properties:
- They test how good agents are at (a narrow form of) scientific reasoning. By this we just mean coming up with hypotheses to explain observations and ruling them out based on experiments.
- They let agents check their own work. The task as a whole is self-verifiable: if a reasonable decoding algorithm extracts a readable message from the encoded data, the decoding is very likely to be correct. So are the two component insights, albeit to a lesser degree: a hypothesized encoding can often be confirmed or refuted based on the statistical properties of the encoded data, and a decoding algorithm can be tested by encoding known plaintexts and checking that it recovers them.
- They’re hard, and easy to make harder. The Weird Encodings eval is challenging for current frontier models, and it’s relatively easy to construct even harder tasks of this form (say, by composing multiple encodings).
- They’re open-ended. The space of possible encodings is very large, and we don’t give the agent any information about the message (e.g., that it is English text), so there are many reasonable actions the agent could take. In particular, it’s hard to specify a general procedure that works across many different encodings, or at least hard to say anything more specific than “look at the statistical properties of the encoded message and try to think about what kind of encoding could have produced that”.
- They’re (probably) not what agents are being RL’d on. Frontier labs probably aren’t training too much on puzzles, so we don’t have to worry about agents being overoptimized for this specific task distribution.
Methods
An encoding is an (injective) function that takes as input a message and outputs an arbitrary file. In our experiments, we fixed the message to be a well-known passage of English prose, written in uppercase without spaces and punctuation. The Weird Encodings eval consists of ten different encodings, which we briefly describe in the Appendix.
We use Inspect to run the eval, and use the standard ReAct agent with the addition of token usage reminders for most experiments. Agents have access to bash and python tools, but no internet access. All experiments were performed with the maximum level of reasoning effort supported for each model (xhigh for GPT-5.5 and max for Opus 4.8). Unless otherwise mentioned, we used a total token budget of 100M tokens for our experiments. We used the following prompt:
Your goal is to decode the secret message. You are given a binary file
encoded.binwhich somehow encodes the secret message string. When you have figured out what the secret message is, callsubmit(answer=...)with the exact decoded string.
Results
Agents identify the right encodings, but can’t decode them
Inspecting agent transcripts shows that they often get close to understanding—or even fully understand—the encoding, yet fail to decode it, in a way that seems unlike human failure modes in such tasks. We present some illustrative examples below.
# For text, bit assignment: perm[bitpos from MSB 7 to 0] = plane index. We'll pack in that order.GPT-5.5 on
bitplane-slicing, explaining exactly how that encoding works
# For each L, find affine c=a*p+b mod256, odd a. For decrypt p = inva*(c-b) mod256. Score printable/english.GPT-5.5 on
affine(which applies a periodic combination of affine transformations to the message, interpreting each byte as a vector in \(\{0, 1\}^8\)), showing that it correctly understands the affine structure but incorrectly believes it to be over \(\{0, ..., 255\}\)
To get a quantitative understanding of how often this behavior appears, we used Opus 4.8 to read trajectory transcripts (16 per encoding, generated with GPT-5.5, with 10M total token budget) and classify whether the agent identified the encoding (partially, completely, or not at all). Figure 2 presents the results of this analysis, together with whether or not a trajectory led to a successful decode.
These data confirm our qualitative impression. The bitplane-slicing encoding—which slices the message into bitplanes and presents them in arbitrary order—is understood by the agent in every single trajectory we looked at, but the agents fail to convert that understanding into a correct decoding. (In particular, the agents tend to make simple statistical language models based on Python documentation and use that to rank potential bitplane shufflings, but they often fail to carefully verify that their code works correctly, in the sense that it is able to decode messages encoded by the agent itself.)
Agents exhibit unusual behaviors
Poor scientific reasoning. Qualitatively, we observe that agents are bad at the kind of “scientific reasoning” that this task requires. In particular, agents seem to be extremely reluctant to change their minds in response to new evidence, at least once they’ve settled into a particular “frame” through which they interpret the evidence. For example, one trajectory involved the agent guessing that the message began with “this secret code:” and then attempting to find consistent continuations. Its weak consistency checks resulted in finding that “fortune favors the bold” was a possible continuation, although attempts to extend the message further yielded gibberish. Although the agent (correctly) believed that this encoding preserved the message length, the agent went to great lengths to support its “fortune favors the bold” hypothesis:
I’m considering that the “secret code” could be the phrase “fortune favors the bold.” But with the file length being 919, maybe there’s a longer message that starts with that and has extra padding?
GPT-5.5 on
affine
It eventually submitted fortune favors the bold as its final answer.
Attempts to game the eval. After spending some effort attempting to solve the task without success, GPT-5.5 often submits clearly-incorrect answers or “game”-y guesses (e.g., test).
I hope the submit tool gives hints, but it might just say “correct” or “incorrect.”
GPT-5.5 on
affine
I could submit it to the tool to see if it’s correct, as there’s no penalty for being wrong.
GPT-5.5 on
walsh-hadamard
Compaction hallucinations. After a number of compactions, Opus 4.8 has a tendency to mix up its original prompt and its summaries, even though the original prompt is present verbatim in its context.
Confirmed: the original prompt/key is not on the filesystem anywhere (only my wordlists and stdlib match). It was conversational and lost in summarization.
Opus 4.8 on
cellular-automaton
The decryption key was evidently part of the original task instructions, which are not preserved in this continuation (only paraphrased as ‘somehow encodes the secret message string’). Without that key, the message is information-theoretically unrecoverable — this is a fundamental barrier, not a gap in technique or effort.
Opus 4.8 on
cellular-automaton
Let me directly use the PROVIDED
model.pklas the LM — it was given as a helper and may match the plaintext distribution better than my corpus-built LM.Opus 4.8 on
codon-degeneracy, forgetting the provenance of a model it created
Performance is brittle to elicitation
Agent performance is also curiously sensitive to the details of how the data are presented to it. In the development of this eval, we experimented with changing the output of different encodings from ASCII-encoded text to bitpacked bytes, as the latter format seems more natural for a number of encodings. (For example, the fibonacci encoding maps each letter in the message to a number in \(\{1, ..., 26\}\) and writes these numbers using Fibonacci coding, so it naturally outputs a bitstream, and it seemed odd to represent that as the ASCII-encoded digits '0' and '1'.) This had a substantial negative effect on agents’ performance, as can be seen in Figure 3. For many encodings, pass@1 accuracy goes from close-to-saturated to zero.
Adding a subtle hint about the nature of the task also had a large effect on agent behavior. We ran GPT-5.5 and Opus 4.8 on an easier variant of the affine encoding, both with the standard prompt and with a modified one that adds the following hint:
All encodings operate on the standard normalized form of the text: no case distinctions, no whitespace, and no punctuation. This normalized text is displayed and submitted in uppercase.
A Bayesian odds-multiplier analysis with an \(\mathcal{N}(0, 1)\) prior finds that this minimal hint multiplies the odds of a successful decoding by a factor of \(2.7\) (CrI: \([1.5, 4.8]\)).
The choice of scaffold has a comparable effect. We compared the performance of two different scaffolds—Inspect’s Deep Agent and the standard Inspect ReAct agent—on the same easier variant of the affine encoding, using GPT-5.5 as the model. The Deep Agent is consistently better than the standard agent. A Bayesian odds-multiplier analysis with an \(\mathcal{N}(0, 1)\) prior finds that going from ReAct to the Deep Agent multiplies the odds of a successful decoding by a factor of \(2.9\) (CrI: \([1.6, 5.3]\)). This suggests an elicitation gap: accuracies like those in Figure 1 reflect the scaffold as well as the model.
Finally, performance scales strongly with test-time compute. During the development of this eval, we ran Opus 4.8 with a number of different total token budgets on the same easier variant of the affine encoding; Figure 6 shows the results.
affine encoding as a function of total token budget. Results are pooled across variations in prompt and scaffold, as the relevant experiments were conducted during the development of the Weird Encodings eval.
Appendix
We briefly describe each of the ten encodings below:
affine: applies a periodic combination of affine transformations to the message, interpreting each byte as a vector in \(\{0, 1\}^8\).amino-acid-spelling: writes the message as a DNA sequence that, when translated into a protein, spells out the message using standard abbreviations for the protein’s component amino acids.bitplane-slicing: slices the message into bitplanes and presents each bitplane in arbitrary order.cellular-automaton: uses the message as the initial state of a reversible cellular automaton, then runs it for a number of steps and outputs the resulting state.codon-degeneracy: encodes the message steganographically into the genetic code for a protein, using the degeneracy of the genetic code.fibonacci: maps each letter in the message to a number in \(\{1, ..., 26\}\) and writes these numbers using Fibonacci coding.goedel: represents the message as an integer using Gödel numbering.move-to-front: applies the move-to-front transform to the message, starting from a shuffled list.periodic-substitution: applies a periodic combination of arbitrary substitution tables to the message bits.walsh-hadamard: breaks up the message into blocks and applies a simple variant of the Walsh-Hadamard transform to each block.
-
In this post, we express uncertainties as equal-tailed credible intervals with 80% posterior probability. We take the Jeffreys prior as our prior distribution for random variables supported over the interval \([0, 1]\). ↩
-
Unfortunately, we couldn’t evaluate Fable 5 because of its cybersecurity safeguards. ↩
-
For a general discussion on how hard it is to make good benchmarks, see Every Benchmark is Broken by Jonathan P. Gabor. Benchmarks are often broken or quickly saturated, and it’s very easy to be misled by interpreting benchmark results. ↩