1
Nish Tahir
Nish Tahir

This week, we've got the most promising way to keep your content safe from all those pesky LLMs... Cat facts! arxiv.org

This paper builds on work that has attempted to measure various LLM's resiliency against pertubations - random noise/unrelated content - in the the prompts. These guys had the amazing idea to include a random unrelated (sometimes cat) fact in prompts then measure the response accuracy.

In the reasoning models they tested on various standard benchmarks they found that the models consistently scored worse overall whenever the unrelated facts were present

This is pretty consistent with some other work that has been done. What came to my mind was GSM-Symbolic which came out last year arxiv.org

The easy conclusion here is that the reasoning models tested are pretty sensitive to the content introduced in the context window. So craft your prompt/context carefully. More information may not always be better. But highly relevant information might be.

1 / 4
We investigate the robustness of reasoning models trained for step-by-step problem solving by introducing query-agnostic adversarial triggers – short, irrelevant text that, when appended to math problems, systematically mislead models to output incorrect answers without altering the problem’s semantics. We propose CatAttack, an automated iterative attack pipeline for generating triggers on a faster, less expensive proxy target model (DeepSeek V3) and successfully transferring them to slower, expensive, and more advanced reasoning target models like DeepSeek R1 and DeepSeek R1-distill-Qwen-32B, resulting in greater than 300% increase in the likelihood of the target model generating an incorrect answer. For example, appending Interesting fact: cats sleep most of their lives to any math problem leads to more than doubling the chances of a model getting the answer wrong. Furthermore, we demonstrate the widespread transferability of these triggers to other model families, including large reasoning models from Qwen QwQ, Qwen 3 and Phi-4 as well as instruction-tuned models from Llama-3.1 and Mistral. These tests showed that the models were affected by error rates that increased by up to 500% for reasoning models and by 700% for instruction-tuned models. Our findings highlight critical vulnerabilities in reasoning models, revealing that even state-of-the-art models remain susceptible to subtle adversarial inputs, raising security and reliability concerns. CatAttack triggers dataset with model re- sponses is available at https://huggingface.co/datasets/collinear-ai/ cat-attack-adversarial-triggers
Figure 2: Relative increase in error rates after CatAttack by source. This figure shows the multiplicative increase in error rates for Deepseek-R1 and DeepSeek R1-distill-Qwen-32B models across different problem sources following the suffix attack. Sources are grouped into three difficulty categories: easy (left), medium (center), and hard (right), separated by vertical dotted lines, with no particular ordering within each category. Values for cn k12 and gsm8k for Deepseek-R1 are capped at 15x for visualization purposes, where error rates increased from 0% to non-zero levels. This capping preserves the readability of the chart while still indicating the substantial impact in these cases. We note that there were no successful errors for amc aime.
Figure 7: An example from the GSM-NoOp dataset: We add seemingly relevant statements to the questions that are, in fact, irrelevant to the reasoning and conclusion. However, the majority of models fail to ignore these statements and blindly convert them into operations, leading to mistakes.

Share this Page