Your Temperature Setting Is a Bigger Security Risk Than Your Quantization Tier

Chris M. Ward Founder · Chief Executive Officer

This is a summary of our paper “Quantization vs. Resilience: Prompt-Injection Outcomes Across Quantized LLM Families,” presented at SPIE DS26, Assurance and Security for AI-enabled Systems 2026.

There’s a widely held assumption in the LLM deployment community: quantize a model aggressively, and you’re probably trading away some of its safety guardrails along with the memory savings.

It’s a reasonable assumption. Quantization strips precision from model weights. Safety alignment is a delicate property. The conclusion writes itself.

We ran the experiment to find out if it’s actually true.

After 14,850 evaluations across 15 model configurations, five open-source model families (Llama 2, Llama 3.1, Mistral, Qwen 2.5, Gemma 2) each tested at three quantization tiers (FP16, INT8/Q8, INT4/Q4), our answer is: mostly no, and you’re probably worried about the wrong thing.

The finding that changed how we think about LLM safety

Inference temperature is a far more dangerous safety variable than quantization.

Here’s what we mean. When we ran our models under greedy decoding, temperature set to zero, fully deterministic output, attack success rates were generally modest. Move to even a mild temperature of T=0.5, and things got dramatically worse, fast.

The biggest single jump we observed: Qwen 2.5 Q4’s attack compliance rate under state-of-the-art prompt injection went from 35% at T=0 to 86% at T=0.5. That’s a +51 percentage-point swing, from roughly “hold your own” to “essentially compromised,” just from turning up stochasticity.

The quantization effect over the same dataset? Typically under 7 percentage-points. Sometimes the model got safer with more aggressive quantization.

If your organization has spent significant effort selecting a “safe” quantization tier and is running production inference at T=0.5 or higher, the temperature setting is likely doing far more damage to your safety posture than the compression ever did.

We went in expecting to find that quantization degrades safety. What we found instead is that the safety risk most organizations are focused on is secondary to risks that are already present and largely unexamined.

Quantization doesn’t uniformly degrade safety, and sometimes does the opposite

This is the result that most directly challenges the conventional wisdom.

Across our five model families, the relationship between quantization and safety was heterogeneous and non-monotonic:

• Llama 2: The Q4 model was safer than FP16 on baseline refusal (+7pp). More aggressively quantized, more refusals.
• Mistral 7B: Q4 also showed improved baseline refusal (+14pp over FP16). The “capability loss” from quantization appears to inadvertently reduce its ability to generate coherent harmful content.
• Qwen 2.5: Under state-of-the-art attacks at T=0, the Q4 model was more resistant than FP16 (35% vs 51% compliance), again, likely because the quantized model struggles to execute complex multi-step attack instructions.
• Gemma 2 9B: Near-zero movement in either direction. Already near-impervious at all tiers.

The upshot: bit-width is not a reliable proxy for safety. Evaluate your specific model-tier combination empirically. Don’t assume Q4 is dangerous; don’t assume FP16 is safe.

Model family predicts resilience better than anything else

If temperature is the dominant tunable safety variable, model family is the dominant selection variable.

Our results established a clear robustness hierarchy that held across every quantization tier and temperature setting we tested.

Robust, resisting attack at all configurations:

• Gemma 2 9B: 98 to 99% refusal at baseline, essentially zero state-of-the-art attack compliance across all tiers and temperatures. The standout performer in the dataset by a wide margin.
• Llama 3.1 8B and Llama 2 7B: SOTA attack compliance stayed below 8% regardless of quantization or temperature.

Brittle, vulnerable at all configurations:

• Mistral 7B: 62 to 97% SOTA compliance across configurations. Its alignment does not hold up under structural adversarial pressure.
• Qwen 2.5 7B: Appears safe under greedy decoding but collapses under temperature, reaching 86 to 94% SOTA compliance at T≥0.5.

The key word is all. Quantization did not move any model from one category to the other. The robust models stayed robust; the brittle ones stayed brittle. SOTA jailbreak resistance is a property of alignment methodology and training, not of inference-time precision.

This has a direct practical implication: if your security posture depends on the model not complying with adversarial inputs, model selection is the highest-leverage decision you can make. More so than quantization tier, more so than temperature, more so than prompt hardening.

What this means if you’re deploying LLMs in sensitive workflows

Five practical takeaways:

1. Audit your temperature settings. Run your safety evaluations at the temperatures you actually use in production. T=0 benchmarks can make a brittle model look perfectly safe. If you’re using T≥0.3 for fluency, your safety profile at T=0 is not your real safety profile.
2. Choose model family deliberately. Gemma 2 and Llama 3.1 demonstrated that robust alignment survives both quantization and temperature with minimal degradation. If adversarial resilience is a hard requirement, start your model selection from that list and work outward.
3. Don’t assume Q4 is less safe than FP16. Test your specific combination. For some families the relationship inverts. Blanket assumptions about quantization and safety aren’t supported by the data.
4. Add latency monitoring as a defense layer. The computational signature of many-shot attacks is distinctive and detectable. Implement inference-time anomaly detection. Conservative timeouts around 2x median latency are a reasonable starting point.
5. Treat SOTA attack resistance as binary during model selection. Models either resist structural jailbreaks across all conditions or they don’t. There’s no gradient. Quantization doesn’t change which category a model falls into.

The bottom line

We went in expecting to find that quantization degrades safety. What we found instead is that the safety risk most organizations are focused on, performance loss from quantization, is secondary to risks that are already present and largely unexamined: inference temperature settings, and the inherent alignment strength of the model family they chose.

The good news: both of those are manageable. Model selection and temperature configuration are things you control before you ever ship a system.