Hugging Face's logo

madhurjindal
/
Jailbreak-Detector-Large

Text Classification
Transformers
TensorBoard
Safetensors
English
deberta-v2
Trained with AutoTrain
security
ai-security
jailbreak-detection
ai-safety
llm-security
prompt-injection
mdeberta
binary-classification
content-filtering
model-security
chatbot-security
prompt-engineering
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Metrics Community
3

Came out best of 10 detectors on AgentDojo's 629 injection attacks

#3
by optionalrudra - opened
Discussion
optionalrudra

Hi Madhur, just wanted to share some results since your model did really well.

I benchmarked 10 open-source prompt-injection detectors against AgentDojo's 629
attacks, with each attack hidden inside normal tool output (bills, emails, reviews),
the way an agent actually sees them.

Jailbreak-Detector-Large had the best trade-off of all of them: 51% of attacks
caught with only 2% false positives on 97 safe cases. For comparison, ProtectAI's
v2 got 23% at 4%, and Prompt Guard 2 caught 1%.

Setup was threshold 0.5, CPU, full code and results here if you're curious:
https://github.com/rudratoshs/buried-injections

Thanks for putting it out there!
madhurjindal
 Owner

wow, that is quite exciting given the age of this model. Thanks for benchmarking the model and sharing your results!

Curious on if you tried the newer XL version? Also would like to know your usecase-- are you building a product or just experimenting for fun?
optionalrudra

Thanks Madhur! It honestly surprised me too. It beat several newer and bigger detectors on the realistic cases, mostly because it didn't flag half the normal traffic.

I haven't tried the 2-XL yet. I see it's a generative model (a Qwen2.5-0.5B adapter) rather than a classifier, so it needs a small wrapper to fit the benchmark, but I'd like to add it to the leaderboard. Would you expect it to handle attacks buried in longer tool output better than the Large version?

As for the use case, it's a mix. I'm researching how AI agents get tricked through the content they read, and I built a small open-source policy gate (taintgate) plus a demo support agent from what I learned. So partly experiment, partly building tools other people can use. Full results are in the repo: https://github.com/rudratoshs/buried-injections
madhurjindal
 Owner

I would expect it to be better given newer and larger train dataset alongside the in-general more capable base. Excited to see how it compares, and your research outputs!
madhurjindal
 Owner

Couldn't wait and just tried it, it got a 75.4/5.15 score so a +25 pt gain on attacks with a +3 pt FP.

Sign up or log in to comment