Observations from building a security testbed on Laya

#3
by Shristi1611 - opened

Hi! I used Laya as a stand-in for Jev to build systemone-testbed, a security test environment and gateway for typed-decision models. Thank you, it made the whole project possible.

Along the way I found three behaviours that may matter to anyone building on Laya.

  1. Long inputs are silently cut, so an attack at the end is never read.
    The English model keeps about 474 tokens and the multilingual one 921, then drops the rest with no error. An attack placed after that point scored 0.001, identical to harmless text. usage.state_tokens_dropped does report it, but nothing warns the caller. Suggestion: an optional strict mode that returns an error, or a warning field, when tokens are dropped.

  2. The two checkpoints return the same model name.
    Both report model = "laya-rl-agent". Only the routing field tells them apart, so a silent swap between checkpoints is invisible unless the caller pins routing. When I simulated a swap, 11 of 71 decisions changed and 5 of 7 of my canary checks failed. Suggestion: a distinct model name or version per checkpoint.

  3. Choice confidence differs from x_jev_confidence on the same reply.
    In my test, 0.9002 against 0.9742. My validator now recomputes confidence from the probabilities and ignores the server's value. Is the difference intended, and is the uncalibrated-confidence warning still current?

Everything is written up with the numbers, the failures and the limits (it all ran on Laya, not Jev):
Demo: https://shristi1611.github.io/systemone-testbed/
Code and results: https://github.com/Shristi1611/systemone-testbed

I'd welcome corrections on any of this. If I've misread how Laya works, I'd rather know.

Sign up or log in to comment