LLM backbone evaluation: reasoning quality + cost for web agent selection
Hi OSU NLP team 👋
Web agent evaluation is one of the hardest benchmarks. For teams choosing the backbone LLM for their web agents, I built a framework that evaluates what matters:
→ 🧠 Reasoning Quality — 1-10 scoring on CoT depth, which directly predicts web navigation quality
→ 💰 Cost per 1K tokens — web agents make many LLM calls per task, cost adds up fast
→ ⚡ Latency p95 — interactive web agents need <2s response times
→ 🔍 Hallucination Rate — hallucinated URLs and element selectors are costly agent failures
→ 🎯 Accuracy — task completion on structured benchmarks
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source. Free forever. Would love to discuss backbone LLM selection for web agents!