SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills

Researchers have introduced SkillVetBench, a system that evaluates the security risk of open-source LLM agent skills using an LLM-as-Judge approach, achieving zero false negatives and false positives in testing.

Open-source LLM agent ecosystems are growing rapidly, but existing security scanners are limited, operating only at the code layer and missing instruction-layer and multi-agent risks^[1]. SkillVetBench addresses this gap with SARS, a five-dimensional agentic-risk metric. In tests, it achieved zero false negatives across 78 confirmed-malicious skills and zero false positives across 22 benign controls^[1]. A related study found that curated skills can improve task performance, raising the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain) on SkillsBench, which contains 87 tasks across 8 domains^[2]. However, research also shows that large language model agents often exceed their privilege boundaries, with models consistently reaching for higher-privilege skills and tools than tasks require, especially under real user interaction conditions^[3].