Agent Seer Evaluator reproduces and extends Apple's Agent Seer specification-driven testing framework across production generative-media MCP servers (mcp-veo-go, mcp-nanobanana-go, mcp-lyria-go, and mcp-omni-go). It tests agent orchestration correctness without executing costly diffusion pipelines.
Key Features
- Deterministic AST Pre-Pass: Sub-millisecond parameter and capability linter catching 100% of illegal model-parameter combinations at zero compute cost.
- Capability Matrix Grounding: Injects machine-readable runtime feature registries into the LLM judge prompt, eliminating false passes and expanding discrimination gaps by 36 percent.
- High-Grounding Synthetic Mocks: Seeds multi-turn tool outputs from real server execution traces, achieving 84 percent high-grounded responses without live cloud API execution.
- Decomposed 14-Subdimension Rubric: Multi-turn hierarchical judge engine enforcing non-linear cascading penalties on parameter errors.
- Multi-Server Narrative Pipeline: Tests complex four-server commercial production workflows across image, video, audio, and multiplexing services.
Explore the source code on GitHub: github.com/ghchinoy/ai-paper-reproductions/tree/main/agent-seer-mcp-tool-calling.