Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.
Connect Runs on your machine
claude mcp add evalsmith-mcp -e EVALSMITH_JUDGE_BASE_URL=<EVALSMITH_JUDGE_BASE_URL> -e EVALSMITH_JUDGE_MODEL=<EVALSMITH_JUDGE_MODEL> -e EVALSMITH_JUDGE_API_KEY=<EVALSMITH_JUDGE_API_KEY> -e LOG_LEVEL=<LOG_LEVEL> -- npx -y [email protected]{
"mcpServers": {
"evalsmith-mcp": {
"command": "npx",
"args": [
"-y",
"[email protected]"
],
"env": {
"EVALSMITH_JUDGE_BASE_URL": "<EVALSMITH_JUDGE_BASE_URL>",
"EVALSMITH_JUDGE_MODEL": "<EVALSMITH_JUDGE_MODEL>",
"EVALSMITH_JUDGE_API_KEY": "<EVALSMITH_JUDGE_API_KEY>",
"LOG_LEVEL": "<LOG_LEVEL>"
}
}
}
}{
"mcpServers": {
"evalsmith-mcp": {
"command": "npx",
"args": [
"-y",
"[email protected]"
],
"env": {
"EVALSMITH_JUDGE_BASE_URL": "<EVALSMITH_JUDGE_BASE_URL>",
"EVALSMITH_JUDGE_MODEL": "<EVALSMITH_JUDGE_MODEL>",
"EVALSMITH_JUDGE_API_KEY": "<EVALSMITH_JUDGE_API_KEY>",
"LOG_LEVEL": "<LOG_LEVEL>"
}
}
}
}{
"servers": {
"evalsmith-mcp": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"[email protected]"
],
"env": {
"EVALSMITH_JUDGE_BASE_URL": "<EVALSMITH_JUDGE_BASE_URL>",
"EVALSMITH_JUDGE_MODEL": "<EVALSMITH_JUDGE_MODEL>",
"EVALSMITH_JUDGE_API_KEY": "<EVALSMITH_JUDGE_API_KEY>",
"LOG_LEVEL": "<LOG_LEVEL>"
}
}
}
}Replace the values in angle brackets with your own. Keys stay on your machine; we never see them.
Tools
This server runs locally, so its tools are listed by your MCP client once it starts.
Environment variables
| Name | Description |
|---|---|
EVALSMITH_JUDGE_BASE_URLOptional | OpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost) |
EVALSMITH_JUDGE_MODELOptional | Model used by the optional LLM judge |
EVALSMITH_JUDGE_API_KEYOptional, secret | API key for the judge endpoint, if it needs one |
LOG_LEVELOptional | pino log level; logs go to stderr |
Details
| Package | [email protected] (NPM, stdio) |
| First published | 28 Oct 2026 |
| Last updated | 28 Oct 2026 |
| Registry name | io.github.diegoperezg7/evalsmith-mcp |
More ai and media servers See all
Listing data from the official MCP registry. Status and tools from NestDaddy's daily check. Browse all MCP servers.