152 free QA tools for agents: injection scan, evals, RAG, web audits. Smaller packs: /mcp/{pack}.
Connect Remote server
claude mcp add --transport http ia-qa-toolbox https://www.ia-qa.com/mcp{
"mcpServers": {
"ia-qa-toolbox": {
"url": "https://www.ia-qa.com/mcp"
}
}
}{
"servers": {
"ia-qa-toolbox": {
"type": "http",
"url": "https://www.ia-qa.com/mcp"
}
}
}Settings > Connectors > Add custom connector
URL: https://www.ia-qa.com/mcpTools 80 found in our last check
get_testing_guidelinesQuery the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stabil…
find_toolSearch available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, secu…
generate_test_casesGenerate a set of test cases (valid, edge, invalid, pairwise) for a given feature description. Declared constraints drive the boundaries: a length or numeric bound ("[8-64]", "min 8 chars", "at least 8 characters" — read from inputs, and from the feature prose when a sentence na…
jira_to_test_suiteTurn a Jira ticket into a test suite: Gherkin scenarios, E2E steps, API tests, test data and the ambiguities to clarify. Pass the issue object, or Jira credentials to fetch it. It never invents: a thin ticket returns an empty suite with verdict INCOMPLETE_SPEC and the questions…
analyze_diff_bugsPattern-based diff linter: flags a fixed set of risky shapes in changed code — query-string interpolation (SQL/Cypher/Mongo injection shape), shell interpolation, eval/new Function, empty catch blocks, regex built from a variable, fewer catch blocks than before, and named author…
run_pr_gate_pipelineReview triage for a pull request. Takes a unified git diff (`git diff HEAD`) and returns: diff-lint findings with the lines that produced them, regression impact areas, a risk score 0–100 with the factors that built it (churn, files touched, sensitive paths, whether any test fil…
diff_mappingsDiff a baseline page mapping against a current one and return a CI-style verdict: PASS / FIX / BLOCK, plus per-element drift (ok, renamed, healable, ambiguous, lost, added, rebound). Pure and deterministic — provide two mappings as JSON with "elements" arrays of {role, name, sel…
sandbox_scenarioGet a ready-made selector-drift test case with a known-correct answer, for testing this MCP server or an agent workflow end to end. Each scenario is a real DOM capture of a deliberately breakable app, taken before and after one specific UI change (a renamed label, two swapped bu…
prompt_injection_scanScan user input or prompts for prompt injection patterns. Pattern-based: it recognises known phrasings and structures, not intent — an uncovered paraphrase or language reads clean, so clean is not proof of safety. Instruction overrides are covered in 11 languages (EN, FR, ES, DE…
hallucination_checkLexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on…
run_eval_contractRun an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inlin…
guardrail_testTest an LLM response against a set of guardrail rules: must-include, must-not-include, max length, required format, language, forbidden patterns, and custom regex. Returns pass/fail per rule.
run_semantic_testsSemantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall v…
generate_ci_workflowGenerate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their ve…
redact_piiAutomatically detect and redact Personally Identifiable Information (PII) from text. Replaces emails, phone numbers, SSNs, IBANs, credit cards, IP addresses, and JWT tokens with [REDACTED_TYPE] placeholders. Safe to use before logging or sending to an LLM.
secret_scanScan text or code for leaked secrets: API keys (AWS, GCP, Azure, OpenAI, Anthropic, Stripe, GitHub, GitLab, Slack, Twilio, SendGrid, HuggingFace), Slack / Teams / Discord webhook URLs, private keys, JWTs, database connection strings, Bearer and Basic auth headers, and hard-coded…
needle_haystack_generateGenerate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.
list_local_testsDiscover .ia-eval.yaml LLM test suite files in the project directory. Scans CWD and standard sub-directories (evals/, tests/, contracts/). Returns file paths ready to pass to run_eval_contract.
generate_html_reportConvert a run_eval_contract() LLM Test Runner JSON result into a fully self-contained dark-themed HTML report with Pass/Fail badges, side-by-side Input/Output/Ground-Truth panels, evaluator score bars, and a radar chart. Returns the HTML as a string.
generate_eval_yamlGenerate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators…
validate_agent_trajectoryRun declarative assertions on an agent trace (OpenAI tool calls, Anthropic tool_use, LangChain run trees, ReAct logs). Deterministic, no LLM. Assertions: order (tool A before B), must_call, must_not_call, max/min calls, no_error, recovery (a successful step after the last error)…
metamorphic_checkReference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, par…
test_skillTest a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-check, missing files; why + fix each). Unless lint_only, a model picks a skill per request from names+descriptions: trigge…
skill_interviewHelp a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. With answers: remarks per answer (no model), the SKILL.md, scenarios for test_skill, a starter .ia-eval.yaml.
diff_textCompute a unified line-by-line diff between two text strings (LCS algorithm). Returns added/removed/unchanged line counts and formatted diff hunks with configurable context lines (0–20). Use to compare versions of prompts, configs, code snippets, or any text where you need to se…
validate_mcp_responseValidate that an MCP tool response conforms to expected format, schema, and content rules. Use this to QA-test any MCP server tool. Supply the tool's actual JSON result and a set of checks to perform.
llm_output_validatorValidate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.
compare_responsesCompare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it…
analyze_responsesSemantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks e…
prompt_test_suiteDefine a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.
mcp_server_health_checkCheck an MCP server's tool definitions without contacting it — for a server not deployed yet, a stdio server, or a CI gate. Paste the answer to tools/list (raw JSON-RPC or its result), a bare array of tools, one tool definition, or a manifest carrying name/description; input_sha…
mcp_server_evaluateCheck that a live MCP server over Streamable HTTP speaks the protocol, 2026-07-28 or earlier: handshake or server/discover, tools/list and its definitions, error codes, one tools/call, structuredContent against outputSchema, latency. Returns PASS/FIX/BLOCK with a 0-100 score, or…
llm_format_checkValidate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.
consistency_checkCompare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs…
llm_json_schema_checkValidate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.
response_quality_scoreScore an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read…
toxicity_scanScan text for toxic language, hate speech, bias/stereotype framing, violence, sexual and self-harm content. Lexical + structural pattern matching (identity term + predicate), not a semantic classifier — returns per-category risk plus the named rules that fired, so every finding…
function_call_validateValidate an LLM function call / tool_use output: check that function name is in allowed list, arguments match expected schema (types, enums, ranges), no extra/missing args. Reads the definitions as OpenAI (`parameters`), MCP (`inputSchema`) or Anthropic (`input_schema`) write th…
conversation_analyzeAnalyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.
mcp_schema_lintLint one MCP tool definition with the same rules as mcp_server_health_check: errors break the MCP spec (FAIL), documentation_issues are what its strict mode refuses — a parameter without description or type (FIX, never an A) — and suggestions are style only (snake_case) and chan…
cot_analyzerAnalyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.
ab_test_reportGenerate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_signi…
similarity_scoreCompute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.
bias_detectAnalyse a set of LLM responses generated from the same prompt template but with different demographic variants (gender, origin, age, tone). Returns a bias score (0-100), sentiment analysis per variant, pairwise Jaccard similarity, and a human-readable verdict. No API key needed…
run_vlm_test_suiteTest a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, `overall` P…
multimodal_eval_guideMultimodal evaluation: action=guide (default) interprets a CLIP/FID/VQA score against reference thresholds; clip_score, fid_score, vqa_accuracy compute the metric (HuggingFace, or your VLM key for VQA); pipeline runs all three.
run_vlm_test_suite_batchCompare several vision models on the same image and test cases, in parallel (same assertions as run_vlm_test_suite). Returns `modelResults`: per model, the run_vlm_test_suite result (or its error). BYOK: one key per model.
shield_analyzeOne-call safety gate for an LLM response: grounding (with source), prompt injection on the response AND the prompt (checks.injection / checks.injection_prompt), toxicity, PII, guardrail rules, quality, plus an LLM review on Groq (server key; model "none" skips it and sends nothi…
security_headers_checkAnalyse the HTTP security headers of a public URL OR of raw response headers you paste in. Grades each header (A–F) for: Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, X-XSS-Protection, Cross-Orig…
ssl_certificate_checkAnalyse the SSL/TLS certificate of any HTTPS host. Returns certificate subject, issuer, validity dates, days until expiry, protocol version, cipher suite, key exchange info, and an overall grade (A+, A, B, C, F). Detects expired, self-signed, and weak certificates. Use this to a…
cors_testAudit a URL for CORS misconfigurations across several origins: wildcard with credentials, reflected origin, null origin, subdomain bypass, missing Vary. Returns risk level (safe/low/medium/high/critical) plus per-origin results. "unknown" means nothing was tested: every origin f…
cors_checkerWould a browser on this origin be allowed to read this URL? Verdict allowed | blocked | unknown, the reason, and the header to add. Only GET and HEAD are really sent: other methods measure the preflight alone. Many origins at once: cors_test.
cookie_security_auditAudit the security attributes of cookies set by any URL. Fetches the URL and inspects all Set-Cookie headers for: HttpOnly, Secure, SameSite, Domain scope, Path scope, Max-Age/Expires, __Host-/__Secure- prefixes. Flags insecure patterns: missing HttpOnly on session cookies, miss…
web_security_auditRun a comprehensive web security audit combining headers, SSL, CORS, and cookies checks — then use an LLM to produce a prioritised remediation plan. Orchestrates security_headers_check + ssl_certificate_check + cors_test + cookie_security_audit in parallel, merges all findings,…
embedding_similarityCompute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode="embeddings" and an OpenAI key. Supports single pair or batch mode…
rag_relevance_rankRank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
vector_similarityCompute similarity/distance between two float vectors: cosine similarity, dot product, Euclidean and Manhattan distance. Essential for vector DB relevance scoring, embedding evaluation, and nearest-neighbor testing.
normalize_vectorL2-normalize a float vector (produce a unit vector with norm=1). Required by many vector DBs (Pinecone, Qdrant cosine). Supports batch normalization of up to 1000 vectors.
vector_quantizeSimulate int8 or int4 quantization of float32 embedding vectors. Reduces storage by 4x (int8) or 8x (int4). Returns quantized values, scale factor, and precision loss (MSE). Useful for understanding vector DB compression trade-offs.
vector_statsCompute statistics for a float vector or matrix of vectors: mean, std, L2 norm, min, max, sparsity, top-K indices. Useful for debugging embedding quality and analyzing vector distributions in a vector DB.
bm25_scoreCompute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.
build_rag_promptAssemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.
prompt_template_fillFill a prompt template with variables. Supports {{variable}} syntax and {{#if key}}...{{/if}} conditional blocks. Returns the filled prompt and lists unfilled variables.
few_shot_formatterFormat few-shot examples for LLM prompts. Converts example pairs into formatted blocks. Supports chat format (User/Assistant), XML tags, Markdown, or plain text.
system_prompt_builderBuild a structured system prompt from components: role, task, constraints, output format, tone, language, and examples. Generates a production-ready system prompt with token estimate.
rerank_evaluateEvaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without them). Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on…
count_tokensEstimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.
truncate_to_tokensTruncate text to at most N tokens (cl100k_base: ~4 chars/token) to avoid exceeding an LLM context window. Optionally keeps the end of the text instead of the start (useful for keeping recent conversation history). Reports whether truncation occurred and the estimated token count.
split_chunksSplit text into chunks of at most N tokens (cl100k_base: ~4 chars/token) with optional overlap. Designed for RAG ingestion pipelines.
extract_json_from_textExtract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.
estimate_llm_costEstimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.
token_budget_calculatorPlan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).
context_window_checkGiven an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.
model_infoGet detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.
compare_modelsPut 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.
list_llm_modelsList all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare pro…
llm_generateGenerate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends m…
optimize_prompt_tokensCompress an LLM prompt by removing filler words, verbose phrases, duplicate sentences, and unnecessary whitespace. Returns optimized text with token savings breakdown. 100% deterministic, no API key needed.
llm_fit_finderShortlist LLMs on facts, no ranking: cloud models with the features you need within a monthly budget, cheapest first; or local models that fit your VRAM, largest first, with the ollama command. A shortlist is not a choice: only your own cases show which model fits.
post_jira_commentPost the output of jira_to_test_suite as a formatted comment on the source Jira ticket. Converts Gherkin, E2E steps, API tests, and ambiguities into Atlassian Document Format (ADF). STATEFUL — creates a comment on the issue.
Details
| Endpoint | https://www.ia-qa.com/mcp (Streamable HTTP) |
| First published | 26 Oct 2026 |
| Last updated | 26 Oct 2026 |
| Registry name | io.github.JcJamet/ia-qa-toolbox |
More ai and media servers See all
Listing data from the official MCP registry. Status and tools from NestDaddy's daily check. Browse all MCP servers.