RobotsGate

Connect Remote server

claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp --header "Authorization: Bearer <YOUR_API_KEY>"

{
  "mcpServers": {
    "robotsgate": {
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp",
      "headers": {
        "Authorization": "Bearer <YOUR_API_KEY>"
      }
    }
  }
}

{
  "servers": {
    "robotsgate": {
      "type": "http",
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp",
      "headers": {
        "Authorization": "Bearer <YOUR_API_KEY>"
      }
    }
  }
}

Settings > Connectors > Add custom connector
URL: https://robotsgate.mike-tusa.workers.dev/mcp

Replace the values in angle brackets with your own. Keys stay on your machine; we never see them.

Tools 4 found in our last check

• list_crawlers

  Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-on…

• generate_robots

  Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-o…

• validate_robots

  Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC…

• check_site

  Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warni…

Headers

Details

More search and web servers See all

• Live Sports Scores & Betting Odds MCP Server

  Keyless sports scores, betting odds, injuries & news: NFL NBA MLB NHL soccer. No API key.

• Knowra

  Search arXiv papers and read Scout reports with methods, evidence and verification points.

• Contract Compass

  Korean public procurement law: rule-engine rulings, statutes search, live court precedents

• Korea Realty

  Korean real estate: court auctions, 10M+ MOLIT records, subscription notice facts, loan/DSR rules

• 校招宝 · 中国校园招聘岗位数据

  中国校园招聘（秋招/春招/实习/提前批）在招岗位、企业档案、投递截止与求职攻略数据。免 API Key、免注册、只读。

• IBDÅ

  Ask about IBDÅ, a design-led R&D and innovation studio in Dubai, and prepare a project enquiry.