Crawl Census

Connect Remote server

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp

{
  "mcpServers": {
    "crawl-census": {
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

{
  "servers": {
    "crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Settings > Connectors > Add custom connector
URL: https://crawlcensus.com/mcp

Tools 7 found in our last check

• scan_site

  Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.

• site_report

  Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

• census_stats

  Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

• crawl_preflight

  Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: a…

• agent_profile

  What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where…

• census_facts

  Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question…

• submit_domains

  Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census wil…

Details

More search and web servers See all

• Live Sports Scores & Betting Odds MCP Server

  Keyless sports scores, betting odds, injuries & news: NFL NBA MLB NHL soccer. No API key.

• Knowra

  Search arXiv papers and read Scout reports with methods, evidence and verification points.

• Contract Compass

  Korean public procurement law: rule-engine rulings, statutes search, live court precedents

• Korea Realty

  Korean real estate: court auctions, 10M+ MOLIT records, subscription notice facts, loan/DSR rules

• 校招宝 · 中国校园招聘岗位数据

  中国校园招聘（秋招/春招/实习/提前批）在招岗位、企业档案、投递截止与求职攻略数据。免 API Key、免注册、只读。

• IBDÅ

  Ask about IBDÅ, a design-led R&D and innovation studio in Dubai, and prepare a project enquiry.