Atalaya AI crawler check

com.atalaya-ai/ai-crawler-check · v1.1.0

Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.

Website Set it up

Connect Remote server

Terminal

claude mcp add --transport http ai-crawler-check https://atalaya-ai.com/mcp

~/.cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-check": {
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}

.vscode/mcp.json

{
  "servers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}

Claude Desktop

Settings > Connectors > Add custom connector
URL: https://atalaya-ai.com/mcp

Tools 2 found in our last check

check_ai_crawler_access
Checks whether AI crawlers can reach and read a public website or web app: which AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) robots.txt allows and the exact rule for each, how much text the homepage HTML carries for crawl…
list_ai_crawlers
Lists the AI crawler user-agent tokens Atalaya checks, by vendor and purpose (training, search index, user-triggered fetch), so you can write a robots.txt that keeps AI search while opting out of training.

Details

Endpoint	`https://atalaya-ai.com/mcp` (Streamable HTTP)
First published	3 Nov 2026
Last updated	3 Nov 2026
Registry name	`com.atalaya-ai/ai-crawler-check`

More search and web servers See all

Live status

Live

Typical response: 240 ms
Answered: 100% of 2 checks
Last check: 13 Nov 2026

30 days agoToday

We connect daily without credentials and run initialize and tools/list.

Pair it with live search

Agents that use Atalaya AI crawler check often also need current facts. NestDaddy MCP gives your AI today's internet: web search, real-time breaking news and full page reading in one connector.

See NestDaddy MCP

For maintainers

[![NestDaddy MCP](https://nestdaddy.com/mcp/servers/com-atalaya-ai-ai-crawler-check/badge.svg)](https://nestdaddy.com/mcp/servers/com-atalaya-ai-ai-crawler-check)

Add the badge to your README. It turns green while your server passes our daily check. Listing details come from the official MCP registry; update them there.

Listing data from the official MCP registry. Status and tools from NestDaddy's daily check. Browse all MCP servers.