A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
Connect Runs on your machine
claude mcp add polite-crawler -e POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR> -e POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT> -e POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE> -e POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED> -e POLITE_CRAWLER_SIGN_KEY=<POLITE_CRAWLER_SIGN_KEY> -e POLITE_CRAWLER_SIGNATURE_AGENT=<POLITE_CRAWLER_SIGNATURE_AGENT> -- npx -y [email protected]{
"mcpServers": {
"polite-crawler": {
"command": "npx",
"args": [
"-y",
"[email protected]"
],
"env": {
"POLITE_CRAWLER_DIR": "<POLITE_CRAWLER_DIR>",
"POLITE_CRAWLER_CONTACT": "<POLITE_CRAWLER_CONTACT>",
"POLITE_CRAWLER_ALLOW_PRIVATE": "<POLITE_CRAWLER_ALLOW_PRIVATE>",
"POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
"POLITE_CRAWLER_SIGN_KEY": "<POLITE_CRAWLER_SIGN_KEY>",
"POLITE_CRAWLER_SIGNATURE_AGENT": "<POLITE_CRAWLER_SIGNATURE_AGENT>"
}
}
}
}{
"mcpServers": {
"polite-crawler": {
"command": "npx",
"args": [
"-y",
"[email protected]"
],
"env": {
"POLITE_CRAWLER_DIR": "<POLITE_CRAWLER_DIR>",
"POLITE_CRAWLER_CONTACT": "<POLITE_CRAWLER_CONTACT>",
"POLITE_CRAWLER_ALLOW_PRIVATE": "<POLITE_CRAWLER_ALLOW_PRIVATE>",
"POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
"POLITE_CRAWLER_SIGN_KEY": "<POLITE_CRAWLER_SIGN_KEY>",
"POLITE_CRAWLER_SIGNATURE_AGENT": "<POLITE_CRAWLER_SIGNATURE_AGENT>"
}
}
}
}{
"servers": {
"polite-crawler": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"[email protected]"
],
"env": {
"POLITE_CRAWLER_DIR": "<POLITE_CRAWLER_DIR>",
"POLITE_CRAWLER_CONTACT": "<POLITE_CRAWLER_CONTACT>",
"POLITE_CRAWLER_ALLOW_PRIVATE": "<POLITE_CRAWLER_ALLOW_PRIVATE>",
"POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
"POLITE_CRAWLER_SIGN_KEY": "<POLITE_CRAWLER_SIGN_KEY>",
"POLITE_CRAWLER_SIGNATURE_AGENT": "<POLITE_CRAWLER_SIGNATURE_AGENT>"
}
}
}
}Replace the values in angle brackets with your own. Keys stay on your machine; we never see them.
Tools
This server runs locally, so its tools are listed by your MCP client once it starts.
Environment variables
| Name | Description |
|---|---|
POLITE_CRAWLER_DIROptional | The folder of the store. The default is ~/.polite-crawler. |
POLITE_CRAWLER_CONTACTOptional | An email address or URL for the From header of each request. |
POLITE_CRAWLER_ALLOW_PRIVATEOptional | Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them. |
POLITE_CRAWLER_ALLOW_USER_PROVIDEDOptional | Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm. |
POLITE_CRAWLER_SIGN_KEYOptional | The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT. |
POLITE_CRAWLER_SIGNATURE_AGENTOptional | Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set. |
Details
| Package | [email protected] (NPM, stdio) |
| First published | 13 Nov 2026 |
| Last updated | 13 Nov 2026 |
| Registry name | io.github.mark1russell7/polite-crawler |
More search and web servers See all
Listing data from the official MCP registry. Status and tools from NestDaddy's daily check. Browse all MCP servers.