Grant Discovery Bot

A public-data crawler in service of nonprofits.

What this crawler does

Grant Discovery Bot collects publicly available information about grant programs, foundations, and federal awards from sources such as the IRS, Grants.gov, SAM.gov, USASpending, and ProPublica’s Nonprofit Explorer. Its purpose is to help nonprofit organizations identify funding opportunities they would otherwise need expensive commercial databases to find.

It is operated by TheAlmoner LLC, an independent project dedicated to making philanthropic information accessible to the organizations that serve the poor and afflicted.

How to identify this crawler

Every request the bot makes carries two unambiguous identifiers:

User-Agent header: Mozilla/5.0 (compatible; GrantDiscoveryBot/1.0; +https://bot.aigranthelper.com)
From header: bot@aigranthelper.com

The token GrantDiscoveryBot inside the User-Agent string is the technical identifier — this is what appears in your access logs and what robots.txt rules should match. “Grant Discovery Bot” is the human-readable name for the same crawler; they refer to one and the same program.

Both values are stable and will not churn between versions. The crawler runs from a single fixed egress IP, advertised in machine-readable form at /ips.json. Each IP reverse-resolves to a hostname of the form crawl-N.aigranthelper.com, so a webmaster auditing access logs can confirm the IP belongs to this operator.

Sub-identities

The same operator runs a lightweight URL-reachability probe under a distinct sub-identity, following the Googlebot / Googlebot-Image naming convention. It performs only HEAD or short-GET requests to check whether source URLs are still live; it never fetches body content for storage.

Reachability probe User-Agent: Mozilla/5.0 (compatible; GrantDiscoveryBot-Reachability/1.0; +https://bot.aigranthelper.com)

This sub-identity shares everything else with the main bot — same operator, same From address, same egress IPs, same opt-out path. A robots.txt rule targeting GrantDiscoveryBot also covers it through token-prefix match.

A second sub-identity exists for compatibility. Two connector families reach portals that reject or degrade requests carrying a bot-shaped user agent and publish no API. Those families — and never the foundation-website crawl — send a browser-shaped compatibility string:

Compatibility User-Agent (state-portal and municipal-association connectors only): Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 GrantDiscoveryBot/1.0 (+https://bot.aigranthelper.com)

It is not an anonymous browser: the GrantDiscoveryBot/1.0 token, the same From address and the same fixed egress IP ride on every one of those requests, so it is identifiable in an access log, and a robots.txt rule targeting GrantDiscoveryBot matches it exactly as it matches the main bot. Because the string opens with a generic browser prefix, it is deliberately never submitted to crawler IP-validation forms; those describe the main GrantDiscoveryBot identity above.

Two connector families send it, and a third would be declared here before it was deployed:

State grant and procurement portals. Every source the crawler’s state-portal strategies fetch: BidNet (bidnetdirect.com) and South Carolina Business Opportunities (scbo.sc.gov), which return 403 to bot-shaped agents; AmpliFund and Submittable grant portals; the CKAN, Socrata and OpenDataSoft open-data endpoints several states publish their grant registers through; plain JSON and HTML grant-listing endpoints; state grant-page sitemaps and vendor-portal probes; and the USASpending public API. These are several dozen state and federal government hosts today, and the set changes as states are added — rather than keep a copy here that would fall behind, the operator will send the current host list to anyone who asks at the contact address below.
State municipal-association grant directories. Five, named individually:
- akfederalfunding.org — Alaska Municipal League
- flcities.com — Florida League of Cities
- gacities.com — Georgia Municipal Association
- oml.org — Oklahoma Municipal League
- vlct.org — Vermont League of Cities and Towns

Foundation websites, the IRS, Grants.gov, SAM.gov and ProPublica are crawled under the main GrantDiscoveryBot identity, and URL-reachability probes under the Reachability sub-identity above.

Two declared exceptions. Two clients send the From address but not the GrantDiscoveryBot token. Both are named here, with the exact string each sends, rather than left for a log audit to find:

DuckDuckGo search client. Search-engine queries to html.duckduckgo.com go out under a generic browser User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36. Correcting it is tracked, and this entry comes off the page when it is fixed.
The operator’s own corrections API. A first-party client reads aigranthelper.com/internal/corrections/ — a documented endpoint of the operator’s own service, not a crawl of anyone else’s site — under GrantSpider/1.0 (+https://bot.aigranthelper.com). It reaches no third-party host and so will never appear in your log; it is listed so the claim above is exhaustive rather than approximate.

Rendering policy

The bot uses plain HTTPS clients only — no headless browsers, no JavaScript execution. Two of its clients send a browser-shaped string rather than the bot-shaped one, and both are declared under Sub-identities above: the compatibility User-Agent, which still carries the GrantDiscoveryBot token, and the DuckDuckGo search client, which carries only the From address. No headless browser and no JavaScript engine is involved in either. This matches the convention followed by Googlebot, Bingbot, ClaudeBot, GPTBot, and other reputable crawlers.

For the small set of portals that cannot be reached by plain HTTPS because the underlying data is rendered exclusively client-side and no public API is available, the operator reserves the right to deploy a clearly-identified rendering sub-identity:

Rendering sub-identity (reserved; not currently in use): Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 GrantDiscoveryBot-Render/1.0 (+https://bot.aigranthelper.com)

If and when this sub-identity is deployed, the affected portals will be enumerated here — following the convention of Google’s Web Rendering Service, which is publicly disclosed at Google’s crawler documentation. As of today, the list is empty and no headless browser runs in production.

How this crawler behaves

robots.txt is honored. Disallow rules, the GrantDiscoveryBot user-agent token, and Crawl-delay directives all apply. The /robots.txt fetch that precedes the crawl is itself made under the same identity as the request it gates — same User-Agent, same From address — so the first line we write in your log names us rather than an anonymous HTTP library.
Rate is conservative. No more than one request to a given host every 5–10 seconds; exponential backoff on 429 and 503 responses.
No headless browsers by default. Plain HTTPS clients; no JavaScript execution. Every request carries the From address, the /robots.txt pre-flight included, and every request carries the GrantDiscoveryBot token except the two clients declared under Sub-identities above — the browser-shaped compatibility requests included. Any portal that genuinely cannot be reached without rendering will be served by the clearly-identified GrantDiscoveryBot-Render/1.0 sub-identity (see Rendering policy above) and listed there by name.
Opt-out is immediate. A request to the contact address below removes a host from the crawl queue within one business day.

How to block or limit this crawler

To stop the crawler from visiting your site, add the following to /robots.txt. The user-agent token GrantDiscoveryBot is what the bot literally sends, so rules must match that token:

User-agent: GrantDiscoveryBot
Disallow: /

To allow access only to specific paths, use standard Disallow rules:

User-agent: GrantDiscoveryBot
Disallow: /private/
Disallow: /api/
Allow: /

To request a longer minimum delay between requests:

User-agent: GrantDiscoveryBot
Crawl-delay: 30

Contact

For any concern — opt-out requests, audit-log inquiries, questions about the data we collect — write to bot@aigranthelper.com. Mail is read by the operator; a human replies.