RanktideBot crawler
What Ranktide's site audit crawler does, how it identifies itself, and how to allow or block it.
RanktideBot is the crawler behind Ranktide's site audits. It only visits a site when a Ranktide user starts an audit of that site. It reads pages to check for SEO issues like broken links, missing titles and redirect chains. It doesn't index content, and it doesn't collect data to train AI models.
How it identifies itself
Every request sends this user agent:
RanktideBot/1.0 (+https://ranktide.io/docs/crawler)Requests are also signed with Web Bot Auth, an HTTP message signature standard. Anyone can fake a user agent, but not a signature. The public key is published at:
https://app.ranktide.io/.well-known/http-message-signatures-directoryThe crawler runs on Cloudflare Workers, so it has no fixed IP addresses. Verify it with the signature instead of an IP list.
robots.txt
An audit starts with one HEAD request to the start URL, following redirects, to find the site's real address. The crawler then reads robots.txt and skips any page it disallows. Rules for RanktideBot take priority over rules for *.
To keep it out of your whole site:
User-agent: RanktideBot
Disallow: /Crawl rate
The crawler sends about one request per second to a site. An audit fetches a limited number of pages, set by the user who starts it.
When a site answers with 429 Too Many Requests, the crawler waits for the time in the Retry-After header, slows down, and tries that page once more. A page refused again is reported as rate limited, and the crawl moves on. If the site refuses several pages in a row, or keeps the crawler waiting for hours in total, the audit stops early.
The crawler doesn't fetch images, scripts, fonts or other files linked from a page.
Allow the crawler through a firewall
If your firewall or bot protection blocks RanktideBot, allow requests whose user agent contains RanktideBot. On Cloudflare, create a WAF custom rule that skips bot protection for that user agent.
Shopify stores
Shopify rate limits crawlers it hasn't authorised and sometimes blocks them. To let RanktideBot through:
- In Shopify admin, open Online Store → Preferences → Crawler access and create a signature for your store's domain.
- In Ranktide, open the project's settings, then Integrations, and paste the signature under Shopify.
Ranktide sends the signature only to the domain it was issued for.
Other ways audits load pages
Two optional audit features load pages without the RanktideBot user agent or signature:
- Render JavaScript loads pages in a headless browser, usually Cloudflare Browser Run.
- Lighthouse checks run from DataForSEO's servers with Lighthouse's own Chrome user agent.
Contact
For questions about RanktideBot visiting your site, email support@ranktide.io. To report a security issue, email security@ranktide.io.
Something missing or wrong on this page? Email support@ranktide.io.