CrawlAlarmBot

If you found this page in your logs, someone used the CrawlAlarm checker on a URL on your site. We only fetch the URLs people submit, and we don't follow links from page to page.

Mozilla/5.0 (compatible; CrawlAlarmBot/1.0; +https://www.crawlalarm.com/bot)

What it requests

The submitted page, any redirects from it, and /robots.txt. It runs only when someone submits a check.

Does it obey robots.txt?

Not as an instruction to itself. Its job is to report what robots.txt tells Googlebot, so it reads the rules for Googlebot and still fetches the page to check for other problems. It never claims to be Googlebot.

Stopping it

Block user agents containing CrawlAlarmBot in your firewall or server configuration.