BoostMasterBot crawler

If you are here because you saw BoostMasterBot/1.0 (+https://boostmaster.com.tr/bot) in your server logs: this is BoostMaster's page diagnostics crawler.

In short: it only visits sites a user added as their own project, obeys your robots.txt, only reads pages, and limits its request rate.

What does it do?

• It visits a site only after a BoostMaster user adds it as a project. It does not discover sites and does not follow links to walk your whole site.
• It fetches: robots.txt, sitemap files the user added, pages listed in those sitemaps, and, when the user asks, the IndexNow key file.
• It only makes read (GET) requests. It never submits forms, never tries to log in and does not run JavaScript.
• It reads at most 10 MB per page and follows at most 5 redirects.
• It looks for technical conditions that prevent indexing: status codes, robots rules, canonicals, content sufficiency and similar.

It obeys robots.txt

• It checks the User-agent: BoostMasterBot group first, otherwise User-agent: *. It does not download a disallowed page; the same rule applies to sitemap files.
• If robots.txt returns a server error (5xx), it does not crawl pages.
• robots.txt is cached for at most one hour; your rule changes apply within an hour.

Rate limits

Both the number of simultaneous requests to a site and the time between requests are limited. Defaults: at most 4 concurrent requests and at least 250 ms between request starts. The user who added the site can make these slower for their own site.

To block it

Add this to your robots.txt:

User-agent: BoostMasterBot
Disallow: /

Once blocked, the user who added the site sees a "our crawler is blocked" notice for those pages.

Where do requests come from?

All requests originate from these addresses: 187.77.111.65, 2a02:4780:c:49eb::1

Contact

If you see a problem with our crawler, write to [email protected].