NewBacklinkBot

NewBacklinkBot is the crawler used by the tools on newbacklink.com. If you found it in your server logs, this page explains why it visited and how to control it.

User agent#

NewBacklinkBot identifies itself with this user agent string:

Mozilla/5.0 (compatible; NewBacklinkBot/1.0; +https://newbacklink.com/bot/)

When it visits#

NewBacklinkBot does not crawl the web on its own schedule. It visits a site only when:

• a person uses one of our tools on that site; or
• it re-checks a guest post page that the Guest Post Finder has already found, which happens periodically.

A visit from NewBacklinkBot usually means that someone, possibly you, ran a check on your site.

What it fetches#

For most tools, NewBacklinkBot fetches a single URL, such as the page being checked, a robots.txt file, a sitemap or the steps of a redirect chain.

The Broken Link Checker is the exception. It crawls up to 100 pages of the requested site. During that crawl it:

• obeys robots.txt rules for NewBacklinkBot, or for * when there are no rules for NewBacklinkBot;
• limits the number of requests it sends at the same time.

The website speed test is different again. It loads the page in a real headless Chrome browser using Lighthouse, the open-source tool behind PageSpeed Insights, on our own server. Because it has to measure the page the way a visitor experiences it, that browser runs the page's JavaScript and uses Lighthouse's own user agent, which contains Chrome-Lighthouse, instead of NewBacklinkBot. If our Lighthouse server is unavailable, the test is run through Google PageSpeed Insights instead, and that request comes from Google.

What it never does#

NewBacklinkBot never submits forms, never logs in to any site and never executes JavaScript. It reads the HTML and HTTP responses that your server returns.

Our server does not fetch private network addresses, so NewBacklinkBot cannot be used to reach internal or local systems.

Browser user agent retries#

We want to be clear about one behavior. Some sites return a 403, 429 or 503 response to bots they do not recognize. When that happens, the SEO Checker and some other single-page tools retry the request once using a standard browser user agent.

We do this because the person running the check asked to see the page as a visitor would see it. The retry is a single request for the same page. It is not repeated and it does not continue into a crawl.

The Broken Link Checker and the Guest Post Finder's periodic re-checks never do this. They always identify themselves as NewBacklinkBot.

How to block NewBacklinkBot#

Add these lines to your site's robots.txt file:

User-agent: NewBacklinkBot
Disallow: /

The Broken Link Checker crawl respects this rule and will not crawl your site. Single-page checks started by a person may still fetch the requested page once, in the same way that a person visiting the page would.

To block only part of your site, use a narrower Disallow path instead of /.

Report a problem#

If NewBacklinkBot is causing problems on your site, or if you would like a page removed from Guest Post Finder re-checks, please tell us through the contact form. Include your domain and, if possible, the dates and times of the requests from your server logs. More about how we handle data is in the privacy policy.