Log File Analyzer

How to use the Log File Analyzer#

1. Get your raw access log (not the error log) from your hosting panel — see the section below for where it lives.
2. Copy a slice of it — a recent day or week is ideal — and paste it into the box. You can paste up to about 3 MB.
3. Optionally add your sitemap URL. We then cross-reference it with the log to show which of your URLs no search engine crawled.
4. Run the analysis.
5. Read the results: every crawler found, the URLs they hit most, wasted crawl budget, and — with a sitemap — your crawl coverage.

Nothing is uploaded or stored beyond the short cache used to show your result again. The log is parsed in memory.

Where to find your access log#

• Hostinger (hPanel): Files → move to the file manager, or Websites → Manage → Advanced → Access logs. cPanel hosts have a Raw Access or Metrics → Raw Log section.
• cPanel: Metrics → Raw Access, then download the log for your domain.
• Nginx / Apache on a VPS: usually /var/log/nginx/access.log or /var/log/apache2/access.log (and their .1, .gz rotations).
• Cloudflare or another CDN in front of your site: the origin log only shows requests the CDN passed through. For a complete picture, use the CDN's own logs (Cloudflare Logpush) or Google Search Console's Crawl Stats report as well.

The tool reads the standard Combined Log Format that Apache and Nginx use by default — each line ends with the request line, a status code, and the user agent in quotes. Common Log Format (without a user agent) can't be used for crawler analysis, because the crawler is identified by its user agent.

Understanding your results#

Crawlers#

Every bot we recognise, busiest first, with its type — Search (Googlebot, Bingbot…), AI (GPTBot, ClaudeBot, PerplexityBot, CCBot…), Social (the preview fetchers behind Facebook, X, LinkedIn…) or SEO tool (AhrefsBot, SemrushBot…). For each you see how many requests it made, how many distinct URLs it reached, and the split of successful (2xx), redirected (3xx) and failed (4xx/5xx) responses.

Most-crawled URLs and wasted crawl budget#

Search engines spend a limited amount of effort — a "crawl budget" — on each site. If your top-crawled URLs are duplicates, faceted-search links or old redirects, that budget is not reaching your important pages. The Wasted crawl budget table lists the redirects and errors crawlers keep requesting; each one is a hit that could have gone to a real page.

What was NOT crawled#

This is the point of a log analysis. When you add your sitemap, we list the URLs in it that no search engine requested in the log you pasted. On a short window that is normal for deep pages — Googlebot visits a small site only a handful of times a day. But if important pages stay in this list across a week or more of logs, they are not getting crawled. Usually the fix is more internal links to them, removing a robots.txt block, or making them load faster.

Are these bots real?#

A user agent is trivial to fake, and scrapers routinely pose as Googlebot. This tool trusts the user-agent string in your log. To confirm a request is really from Googlebot, verify its IP: a reverse DNS lookup must resolve to a googlebot.com or google.com host name, and a forward lookup of that name must return the same IP. Google explains this in its guide to verifying Googlebot. Bing has an equivalent check.

How this tool works#

We parse each line of the log with the Combined Log Format pattern, read the IP, timestamp, request path, status code and user agent, and match the user agent against a list of known search-engine, AI, social and SEO-tool crawlers. Human browser traffic is ignored — this tool is about crawlers. If you supply a sitemap, we fetch it (following one level of sitemap index), collect its URLs and compare their paths with the paths search engines actually requested.

Everything comes from the log you paste. We do not see your live server, and the only thing we fetch is the sitemap you optionally provide.

Limitations#

• We can only report what is in the log you paste. A one-hour slice of a small site may contain few crawler hits, or none.
• Requests served entirely by a CDN cache may never reach your origin log. For those, use the CDN's logs or Search Console.
• Bot identification is by user agent, which can be spoofed — see the verification note above.
• We read up to the first 60,000 lines of a paste. For a bigger log, analyse a narrower date range.

Tips#

• Analyse a week of logs, not an hour, before concluding a page is not being crawled.
• If redirects dominate your crawl budget, update the internal links that point at the old URLs so crawlers reach the final page directly. Trace them with the Redirect Checker.
• If crawlers keep hitting 404s, find where those dead links come from with the Broken Link Checker.
• Check that the pages you want crawled are actually allowed in your robots.txt, and listed in a valid sitemap.
• Decide your policy for AI crawlers deliberately — this tool shows you which ones are visiting, so you can choose whether to allow or block each one.

Frequently asked questions#

What log format does this tool need?#

The standard Combined Log Format used by Apache and Nginx by default: each line ends with the request line, the status code, and the user agent in quotes. Common Log Format has no user agent, so it can't be used to identify crawlers.

Does the log data get uploaded or stored?#

The log is parsed in memory to produce your result. Like every tool here, the result is cached briefly so the same paste returns instantly, then expires. We do not build a profile of your site from it.

How is this different from Google Search Console's Crawl Stats?#

Search Console's Crawl Stats report is Google's own record of how Googlebot crawled your site — it is authoritative for Googlebot and needs no log file, so use it too. A log analysis is broader: it covers every crawler that reached your server (Bing, AI bots, scrapers), shows the exact URLs and status codes, and works for any host. The two complement each other.

Why are some of my pages listed as "not crawled"?#

Because no search engine requested them in the log you pasted. Over a short window that is expected for deep or new pages. If a page stays uncrawled across a long log, add internal links to it, confirm it isn't blocked in robots.txt, and make sure it responds quickly.

Can it tell if a "Googlebot" hit is fake?#

Not on its own — it trusts the user agent. It does show how many distinct IPs claimed to be Googlebot, which is a starting point. Confirm each one with the reverse-DNS check described above.

Does it work behind Cloudflare or another CDN?#

Your origin log only contains requests the CDN forwarded; cached hits are missing. Analyse the CDN's own logs for a complete picture, and cross-check with Search Console.