1001Ferramentas
Tools Categories Guides FAQ
About Contact Privacy Policy Terms of Use
🕷️ SEO

Crawl Budget Log Analyzer

Parses Apache and Nginx access logs and reports how much crawling each bot does, which URLs Googlebot hits and where crawl budget is wasted.

Common Log Format and Combined are accepted. The file is parsed in this tab and nothing is uploaded.

How to read the panels

The agent column comes from the user-agent string, matched by substring in a fixed order so that Googlebot-Image is never counted as plain Googlebot. A line whose user-agent has no known signature but does contain bot, spider or crawler lands in Other bots; everything else is treated as human traffic and is filtered out by default.

The waste panel is the one worth acting on: every redirect and every error served to a Google crawler is a request that produced no indexable page. A high share of static files in the resource panel usually means images and assets are eating requests that should be reaching product and article pages.

Reading access.log to find where crawl budget leaks

The server log is the only place that records what search engines actually requested from your site, line by line, with no sampling. Paste the contents of your access.log here and the page splits every request by agent, by response code, by URL and by hour, showing how much of the crawling reached useful pages and how much drained into redirects, errors and static files.

Parsing accepts Common Log Format and Combined, the two layouts Apache and Nginx write by default. The agent comes from the user-agent string, matched by substring in a fixed order, so Googlebot-Image, Googlebot-News and AdsBot show up on their own rows instead of being folded into plain Googlebot. Anything without a known signature that still carries bot, spider or crawler lands in Other bots.

The panel that usually drives a decision is the waste one: every URL where a Google crawler got a 301, a 404 or a 500 is burning requests without returning an indexable page. Priority multiplies hits by the weight of the band, 2 for 5xx, 1.5 for 4xx and 1 for 3xx, so you can attack what hurts most first. Up to 50,000 lines are analyzed per run and nothing is uploaded anywhere.

Frequently asked questions

Where do I find this log file?
On your own server it usually sits at /var/log/nginx/access.log or /var/log/apache2/access.log. On shared hosting, look for raw access logs in cPanel or the equivalent panel, which hands you one .gz per day. Behind a CDN such as Cloudflare, use the log export in that panel instead, since many requests never reach the origin server at all.
Can I tell whether the Googlebot in the log is genuine?
Not here. Classification reads only the user-agent text, which any script can copy. To confirm it, run the two-step check Google documents: a reverse DNS lookup on the IP, which must end in googlebot.com or google.com, and a forward lookup on the name you got, which must resolve back to the same IP. It is worth doing that for the IPs with the most hits.
What is the line limit and what happens past it?
The first 50,000 lines are processed and a warning shows the total received, so you know the numbers cover only part of the file. For longer periods, the better route is to analyze one day at a time and compare the panels, since what matters is usually the change between days rather than the raw total.

Related Tools

🔍

Meta Tag Analyzer

Paste a page HTML and see all meta tags (description, keywords, Open Graph, Twitter, viewport, robots) extracted and organized. Everything in your browser.
📏

SEO Title Length

Analyzes title tag length in chars and pixel estimate to suggest optimal truncation for Google.
📝

Title and Description Analyzer

Check whether your <title> (50-60 chars) and meta description (150-160 chars) are at the ideal length for Google. Shows snippet preview. Everything in your browser.