Search Console shows you what Google thinks of your site; the server log shows what Googlebot actually does. Which URLs are crawled how often, which redirect chains and 404s the bot spends time on, and when it arrives at a newly published page are only fully visible in the log. Logs don’t lie. This guide explains how to prepare access logs for SEO crawl analysis, filter out fake Googlebots and pull quick findings from the command line.
How log analysis differs from Search Console
| Question | Search Console | Server log |
|---|---|---|
| Which URLs were crawled? | A sampled list | Every request, the full list |
| Distribution by my own directory groups | Limited | In whatever breakdown you want |
| URLs in the sitemap that are never crawled | Indirect | Direct comparison |
| Parameterized URL waste | Hard to see | Count per parameter |
| Google’s content evaluation | Available (index status) | None |
The two sources complement each other. The Crawl Stats report in Search Console (under Settings) gives the distribution of total requests, average response time, response codes, file types and Googlebot types; at URL level it shows only a sample list. When you see an abnormal trend in the report, you look for the cause at URL level in the log.
Look at crawl budget at the right scale too. Google’s crawl budget guide for large sites states that the topic mainly concerns very large sites (roughly over a million unique pages) or sites whose content changes very quickly (tens of thousands of pages updated daily). On a corporate site with a few hundred pages, budget isn’t a problem. The real gain is diagnosis: errors, chains and discovery delays.
Log format and required fields
The default combined format of Nginx and Apache is enough for basic analysis:
66.249.66.1 - - [10/Nov/2026:10:15:32 +0300] "GET /guides/example/ HTTP/1.1" 200 15342 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Split by spaces, $1 is the client IP, $4 the time, $7 the request path, $9 the status code and $10 the response size; split by double quotes, $6 is the user agent. If response time and the virtual host name are also logged, the analysis goes deeper. An example format for Nginx:
log_format seo '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time $host';
access_log /var/log/nginx/example.com.access.log seo;
Add new fields at the end of the line; that way existing awk field numbers don’t break. A single field added in the middle makes all the old scripts silently produce wrong results. Storing, rotating and centrally collecting logs is a separate operational topic; the focus here is interpreting logs from an SEO perspective. For the collection side, see the web server log management article.
Real IP behind a CDN
If the site is behind a CDN such as Cloudflare, the server log shows the CDN’s IPs as the client and Googlebot verification isn’t possible. This is the first place to look in the console: if all the IPs in the log come from a few ranges, the realip configuration is missing. In Nginx it’s solved with ngx_http_realip_module:
# For each of the IPv4/IPv6 ranges Cloudflare publishes:
set_real_ip_from 173.245.48.0/20;
# ... other ranges ...
real_ip_header CF-Connecting-IP;
Cloudflare updates the range list; a list copied by hand goes stale over time, so update it automatically. The header must be trusted only on requests from these ranges; otherwise anyone can spoof their IP with the header. Also, requests answered from the CDN cache never reach the origin server; for the full picture, take the CDN-side logs into account too. How the cache layers work together is covered in the Cloudflare, WP Rocket and Redis article.
Verifying real Googlebot
The user agent string can be faked; browsers, SEO tools and malicious bots frequently declare themselves “Googlebot”. Google’s verification method has two steps: the IP’s reverse DNS record must end in googlebot.com or google.com, and a forward DNS lookup of that name must return the same IP. Alternatively, you can compare against the IP range JSON files Google publishes; for large numbers of IPs this method is faster.
# 1) Unique IPs declaring Googlebot
grep -i 'googlebot' access.log | awk '{print $1}' | sort -u > gb_ips.txt
# 2) Reverse + forward DNS verification
while read -r ip; do
ptr=$(host "$ip" | awk '/domain name pointer/ {print $NF}' | sed 's/\.$//')
case "$ptr" in
*.googlebot.com|*.google.com)
if host "$ptr" | grep -qwF "$ip"; then echo "OK $ip $ptr"; else echo "FAKE $ip $ptr"; fi ;;
*) echo "FAKE $ip ${ptr:-no-ptr}" ;;
esac
done < gb_ips.txt > gb_verify.txt
# 3) Separate requests from verified IPs only
awk '$1=="OK"{print $2}' gb_verify.txt > gb_ok.txt
awk 'NR==FNR{ok[$1];next} ($1 in ok)' gb_ok.txt access.log > googlebot.log
Because the loop does a DNS lookup for each IP, it’s slow for thousands of IPs; cache the results or switch to the IP range method. For IPv6 addresses, the notation in the log and in the host output may differ; check non-matching ones by hand. Addresses ending in googleusercontent.com belong to user-triggered Google tools and aren’t included in crawl analysis.
Quick analysis on the command line
On the verified googlebot.log file, the first table takes a few lines. Order matters: first the status code distribution, then the directory breakdown, and finally individual URLs.
# Status code distribution
awk '{print $9}' googlebot.log | sort | uniq -c | sort -rn
# Daily request count (the log must be in chronological order)
awk '{print substr($4,2,11)}' googlebot.log | uniq -c
# Distribution by first-level directory
awk '{split($7,a,"?"); split(a[1],p,"/"); print (p[2]=="" ? "/" : "/" p[2] "/")}' googlebot.log \
| sort | uniq -c | sort -rn | head -20
# Most crawled parameter names
awk '$7 ~ /\?/ {split($7,a,"?"); print a[2]}' googlebot.log \
| tr '&' '\n' | cut -d= -f1 | sort | uniq -c | sort -rn | head
# Mobile / desktop Googlebot ratio
awk -F'"' '{print ($6 ~ /Mobile/) ? "smartphone" : "desktop"}' googlebot.log | sort | uniq -c
# Most crawled URLs returning 3xx and 4xx
awk '$9 ~ /^[34]/ {print $9, $7}' googlebot.log | sort | uniq -c | sort -rn | head -30
If rotated log files are compressed, you can feed the output of zcat -f access.log* to the same commands. For data over a few gigabytes, it’s practical to load the logs into a database (for example DuckDB or ClickHouse) to run the same queries in SQL. The tool changes. The method doesn’t.
Comparing with the sitemap
The single most informative analysis is comparing the URLs in the sitemap with the URLs crawled:
# Paths in the sitemap
curl -s https://www.example.com/page-sitemap.xml https://www.example.com/post-sitemap.xml \
| grep -o '<loc>[^<]*' | sed -E 's#<loc>https?://[^/]+##' | sort -u > sitemap_paths.txt
# Crawled paths (without parameters, 200 only)
awk '$9==200 {split($7,a,"?"); print a[1]}' googlebot.log | sort -u > crawled_paths.txt
comm -23 sitemap_paths.txt crawled_paths.txt > in_sitemap_not_crawled.txt
comm -13 sitemap_paths.txt crawled_paths.txt > crawled_not_in_sitemap.txt
The first list shows important pages that weren’t visited at all during the analysis period; the second shows URLs outside the sitemap that consume crawl resources. The analysis window should be at least a few weeks; in a few days of logs, rarely crawled pages wrongly appear “not crawled” and you raise a false alarm.
Crawl distribution and areas of waste
When interpreting the findings, look for these patterns we come across again and again in the field:
- Parameter explosion: If filter, sorting or calendar parameters make up a significant share of total requests, there’s an infinite URL space.
- Redirect chains: If the same old URLs keep getting 301s, internal links and the sitemap still point to the old addresses.
- Persistent 404s: If requests keep coming to deleted pages, find their source (internal link, external link, old sitemap); if there’s equivalent content, redirect.
- 5xx and slow responses: Google lowers its crawl rate when the server returns errors or slows down. If the
$request_timefield is logged, list slow URLs separately. - Admin and technical paths: Heavy requests to paths such as
/wp-json/,/feed/and?replytocom=show that the signals need reviewing. - Discovery delay: Measure the time between a new post’s publish time and the first Googlebot request; long times point to internal linking and sitemap problems.
Actions by finding
| Finding | Likely cause | Action |
|---|---|---|
| Parameterized URLs crawled heavily | Filter links are crawlable | Canonical; narrow with robots.txt if needed |
| Many 301s | Internal links point to the old URL | Update links and the sitemap to the final URL |
| Persistent 404s | Deleted content, broken links | 301 if there’s a counterpart, otherwise leave 404/410 and fix the internal link |
| 5xx or high response time | Insufficient resources, heavy queries | Improve the server/cache, profile slow URLs |
| Sitemap URL not crawled | Weak internal linking, deep position | Link from related pages, review the menu/category structure |
| High fake Googlebot traffic | Scraper bots | WAF/rate limiting rules (excluding verified Googlebot) |
Most actions touch indexing signals; for choosing between robots.txt, canonical and noindex, see the indexing signals guide. After the change, rerun the same commands a few weeks apart and verify that the distribution shifts in the expected direction. Don’t count it as fixed without measuring.
The log format, real IP configuration and cache layers are set up on the server side; we handle this setup in Nginx web server and reverse proxy work, and interpreting the findings as part of technical SEO and site architecture.
Frequently asked questions
Is log analysis worth doing on a small corporate site?
According to Google, crawl budget is mostly a constraint on very large or very frequently changing sites. On a small site, the value of log analysis is diagnosis more than budget: it shows how long Googlebot takes to find new pages, how many requests it spends on 404s and redirect chains, and whether it gets server errors.
Is every request that says Googlebot in the user agent really Google?
No. The user agent is the client’s declaration and is easily faked. Real Googlebot is verified by the reverse DNS record of the IP address ending in googlebot.com or google.com and a forward lookup of that name returning the same IP, or by the IP ranges Google publishes.
Why do the logs always show Cloudflare IPs on a site behind Cloudflare?
Because the web server receives the connection directly from Cloudflare, the remote address is Cloudflare’s IP. You need to log the real client IP by trusting the CF-Connecting-IP header only on requests from Cloudflare IP ranges, with the realip module in Nginx or mod_remoteip in Apache.
Does the Search Console Crawl Stats report replace log analysis?
Partly. The report gives the distribution of total requests, response codes, file types and Googlebot types; at URL level it shows only a sample list. For a full list by URL, a breakdown by your own directory groups and a comparison with the sitemap, you need raw logs. Used together, they verify each other.