At 3 a.m. the site is returning 502s, and the only evidence you have is the web server logs. Whether they’re useful at that moment depends on three things: do they contain the right fields, are they rotated before the disk fills up, and are they somewhere you can find what you’re looking for within minutes? Below we cover the format of Nginx, Apache and PHP-FPM logs, logrotate settings, central collection options and the queries we run during an incident. The audience is system administrators and DevOps teams managing one or more Linux web servers.
Log types in the web layer
A request passes through several layers along the way, and each layer keeps its own record. Knowing which question to look for in which log saves minutes during an incident:
| Log | Typical location | Which question it answers |
|---|---|---|
| Nginx/Apache access log | /var/log/nginx/access.log |
Who requested what, and when; response code and time |
| Nginx/Apache error log | /var/log/nginx/error.log |
Upstream connection errors, permission and configuration problems |
| PHP-FPM process log | /var/log/php8.2-fpm.log |
Did the pool fill up, were workers restarted? |
| PHP-FPM slowlog | Defined in the pool configuration | Which function slow requests were waiting in |
| PHP application error log | The file set by the error_log setting |
Warnings, fatal errors, application exceptions |
Locations vary by distribution and panel software; in panels such as Virtualmin, each virtual server’s logs are usually under its own home directory. A situation we often come across in the field: site logs created by the panel fall outside the scope of the packaged logrotate definition and grow for months. Document where all the logs for each site are, and check that they’re all included in the same rotation policy.
Two settings are often forgotten on the PHP-FPM side. The slowlog isn’t written unless request_slowlog_timeout is defined. catch_workers_output = yes makes errors that workers write to standard output land in the FPM log. We covered pool saturation and slow requests together with performance settings in the article on PHP-FPM and MariaDB performance problems.
Log format and JSON logging
The default combined format writes the client address, request line, status code, size, referrer and user agent. But the two pieces of information most sought in troubleshooting are missing: how long the request took and how long the application behind it took to respond. Without durations, the slowness debate turns into guesswork. If the client address isn’t passed correctly behind a reverse proxy or CDN, all requests look as if they come from the same IP; we explained this under the real_ip heading in the Nginx reverse proxy configuration article.
If you’re planning central collection, write logs in JSON format from the start; the hassle of writing regular expressions for parsing disappears. In Nginx 1.11.8 and later, the escape=json parameter escapes special characters in values in a JSON-compatible way:
log_format json_log escape=json '{'
'"time":"$time_iso8601",'
'"host":"$host",'
'"remote_addr":"$remote_addr",'
'"request_id":"$request_id",'
'"method":"$request_method",'
'"uri":"$uri",'
'"status":$status,'
'"bytes":$body_bytes_sent,'
'"request_time":$request_time,'
'"upstream_time":"$upstream_response_time",'
'"upstream_status":"$upstream_status",'
'"referer":"$http_referer",'
'"user_agent":"$http_user_agent"'
'}';
access_log /var/log/nginx/access.json json_log;
The choices here are deliberate. $status, $body_bytes_sent and $request_time are always numeric, so they’re written without quotes; query tools can compare them as numbers. $upstream_response_time, on the other hand, is kept in quotes because it can be - when no upstream was contacted, or a comma-separated list when more than one upstream was tried. Using $uri instead of $request_uri leaves out the query string; the reason is in the sensitive data section below. $request_id generates a request-specific ID; if you pass it to the application with proxy_set_header X-Request-ID $request_id; and have it written to the application logs too, you can trace a single request across all layers.
In Apache, adding the %D field (the request’s duration in microseconds) to LogFormat for the same purpose at least records the response time.
logrotate configuration
An access log that isn’t rotated can fill the disk within a few weeks on a busy site. When the disk fills up, it’s not just logging that stops; database and session files can’t be written either. The site goes down completely. The logrotate definitions that come with packages cover the default directories; logs written to custom directories need a separate definition:
# /etc/logrotate.d/nginx-json
/var/log/nginx/*.json {
daily
rotate 30
missingok
notifempty
compress
delaycompress
dateext
create 0640 www-data adm
sharedscripts
postrotate
[ -f /run/nginx.pid ] && kill -USR1 "$(cat /run/nginx.pid)"
endscript
}
The USR1 signal tells Nginx to reopen its log files; the process isn’t restarted. If the signal isn’t sent, Nginx keeps writing to the renamed old file and the new file stays empty. delaycompress delays compressing the previous file by one round; a process that keeps writing until the signal is handled has no problems. The USR1 signal is used for the same purpose with PHP-FPM. User and group names vary by distribution (www-data on Debian/Ubuntu, nginx on RHEL-based systems); adjust them for your own system.
Verify the configuration in debug mode before putting it live:
logrotate -d /etc/logrotate.d/nginx-json # only shows what it would do
logrotate -f /etc/logrotate.d/nginx-json # forces a rotation
rotate 30 sets the number of files kept on the server; it shouldn’t be your entire retention policy. Long-term retention is done on the central system; only a short window remains on the local disk for quick access.
Central collection options
On a single server, grep and awk do the job. When multiple servers, several nodes behind a load balancer or the possibility of an attacker deleting logs on a compromised server come into play, central collection is needed. A log sitting on the server is open to the attacker too. Three common approaches:
| Option | Strength | Limitation |
|---|---|---|
| A central syslog server with rsyslog | Ready on almost every Linux; simple and lightweight | No querying or visualization; stores as files |
| Grafana Loki | Low storage cost since it indexes only labels; integrated with Grafana | Slower than indexed systems for full-text search |
| Elasticsearch/OpenSearch (ELK) | Powerful full-text search and analysis | High memory and disk needs; heavy operational load |
Nginx can also send logs directly to syslog: access_log syslog:server=10.0.20.5:514,tag=nginx json_log;. But syslog delivery over UDP gives no delivery guarantee; logs are silently lost when there’s a network problem. That’s why the setup we recommend is to keep writing logs to a local file and ship them by reading from the file with a collector agent (rsyslog’s imfile module, Grafana Alloy, Filebeat or Fluent Bit). When the central system is unreachable, the agent picks up where it left off.
For a small or medium-sized web infrastructure, Loki + Grafana is usually a balanced starting point in terms of cost and operational load; the response code distribution and response times from the access logs can be shown side by side with uptime and performance dashboards. You can design these dashboards together with alert rules as part of web monitoring and uptime monitoring. If web server records will be correlated with other systems and used for incident tracking, bring them into the scope of SIEM setup and security logs.
Log queries for troubleshooting
The queries below work with the JSON-format Nginx log and answer the first questions asked during an incident: which codes are being returned, which URL is failing, what’s slow, who’s generating the load? jq is available in most distributions’ package repositories.
# Status code distribution in the latest log
jq -r '.status' /var/log/nginx/access.json | sort | uniq -c | sort -rn
# Most frequent URIs returning 5xx
jq -r 'select(.status >= 500) | .uri' /var/log/nginx/access.json \
| sort | uniq -c | sort -rn | head -20
# Requests taking longer than 2 seconds
jq -c 'select(.request_time > 2) | {time, uri, request_time, upstream_time}' \
/var/log/nginx/access.json | tail -50
# Client addresses making the most requests
jq -r '.remote_addr' /var/log/nginx/access.json | sort | uniq -c | sort -rn | head
If you use Loki, you can ask the same questions by time range with LogQL:
{job="nginx"} | json | status >= 500
sum by (status) (count_over_time({job="nginx"} | json [5m]))
Keep this in mind when interpreting: for codes such as 502 and 504, the Nginx access log only shows the result. The cause (connection refused, timeout, upstream closed early) is in the error log. When 5xx errors rise, the first place to look in the console is the error log lines in the same window as the timestamp in the access log; then search the PHP-FPM log for the “max_children reached” warning. Analyzing search engine crawler behavior from logs is a separate topic; we covered it in the article on Googlebot crawl analysis with server logs.
Retention period and sensitive data control
Since access logs contain IP addresses, user agents and visited URLs, they can qualify as personal data. In that case the retention period stops being a technical preference; it becomes a decision that must be defined in the organization’s data processing inventory. In Türkiye, the general principles of Law No. 6698 (KVKK) and, depending on the nature of the service, the traffic data retention obligations under Law No. 5651 should be evaluated together. Set the period with your legal department, and configure local rotation and central retention accordingly.
As important as the retention period is the data written to logs unintentionally. Checklist:
- Does the query string carry a password reset token, session ID, API key or email address? If so, use
$uriinstead of$request_uriin the access log, or fix the application so it doesn’t carry this information in the query string. - Have
AuthorizationorCookieheaders been added to the log format? Remove headers added for debugging and forgotten. - Does the application dump the request body (including form data) into the logs on error?
- Is PHP
display_errorsoff andlog_errorson in production? Error messages can contain database connection details. - Do log file permissions give read access only to the authorized user and group? Are there log files under the web root?
- Is access to the central logging system role-based, and is access recorded?
- Are logs whose retention period has expired really deleted from both the server and the central system?
The ngx_http_log_module documentation is the reference for the full list of Nginx log variables, and the Grafana Loki documentation for LogQL syntax. If logging, reverse proxy and security settings need to be reviewed together, see the Nginx web server and reverse proxy service page.
Frequently asked questions
Does switching access logs to JSON format affect performance?
Usually there’s no noticeable effect; the cost of Nginx formatting a log line is small next to the cost of processing a request. The real difference is on disk: since field names are repeated on every line, JSON logs are larger, but compression largely closes this gap.
Why doesn’t Nginx write to the empty log file after rotation?
Nginx writes through an open file handle; when logrotate renames the file, the process keeps writing to the old file. If a USR1 signal isn’t sent to the Nginx master process in the postrotate section, the new file isn’t opened. Check the signal step and the path of the pid file.
Should Loki or ELK be preferred?
If the main need is to filter by time range and labels and derive counts and rates, Loki does the job with fewer resources. If frequent searching within free text, complex analysis and large-volume security analysis are needed, an Elasticsearch- or OpenSearch-based setup is more suitable; but it needs more memory, disk and operational effort.
How long should I keep web server logs?
There’s no single right period. Determine it with your legal department by looking at the proportionality principle in KVKK, the obligations of Law No. 5651 depending on the nature of the service, the need to investigate security incidents retrospectively and contractual terms. On the technical side, the common approach is to keep a short period on the local server and a policy-compliant period on the central system.