WAF false positive and a duplicate rule ID that took down a balancer

A customer's mobile app could not update user profiles or delete transactions: every PUT and DELETE request received a 403. This is a textbook WAF false positive on a balancer running Coraza WAF. While patching it, if a new rule is given the same ID as an existing one, the next nginx reload can make every website on the balancer stop serving, even though nginx -t and systemctl reload both report success. This guide follows the actual order of handling: classify the blocked requests, lift fail2ban's mistaken bans, write narrow exceptions, and use a safe reload procedure so that a WAF false positive for one customer does not turn into an incident for every customer.

Background on Coraza WAF, OWASP CRS 4, anomaly scoring and how to organize exception files is in Coraza WAF and OWASP CRS 4 on Nginx.

Symptoms of a WAF false positive

• The mobile app (User-Agent like okhttp/4.x) reports errors when updating the profile, changing notification settings and deleting transactions, which are exactly the endpoints that use PUT and DELETE.
• The error is a 403 from the balancer. The request is blocked right at the balancer, so it never reaches the PHP backend on the minion.
• Some real users' IPs are on fail2ban's ban list, meaning they are blocked completely, not just on a few endpoints.

All three signs point to the balancer layer: the errors are tied to a group of methods, the 403 is generated before the backend, and IPs are fully blocked after a series of errors. That is a WAF false positive combined with fail2ban counting 403s.

Investigation process

Step 1: separate the WAF false positive from real attacks in the audit log

The Coraza audit log (for example /var/log/coraza/audit.log) writes one JSON line per transaction. Filter by the customer's domains, then group by rule ID and by path. Not every blocked request is a WAF false positive:

CRS rule 911100 only allows GET HEAD POST OPTIONS by default. Rule 933150 matches the names of sensitive PHP functions in parameters. For a properly designed REST API, both are predictable WAF false positive cases. But without careful filtering, it is very easy to also "patch" the requests scanning for .env, because they return the very same 403.

# count blocked requests by rule ID for one host
# Coraza v3 writes the rule ID inside the error_message string, the data field is empty;
# RelevantOnly also logs non-blocked transactions, so filter on is_interrupted
jq -r 'select(.transaction.is_interrupted
              and (.transaction.request.headers.host // .transaction.request.headers.Host)[0] == "api-app.example.com")
       | .messages[]?.error_message' /var/log/coraza/audit.log \
  | grep -oE '\[id "[0-9]+"\]' | sort | uniq -c | sort -rn

Step 2: fail2ban turns a WAF false positive into an IP ban

A jail such as nginx-coraza counts the 403s returned by the WAF and bans IPs. If the jail is shared by every site on the balancer, the ban list mixes IPs from many sites. Match each IP against the 403 log of the customer's own domain, then classify:

• IPs that only hit rules 911100/933150 on API endpoints with the app's User-Agent: real users, lift the ban.
• IPs scanning /config/.env, /api/.env or exploiting wp-json: bots, keep the ban.
• IPs banned for violations on other sites: leave them alone.

Each WAF false positive does more than break one request. It can turn a real user into an "attacker" in fail2ban's eyes.

Step 3: patch, reload, and the duplicate rule ID trap

A common routine: add two exception rules at the end of the tuning file, view the end of the file with tail, see the latest IDs and number the new ones after them. nginx -t reports OK, systemctl reload nginx reports success. But if that ID is already used somewhere in the middle of the file, the journal records:

systemd[1]: Reloading nginx.service ...
nginx: [notice] coraza: rules collected inline/local: 0/1 (WAFs will be created in worker processes)
systemd[1]: Reloaded nginx.service ...
nginx: [error] coraza: failed to create WAF: invalid WAF config from file:
       ... failed to compile the directive "secrule": duplicated rule id 10001
nginx: [error] coraza: failed to build loc WAF in worker
nginx: [alert] worker process ... exited with fatal code 2 and cannot be respawned

From that moment on, every site on the balancer stops serving, and it is usually only discovered when users report it.

Root causes

1. A WAF false positive on a REST API. The CRS default method policy and the list of PHP function names know nothing about an API that uses PUT/DELETE and has a field named number_format. fail2ban amplifies the error: 403s caused by the WAF false positive are counted the same as 403s caused by attacks.
2. A duplicate rule ID takes down the balancer. Three factors combine:
   • Coraza does not compile rules during the config test. The line WAFs will be created in worker processes says it plainly: rules are only built when a worker starts. So nginx -t cannot catch a duplicate ID.
   • Reload is "graceful": the master lets the old workers exit and spawns new ones. The new workers exit with fatal code 2 and the master does not respawn them. The nginx master is still alive, but no worker is left to serve.
   • systemctl reload still returns success, so anyone looking only at the exit code thinks everything is fine.

The blast radius is the entire balancer, because every site shares one nginx process and one Coraza exception file.

Remediation

When the balancer has lost its workers

# find duplicate IDs across every loaded rule file
grep -hoE 'id:[0-9]+' /etc/coraza/rules/*.conf /etc/coraza/crs-setup.conf /etc/coraza/crs/rules/*.conf \
  | sort | uniq -d

# change the IDs of the newly added rules, test and reload again
sed -i 's/id:10001,/id:10101,/; s/id:10002,/id:10102,/' /etc/coraza/rules/local-tuning.conf
nginx -t && systemctl reload nginx
sleep 2; pgrep -c -f "nginx: worker"

Then curl the home pages of the customer's domains and a few random other domains on the balancer to confirm.

Patch the WAF false positive with the right scope

The principle is to remove as narrowly as possible: open the methods only for the API hosts, and drop rule 933150 only for one endpoint:

# WAF false positive 1: REST API legitimately using PUT/DELETE (rule 911100)
SecRule REQUEST_HEADERS:Host "@within api-app.example.com,api-admin.example.com" \
    "id:10101,phase:1,pass,nolog,setvar:tx.allowed_methods=GET HEAD POST PUT DELETE PATCH OPTIONS"

# WAF false positive 2: JSON field "number_format" treated as PHP function injection (rule 933150)
SecRule REQUEST_URI "@beginsWith /api/v1/me/preferences" \
    "id:10102,phase:2,pass,nolog,ctl:ruleRemoveById=933150"

The first rule runs in phase 1 so that the tx.allowed_methods variable is reset before rule 911100 checks the method. Test again with a PUT /api/v1/me request, confirm the audit log no longer records a block, and only then lift the bans on real users' IPs:

fail2ban-client set nginx-coraza unbanip <user-ip>

When the customer asks to turn off the WAF

Sometimes a customer asks to turn the WAF off entirely for their domain, even after the WAF false positive cases have been patched. If you must do it, turn it off per host and verify with a simple SQLi payload:

SecRule REQUEST_HEADERS:Host "@within example.com,www.example.com" \
    "id:10201,phase:1,pass,nolog,ctl:ruleEngine=Off"

curl -s -o /dev/null -w '%{http_code}\n' "https://example.com/?id=1'%20UNION%20SELECT%201"     # 200: disabled
curl -s -o /dev/null -w '%{http_code}\n' "https://other-site.example/?id=1'%20UNION%20SELECT%201" # 403: still enabled

This decision must come with a written risk statement (the domain no longer has SQLi, XSS or RCE filtering at the balancer), an expiry date and a reminder to turn it back on. Turning it back on only requires deleting the engine-off rule, keeping the narrow exceptions, then testing and reloading.

Checklist for fixing a WAF false positive on a balancer

• Before adding a rule: scan the IDs in every tuning file, crs-setup.conf and crs/rules/*.conf, not just the end of the file.
• Agree on ID ranges by purpose (for example one block of a hundred IDs per customer or per exception type), documented in a comment at the top of the file.
• Make a timestamped backup of the exception file before every edit, with notes on how to roll back.
• Use a single reload script: check for duplicate IDs, nginx -t, reload, wait a few seconds, count the workers, curl a few sample domains. If any step fails, copy the backup back and reload.
• After reloading: read journalctl -u nginx right away and search for failed to create WAF. Do not trust nginx -t or the exit code of systemctl reload.
• Set up an external HTTP probe for each balancer, so users are not the first monitoring system.
• fail2ban: do not ban immediately when the only evidence is a 403 from a rule known to cause false positives; consider splitting jails by site group.
• Every per-host ruleEngine=Off rule must have an expiry date.

Lessons learned

• A mistake in shared configuration has the blast radius of the whole balancer. When you fix something for one customer, test as if you were fixing it for everyone.
• nginx -t only checks what nginx understands. When a module builds its configuration inside the worker, errors only show up when the worker starts.
• Classify before you remove. Most blocked requests are usually bots. A real WAF false positive only needs a few very narrow exceptions, not turning the WAF off.
• Turning off the WAF is a time-limited decision. The customer has the right to ask, but the operations team must record the risk and follow up until it is turned back on.

References: the Coraza WAF documentation and the OWASP Core Rule Set documentation. Other blocking measures at the balancer are covered in Standardizing Nginx security: security snippets and connection limits.

Next article

Image upload errors 419 and 502 on Kubernetes. One large image, several wrong hypotheses and two memory limits stacked on top of each other in the same pod.