The CrowdSec local notifier (cs_notifier.sh) sent an immediate, fully-styled HTML email for every single locally-detected attack ban (origin = crowdsec). During active scanning/probing periods this produced dozens to hundreds of emails per week, which was far more volume than expected and made the inbox unmanageable.
No infrastructure change caused this — it was the original by-design behavior of the notifier (immediate per-decision alerting), which became unmanageable at the observed local attack volume.
Reviewed full per-IP email-sending loop in cs_notifier.sh
Needed to understand existing LAPI polling, origin routing, and email format before changing behavior
Confirmed: every "crowdsec" origin decision triggered an immediate styled HTML email; CAPI decisions were already batched into a midnight digest
cs_notifier.sh: append local "crowdsec" decisions to local_pending.json (no email) cs_local_digest.sh (new): read local_pending.json, send one batched email, clear log
Decouples fast polling (every 5 min) from email sending (every 4 hours) so volume drops without losing any ban data
cs_notifier.sh simplified — CAPI routing logic unchanged, local routing now just appends to a pending file
sudo bash -c 'echo '"'"'{"value":"1.2.3.4","scenario":"crowdsecurity/http-probing","duration":"4h","type":"ban","scope":"Ip","origin":"crowdsec","start_at":"2026-06-09T14:00:00Z"}'"'"' >> /var/log/crowdsec-notifier/local_pending.json'
sudo bash /usr/scripts/crowdsec/cs_local_digest.sh
sudo tail -20 /var/log/crowdsec-notifier/notifier.log
Verify digest builds, sends, and clears the pending log without waiting for a real attack
First attempts: email format didn't match expectations, and the "Banned IPs — Past 4 Hours" table rendered with an empty body — IP rows were missing entirely
echo "$SNAPSHOT" | python3 - << 'PYEOF' ... (reads sys.stdin) ... PYEOF
Table rows were empty even though local_pending.json had data — needed to isolate whether the bug was in JSON parsing or in HTML assembly
Confirmed: this pattern produced zero output. A heredoc attached to python3 - becomes stdin itself, silently overriding the piped data — sys.stdin in Python was reading the heredoc body, not $SNAPSHOT
python3 << PYEOF
snapshot = """$SNAPSHOT"""
for line in snapshot.strip().splitlines():
...
PYEOF
Pass $SNAPSHOT directly into the heredoc as a shell-expanded Python string instead of piping
Re-ran full test — table populated correctly with IP, scenario, type, duration, scope, detected time, and CTI/WHOIS/AbuseIPDB links; log confirmed send + local_pending.json cleared
Notification volume: the original notifier sent one full HTML email per locally-detected ban with no batching, which scales linearly with attack frequency and quickly becomes unmanageable.
The "Banned IPs — Past 4 Hours" table rendered with headers but an empty <tbody>. The script logged a non-zero pending count, so data existed — the failure was in row generation, not data collection.
echo "$SNAPSHOT" | python3 - << 'PYEOF'
import sys, json
for line in sys.stdin: # <- reads the HEREDOC body, not $SNAPSHOT
...
PYEOF
When a heredoc is attached directly to python3 -, the heredoc body becomes the process's stdin, completely overriding any piped input. sys.stdin received the (empty) heredoc content instead of $SNAPSHOT, so the loop never executed.
Resolution: Replaced immediate per-ban emails with a 5-minute polling script that appends local bans to local_pending.json, plus a new cs_local_digest.sh that runs every 4 hours, builds one batched email (stat cards + full IP table), sends it, and clears the log — and fixed the row-generation bug by passing $SNAPSHOT as an inline Python variable inside the heredoc instead of piping it.
/usr/scripts/crowdsec/cs_notifier.sh
/usr/scripts/crowdsec/cs_local_digest.sh
chmod +x /usr/scripts/crowdsec/cs_notifier.sh chmod +x /usr/scripts/crowdsec/cs_local_digest.sh mkdir -p /var/log/crowdsec-notifier
Every 5 minutes: bash /usr/scripts/crowdsec/cs_notifier.sh
Every 4 hours (cron: 0 0,4,8,12,16,20 * * *): bash /usr/scripts/crowdsec/cs_local_digest.sh
sudo bash -c 'echo '"'"'{"value":"1.2.3.4","scenario":"crowdsecurity/http-probing","duration":"4h","type":"ban","scope":"Ip","origin":"crowdsec","start_at":"2026-06-09T14:00:00Z"}'"'"' >> /var/log/crowdsec-notifier/local_pending.json'
sudo bash /usr/scripts/crowdsec/cs_local_digest.sh
sudo tail -20 /var/log/crowdsec-notifier/notifier.log
| Lesson | Detail |
|---|---|
| Batch high-frequency alerts | Per-event email alerting doesn't scale with attack frequency. A 4-hour batched digest preserves full visibility (every IP, scenario, type, duration, scope, and threat-intel links) while cutting email volume from hundreds/week to a handful/day. |
| Skip-if-empty for digests | cs_local_digest.sh exits silently with no email if local_pending.json is empty — avoids inbox clutter during quiet periods. |
| Snapshot-before-clear pattern | The pending log is only cleared (> local_pending.json) after a confirmed successful SMTP send. If the send fails, the data stays intact and is retried on the next 4-hour run. |
| python3 - with a heredoc ignores piped stdin | echo "$DATA" | python3 - << EOF ... EOF does NOT work as expected — the heredoc body becomes stdin and the pipe is discarded. Pass shell data into Python via inline variable interpolation inside the heredoc instead (var = """$DATA"""). |
| Iterating on email templates | Several format iterations were needed to match the desired layout exactly. Keeping the rendered HTML output from a real test send was the fastest way to confirm/correct structure. |
| Item | Old value | New value |
|---|---|---|
| cs_notifier.sh behavior | Sends immediate per-IP email on each local ban |
Appends local bans to local_pending.json (no email) |
| New scheduled job | (none) |
cs_local_digest.sh every 4 hours — cron: 0 0,4,8,12,16,20 * * * |
| New log file | (none) |
/var/log/crowdsec-notifier/local_pending.json |
| server_nas16.md | 2-script architecture (cs_notifier.sh, cs_digest.sh) |
3-script architecture — updated (cs_notifier.sh, cs_local_digest.sh, cs_digest.sh) |