commit 16746fc2b8aa1ce75b3e7b8ab2e25bcb6b6cbfa2 Author: Rufus King Date: Sun Jul 26 21:21:03 2026 -0400 Initial import diff --git a/CrowdSec/crowdsec_digest_summary.html b/CrowdSec/crowdsec_digest_summary.html new file mode 100644 index 0000000..a9bf925 --- /dev/null +++ b/CrowdSec/crowdsec_digest_summary.html @@ -0,0 +1,527 @@ + + + + + + +KingDezigns — CrowdSec Local Alert Digest Summary + + + + + +

+ + +

KingDezigns — CrowdSec Local Alert Digest summary

+ 📅 2026-06-09 + 🖥 NAS16 (CrowdSec notifier) / HAS (CrowdSec LAPI) + ⚙️ OpenMediaVault / OMV Scheduled Jobs + 🐳 Bash + Python3 + SMTP_SSL + ☁️ CrowdSec LAPI stream +

+ + + +

The problem

+ +

The CrowdSec local notifier (cs_notifier.sh) sent an immediate, fully-styled HTML email for every single locally-detected attack ban (origin = crowdsec). During active scanning/probing periods this produced dozens to hundreds of emails per week, which was far more volume than expected and made the inbox unmanageable.

+ +

+ "CrowdSec LIVE ATTACK Detected on HAS - $IP" (one email per ban) + Hundreds of emails received weekly +

+ +

No infrastructure change caused this — it was the original by-design behavior of the notifier (immediate per-decision alerting), which became unmanageable at the observed local attack volume.

+ + + +

Build & test steps

+ +

1 Reviewed original cs_notifier.sh

Reviewed full per-IP email-sending loop in cs_notifier.sh

Needed to understand existing LAPI polling, origin routing, and email format before changing behavior

Confirmed: every "crowdsec" origin decision triggered an immediate styled HTML email; CAPI decisions were already batched into a midnight digest

+ +

2 Split notifier into polling + digest

cs_notifier.sh: append local "crowdsec" decisions to local_pending.json (no email)
+cs_local_digest.sh (new): read local_pending.json, send one batched email, clear log

Decouples fast polling (every 5 min) from email sending (every 4 hours) so volume drops without losing any ban data

cs_notifier.sh simplified — CAPI routing logic unchanged, local routing now just appends to a pending file

+ +

3 Manual test — inject fake ban

sudo bash -c 'echo '"'"'{"value":"1.2.3.4","scenario":"crowdsecurity/http-probing","duration":"4h","type":"ban","scope":"Ip","origin":"crowdsec","start_at":"2026-06-09T14:00:00Z"}'"'"' >> /var/log/crowdsec-notifier/local_pending.json'
+sudo bash /usr/scripts/crowdsec/cs_local_digest.sh
+sudo tail -20 /var/log/crowdsec-notifier/notifier.log

Verify digest builds, sends, and clears the pending log without waiting for a real attack

First attempts: email format didn't match expectations, and the "Banned IPs — Past 4 Hours" table rendered with an empty body — IP rows were missing entirely

+ +

4 Isolated row-generation bug

echo "$SNAPSHOT" | python3 - << 'PYEOF'
+... (reads sys.stdin) ...
+PYEOF

Table rows were empty even though local_pending.json had data — needed to isolate whether the bug was in JSON parsing or in HTML assembly

Confirmed: this pattern produced zero output. A heredoc attached to python3 - becomes stdin itself, silently overriding the piped data — sys.stdin in Python was reading the heredoc body, not $SNAPSHOT

+ +

5 Fixed with inline variable substitution

python3 << PYEOF
+snapshot = """$SNAPSHOT"""
+for line in snapshot.strip().splitlines():
+    ...
+PYEOF

Pass $SNAPSHOT directly into the heredoc as a shell-expanded Python string instead of piping

Re-ran full test — table populated correctly with IP, scenario, type, duration, scope, detected time, and CTI/WHOIS/AbuseIPDB links; log confirmed send + local_pending.json cleared

+ +

+ + + +

Root cause

+ +

Notification volume: the original notifier sent one full HTML email per locally-detected ban with no batching, which scales linearly with attack frequency and quickly becomes unmanageable.

+ +

⚠ Key clue (table-rendering bug during testing)

The "Banned IPs — Past 4 Hours" table rendered with headers but an empty <tbody>. The script logged a non-zero pending count, so data existed — the failure was in row generation, not data collection.

echo "$SNAPSHOT" | python3 - << 'PYEOF'
+import sys, json
+for line in sys.stdin:   # <- reads the HEREDOC body, not $SNAPSHOT
+    ...
+PYEOF

When a heredoc is attached directly to python3 -, the heredoc body becomes the process's stdin, completely overriding any piped input. sys.stdin received the (empty) heredoc content instead of $SNAPSHOT, so the loop never executed.

+ +

Resolution: Replaced immediate per-ban emails with a 5-minute polling script that appends local bans to local_pending.json, plus a new cs_local_digest.sh that runs every 4 hours, builds one batched email (stat cards + full IP table), sends it, and clears the log — and fixed the row-generation bug by passing $SNAPSHOT as an inline Python variable inside the heredoc instead of piping it.

+ + + + + + +

Scripts / files created

Replace (updated)

/usr/scripts/crowdsec/cs_notifier.sh

Add (new)

/usr/scripts/crowdsec/cs_local_digest.sh

Setup

chmod +x /usr/scripts/crowdsec/cs_notifier.sh
+chmod +x /usr/scripts/crowdsec/cs_local_digest.sh
+mkdir -p /var/log/crowdsec-notifier

Run — cs_notifier.sh (existing OMV job, unchanged schedule)

Every 5 minutes:
+bash /usr/scripts/crowdsec/cs_notifier.sh

Run — cs_local_digest.sh (new OMV scheduled job)

Every 4 hours (cron: 0 0,4,8,12,16,20 * * *):
+bash /usr/scripts/crowdsec/cs_local_digest.sh

Manual test

sudo bash -c 'echo '"'"'{"value":"1.2.3.4","scenario":"crowdsecurity/http-probing","duration":"4h","type":"ban","scope":"Ip","origin":"crowdsec","start_at":"2026-06-09T14:00:00Z"}'"'"' >> /var/log/crowdsec-notifier/local_pending.json'
+sudo bash /usr/scripts/crowdsec/cs_local_digest.sh
+sudo tail -20 /var/log/crowdsec-notifier/notifier.log

+ + + +

Lessons learned

+ + + + + + + + + + + + + + + + + + + + + + + + + + +

Lesson	Detail
Batch high-frequency alerts	Per-event email alerting doesn't scale with attack frequency. A 4-hour batched digest preserves full visibility (every IP, scenario, type, duration, scope, and threat-intel links) while cutting email volume from hundreds/week to a handful/day.
Skip-if-empty for digests	cs_local_digest.sh exits silently with no email if local_pending.json is empty — avoids inbox clutter during quiet periods.
Snapshot-before-clear pattern	The pending log is only cleared (`> local_pending.json`) after a confirmed successful SMTP send. If the send fails, the data stays intact and is retried on the next 4-hour run.
python3 - with a heredoc ignores piped stdin	`echo "$DATA" \| python3 - << EOF ... EOF` does NOT work as expected — the heredoc body becomes stdin and the pipe is discarded. Pass shell data into Python via inline variable interpolation inside the heredoc instead (`var = """$DATA"""`).
Iterating on email templates	Several format iterations were needed to match the desired layout exactly. Keeping the rendered HTML output from a real test send was the fastest way to confirm/correct structure.

+ + + +

Network map — items to update

+ + + + + + + + + + + + + + + + + + + + + + + + + + +

Item	Old value	New value
cs_notifier.sh behavior	`Sends immediate per-IP email on each local ban`	`Appends local bans to local_pending.json (no email)`
New scheduled job	`(none)`	`cs_local_digest.sh every 4 hours — cron: 0 0,4,8,12,16,20 * * *`
New log file	`(none)`	`/var/log/crowdsec-notifier/local_pending.json`
server_nas16.md	`2-script architecture (cs_notifier.sh, cs_digest.sh)`	`3-script architecture — updated (cs_notifier.sh, cs_local_digest.sh, cs_digest.sh)`

+ + + + + +

+ + diff --git a/CrowdSec/cs_local_digest.sh b/CrowdSec/cs_local_digest.sh new file mode 100644 index 0000000..d954ce1 --- /dev/null +++ b/CrowdSec/cs_local_digest.sh @@ -0,0 +1,237 @@ +#!/bin/bash +# ============================================================= +# CrowdSec Local Attack — 4-Hour Digest Mailer +# Host: NAS16 (192.168.150.40) +# Schedule: Every 4 hours via OMV Scheduled Jobs +# Recommended cron: 0 0,4,8,12,16,20 * * * +# +# Reads: /var/log/crowdsec-notifier/local_pending.json +# Sends: One batched HTML email: +# - Dark header + red status banner +# - 3 stat cards: Local Bans / Origin / CAPI Today +# - Single table: IP Address / Scenario / Type / +# Duration / Scope / Detected At / Links +# - Amber CAPI footer note +# - Dark footer +# Clears: local_pending.json after a successful send. +# Skips: Send entirely if there are zero pending bans. +# ============================================================= + +SMTP_HOST="smtppro.zoho.com" +SMTP_PORT="465" +SMTP_USER="rufus.king@kingdezigns.com" +SMTP_PASS="Valerie8545" +FROM_EMAIL="rufus.king@kingdezigns.com" +TO_EMAIL="rufus.king@kingdezigns.com" + +LOCAL_PENDING="/var/log/crowdsec-notifier/local_pending.json" +CAPI_DIGEST="/var/log/crowdsec-notifier/capi_digest.json" +LOG_FILE="/var/log/crowdsec-notifier/notifier.log" + +mkdir -p /var/log/crowdsec-notifier +touch "$LOCAL_PENDING" +touch "$CAPI_DIGEST" + +log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG_FILE"; } + +log "--- cs_local_digest run start ---" + +# --- Count pending bans --- +PENDING_COUNT=$(grep -c . "$LOCAL_PENDING" 2>/dev/null || echo 0) +PENDING_COUNT=$(echo "$PENDING_COUNT" | tr -d ' ') + +if [[ "$PENDING_COUNT" -eq 0 ]]; then + log "No local bans in this window. Skipping email." + log "--- cs_local_digest run complete (no bans) ---" + exit 0 +fi + +log "Building digest for $PENDING_COUNT local ban(s)." + +NOW=$(date '+%Y-%m-%d %H:%M:%S UTC') + +CAPI_TODAY=$(wc -l < "$CAPI_DIGEST" 2>/dev/null | tr -d ' ') +CAPI_TODAY=${CAPI_TODAY:-0} + +# Snapshot the pending file before clearing +SNAPSHOT=$(cat "$LOCAL_PENDING") + +# --- Build table rows --- +# NOTE: $SNAPSHOT is expanded inline inside the Python heredoc. +# Do NOT pipe into python3 - with a heredoc: the heredoc becomes stdin, +# silently replacing the pipe. Pass data as an inline variable instead. +TABLE_ROWS=$(python3 << PYEOF +import json + +snapshot = """$SNAPSHOT""" +rows = [] +for line in snapshot.strip().splitlines(): + line = line.strip() + if not line: + continue + try: + d = json.loads(line) + except Exception: + continue + + ip = d.get("value", "unknown") + scenario = d.get("scenario", "unknown") + duration = d.get("duration", "unknown") + ban_type = d.get("type", "ban").upper() + scope = d.get("scope", "Ip").upper() + ban_time = d.get("start_at", "") + if ban_time: + ban_time = ban_time[:19].replace("T", " ") + " UTC" + else: + ban_time = "unknown" + + rows.append(f""" + 🌐 {ip} + {scenario} + {ban_type} + {duration} + {scope} + {ban_time} + + CTI + WHOIS + Abuse + + """) + +print("\n".join(rows)) +PYEOF +) + +log "Generated ${PENDING_COUNT} table row(s)." + +# --- Write the full email HTML --- +HTML_FILE=$(mktemp /tmp/cs_local_digest_XXXXXX.html) +cat > "$HTML_FILE" << HTMLEOF + + + + + + +

+ + + + + + + + + + + + + + + + + + + + +

+ + + +

CrowdSec Intrusion Prevention

🛡 HAS — homeassistant

$NOW | Machine: homeassistant

🚨 LOCAL BANS — 4H DIGEST

+ 🚨 $PENDING_COUNT locally-detected attack(s) banned in the past 4 hours. +

+ + + + + + +

Local Bans

$PENDING_COUNT

this window

Origin

⚡ LOCAL

crowdsec detection

CAPI Today

$CAPI_TODAY

cloud blocklist bans

+ 📌 Banned IPs — Past 4 Hours +

+ + + + + + + + + + + + + +$TABLE_ROWS + +

IP Address	Scenario	Type	Duration	Scope	Detected At	Links

+ ⛅ Cloud blocklist (CAPI): $CAPI_TODAY pre-emptive ban(s) recorded today. Full CAPI digest delivered at midnight by cs_digest.sh. +

+ CrowdSec Intrusion Prevention | homeassistant (HAS) | KingDezigns Infrastructure +

+ + +HTMLEOF + +# --- Send the email --- +python3 << PYEOF >> "$LOG_FILE" 2>&1 +import smtplib, ssl +from email.mime.multipart import MIMEMultipart +from email.mime.text import MIMEText + +with open("$HTML_FILE") as f: + html_body = f.read() + +msg = MIMEMultipart("alternative") +msg["Subject"] = "CrowdSec 4H Local Digest — $PENDING_COUNT Ban(s) on HAS" +msg["From"] = "CrowdSec KingDezigns <$FROM_EMAIL>" +msg["To"] = "$TO_EMAIL" +msg.attach(MIMEText(html_body, "html")) + +ctx = ssl.create_default_context() +with smtplib.SMTP_SSL("$SMTP_HOST", $SMTP_PORT, context=ctx, timeout=15) as s: + s.login("$SMTP_USER", "$SMTP_PASS") + s.sendmail("$FROM_EMAIL", ["$TO_EMAIL"], msg.as_string()) +print("sent") +PYEOF + +SEND_STATUS=$? +rm -f "$HTML_FILE" + +if [[ $SEND_STATUS -eq 0 ]]; then + log "4-hour digest sent. $PENDING_COUNT ban(s) reported. Clearing local_pending.json." + > "$LOCAL_PENDING" +else + log "ERROR: Digest email failed. local_pending.json NOT cleared — will retry next run." +fi + +log "--- cs_local_digest run complete ---" diff --git a/CrowdSec/cs_notifier.sh b/CrowdSec/cs_notifier.sh new file mode 100644 index 0000000..fca900f --- /dev/null +++ b/CrowdSec/cs_notifier.sh @@ -0,0 +1,100 @@ +#!/bin/bash +# ============================================================= +# CrowdSec Local Attack Notifier +# Host: NAS16 (192.168.150.40) +# Target: CrowdSec LAPI on HAS (192.168.150.30:8080) +# Schedule: Every 5 minutes via OMV Scheduled Jobs +# +# Local (crowdsec origin) decisions are appended to: +# /var/log/crowdsec-notifier/local_pending.json +# and batched into a 4-hour digest by cs_local_digest.sh. +# +# CAPI decisions continue to be appended to capi_digest.json +# and sent as a midnight summary by cs_digest.sh (unchanged). +# ============================================================= + +LAPI_URL="http://192.168.150.30:8080" +API_KEY="K1nZ5TYeyl18adqw7+dop/po9gHZNQvsuLz78lla/34" + +STARTUP_FLAG="/var/log/crowdsec-notifier/stream_initialized.flag" +LOCAL_PENDING="/var/log/crowdsec-notifier/local_pending.json" +CAPI_DIGEST="/var/log/crowdsec-notifier/capi_digest.json" +LOG_FILE="/var/log/crowdsec-notifier/notifier.log" + +mkdir -p /var/log/crowdsec-notifier +touch "$CAPI_DIGEST" +touch "$LOCAL_PENDING" + +log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG_FILE"; } + +log "--- cs_notifier run start ---" + +# --- First run: initialize stream cursor silently --- +if [[ ! -f "$STARTUP_FLAG" ]]; then + log "First run — initializing stream cursor. No emails will be sent." + INIT_RESPONSE=$(curl -s --max-time 60 \ + -H "X-Api-Key: $API_KEY" \ + "${LAPI_URL}/v1/decisions/stream?startup=true") + + if [[ -z "$INIT_RESPONSE" ]]; then + log "ERROR: Empty response from LAPI during startup. Will retry next run." + exit 1 + fi + + NEW_COUNT=$(echo "$INIT_RESPONSE" | python3 -c " +import sys, json +data = json.load(sys.stdin) +print(len(data.get('new') or [])) +" 2>/dev/null) + + log "Stream cursor initialized. Skipped $NEW_COUNT existing decisions. Ready for live monitoring." + touch "$STARTUP_FLAG" + log "--- cs_notifier run complete (init) ---" + exit 0 +fi + +# --- Normal run: fetch only new decisions since last call --- +RESPONSE=$(curl -s --max-time 30 \ + -H "X-Api-Key: $API_KEY" \ + "${LAPI_URL}/v1/decisions/stream?startup=false") + +HTTP_CHECK=$(echo "$RESPONSE" | python3 -c "import sys,json; json.load(sys.stdin); print('ok')" 2>/dev/null) +if [[ "$HTTP_CHECK" != "ok" ]]; then + log "ERROR: Invalid response from LAPI stream endpoint" + exit 1 +fi + +# --- Parse and route new decisions --- +NEW_LOCAL_COUNT=0 +NEW_CAPI_COUNT=0 + +while IFS= read -r decision; do + [[ -z "$decision" ]] && continue + ORIGIN=$(echo "$decision" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('origin',''))" 2>/dev/null) + + if [[ "$ORIGIN" == "crowdsec" ]]; then + # Append to pending local digest — cs_local_digest.sh will send every 4 hours + echo "$decision" >> "$LOCAL_PENDING" + NEW_LOCAL_COUNT=$((NEW_LOCAL_COUNT + 1)) + elif [[ "$ORIGIN" == "CAPI" ]]; then + # Append to CAPI digest — cs_digest.sh will send at midnight + echo "$decision" >> "$CAPI_DIGEST" + NEW_CAPI_COUNT=$((NEW_CAPI_COUNT + 1)) + fi + +done < <(echo "$RESPONSE" | python3 -c " +import sys, json +data = json.load(sys.stdin) +new = data.get('new') or [] +for item in new: + print(json.dumps(item)) +") + +CAPI_TODAY=$(wc -l < "$CAPI_DIGEST" 2>/dev/null | tr -d ' ') +CAPI_TODAY=${CAPI_TODAY:-0} +LOCAL_PENDING_COUNT=$(wc -l < "$LOCAL_PENDING" 2>/dev/null | tr -d ' ') +LOCAL_PENDING_COUNT=${LOCAL_PENDING_COUNT:-0} + +log "New local: $NEW_LOCAL_COUNT (pending total: $LOCAL_PENDING_COUNT) | New CAPI: $NEW_CAPI_COUNT | CAPI today: $CAPI_TODAY" + +log "--- cs_notifier run complete ---" diff --git a/ddns/ddns_summary.html b/ddns/ddns_summary.html new file mode 100644 index 0000000..7a9a070 --- /dev/null +++ b/ddns/ddns_summary.html @@ -0,0 +1,406 @@ + + + + + +KingDezigns — DDNS Updater Summary + + + + + +

+ +

KingDezigns — DDNS Updater Summary

+ 📅 May 19, 2025 + 🖥 NAS16 (Raspberry Pi) + ⚙️ OpenMediaVault + ☁️ DreamHost DNS API + 🌐 20 A Records · 4 Domains +

+ + + +

What was built

A self-contained Bash script that keeps all KingDezigns DreamHost A records synchronized with the current home WAN IP. It runs every 15 minutes via OMV Scheduled Tasks. It only calls the DreamHost API when the IP actually changes (or during a nightly sanity check), so there is zero risk of being banned for spamming.

+ 20 A records managed + 4 domains + Max 2 API calls per event + HTML email notifications + Every 15 min via cron + Daily sanity check 00:00–00:14 +

+ + + +

Logic flow map

+ +

⏱ Script runs (every 15 min via OMV cron)

↓

+ +

🌐 Get current WAN IP from api.ipify.org / ifconfig.me / icanhazip.com

↓

+ +

📄 Read cached previous IP from /var/log/ddns/last_wan_ip.txt

↓

+ +

🕛 Is the time between 00:00 and 00:14? (daily sanity-check window)

↓

+ + +

IP changed OR daily window

▶ Query DreamHost — check each A record

📡 Call dns-remove_record + dns-add_record for each mismatched record

⏳ Wait 3–5 minutes (random)

🔍 Re-query DreamHost — verify each record

+ + +

Still wrong

🔁 Attempt 2 of 2 — update remaining records

⏳ Wait 3–5 min · Re-verify

Still failing

✗ STOP (max attempts reached)

📧 Email: partial failure + manual fix instructions + WAN IP

Now correct

✓ Mark success

📧 Email: all records updated

All correct

✓ All records verified

📧 Email: success — all records updated

+ +

💾 Write new WAN IP to cache file

+ +

IP unchanged + not daily window

💾 Overwrite cache with same IP (refreshes timestamp)

🔇 Exit silently — no API calls, no email

+ +

⚠ DreamHost API — how updates work

DreamHost has no "update" DNS command. To change an A record you must remove the old value then add the new one. The script always fetches the live DreamHost value immediately before each remove/add pair so it never tries to remove a value that isn't there.

+ + + +

Domains & records managed

+ + + + + + + + + + + + + + + + + + + + + + + + +

Record (DreamHost "record" field)	Zone / Domain	Type
`albranch.org`	albranch.org	A (root / @)
`www.albranch.org`	albranch.org	A
`indigorhayne.com`	indigorhayne.com	A (root / @)
`www.indigorhayne.com`	indigorhayne.com	A
`rufusking.com`	rufusking.com	A (root / @)
`www.rufusking.com`	rufusking.com	A
`kingdezigns.com`	kingdezigns.com	A (root / @)
`www.kingdezigns.com`	kingdezigns.com	A
`*.web.kingdezigns.com`	kingdezigns.com	A (wildcard)
`cloud.kingdezigns.com`	kingdezigns.com	A
`files08.kingdezigns.com`	kingdezigns.com	A
`files16.kingdezigns.com`	kingdezigns.com	A
`has.kingdezigns.com`	kingdezigns.com	A
`nginx.kingdezigns.com`	kingdezigns.com	A
`omv08.kingdezigns.com`	kingdezigns.com	A
`omv16.kingdezigns.com`	kingdezigns.com	A
`pihole.kingdezigns.com`	kingdezigns.com	A
`plex.kingdezigns.com`	kingdezigns.com	A
`vault.kingdezigns.com`	kingdezigns.com	A
`webmin.kingdezigns.com`	kingdezigns.com	A

+ + + +

Deployment — step by step

+ +

1 Install dependencies

sudo apt-get update
+sudo apt-get install -y curl python3 mailutils

curl fetches WAN IP and calls the DreamHost API. python3 parses the JSON responses. mailutils provides the mail command for sending HTML notifications.

+ +

2 Copy the script to its permanent location

sudo cp ddns_update.sh /usr/local/sbin/ddns_update.sh

This puts it in a root-only system bin directory alongside other admin utilities.

+ +

3 Set ownership and permissions

sudo chown root:root /usr/local/sbin/ddns_update.sh
+sudo chmod 700 /usr/local/sbin/ddns_update.sh

The script reads/writes to /var/log/ddns/ and calls sendmail — it must run as root. Mode 700 prevents other users from reading the embedded API key.

+ +

4 Create the log directory

sudo mkdir -p /var/log/ddns
+sudo chmod 700 /var/log/ddns

The script will create this automatically on first run, but creating it manually with 700 ensures no other user can read the cached IP or logs before then.

+ +

5 Test a manual run

sudo /usr/local/sbin/ddns_update.sh

Run once by hand to verify API connectivity, confirm all 20 records are found, and check that the email arrives at rufus.king@kingdezigns.com. Watch the output — it prints everything it does.

On the very first run there is no cached IP, so it will treat every record as new and perform a full update cycle.

+ +

6 Verify the log file after test run

sudo cat /var/log/ddns/ddns_run.log

You should see the WAN IP detected, DreamHost responses for each record, and "DDNS run complete" at the bottom.

+ +

7 Add OMV Scheduled Task

OMV → System → Scheduled Tasks → + (Add)
+
+  Enabled:   ✓
+  Minute:    */15
+  Hour:      *
+  Day:       *
+  Month:     *
+  Weekday:   *
+  User:      root
+  Command:   /usr/local/sbin/ddns_update.sh

This fires the script every 15 minutes around the clock. OMV uses standard cron under the hood — you can verify with sudo crontab -l after saving.

Script is now live and fully automated.

+ + + +

Script & files created

Script location

/usr/local/sbin/ddns_update.sh

IP cache (previous WAN IP)

/var/log/ddns/last_wan_ip.txt

Run log (last 5000 lines, auto-trimmed)

/var/log/ddns/ddns_run.log

Notification email

rufus.king@kingdezigns.com

Manual run command

sudo /usr/local/sbin/ddns_update.sh

Force a fresh update (clears cache)

sudo rm /var/log/ddns/last_wan_ip.txt && sudo /usr/local/sbin/ddns_update.sh

Watch the live log

sudo tail -f /var/log/ddns/ddns_run.log

+ + + +

Email notification behavior

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Scenario	Email sent?	Subject line
IP unchanged, not daily window	No	Silent exit — no email
IP changed, all records updated successfully	Yes	[DDNS] ✅ All Records Updated — <new IP>
IP changed, some records failed after 2 attempts	Yes — action needed	[DDNS] ⚠️ PARTIAL FAILURE — Manual action needed — <new IP>
Daily sanity check (00:00–00:14), all records match	Yes	[DDNS] ✅ All Records Updated — <IP>
Daily sanity check, mismatch found & fixed	Yes	[DDNS] ✅ All Records Updated — <new IP>
WAN IP discovery fails (all 3 sources down)	No	Script aborts — will retry next 15-min run

All emails are sent as HTML using the KingDezigns branded template — same typography and color system as this document. They include a record-by-record results table and, on failure, a direct link to the DreamHost DNS panel with the correct IP pre-noted for manual entry.

+ + + +

Important notes & gotchas

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Item	Detail
API key is embedded in the script	The script is chmod 700 / root only. Do not copy or share the file without scrubbing the key first.
DreamHost has no "update" API command	Every update is a remove + add pair. The script always reads the live DreamHost value first so it never removes the wrong IP.
Wildcard record `*.web.kingdezigns.com`	DreamHost supports wildcard A records via API. The script passes the literal string `*.web.kingdezigns.com` to `dns-list_records` and it will match correctly.
Mail must be configured on NAS16	Run `sudo apt-get install mailutils` and ensure your MTA (Postfix or ssmtp) is configured to relay outbound email. Test with: `echo "test" \| mail -s "test" rufus.king@kingdezigns.com`
First run behavior	On the very first run there is no cached IP. The script treats every record as needing an update and performs a full cycle. This is expected and correct.
Daily window fires once per day	The 00:00–00:14 window aligns with the every-15-min schedule — only one run per day falls in that window. No extra logic needed.
Log rotation	The run log auto-trims itself to 5000 lines on every execution. No logrotate config needed.

+ + + + +

+ + diff --git a/ddns/ddns_update.sh b/ddns/ddns_update.sh new file mode 100644 index 0000000..2588b81 --- /dev/null +++ b/ddns/ddns_update.sh @@ -0,0 +1,662 @@ +#!/bin/bash +# ============================================================================= +# ddns_update.sh — DreamHost Dynamic DNS Updater (v2 — hardened 2026-07-23) +# KingDezigns Home Network — NAS16 +# +# Schedule every 15 minutes via OMV Scheduled Tasks: +# Minute: */15 Hour/Day/Month/Weekday: * +# Command: /usr/scripts/ddns/ddns_update.sh +# +# Manages A records for: +# albranch.org, indigorhayne.com, rufusking.com, kingdezigns.com (+subs) +# +# v2 CHANGES (root cause: 2026-07-23 DreamHost API rate-limit outage — see +# KingDezigns troubleshooting summary for full diagnosis): +# 1. Zone is fetched ONCE per run (1 API call) instead of once per record +# (was 23 calls just to check status). All checks are done locally +# against that snapshot. +# 2. Records are ADDED before the old value is REMOVED (was remove-then-add, +# which left a domain with NO A record at all if the add step failed). +# 3. A sleep is inserted between each record's mutating calls to stay well +# under DreamHost's documented add/remove throttling. +# 4. A max number of records are mutated per run (BATCH_LIMIT). Anything +# left over is written to a pending file and picked up automatically on +# the next 15-min run — no need to wait for the old remove-then-add loop +# to hammer all 23 records in one shot. +# 5. If several records in a row come back with an API error, the run +# aborts immediately (fail-fast) instead of grinding through the rest — +# that's the signature of a rate limit, and keeps retrying makes it worse. +# 6. The real DreamHost error reason (the "data" field) is captured and +# logged/emailed, not just the generic "error" result. +# 7. curl now uses --data-urlencode for every parameter instead of manual +# string concatenation (safer for records like *.web.kingdezigns.com). +# +# Behavior: +# Every 15-min run, an update pass happens if ANY of the following is true: +# - WAN IP changed since last cached value +# - Daily sanity-check window (00:00–00:14) +# - There are records left over (pending) from a previous run's batch +# limit or a rate-limit abort +# Each pass re-fetches the live zone once and compares it locally — so a +# "pending" record that was fixed manually (or by a later successful +# attempt) will simply drop out on its own. +# ============================================================================= + +# ── Configuration ───────────────────────────────────────────────────────────── +HOSTNAME_LABEL=$(hostname -s 2>/dev/null || echo "NAS16") +REPORT_TO="rufus.king@kingdezigns.com" +REPORT_FROM="ddns-monitor@nas16.local" # Display only — Postfix uses your Gmail relay +API_KEY="4ACNN8PRYNQCMCWW" +LOG_FILE="/var/log/ddns/ddns_run.log" +IP_CACHE="/var/log/ddns/last_wan_ip.txt" +PENDING_FILE="/var/log/ddns/pending_records.txt" # recordfailed_attempt_count + +# Rate-limit hardening knobs — tuned to stay well under DreamHost's documented +# dns-add_record throttle (~10 successful calls / 5 min, ~50 / hour). +BATCH_LIMIT=8 # max records mutated (add+remove) per script run +SLEEP_BETWEEN_RECORDS=25 # seconds between each record's add/remove pair +FAIL_FAST_THRESHOLD=3 # abort the run after this many consecutive API errors +STALE_THRESHOLD=6 # attempts before a pending record is flagged "stale" in the email + +# Note: Email sent via OMV-configured Postfix (Gmail relay). No SMTP config needed here. +# ────────────────────────────────────────────────────────────────────────────── + +RECORDS=( + "albranch.org" + "www.albranch.org" + "indigorhayne.com" + "www.indigorhayne.com" + "rufusking.com" + "www.rufusking.com" + "kingdezigns.com" + "www.kingdezigns.com" + "*.web.kingdezigns.com" + "cloud.kingdezigns.com" + "files08.kingdezigns.com" + "files16.kingdezigns.com" + "has.kingdezigns.com" + "nginx.kingdezigns.com" + "omv08.kingdezigns.com" + "omv16.kingdezigns.com" + "pihole.kingdezigns.com" + "plex.kingdezigns.com" + "vault.kingdezigns.com" + "webmin.kingdezigns.com" + "office.kingdezigns.com" + "stocks.kingdezigns.com" + "collabora.kingdezigns.com" +) + +DH_API="https://api.dreamhost.com/" +REPORT_TIME=$(date "+%Y-%m-%d %H:%M:%S") + +mkdir -p /var/log/ddns + +# Trim log to last 5000 lines +if [[ -f "$LOG_FILE" ]]; then + tail -n 5000 "$LOG_FILE" > "${LOG_FILE}.tmp" && mv "${LOG_FILE}.tmp" "$LOG_FILE" +fi + +log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; } + +log "===== DDNS Update started on $HOSTNAME_LABEL (v2 hardened) =====" + +# ── Helpers ─────────────────────────────────────────────────────────────────── + +unique_id() { + cat /proc/sys/kernel/random/uuid 2>/dev/null \ + || python3 -c "import uuid; print(uuid.uuid4())" 2>/dev/null \ + || echo "$(date +%s%N)-$$" +} + +get_wan_ip() { + local ip + for src in "https://api.ipify.org" "https://ifconfig.me/ip" "https://icanhazip.com"; do + ip=$(curl -s --max-time 8 "$src" 2>/dev/null | tr -d '[:space:]') + if [[ "$ip" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]]; then + echo "$ip"; return 0 + fi + done + return 1 +} + +# Fetches the ENTIRE DreamHost DNS zone in a single API call and prints one +# "recordvalue" line per A record on stdout. On any failure (including a +# throttled/erroring API), prints a reason to stderr and returns non-zero — +# callers must NOT treat that as "record doesn't exist". +fetch_zone_snapshot() { + curl -s --max-time 20 -G "$DH_API" \ + --data-urlencode "key=${API_KEY}" \ + --data-urlencode "cmd=dns-list_records" \ + --data-urlencode "format=json" \ + --data-urlencode "unique_id=$(unique_id)" \ + | python3 -c " +import sys, json +try: + data = json.load(sys.stdin) + if data.get('result') != 'success': + sys.stderr.write('LIST_ERROR: ' + str(data.get('data', 'unknown')) + '\n') + sys.exit(2) + for e in data.get('data', []): + if e.get('type', '') == 'A': + sys.stdout.write(e.get('record', '') + '\t' + e.get('value', '') + '\n') +except Exception as ex: + sys.stderr.write('LIST_EXCEPTION: ' + str(ex) + '\n') + sys.exit(2) +" +} + +# Both print "resultdata" on stdout — always something printable, so a +# JSON/network failure shows up as an explicit "exception" row in the log +# instead of silently looking like an empty/unknown result. +dh_add_record() { + local record="$1" ip="$2" + curl -s --max-time 15 -G "$DH_API" \ + --data-urlencode "key=${API_KEY}" \ + --data-urlencode "cmd=dns-add_record" \ + --data-urlencode "format=json" \ + --data-urlencode "unique_id=$(unique_id)" \ + --data-urlencode "record=${record}" \ + --data-urlencode "type=A" \ + --data-urlencode "value=${ip}" \ + | python3 -c " +import sys, json +try: + d = json.load(sys.stdin) + print(str(d.get('result', 'unknown')) + '\t' + str(d.get('data', ''))) +except Exception as ex: + print('exception\t' + str(ex)) +" 2>/dev/null +} + +dh_remove_record() { + local record="$1" ip="$2" + curl -s --max-time 15 -G "$DH_API" \ + --data-urlencode "key=${API_KEY}" \ + --data-urlencode "cmd=dns-remove_record" \ + --data-urlencode "format=json" \ + --data-urlencode "unique_id=$(unique_id)" \ + --data-urlencode "record=${record}" \ + --data-urlencode "type=A" \ + --data-urlencode "value=${ip}" \ + | python3 -c " +import sys, json +try: + d = json.load(sys.stdin) + print(str(d.get('result', 'unknown')) + '\t' + str(d.get('data', ''))) +except Exception as ex: + print('exception\t' + str(ex)) +" 2>/dev/null +} + +# ── Get WAN IP ──────────────────────────────────────────────────────────────── + +log "Checking current WAN IP..." +if ! WAN_IP=$(get_wan_ip); then + log "ERROR: All WAN IP sources failed. Aborting — will retry next run." + exit 1 +fi +log "Current WAN IP: $WAN_IP" + +PREV_IP="" +[[ -f "$IP_CACHE" ]] && PREV_IP=$(tr -d '[:space:]' < "$IP_CACHE") +log "Cached previous IP: ${PREV_IP:-}" + +# ── Load pending records left over from a previous run ──────────────────────── + +declare -A PENDING_ATTEMPTS +PENDING_COUNT=0 +if [[ -f "$PENDING_FILE" ]]; then + while IFS=$'\t' read -r p_rec p_attempts; do + [[ -z "$p_rec" ]] && continue + PENDING_ATTEMPTS["${p_rec,,}"]="${p_attempts:-0}" + (( PENDING_COUNT++ )) + done < "$PENDING_FILE" +fi +[[ $PENDING_COUNT -gt 0 ]] && log "Found $PENDING_COUNT record(s) pending from a previous run." + +# ── Daily window check (00:00–00:14) ───────────────────────────────────────── + +HOUR=$(date '+%H') +MINUTE=$(date '+%M') +IN_DAILY_WINDOW=false +if [[ "$HOUR" == "00" && 10#$MINUTE -lt 15 ]]; then + IN_DAILY_WINDOW=true + log "Daily sanity-check window active (00:00–00:14)." +fi + +# ── Decide whether to run ───────────────────────────────────────────────────── + +DO_UPDATE=false +TRIGGER_REASON="" + +if [[ "$WAN_IP" != "$PREV_IP" ]]; then + DO_UPDATE=true + TRIGGER_REASON="WAN IP changed (${PREV_IP:-none} → ${WAN_IP})" +elif [[ "$IN_DAILY_WINDOW" == "true" ]]; then + DO_UPDATE=true + TRIGGER_REASON="Daily sanity-check (IP unchanged at ${WAN_IP})" +elif [[ $PENDING_COUNT -gt 0 ]]; then + DO_UPDATE=true + TRIGGER_REASON="Retrying $PENDING_COUNT pending record(s) from a previous run" +fi + +if [[ "$DO_UPDATE" == "false" ]]; then + log "IP unchanged ($WAN_IP) and no pending records. No action needed — exiting silently." + echo "$WAN_IP" > "$IP_CACHE" + log "===== DDNS Update complete (no change) on $HOSTNAME_LABEL =====" + exit 0 +fi + +log "Trigger: $TRIGGER_REASON" + +# ── Fetch the full zone ONCE ────────────────────────────────────────────────── + +log "Fetching current DNS zone from DreamHost (single API call)..." +ZONE_ERR_FILE=$(mktemp) +ZONE_RAW=$(fetch_zone_snapshot 2> "$ZONE_ERR_FILE") +ZONE_RC=$? +if [[ $ZONE_RC -ne 0 ]]; then + ZONE_ERR=$(cat "$ZONE_ERR_FILE") + rm -f "$ZONE_ERR_FILE" + log "ERROR: could not fetch DNS zone snapshot ($ZONE_ERR) — likely DreamHost API throttling/outage." + log "Not touching pending state or the IP cache — will retry in full on the next scheduled run." + log "===== DDNS Update aborted on $HOSTNAME_LABEL =====" + # Lightweight failure alert — deliberately not the full report template. + { + echo "To: $REPORT_TO" + echo "From: $REPORT_FROM" + echo "Subject: [DDNS] $HOSTNAME_LABEL — API UNREACHABLE | $REPORT_TIME" + echo "Content-Type: text/plain; charset=UTF-8" + echo "" + echo "DDNS update on $HOSTNAME_LABEL could not even fetch the DNS zone from DreamHost." + echo "Reason: $ZONE_ERR" + echo "" + echo "This usually means the DreamHost API is rate-limiting this key right now." + echo "No changes were attempted. The script will retry automatically on the next" + echo "15-minute run — no action needed unless this repeats for more than an hour." + } | sendmail -t + exit 1 +fi +rm -f "$ZONE_ERR_FILE" + +declare -A ZONE +ZONE_RECORD_COUNT=0 +while IFS=$'\t' read -r z_rec z_val; do + [[ -z "$z_rec" ]] && continue + ZONE["${z_rec,,}"]="$z_val" + (( ZONE_RECORD_COUNT++ )) +done <<< "$ZONE_RAW" +log "Zone snapshot retrieved — $ZONE_RECORD_COUNT A record(s) found on DreamHost." + +# ── Determine which records need updating (all local — no API calls) ───────── + +RECORDS_OK=() +NEEDS_UPDATE=() +for rec in "${RECORDS[@]}"; do + cur_ip="${ZONE[${rec,,}]:-}" + if [[ "$cur_ip" == "$WAN_IP" ]]; then + RECORDS_OK+=("$rec") + else + log " [$rec] needs update — DreamHost shows '${cur_ip:-}'" + NEEDS_UPDATE+=("$rec") + fi +done + +log "Checked all ${#RECORDS[@]} records against target IP $WAN_IP — ${#RECORDS_OK[@]} already correct, ${#NEEDS_UPDATE[@]} need updating." + +RECORDS_FAILED=() +RECORDS_DEFERRED=() +ATTEMPTED_COUNT=0 +RATE_LIMIT_ABORTED=false + +if [[ ${#NEEDS_UPDATE[@]} -eq 0 ]]; then + log "Nothing to update." + rm -f "$PENDING_FILE" +else + # Split into this run's batch vs. overflow (batch-limit deferral) + THIS_RUN_BATCH=("${NEEDS_UPDATE[@]:0:$BATCH_LIMIT}") + OVERFLOW=("${NEEDS_UPDATE[@]:$BATCH_LIMIT}") + + if [[ ${#OVERFLOW[@]} -gt 0 ]]; then + log "Batch limit is $BATCH_LIMIT records/run — deferring ${#OVERFLOW[@]} record(s) to the next run: ${OVERFLOW[*]}" + RECORDS_DEFERRED+=("${OVERFLOW[@]}") + fi + + CONSECUTIVE_ERRORS=0 + FIRST=true + for rec in "${THIS_RUN_BATCH[@]}"; do + if [[ "$RATE_LIMIT_ABORTED" == "true" ]]; then + RECORDS_DEFERRED+=("$rec") + continue + fi + + if [[ "$FIRST" == "false" ]]; then + sleep "$SLEEP_BETWEEN_RECORDS" + fi + FIRST=false + + old_ip="${ZONE[${rec,,}]:-}" + log " Updating [$rec] ${old_ip:-} → $WAN_IP" + + IFS=$'\t' read -r add_result add_data <<< "$(dh_add_record "$rec" "$WAN_IP")" + log " add: $add_result${add_data:+ ($add_data)}" + (( ATTEMPTED_COUNT++ )) + + if [[ "$add_result" == "success" ]]; then + RECORDS_OK+=("$rec") + CONSECUTIVE_ERRORS=0 + + # Clean up the stale old value now that the new one is live. + if [[ -n "$old_ip" && "$old_ip" != "$WAN_IP" ]]; then + IFS=$'\t' read -r rm_result rm_data <<< "$(dh_remove_record "$rec" "$old_ip")" + log " remove: $rm_result${rm_data:+ ($rm_data)} (cleanup of old value $old_ip)" + if [[ "$rm_result" != "success" ]]; then + log " NOTE: old A record ($old_ip) for [$rec] could not be removed — DNS may briefly round-robin between old and new IP until a future cleanup pass." + fi + fi + else + log " WARNING: update failed for [$rec] — $add_data" + RECORDS_FAILED+=("${rec}::${add_data}") + (( CONSECUTIVE_ERRORS++ )) + if [[ $CONSECUTIVE_ERRORS -ge $FAIL_FAST_THRESHOLD ]]; then + log "WARNING: $FAIL_FAST_THRESHOLD consecutive API errors — this looks like DreamHost rate-limiting. Aborting remaining updates this run rather than making it worse." + RATE_LIMIT_ABORTED=true + fi + fi + done +fi + +# ── Persist pending state for next run ──────────────────────────────────────── + +declare -A NEW_PENDING +for rec in "${RECORDS_DEFERRED[@]}"; do + key="${rec,,}" + NEW_PENDING["$key"]="${PENDING_ATTEMPTS[$key]:-0}" # not actually attempted — don't increment +done +for entry in "${RECORDS_FAILED[@]}"; do + rec="${entry%%::*}" + key="${rec,,}" + prev="${PENDING_ATTEMPTS[$key]:-0}" + NEW_PENDING["$key"]=$(( prev + 1 )) +done + +STALE_RECORDS=() +if [[ ${#NEW_PENDING[@]} -gt 0 ]]; then + : > "$PENDING_FILE" + for key in "${!NEW_PENDING[@]}"; do + echo -e "${key}\t${NEW_PENDING[$key]}" >> "$PENDING_FILE" + if [[ ${NEW_PENDING[$key]} -ge $STALE_THRESHOLD ]]; then + STALE_RECORDS+=("$key (${NEW_PENDING[$key]} failed attempts)") + fi + done +else + rm -f "$PENDING_FILE" +fi + +# ── Persist WAN IP cache ─────────────────────────────────────────────────────── +echo "$WAN_IP" > "$IP_CACHE" +log "IP cache updated to $WAN_IP" + +log "Update complete — OK: ${#RECORDS_OK[@]} Failed: ${#RECORDS_FAILED[@]} Deferred: ${#RECORDS_DEFERRED[@]} API calls used: $(( ZONE_RECORD_COUNT >= 0 ? 1 : 1 ))+$(( ATTEMPTED_COUNT * 2 ))" + +# ── Skip email entirely if there was truly nothing to report ───────────────── +if [[ ${#NEEDS_UPDATE[@]} -eq 0 ]]; then + log "===== DDNS Update complete (all records already correct) on $HOSTNAME_LABEL =====" + exit 0 +fi + +# ── Determine overall status ────────────────────────────────────────────────── +if [[ ${#RECORDS_FAILED[@]} -eq 0 && ${#RECORDS_DEFERRED[@]} -eq 0 ]]; then + OVERALL_STATUS="SUCCESS" +elif [[ ${#RECORDS_FAILED[@]} -eq 0 && ${#RECORDS_DEFERRED[@]} -gt 0 ]]; then + OVERALL_STATUS="DEFERRED" +elif [[ ${#RECORDS_FAILED[@]} -gt 0 && ( $ATTEMPTED_COUNT -gt ${#RECORDS_FAILED[@]} || ${#RECORDS_DEFERRED[@]} -gt 0 ) ]]; then + OVERALL_STATUS="PARTIAL" +else + OVERALL_STATUS="FAILED" +fi + +# ── Build HTML email — matching KingDezigns email style guide ──────────────── + +case "$OVERALL_STATUS" in + SUCCESS) + BADGE_HTML="

✅ ALL UPDATED

" + BANNER_HTML="✅ All A records updated successfully — no action required." + ;; + DEFERRED) + BADGE_HTML="

🕓 BATCHED

" + BANNER_HTML="🕓 Some records queued for the next run (rate-limit batching) — no errors, no action required." + ;; + PARTIAL) + BADGE_HTML="

⚠️ PARTIAL

" + BANNER_HTML="⚠️ Some records failed to update — they will be retried automatically." + ;; + FAILED) + BADGE_HTML="

🚨 FAILED

" + BANNER_HTML="🚨 Updates failed — likely DreamHost rate-limiting. Will retry automatically; check if this persists past an hour." + ;; +esac + +# Record rows: OK +RECORD_ROWS_HTML="" +for rec in "${RECORDS_OK[@]}"; do + RECORD_ROWS_HTML+=" + + $rec + ✓ UPDATED + $WAN_IP + " +done + +# Record rows: FAILED (with real DreamHost reason) +for entry in "${RECORDS_FAILED[@]}"; do + rec="${entry%%::*}" + reason="${entry#*::}" + RECORD_ROWS_HTML+=" + + $rec + ✗ FAILED + $reason + " +done + +# Record rows: DEFERRED (queued for next run — not a failure) +for rec in "${RECORDS_DEFERRED[@]}"; do + RECORD_ROWS_HTML+=" + + $rec + 🕓 QUEUED + retrying next run + " +done + +# Manual action block — only shown when there are real (non-deferred) failures +MANUAL_BLOCK="" +if [[ ${#RECORDS_FAILED[@]} -gt 0 ]]; then + FAILED_LIST_HTML="" + for entry in "${RECORDS_FAILED[@]}"; do + rec="${entry%%::*}" + reason="${entry#*::}" + FAILED_LIST_HTML+="

• $rec ($reason)

" + done + MANUAL_BLOCK=" + +

⚠️ Failed Records — Will Auto-Retry

These records did not update this run. They stay queued and will be retried automatically on the next 15-minute run — no manual action needed unless they keep failing for more than an hour.

+ $FAILED_LIST_HTML +

+ + + + + + + + + + +

Target A record value	$WAN_IP

DreamHost DNS Panel (manual fallback)	panel.dreamhost.com → Manage Websites → DNS

+ " +fi + +# Stale block — records that have failed repeatedly across multiple runs +STALE_BLOCK="" +if [[ ${#STALE_RECORDS[@]} -gt 0 ]]; then + STALE_LIST_HTML="" + for s in "${STALE_RECORDS[@]}"; do + STALE_LIST_HTML+="

• $s

" + done + STALE_BLOCK=" + +

🔴 Stale — Failing Across Multiple Runs

These records have failed $STALE_THRESHOLD+ times across separate runs — this is no longer likely to be a transient rate limit. Worth checking manually.

+ $STALE_LIST_HTML +

+ " +fi + +# ── Compose full HTML email ─────────────────────────────────────────────────── +HTML_BODY=$(cat < + + + + + +

+ + + + + + + $BANNER_HTML + + + + + + $MANUAL_BLOCK + + + $STALE_BLOCK + + + + + + + +

+ + + + + +

DreamHost Dynamic DNS Monitor

🌐 $HOSTNAME_LABEL

$REPORT_TIME | ${#RECORDS[@]} records monitored | Trigger: $TRIGGER_REASON

$BADGE_HTML

+ 📋 A Record Update Results + ${#RECORDS_OK[@]} correct · ${#RECORDS_FAILED[@]} failed · ${#RECORDS_DEFERRED[@]} queued +

+ + + + + + + $RECORD_ROWS_HTML +

Record	Status	Detail

📊 Run Summary

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Host	$HOSTNAME_LABEL

Run Time	$REPORT_TIME

Trigger	$TRIGGER_REASON

Target WAN IP	$WAN_IP

Previous IP	${PREV_IP:-}

Records Correct	${#RECORDS_OK[@]} / ${#RECORDS[@]}

API Calls This Run	1 zone fetch + $(( ATTEMPTED_COUNT * 2 )) mutation call(s)

Rate-Limit Abort Triggered	$([ "$RATE_LIMIT_ABORTED" == "true" ] && echo "Yes — stopped early" \|\| echo "No")

Overall Status	$OVERALL_STATUS

+ Automated DDNS Monitor | $HOSTNAME_LABEL | KingDezigns Infrastructure +

+ + +EOF +) + +# ── Send email via OMV Postfix (Gmail relay) ────────────────────────────────── +SUBJECT="[DDNS] $HOSTNAME_LABEL — $OVERALL_STATUS | $REPORT_TIME" + +{ + echo "To: $REPORT_TO" + echo "From: $REPORT_FROM" + echo "Subject: $SUBJECT" + echo "MIME-Version: 1.0" + echo "Content-Type: text/html; charset=UTF-8" + echo "" + echo "$HTML_BODY" +} | sendmail -t + +log "Email sent to $REPORT_TO — Overall status: $OVERALL_STATUS" +log "===== DDNS Update complete on $HOSTNAME_LABEL =====" diff --git a/nas/NAS08-to-NAS16-Sync-Summary.html b/nas/NAS08-to-NAS16-Sync-Summary.html new file mode 100644 index 0000000..1ae0349 --- /dev/null +++ b/nas/NAS08-to-NAS16-Sync-Summary.html @@ -0,0 +1,530 @@ + + + + + +KingDezigns — NAS08 → NAS16 Sync Setup Summary + + + + + +

+ +

KingDezigns — NAS08 → NAS16 Sync Setup Summary v2.0

+ 📅 Updated May 18, 2026 + 🖥 NAS08 (192.168.150.35) → NAS16 (192.168.150.40) + ⚙️ OpenMediaVault / Raspberry Pi OS + 🔧 rsync + NFSv4 + HTML Email Report +

+ + + +

The goal

Mirror the Public folder from NAS08 to NAS16, copying only files that have changed. The sync must be lightweight enough for Raspberry Pi hardware, non-destructive (no deletes on NAS16), and scheduled via the OMV Task Scheduler. After every run, an HTML email report is sent summarizing what transferred, what folders were affected, and any errors.

+ Source: NAS08 — 192.168.150.35 + Destination: NAS16 — 192.168.150.40 + ~225,000 files / ~983 GB + NFSv4 file share mount + HTML email report +

NAS16 pulls from NAS08 (rather than NAS08 pushing) so the heavier rsync process runs on the machine with more RAM (16GB vs 8GB), leaving NAS08 as a passive file server during the sync.

+ + + +

Folder paths

+ +

S Source — NAS08

/mnt/kingdezigns08/Public/

NFS share mounted on NAS16 via fstab at boot. Mount verified as non-empty before rsync runs.

+ +

D Destination — NAS16

/export/kingdezignsnas-16/Public/

Local path on NAS16. Script verifies directory exists before writing.

+ +

⚠ NAS08-only items (copied on first run)

16 - Documents/
+Savings Card Forms.pdf
+kingdezigns08.nas

All three copied successfully on first sync. kingdezigns08.nas included intentionally.

+ +

✕ Permanently excluded

/mnt/kingdezigns08/Public/.Trash-1000/

System trash folder generated by NAS08's desktop environment. Never needed in backup. Excluded via --exclude='.Trash-1000/' in rsync flags.

No longer appears in logs or emails.

+ + + +

Key decisions

+ +

Pull vs Push

NAS16 pulls from NAS08 via the mounted NFS share. rsync runs entirely on NAS16 to keep NAS08 load minimal — NAS08 is passive during the sync window.

+ +

NFSv4 File Share vs SSH

NFS share used instead of SSH. Drive already mounted via fstab on NAS16 — no SSH keys or remote execution needed. NFS maps users by UID. Both machines have rufusking as UID 1000 so permissions resolve correctly over the mount.

+ +

No --delete flag

Files on NAS16 that do not exist on NAS08 are left untouched. NAS16 is a backup, not a strict mirror. This protects NAS16-only files from accidental deletion.

+ +

--update flag (not --checksum)

rsync skips files where the NAS16 copy is newer than NAS08. This is lightweight (uses timestamps, not file hashing) and protects any edits made directly on NAS16.

+ +

Schedule: Every other day at 2:30 AM via OMV Task Scheduler. NAS08 runs its own nas16-backup.sh at 2:00 AM (finishes ~2:05 AM). The 25-minute buffer prevents overlap on heavy nights.

+ + + +

Setup steps performed

+ +

1 Created script directory on NAS16

sudo mkdir /usr/scripts/nas
+sudo chown rufusking:users /usr/scripts/nas

/usr/scripts already existed. /nas subfolder created manually on NAS16. chown required after sudo mkdir — ownership defaults to root otherwise.

Directory created and ownership set successfully.

+ +

2 Copied script from Linux PC to NAS16

scp /path/to/nas08_to_nas16_sync.sh rufusking@192.168.150.40:/usr/scripts/nas/

File transferred successfully.

+ +

3 Set executable permission on NAS16

sudo chmod +x /usr/scripts/nas/nas08_to_nas16_sync.sh

Run directly on NAS16 (not via remote SSH). Makes the script executable by the OMV scheduler.

Permission set. Script ready to run.

+ +

4 NFS export updated on NAS08 — subtree_check → no_subtree_check

# Changed in OMV UI → Services → NFS → Shares (all four shares)
+# Then reloaded on NAS08:
+sudo exportfs -ra
+
+# Then remounted on NAS16 to pick up the new export options:
+sudo umount /mnt/kingdezigns08
+sudo mount /mnt/kingdezigns08

subtree_check was causing intermittent Permission Denied errors on 02 - Photos, 08 - Taxes, and 16 - Documents during scheduled overnight runs. no_subtree_check is the modern recommended default for private NFS setups.

All three folders now accessible. Permission errors resolved.

+ +

5 Fixed log directory ownership on NAS16

sudo chown -R rufusking:users /var/log/nas-sync/

Log directory was created by root during the first automated OMV run. Script now self-corrects ownership at startup so this never needs to be done manually again.

Log writing works cleanly. Self-healing ownership check added to script.

+ +

6 OMV Scheduler entry added on NAS16

Command : /usr/scripts/nas/nas08_to_nas16_sync.sh
+Schedule: 30 2 */2 * *   (2:30 AM every other day)

Scheduled and confirmed running.

+ +

+ + + +

Script — final state

Location on NAS16

/usr/scripts/nas/nas08_to_nas16_sync.sh

Log output

/var/log/nas-sync/nas08_to_nas16_YYYY-MM-DD.log

Log rotation

Logs older than 30 days deleted automatically

rsync flags

--archive          Preserve permissions, timestamps, symlinks, owner/group
+--update           Skip files where NAS16 copy is newer (protects NAS16 edits)
+--human-readable   Sizes in KB/MB/GB
+--itemize-changes  One structured line per transferred file — used for folder activity parsing
+--stats            Summary totals at end of each run
+--exclude          Excludes .Trash-1000/ from all operations

Safety features

NFS mount check     Waits up to 30s for NFS share to be fully responsive before starting
+Source empty check  Aborts if NAS08 share appears empty — prevents accidental wipe of destination
+Dest check          Confirms NAS16 folder exists before writing
+Smart lock file     Stores PID in lock file; auto-clears stale locks if process is no longer running
+Log ownership fix   Self-corrects /var/log/nas-sync/ ownership on every startup
+Exit code           Returns rsync exit code so OMV scheduler can detect failures

Email report sections

Header/banner       SUCCESS / WARNING / FAILED status with color coding
+Stats tiles         Files scanned, transferred, new files, data moved, duration
+Source/dest info    IPs, paths, transfer rate, total source size
+Folder Activity     Collapsed by default — folders with changes show file count,
+                    folders with errors show the error message, no-activity runs
+                    show "all files up to date"
+Full rsync log      Collapsed by default — complete raw output available on demand

Email recipient

rufus.king@kingdezigns.com

+ + + +

Troubleshooting log

+ +

T1 Permission Denied on 02 - Photos, 08 - Taxes, 16 - Documents

Error

rsync: [sender] opendir "/mnt/kingdezigns08/Public/02 - Photos" failed: Permission denied (13)

Investigation: ACLs confirmed correct on NAS08 (rufusking:rwx present on all affected folders and subfolders). UIDs matched on both machines (UID 1000). Folders readable manually via NAS16 terminal. Error only appeared during scheduled overnight runs.

Root cause: subtree_check in the NFS export config on NAS08 was causing intermittent access failures for folders with ACL-based permissions and mixed ownership (some owned by www-data via Nextcloud). subtree_check forces NFS to validate every file access against the export tree — this check fails intermittently on first access after an idle period, especially overnight.

Fix: Changed all four NFS shares on NAS08 from subtree_check to no_subtree_check via OMV UI. Reloaded exports with sudo exportfs -ra. Remounted share on NAS16. All three folders accessible on next run.

Resolved — no Permission Denied errors since fix applied.

+ +

T2 tee: Permission Denied on log file

Error

tee: /var/log/nas-sync/nas08_to_nas16_2026-05-17.log: Permission denied

Root cause: /var/log/nas-sync/ was created by root during the first automated OMV run. Running the script manually as rufusking couldn't write to a root-owned directory.

Fix: sudo chown -R rufusking:users /var/log/nas-sync/. Script updated to self-correct ownership on every startup — checks if the log directory owner matches the running user and auto-chowns if not.

Resolved — script is now self-healing for this condition.

+ +

T3 Stale lock file blocking manual runs

Error

ERROR: Lock file exists at /tmp/nas08_to_nas16.lock. Another sync may still be running. Exiting.

Root cause: Script was killed mid-run during testing, leaving the lock file behind. Original lock logic just checked for file existence with no way to distinguish a live run from a stale one.

Fix: Lock file now stores the script's PID. On startup, if a lock file exists the script checks whether that PID is still running. If not, it removes the stale lock and continues with a warning in the log.

Resolved — stale locks are now self-clearing.

+ +

T4 Folder Activity section showing 0 updated despite files transferring

Root cause: Original folder parser relied on xfr# markers from rsync's --progress flag. When --progress was removed (to clean up captured output), those markers disappeared and the parser never incremented folder counts. Switching to --verbose didn't help because verbose output only shows directory names, not individual files.

Fix: Replaced --verbose with --itemize-changes, which outputs a structured prefix line for every transferred file regardless of size (e.g. >f+++++++++ 02 - Photos/2020/12/photo.jpg). Parser updated to detect lines starting with >f and extract the top-level folder name.

Resolved — folder activity now correctly shows file counts per folder.

+ +

+ + + +

NAS08 NFS export reference

All four shares updated to no_subtree_check on May 18, 2026. The /export root entries are OMV auto-generated read-only entries — leave them as-is.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Share	Networks	Key Options
`/export/kingdezigns-all`	.100 .110 .120 .150	`rw, no_subtree_check, insecure, crossmnt`
`/export/kingdezigns08-secure`	.100 .110 .120 .150	`rw, no_subtree_check, insecure, crossmnt`
`/export/kingdezigns-public`	.100 .110 .120 .150	`rw, no_subtree_check, insecure, crossmnt`
`/export/HomeAssistantBackups`	.150 only	`rw, no_subtree_check, insecure, crossmnt`
`/export` (root)	.100 .110 .120 .150	`ro, root_squash, subtree_check` — OMV auto-generated, do not edit

Note: duplicate 192.168.100.0/24 entry for /export root exists in OMV config — pre-existing, harmless, clean up in OMV UI when convenient.

+ + + +

NAS08 nightly task timeline

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Time	Device	Script	Notes
12:00 AM	NAS08	`/usr/scripts/zfs/nas08_zfs_report.sh`	Daily ZFS health report email.
2:00 AM	NAS08 & NAS16	`/usr/scripts/omv/nas16-backup.sh`	Pre-existing backup script. NAS16 finishes ~1 min, NAS08 ~5 min. Runs every 3 days.
2:30 AM	NAS16	`/usr/scripts/nas/nas08_to_nas16_sync.sh`	NAS08 → NAS16 Public folder sync. 25-min buffer after NAS08 backup finishes.
3:00 AM (Sun)	NAS08	`/usr/scripts/zfs/nas08_zfs_scrub.sh`	Weekly ZFS scrub. Sundays only.
TBD	TBD	Future maintenance tasks	Drive health checks, log cleanups, etc. — to be added later.

+ + + +

Lessons learned

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Lesson	Detail
subtree_check causes intermittent NFS permission failures	On folders with ACL-based or mixed ownership, subtree_check can fail the export tree validation intermittently — especially after idle periods overnight. no_subtree_check is the safe modern default for private home networks.
NFS client cache must be cleared after export changes	Running exportfs -ra on the server isn't enough. The NFS client (NAS16) must unmount and remount to pick up the new export options. Without this, the old subtree_check behavior persists.
ACL permissions and NFS UIDs are separate concerns	Files can have correct ACLs (rufusking:rwx) but still be blocked over NFS if the UID mapping fails. Always verify UIDs match on both machines with id username before assuming ACL issues.
sudo operations reset directory ownership	Any sudo mkdir or sudo mount can leave directories owned by root. Scripts that write to those directories will fail with Permission Denied when run as a regular user. Always chown after sudo directory operations.
Lock files should store PIDs, not just exist	A bare touch lock file can't distinguish a live run from a stale one left by a killed process. Storing the PID and checking if it's still running makes lock files self-healing.
--itemize-changes is the right flag for parsing file transfers	--progress generates xfr# markers but is noisy. --verbose only lists directories. --itemize-changes gives one clean structured line per transferred file (>f prefix) that's easy to parse reliably regardless of file size.
Pull is safer than push for backups	Running rsync on the destination keeps the source device (NAS08) passive and low-load during the sync window.
Schedule buffer matters	Running the sync immediately after another backup script risks overlap on heavy nights. A 25-30 min buffer between jobs keeps things clean.
File count is larger than expected when blocked folders open up	Initial estimate was ~16,000 files. Actual count after Photos/Taxes/Documents became accessible was ~225,000 files across ~983 GB. rsync handled this without issue.

+ + + +

Network map — items to update

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Item	Detail
New script on NAS16	`/usr/scripts/nas/nas08_to_nas16_sync.sh` — document in NAS16 script inventory
New log location on NAS16	`/var/log/nas-sync/` — add to log monitoring notes. 30-day auto-rotation.
OMV Scheduler — NAS16	Task added: `30 2 /2 *` running the sync script
NFS exports changed on NAS08	All four shares updated to `no_subtree_check` in OMV. Document in NAS08 config notes.
NAS08 → NAS16 data flow	NAS08 Public folder now syncing to NAS16 Public every other day at 2:30 AM. 225,000 files / 983 GB in scope.
OMV duplicate export entry (cleanup pending)	192.168.100.0/24 appears twice for the /export root in NAS08's exports. Harmless but should be cleaned up in OMV UI when convenient.

+ + + + +

+ + diff --git a/nas/nas08_to_nas16_sync.sh b/nas/nas08_to_nas16_sync.sh new file mode 100644 index 0000000..825becc --- /dev/null +++ b/nas/nas08_to_nas16_sync.sh @@ -0,0 +1,476 @@ +#!/bin/bash +# ============================================================================= +# NAS08 → NAS16 Sync Script +# Source : NAS08 (192.168.150.35) /mnt/kingdezigns08/Public +# Dest : NAS16 (192.168.150.40) /export/kingdezignsnas-16/Public +# +# Behavior: +# - Copies only changed/new files from NAS08 to NAS16 +# - Does NOT delete files on NAS16 that are missing from NAS08 +# - Preserves file timestamps and symlinks +# - Skips permission syncing (NFS all_squash incompatible with chmod) +# - Skips directory timestamp syncing (NFS all_squash incompatible with utimes) +# - Auto-corrects destination ownership/permissions before sync if needed +# - Sends an HTML status email after every run +# - Logs every run with timestamps +# +# Schedule via OMV Task Scheduler: 30 2 */2 * * +# ============================================================================= + +# --- Configuration ----------------------------------------------------------- +SOURCE="/mnt/kingdezigns08/Public/" +DEST="/export/kingdezignsnas-16/Public/" +LOG_DIR="/var/log/nas-sync" +LOG_FILE="${LOG_DIR}/nas08_to_nas16_$(date +%Y-%m-%d).log" +LOCK_FILE="/tmp/nas08_to_nas16.lock" +REPORT_TO="rufus.king@kingdezigns.com" +REPORT_FROM="nas-sync@kingdezigns16.local" +NOW=$(date '+%Y-%m-%d %H:%M:%S') +NFS_MOUNT="/mnt/kingdezigns08" +DEST_EXPECTED_OWNER="www-data" +DEST_EXPECTED_GROUP="ncshare" + +# --- HTML escape helper ------------------------------------------------------ +html_escape() { sed 's/&/\&/g; s//\>/g; s/"/\"/g'; } + +# --- Safety Checks ----------------------------------------------------------- + +# Ensure log directory exists and is owned by the running user. +mkdir -p "$LOG_DIR" +if [ "$(stat -c '%U' "$LOG_DIR")" != "$(whoami)" ]; then + sudo chown -R "$(whoami)":users "$LOG_DIR" 2>/dev/null || true +fi + +# Prevent overlapping runs — but self-heal stale lock files. +if [ -f "$LOCK_FILE" ]; then + LOCK_PID=$(cat "$LOCK_FILE" 2>/dev/null) + if [ -n "$LOCK_PID" ] && kill -0 "$LOCK_PID" 2>/dev/null; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: Sync already running as PID $LOCK_PID. Exiting." | tee -a "$LOG_FILE" + exit 1 + else + echo "[$(date '+%Y-%m-%d %H:%M:%S')] WARNING: Stale lock file found (PID $LOCK_PID no longer running). Removing and continuing." | tee -a "$LOG_FILE" + rm -f "$LOCK_FILE" + fi +fi +echo $$ > "$LOCK_FILE" + +# Wait for NFS mount to be fully ready (up to 30 seconds) +echo "[$(date '+%Y-%m-%d %H:%M:%S')] Checking NFS mount readiness..." | tee -a "$LOG_FILE" +WAIT=0 +until mountpoint -q "$NFS_MOUNT"; do + sleep 5 + WAIT=$((WAIT + 5)) + if [ $WAIT -ge 30 ]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: NFS mount $NFS_MOUNT not ready after 30 seconds. Aborting." | tee -a "$LOG_FILE" + rm -f "$LOCK_FILE" + exit 1 + fi + echo "[$(date '+%Y-%m-%d %H:%M:%S')] Waiting for NFS mount... (${WAIT}s)" | tee -a "$LOG_FILE" +done +echo "[$(date '+%Y-%m-%d %H:%M:%S')] NFS mount ready." | tee -a "$LOG_FILE" + +# Verify source is mounted and not empty +if [ ! -d "$SOURCE" ] || [ -z "$(ls -A "$SOURCE" 2>/dev/null)" ]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: Source '$SOURCE' is not mounted or is empty. Aborting to prevent data loss." | tee -a "$LOG_FILE" + rm -f "$LOCK_FILE" + exit 1 +fi + +# Verify destination exists +if [ ! -d "$DEST" ]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: Destination '$DEST' does not exist. Aborting." | tee -a "$LOG_FILE" + rm -f "$LOCK_FILE" + exit 1 +fi + +# --- Destination Permission Precheck ----------------------------------------- +# NFS all_squash maps all writes to www-data:ncshare (anonuid=33, anongid=1001). +# If the destination tree is not owned by www-data:ncshare, rsync will fail to +# set permissions and timestamps. Self-correct before the sync runs so this +# never requires manual intervention. + +PERM_CORRECTED=false +echo "[$(date '+%Y-%m-%d %H:%M:%S')] Verifying destination ownership and permissions..." | tee -a "$LOG_FILE" + +DEST_OWNER=$(stat -c '%U' "$DEST" 2>/dev/null) +DEST_GROUP=$(stat -c '%G' "$DEST" 2>/dev/null) + +if [ "$DEST_OWNER" != "$DEST_EXPECTED_OWNER" ] || [ "$DEST_GROUP" != "$DEST_EXPECTED_GROUP" ]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] WARNING: Destination ownership is ${DEST_OWNER}:${DEST_GROUP} — expected ${DEST_EXPECTED_OWNER}:${DEST_EXPECTED_GROUP}. Correcting now..." | tee -a "$LOG_FILE" + sudo chown -R "${DEST_EXPECTED_OWNER}:${DEST_EXPECTED_GROUP}" "$DEST" 2>&1 | tee -a "$LOG_FILE" + sudo chmod -R 2770 "$DEST" 2>&1 | tee -a "$LOG_FILE" + sudo setfacl -R -m "g:${DEST_EXPECTED_GROUP}:rwx" "$DEST" 2>&1 | tee -a "$LOG_FILE" + sudo setfacl -R -d -m "g:${DEST_EXPECTED_GROUP}:rwx" "$DEST" 2>&1 | tee -a "$LOG_FILE" + PERM_CORRECTED=true + echo "[$(date '+%Y-%m-%d %H:%M:%S')] Destination permissions corrected successfully." | tee -a "$LOG_FILE" +else + echo "[$(date '+%Y-%m-%d %H:%M:%S')] Destination ownership OK (${DEST_OWNER}:${DEST_GROUP})." | tee -a "$LOG_FILE" +fi + +# --- Run Sync ---------------------------------------------------------------- + +echo "[$(date '+%Y-%m-%d %H:%M:%S')] ===== NAS08 → NAS16 Sync Started =====" | tee -a "$LOG_FILE" +echo "[$(date '+%Y-%m-%d %H:%M:%S')] Source : $SOURCE" | tee -a "$LOG_FILE" +echo "[$(date '+%Y-%m-%d %H:%M:%S')] Dest : $DEST" | tee -a "$LOG_FILE" + +START_TIME=$(date +%s) + +# --archive : recursive, preserve symlinks, owner, group, devices +# --no-perms : skip chmod — NFS all_squash prevents non-owner chmod +# --omit-dir-times : skip directory utimes — NFS all_squash prevents non-owner utimes on dirs +# File timestamps are still preserved exactly. +# --update : skip files that are newer on destination +RSYNC_OUTPUT=$(rsync \ + --archive \ + --no-perms \ + --omit-dir-times \ + --update \ + --human-readable \ + --itemize-changes \ + --stats \ + --exclude='.Trash-1000/' \ + "$SOURCE" \ + "$DEST" 2>&1) + +RSYNC_EXIT=$? +END_TIME=$(date +%s) +ELAPSED=$(( END_TIME - START_TIME )) +ELAPSED_FMT=$(printf '%dm %ds' $((ELAPSED/60)) $((ELAPSED%60))) + +# Log full rsync output +echo "$RSYNC_OUTPUT" >> "$LOG_FILE" + +# --- Parse rsync stats ------------------------------------------------------- + +FILES_TOTAL=$(echo "$RSYNC_OUTPUT" | grep "Number of files:" | grep -oP '[\d,]+' | head -1 | tr -d ',') +FILES_TRANSFERRED=$(echo "$RSYNC_OUTPUT" | grep "Number of regular files transferred:" | grep -oP '[\d,]+' | head -1 | tr -d ',') +FILES_CREATED=$(echo "$RSYNC_OUTPUT" | grep "Number of created files:" | grep -oP '[\d,]+' | head -1 | tr -d ',') +TOTAL_SIZE=$(echo "$RSYNC_OUTPUT" | grep "Total file size:" | grep -oP '[0-9.,]+\s*[KMGT]?' | head -1) +TRANSFERRED_SIZE=$(echo "$RSYNC_OUTPUT" | grep "Total transferred file size:" | grep -oP '[0-9.,]+\s*[KMGT]?' | head -1) +TRANSFER_RATE=$(echo "$RSYNC_OUTPUT" | grep "sent.*bytes.*received" | grep -oP '[\d.,]+\s*[KMGT]?bytes/sec' | head -1) + +FILES_TOTAL=${FILES_TOTAL:-0} +FILES_TRANSFERRED=${FILES_TRANSFERRED:-0} +FILES_CREATED=${FILES_CREATED:-0} +TOTAL_SIZE=${TOTAL_SIZE:-"unknown"} +TRANSFERRED_SIZE=${TRANSFERRED_SIZE:-"0"} +TRANSFER_RATE=${TRANSFER_RATE:-"unknown"} + +# --- Parse per-folder activity ----------------------------------------------- + +declare -A FOLDER_COUNTS +declare -A FOLDER_ERRORS + +while IFS= read -r LINE; do + if echo "$LINE" | grep -qE '^>f'; then + FILEPATH=$(echo "$LINE" | sed 's/^[^ ]* //') + TOP=$(echo "$FILEPATH" | cut -d'/' -f1) + if [ -n "$TOP" ]; then + FOLDER_COUNTS["$TOP"]=$(( ${FOLDER_COUNTS["$TOP"]:-0} + 1 )) + fi + fi + + if echo "$LINE" | grep -qE "failed:|Permission denied|No such file|error:"; then + ERR_FOLDER=$(echo "$LINE" | grep -oP '"[^"]*"' | head -1 | tr -d '"' | xargs basename 2>/dev/null || echo "unknown") + FOLDER_ERRORS["$ERR_FOLDER"]=$(echo "$LINE" | html_escape) + fi +done <<< "$RSYNC_OUTPUT" + +# Build folder activity rows +FOLDER_ROWS="" +FOLDERS_WITH_CHANGES=0 +FOLDERS_WITH_ERRORS=0 + +for FOLDER in "${!FOLDER_COUNTS[@]}"; do + COUNT=${FOLDER_COUNTS[$FOLDER]} + FOLDERS_WITH_CHANGES=$((FOLDERS_WITH_CHANGES + 1)) + FOLDER_ESC=$(echo "$FOLDER" | html_escape) + FOLDER_ROWS+=" + + 📁 ${FOLDER_ESC} + + ${COUNT} file(s) updated + + " +done + +for FOLDER in "${!FOLDER_ERRORS[@]}"; do + FOLDERS_WITH_ERRORS=$((FOLDERS_WITH_ERRORS + 1)) + FOLDER_ESC=$(echo "$FOLDER" | html_escape) + ERR_ESC=${FOLDER_ERRORS[$FOLDER]} + FOLDER_ROWS+=" + + ⚠ ${FOLDER_ESC} + ${ERR_ESC} + " +done + +if [ -z "$FOLDER_ROWS" ]; then + FOLDER_ROWS=" + + No folder activity — all files are already up to date. + " +fi + +# --- Determine overall status ------------------------------------------------ + +if [ $RSYNC_EXIT -eq 0 ] && [ "$FOLDERS_WITH_ERRORS" -eq 0 ]; then + OVERALL_STATUS="SUCCESS" + BANNER_BG="#1a7f4b" + BANNER_ICON="✅" + BANNER_MSG="Sync completed successfully — no errors detected." + STATUS_BADGE_BG="#1a7f4b" + CARD_HEADER_BG="#f0fdf4" +elif [ "$FOLDERS_WITH_ERRORS" -gt 0 ]; then + OVERALL_STATUS="WARNING" + BANNER_BG="#b45309" + BANNER_ICON="⚠️" + BANNER_MSG="Sync completed with warnings — some folders had issues." + STATUS_BADGE_BG="#b45309" + CARD_HEADER_BG="#fffbeb" +else + OVERALL_STATUS="FAILED" + BANNER_BG="#b91c1c" + BANNER_ICON="🚨" + BANNER_MSG="Sync failed — check the log for details." + STATUS_BADGE_BG="#b91c1c" + CARD_HEADER_BG="#fff1f2" +fi + +# Include permission correction notice in subject if it ran +PERM_NOTE="" +if [ "$PERM_CORRECTED" = true ]; then + PERM_NOTE=" [permissions auto-corrected]" +fi + +SUBJECT="[NAS Sync] NAS08 → NAS16 — ${OVERALL_STATUS}${PERM_NOTE} — ${NOW}" + +# --- Log result -------------------------------------------------------------- + +if [ $RSYNC_EXIT -eq 0 ]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] ===== Sync Completed Successfully =====" | tee -a "$LOG_FILE" +else + echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: rsync exited with code $RSYNC_EXIT. Check log: $LOG_FILE" | tee -a "$LOG_FILE" +fi + +# --- Build HTML email -------------------------------------------------------- + +RAW_LOG_ESC=$(echo "$RSYNC_OUTPUT" | html_escape) + +# Permission correction notice block (only shown if correction ran) +if [ "$PERM_CORRECTED" = true ]; then + PERM_NOTICE_BLOCK=" +

⚠ Destination Permissions Auto-Corrected

Destination ownership was not www-data:ncshare. chown, chmod, and setfacl were applied automatically before the sync ran.

" +else + PERM_NOTICE_BLOCK="" +fi + +HTML=$(cat < + + + + + +

+ + + + + + + + + + + + + + + + + +

+ + + +

NAS File Sync Monitor

🔄 NAS08 → NAS16

${NOW} | Schedule: every other day at 2:30 AM

${BANNER_ICON} ${OVERALL_STATUS}

+ ${BANNER_ICON} ${BANNER_MSG} +

+ +

+ + + +

+ 📄 Sync Results + NAS08 Public → NAS16 Public +

+ ${OVERALL_STATUS} +

+ +

+ + ${PERM_NOTICE_BLOCK} + + + + + + + + + + +

Scanned

${FILES_TOTAL}

Transferred

${FILES_TRANSFERRED}

New Files

${FILES_CREATED}

Data Moved

${TRANSFERRED_SIZE}

Duration

${ELAPSED_FMT}

+ + +

+ + + + + + + + + + + + + + + + + +

Source	NAS08 (192.168.150.35) → ${SOURCE}
Destination	NAS16 (192.168.150.40) → ${DEST}
Transfer Rate	${TRANSFER_RATE}
Total Source	${TOTAL_SIZE}

+ + +

▶ Folder Activity — ${FOLDERS_WITH_CHANGES} updated | ${FOLDERS_WITH_ERRORS} error(s)

+ + + + + + + + ${FOLDER_ROWS} +

Folder	Status

+ + +

▶ Full rsync Log — click to expand

${RAW_LOG_ESC}

+ +

📋 Report Summary

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Host	NAS16 (kingdezigns16)
Run Time	${NOW}
rsync Exit Code	${RSYNC_EXIT}
Permissions Auto-Corrected	${PERM_CORRECTED}
Folders Updated	${FOLDERS_WITH_CHANGES}
Folder Errors	${FOLDERS_WITH_ERRORS}
Log File	${LOG_FILE}

+ Automated NAS Sync Monitor | NAS16 | KingDezigns Infrastructure +

+ + +HTMLEOF +) + +# --- Send email -------------------------------------------------------------- + +send_email() { + if command -v sendmail &>/dev/null; then + { echo "To: ${REPORT_TO}"; echo "From: ${REPORT_FROM}"; echo "Subject: ${SUBJECT}" + echo "MIME-Version: 1.0"; echo "Content-Type: text/html; charset=UTF-8"; echo "" + echo "$HTML"; } | sendmail -t + echo "[$(date '+%Y-%m-%d %H:%M:%S')] Report sent via sendmail to ${REPORT_TO}" | tee -a "$LOG_FILE" + elif command -v mailx &>/dev/null; then + echo "$HTML" | mailx -a "Content-Type: text/html" -s "$SUBJECT" "$REPORT_TO" + echo "[$(date '+%Y-%m-%d %H:%M:%S')] Report sent via mailx to ${REPORT_TO}" | tee -a "$LOG_FILE" + elif command -v mail &>/dev/null; then + echo "$HTML" | mail -a "Content-Type: text/html; charset=UTF-8" -s "$SUBJECT" "$REPORT_TO" + echo "[$(date '+%Y-%m-%d %H:%M:%S')] Report sent via mail to ${REPORT_TO}" | tee -a "$LOG_FILE" + else + echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: No mail transport found." | tee -a "$LOG_FILE" + FALLBACK="/tmp/nas_sync_report_$(date +%Y%m%d_%H%M%S).html" + echo "$HTML" > "$FALLBACK" + echo "[$(date '+%Y-%m-%d %H:%M:%S')] HTML report saved to: $FALLBACK" | tee -a "$LOG_FILE" + fi +} + +send_email + +# --- Cleanup ----------------------------------------------------------------- + +rm -f "$LOCK_FILE" + +# Rotate logs older than 30 days +find "$LOG_DIR" -name "nas08_to_nas16_*.log" -mtime +30 -delete + +echo "[$(date '+%Y-%m-%d %H:%M:%S')] Done — Status: ${OVERALL_STATUS} | Transferred: ${FILES_TRANSFERRED} file(s) | Duration: ${ELAPSED_FMT}" | tee -a "$LOG_FILE" + +exit $RSYNC_EXIT diff --git a/omv/NAS16-Backup-Summary.html b/omv/NAS16-Backup-Summary.html new file mode 100644 index 0000000..dc14de8 --- /dev/null +++ b/omv/NAS16-Backup-Summary.html @@ -0,0 +1,1003 @@ + + + + + +KingDezigns — NAS16 Automated Backup Setup Summary + + + + + +

+ +

KingDezigns — NAS16 Automated Backup Setup Summary

+ 📅 2026-05-12 + 🖥 NAS16 / Raspberry Pi 5 + ⚙️ OpenMediaVault (OMV) + 🌐 Apache · PHP · MariaDB · Webmin + 💾 ZFS RAIDZ2 — Penta SATA HAT +

+ + + +

The goal

+ +

NAS16 serves live websites via Apache/PHP and hosts their databases in MariaDB. A software or hardware failure — corrupt SD card, bad OMV update, runaway script — would take down all hosted sites and lose database state with no documented recovery path and no saved configuration.

+ +

+ No OS backup + No database backup + No automated recovery plan + Web server config undocumented +

+ +

The goal was to create a fully automated, scheduled backup of the OS and configuration layer — including all databases — that could restore NAS16 to full operation after a failure with no manual intervention during normal operation. Website files on the NAS drives are intentionally excluded; those are covered separately by NAS-to-NAS redundancy.

+ +

⚠ Critical — store this document off NAS16

Keep a copy of this document on your workstation, a USB drive, or in print. If NAS16 is down you cannot read files stored on it.

+ + + +

What is backed up

+ +

Backup Script Location

/usr/scripts/omv/nas16-backup.sh

What It Backs Up

OMV config        — omv-confbak export + raw /etc/openmediavault/config.xml
+/etc              — full system config (network, fstab, cron, etc.)
+Apache config     — sites-available, sites-enabled, mods-enabled, conf-available,
+                    conf-enabled, apache2.conf
+PHP config        — all installed PHP version ini + FPM pool configs (/etc/php/*)
+MariaDB databases — mysqldump of every user database (.sql.gz per database)
+                    system schemas excluded (information_schema, performance_schema,
+                    sys, mysql)
+Webmin config     — /etc/webmin (Webmin and Adminer settings)
+Crontab           — root crontab + /etc/cron.d entries
+MANIFEST.txt      — baked-in restore instructions and version info
+
+NOT INCLUDED (by design):
+  Website files on NAS drives — covered by NAS-to-NAS redundancy

Output Paths

/export/kingdezignsnas-16/Backups/NAS16/nas16-backup-YYYY-MM-DD.tar.gz
+/export/kingdezignsnas-16/Backups/NAS16/backup.log

Retention

10 archives retained (~30 days coverage) — older archives auto-deleted on each run

Expected Size

Varies based on database sizes — /etc + Apache + PHP + Webmin are small.
+Primary size driver is MariaDB dumps. Monitor backup.log after first run for actual size.

Schedule via OMV UI

System → Scheduled Jobs → Add
+  Name:          NAS16 Backup
+  Command:       sudo /usr/scripts/omv/nas16-backup.sh
+  Minute:        0
+  Hour:          2
+  Day of month:  */3
+  Month:         *
+  Day of week:   *
+Runs automatically at 2:00 AM every 3 days

Manual Run

sudo /usr/scripts/omv/nas16-backup.sh
+# or: OMV UI → System → Scheduled Jobs → Run

+ + + +

Initial setup steps

+ +

1 Create the scripts directory — NAS16

sudo mkdir -p /usr/scripts/omv
+sudo chown rufusking:rufusking /usr/scripts/omv

Placing scripts in /usr/scripts/omv/ avoids the root-SCP requirement of /usr/local/bin/. The rufusking user can copy files here directly.

+ +

2 Create the backup destination — NAS16

sudo mkdir -p /export/kingdezignsnas-16/Backups/NAS16

The script creates this automatically via mkdir -p on every run, but creating it manually avoids a tee log error on the very first run if the path doesn't exist yet.

+ +

3 Copy the script to NAS16 — workstation

# From your workstation:
+scp nas16-backup.sh rufusking@<NAS16-IP>:/usr/scripts/omv/nas16-backup.sh

+ +

4 Make the script executable — NAS16

chmod +x /usr/scripts/omv/nas16-backup.sh

+ +

5 Run a test backup — NAS16

sudo /usr/scripts/omv/nas16-backup.sh

Watch for any WARNING or ERROR lines in the output. Then verify the archive was created:

ls -lh /export/kingdezignsnas-16/Backups/NAS16/
+cat /export/kingdezignsnas-16/Backups/NAS16/backup.log

+ +

6 Inspect archive contents — NAS16

tar -tzf /export/kingdezignsnas-16/Backups/NAS16/nas16-backup-$(date +%Y-%m-%d).tar.gz | head -60

You should see folders: omv-config/ etc/ apache/ php/ databases/ webmin/ cron/ MANIFEST.txt

+ +

7 Schedule the job in OMV UI — NAS16

OMV UI → System → Scheduled Jobs → Add
+  Enabled:       Yes
+  Name:          NAS16 Backup
+  Command:       sudo /usr/scripts/omv/nas16-backup.sh
+  Minute:        0
+  Hour:          2
+  Day of month:  */3
+  Month:         *
+  Day of week:   *
+→ Save → Apply

+ +

+ + + +

Full recovery procedure

+ +

⚠ Before You Start

You need: a workstation with internet, a replacement SD card (16GB+), Raspberry Pi Imager, SSH access, and your rufusking password. Physically connect all ZFS drives to the Penta SATA HAT before powering on.

+ + +

Phase 1 — Flash the OS

+ +

1 Download Raspberry Pi Imager — workstation

https://www.raspberrypi.com/software/

Download and install for your OS (Windows / Mac / Linux).

+ +

2 Flash Raspberry Pi OS Lite (64-bit) — workstation

Open Raspberry Pi Imager
+→ Choose Device    : Raspberry Pi 5
+→ Choose OS        : Raspberry Pi OS Lite (64-bit)   ← NOT the Desktop version
+→ Choose Storage   : your SD card
+→ Next → Edit Settings:
+    Hostname  : NAS16
+    Username  : rufusking
+    Password  : your usual password
+    Services  : Enable SSH → Use password authentication
+→ Save → Yes → Yes (confirm flash and verify)

Wait for the flash and verify to complete fully before removing the SD card.

+ +

3 Boot the Raspberry Pi

1. Insert SD card into Raspberry Pi 5
+2. Connect Penta SATA HAT with all ZFS drives plugged in
+3. Power on
+4. Wait 2–3 minutes for first boot to complete

+ + +

Phase 2 — Install OpenMediaVault

+ +

⚠ Note

Verify the current OMV install script at https://github.com/OpenMediaVault-Plugin-Developers/installScript before running. Steps below reflect the method as of 2026-05-12.

+ +

4 SSH into NAS16 — workstation

ssh rufusking@<NAS16-IP>

If the static IP is not yet assigned, log into UniFi, find NAS16 in Clients, and SSH to its temporary IP instead.

+ +

5 Update the OS — NAS16

sudo apt-get update && sudo apt-get upgrade -y

Wait for the prompt to return before continuing.

+ +

6 Install OpenMediaVault — NAS16

sudo wget -O - https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/install | sudo bash

Takes 10–20 minutes. The Pi may reboot automatically. If it does, SSH back in and wait for the script to finish.

+ +

7 Verify OMV is running — browser on workstation

http://<NAS16-IP>
+
+Default login:
+  Username: admin
+  Password: openmediavault

Confirm the OMV login page loads. Do not configure anything yet — the restore handles all configuration.

+ + +

Phase 3 — Enable PCIe for Penta SATA HAT

+ +

The Raspberry Pi 5 does not expose PCIe by default. Without enabling it, the Penta SATA HAT and all ZFS drives are completely invisible to the OS. This must be done before any ZFS commands will work.

+ +

8 Edit boot config to enable PCIe — NAS16

sudo nano /boot/firmware/config.txt

Scroll to find the line arm_boost=1 and add these lines immediately after it:

# Enable PCIe
+dtparam=pciex1
+dtparam=pciex1_gen=3

Save and exit:

Ctrl+X  →  Y  →  Enter

+ +

9 Reboot to apply PCIe config — NAS16

sudo reboot

Wait 2–3 minutes, then SSH back in.

+ + +

Phase 4 — Import ZFS Pool

+ +

10 Install ZFS tools — NAS16

sudo apt-get install -y zfsutils-linux

Required on every fresh OS install.

+ +

11 Verify all drives are visible — NAS16

lsblk

All drives attached to the Penta SATA HAT should be listed. If any are missing, check physical connections and reboot.

+ +

12 Import the ZFS pool — NAS16

sudo zpool import kingdezignsnas-16

If the previous OS failed uncleanly, force the import:

# Only use -f if the standard import fails:
+sudo zpool import -f kingdezignsnas-16

+ +

13 Verify pool imported correctly — NAS16

sudo zpool status

All drives must show ONLINE and errors should be No known data errors.

If any drive shows FAULTED or DEGRADED, do not proceed. See ZFS Troubleshooting at the bottom of this document.

+ +

14 Confirm backup files are accessible — NAS16

ls /kingdezignsnas-16/Backups/NAS16/
+
+# List archives sorted newest first:
+ls -lht /kingdezignsnas-16/Backups/NAS16/nas16-backup-*.tar.gz

Use /kingdezignsnas-16/ directly at this stage. The /export/kingdezignsnas-16/ path is recreated by OMV after the config restore in Phase 6.

+ + +

Phase 5 — Extract the Backup Archive

+ +

15 Create restore working directory — NAS16

sudo mkdir -p /tmp/nas16-restore

+ +

16 Extract the most recent backup — NAS16

# Replace YYYY-MM-DD with the actual archive date from step 14
+sudo tar -xzf /kingdezignsnas-16/Backups/NAS16/nas16-backup-YYYY-MM-DD.tar.gz \
+    -C /tmp/nas16-restore

+ +

17 Confirm extraction and read the manifest — NAS16

ls /tmp/nas16-restore/
+ls /tmp/nas16-restore/nas16-backup-*/
+
+cat /tmp/nas16-restore/nas16-backup-*/MANIFEST.txt

You should see: omv-config/ etc/ apache/ php/ databases/ webmin/ cron/ MANIFEST.txt

+ + +

Phase 6 — Restore System Configuration

+ +

18 Restore /etc — NAS16

sudo rsync -a /tmp/nas16-restore/nas16-backup-*/etc/ /etc/

Restores all system configuration: network settings, fstab, cron, and everything else under /etc.

+ +

19 Restore OMV configuration — NAS16

# Copy saved OMV config back into place:
+sudo cp /tmp/nas16-restore/nas16-backup-*/omv-config/config.xml.raw \
+    /etc/openmediavault/config.xml
+
+# Load it into the OMV database:
+sudo omv-confdbadm populate
+
+# Re-apply all settings to the system:
+sudo omv-salt deploy run all

The last command takes several minutes. Do not interrupt it — let it complete fully.

+ +

20 Reboot and verify OMV — NAS16 then browser

sudo reboot

After 2–3 minutes, SSH back in. Then open a browser:

http://<NAS16-IP>

Log in and verify shares, users, and settings are present. Confirm the ZFS pool is mounted and /export/kingdezignsnas-16 is available under Storage → File Systems.

+ + +

Phase 7 — Reinstall Web Stack (Apache, PHP, MariaDB, Webmin)

+ +

⚠ Install First, Restore Config Second

The web stack packages must be reinstalled before restoring configuration files. Restoring config before the packages exist will have no effect and may be overwritten by the installer.

+ +

21 Install Apache and PHP — NAS16

sudo apt-get install -y apache2 php libapache2-mod-php php-mysql php-cli php-curl \
+    php-mbstring php-xml php-zip php-gd

Install the same PHP version that was running before. Check the MANIFEST.txt from the archive for the exact version if unsure.

+ +

22 Install MariaDB — NAS16

sudo apt-get install -y mariadb-server mariadb-client

+ +

23 Secure MariaDB and verify it is running — NAS16

sudo systemctl start mariadb
+sudo systemctl enable mariadb
+sudo systemctl status mariadb

Do NOT run mysql_secure_installation before restoring databases — the debian.cnf maintenance account needs to remain intact for the restore steps to work without a password.

+ +

24 Install Webmin — NAS16

# Add Webmin repository and install:
+curl -o setup-repos.sh https://raw.githubusercontent.com/webmin/webmin/master/setup-repos.sh
+sudo sh setup-repos.sh
+sudo apt-get install -y webmin

Verify the current install method at https://webmin.com/download/ before running — the method may change.

+ +

25 Install Adminer — NAS16

sudo apt-get install -y adminer
+sudo a2enconf adminer
+sudo systemctl reload apache2

Or manually place adminer.php in your web root if you use a custom location:

sudo wget -O /var/www/html/adminer.php https://www.adminer.org/latest.php

+ + +

Phase 8 — Restore Web Stack Configuration

+ +

26 Restore Apache configuration — NAS16

sudo rsync -a /tmp/nas16-restore/nas16-backup-*/apache/sites-available/ \
+    /etc/apache2/sites-available/
+
+sudo rsync -a /tmp/nas16-restore/nas16-backup-*/apache/sites-enabled/ \
+    /etc/apache2/sites-enabled/
+
+sudo rsync -a /tmp/nas16-restore/nas16-backup-*/apache/conf-available/ \
+    /etc/apache2/conf-available/
+
+sudo rsync -a /tmp/nas16-restore/nas16-backup-*/apache/conf-enabled/ \
+    /etc/apache2/conf-enabled/
+
+# Restore main apache2.conf:
+sudo cp /tmp/nas16-restore/nas16-backup-*/apache/apache2.conf \
+    /etc/apache2/apache2.conf

Do not overwrite mods-enabled directly — Apache manages these symlinks internally. Re-enable required mods manually if needed (see step 27).

+ +

27 Re-enable Apache modules — NAS16

# Check what modules were enabled (from your backup):
+ls /tmp/nas16-restore/nas16-backup-*/apache/mods-enabled/
+
+# Re-enable common modules as needed:
+sudo a2enmod rewrite ssl headers proxy proxy_http
+sudo systemctl restart apache2

Enable only the mods you see in the backup snapshot. The full list is in the mods-enabled/ folder inside the archive.

+ +

28 Restore PHP configuration — NAS16

# Replace X.X with your PHP version (e.g. 8.2):
+sudo rsync -a /tmp/nas16-restore/nas16-backup-*/php/X.X/ /etc/php/X.X/

+ +

29 Restore Webmin configuration — NAS16

sudo rsync -a /tmp/nas16-restore/nas16-backup-*/webmin/ /etc/webmin/

+ +

30 Restart web stack services — NAS16

sudo systemctl restart apache2
+sudo systemctl restart mariadb
+sudo systemctl restart webmin
+
+# Verify all are running:
+sudo systemctl status apache2 mariadb webmin

+ + +

Phase 9 — Restore Databases

+ +

31 List backed-up databases — NAS16

ls /tmp/nas16-restore/nas16-backup-*/databases/

You should see one .sql.gz file per database. Each file is a complete dump of that database including schema and data.

+ +

32 Restore each database — NAS16

# For each database (replace DB_NAME with the actual name):
+gunzip -c /tmp/nas16-restore/nas16-backup-*/databases/DB_NAME.sql.gz \
+    | sudo mysql --defaults-file=/etc/mysql/debian.cnf DB_NAME
+
+# If the database does not exist yet, create it first:
+sudo mysql --defaults-file=/etc/mysql/debian.cnf \
+    -e "CREATE DATABASE IF NOT EXISTS DB_NAME CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
+
+# Then restore:
+gunzip -c /tmp/nas16-restore/nas16-backup-*/databases/DB_NAME.sql.gz \
+    | sudo mysql --defaults-file=/etc/mysql/debian.cnf DB_NAME

Repeat for every .sql.gz file in the databases/ folder. Take your time — one at a time to catch any individual errors.

+ +

33 Verify databases restored correctly — NAS16

sudo mysql --defaults-file=/etc/mysql/debian.cnf \
+    -e "SHOW DATABASES;"
+
+# Spot-check row counts in a key table (replace DB_NAME and TABLE_NAME):
+sudo mysql --defaults-file=/etc/mysql/debian.cnf \
+    -e "SELECT COUNT(*) FROM DB_NAME.TABLE_NAME;"

+ + +

Phase 10 — Final Steps & Verification

+ +

34 Restore the crontab — NAS16

sudo crontab /tmp/nas16-restore/nas16-backup-*/cron/root-crontab.txt
+
+# Verify it was applied:
+sudo crontab -l

+ +

35 Reschedule the backup job in OMV UI — NAS16

OMV UI → System → Scheduled Jobs → Add
+  Enabled:       Yes
+  Name:          NAS16 Backup
+  Command:       sudo /usr/scripts/omv/nas16-backup.sh
+  Minute:        0
+  Hour:          2
+  Day of month:  */3
+  Month:         *
+  Day of week:   *
+→ Save → Apply

+ +

36 Copy the backup script back to NAS16 — workstation

scp nas16-backup.sh rufusking@<NAS16-IP>:/usr/scripts/omv/nas16-backup.sh
+chmod +x /usr/scripts/omv/nas16-backup.sh

The script lives outside /etc so it is not automatically restored by the /etc rsync. Always re-deploy it manually.

+ +

37 Verify all websites load — browser

# Test each hosted domain or IP in your browser.
+# Check Apache error log if anything fails:
+sudo tail -50 /var/log/apache2/error.log

+ +

38 Clean up the restore working directory — NAS16

sudo rm -rf /tmp/nas16-restore

+ +

39 Run a fresh backup immediately to create a new baseline — NAS16

# Option A — OMV UI:
+System → Scheduled Jobs → NAS16 Backup → Run
+
+# Option B — SSH on NAS16:
+sudo /usr/scripts/omv/nas16-backup.sh
+
+# Verify it completed successfully:
+ls -lh /export/kingdezignsnas-16/Backups/NAS16/
+cat /export/kingdezignsnas-16/Backups/NAS16/backup.log

+ +

40 Run a ZFS scrub to verify data integrity — NAS16

sudo zpool scrub kingdezignsnas-16
+
+# Check results a few minutes later:
+sudo zpool status

A healthy result:

scan: scrub repaired 0B in 00:xx:xx with 0 errors

+ + +

ZFS Troubleshooting

+ +

— Pool shows DEGRADED — 1 drive failed

sudo zpool status   ← failed drive shows FAULTED or UNAVAIL

RAIDZ2 tolerates up to 2 simultaneous drive failures. The pool is still fully readable and writable. Complete the full restore procedure first, then replace the failed drive.

+ +

— Pool shows DEGRADED — 2 drives failed

sudo zpool status   ← two drives show FAULTED or UNAVAIL

Data is at risk but still accessible. Complete the restore immediately and replace both drives as fast as possible.

+ +

— Pool cannot be imported — not found

# Scan all connected devices for importable pools:
+sudo zpool import
+
+# If kingdezignsnas-16 appears in the list, force import:
+sudo zpool import -f kingdezignsnas-16

If no pools are found at all, PCIe is likely not enabled. Verify Steps 8–9 (boot config) were applied correctly and reboot again before retrying.

+ +

— Pool imported but /export/kingdezignsnas-16 is missing

# Access backup files directly via the ZFS mount point:
+ls /kingdezignsnas-16/Backups/NAS16/

/export/kingdezignsnas-16/ is created by OMV when it mounts the filesystem. Use /kingdezignsnas-16/ as the base path for all restore commands until OMV is fully restored.

+ +

+ + + +

Key differences from NAS08

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Area	NAS08	NAS16
Primary role	Docker containers (Pi-hole, Plex, Vaultwarden, Nextcloud, NPM)	Web server (Apache/PHP), databases (MariaDB), Webmin admin
What's backed up beyond /etc + OMV	Docker Compose files, Pi-hole data, Plex config/metadata	Apache vhosts, PHP config, MariaDB dumps, Webmin config
Website files	N/A	Intentionally excluded — stored on NAS drives, covered by redundancy
Database backup	None (no databases)	Full mysqldump of all user databases, .sql.gz per database
Backup destination	/export/kingdezigns-all/Backups/NAS08/	/export/kingdezignsnas-16/Backups/NAS16/
ZFS pool name	kingdezignsnas	kingdezignsnas-16 (assumed — verify with `sudo zpool list`)
Recovery phases	9 phases, 34 steps	10 phases, 40 steps

+ + + +

Notes & assumptions to verify

+ +

⚠ Verify these before first run

The script was written based on information provided and follows Raspberry Pi OS / Debian conventions. Verify the following on NAS16 before treating any backup as production-ready.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Item	Assumed value	How to verify
ZFS pool name	`kingdezignsnas-16`	`sudo zpool list`
Backup destination path	`/export/kingdezignsnas-16/Backups/NAS16/`	`ls /export/` — confirm share name matches
MariaDB auth method	Uses `/etc/mysql/debian.cnf` (maintenance account, no password when run as root)	Run: `sudo mysql --defaults-file=/etc/mysql/debian.cnf -e "SHOW DATABASES;"` — should work without password
PHP version(s)	Auto-detected from `/etc/php/*/`	`php -v` and `ls /etc/php/`
Apache config location	`/etc/apache2/` (standard Debian)	`apache2 -V` — confirms config path
Webmin config location	`/etc/webmin/`	`ls /etc/webmin/`

+ + + +

Network map — items to update

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Item	Old value	New value
NAS16 — Backup status	`None`	`Automated — every 3 days — 30-day retention`
NAS16 — Backup destination	`None`	`/export/kingdezignsnas-16/Backups/NAS16/`
NAS16 — Backup script	`None`	`/usr/scripts/omv/nas16-backup.sh`
NAS16 — PCIe requirement	`not documented`	`dtparam=pciex1 + dtparam=pciex1_gen=3 required in /boot/firmware/config.txt on fresh OS`
NAS16 — Web stack	`not documented`	`Apache · PHP · MariaDB · Webmin · Adminer`
NAS16 — Recovery plan	`None`	`NAS16-Backup-Summary.html — 40 steps across 10 phases + ZFS troubleshooting`

+ + + + +

+ + diff --git a/omv/SSD-fstrim-Summary.html b/omv/SSD-fstrim-Summary.html new file mode 100644 index 0000000..c8d03fe --- /dev/null +++ b/omv/SSD-fstrim-Summary.html @@ -0,0 +1,521 @@ + + + + + +KingDezigns — SSD Hardening Summary + + + + + +

+ +

KingDezigns — SSD Hardening (TRIM + noatime) summary

+ 📅 2026-05-23 + 🖥 NAS08 & NAS16 + ⚙️ OpenMediaVault + 🔧 Raspberry Pi 5 / Raspberry Pi +

+ + + +

Objective

Harden both NAS08 and NAS16 to extend SSD lifespan by ensuring TRIM is running correctly, noatime is active, and SMART health is being monitored — with all scheduled jobs visible in OMV's scheduler for easy management.

+ NAS08 — 5 drives (1x PNY CS900 + 4x Crucial BX500) + NAS16 — 1 drive (unknown model, standard SMART attrs) +

+ + + +

Investigation steps — both units

+ +

1 Verify TRIM support on all drives

lsblk --discard /dev/sda

Confirm DISC-GRAN and DISC-MAX are non-zero before enabling TRIM.

Both units: DISC-GRAN 4K / DISC-MAX 4G — TRIM fully supported on all drives.

+ +

2 Check if systemd fstrim.timer was already active

systemctl status fstrim.timer

Determine whether a TRIM schedule already existed before creating a new one.

Both units: timer active and enabled, running weekly. NAS08 next run Monday 12:33 AM — NAS16 next run Monday 1:37 AM.

+ +

3 Confirm which volumes were being trimmed

journalctl -u fstrim.service --no-pager

Verify the timer was actually running and catching all mounted volumes.

Both units: / and /boot/firmware trimmed successfully on May 11 and May 18. NAS08 ~400 GiB freed per run. NAS16 ~228 GiB on first run, 3.3 GiB on second (less free space).

+ +

4 Check noatime mount status

findmnt -t ext4,xfs,btrfs -o TARGET,SOURCE,OPTIONS

Confirm noatime was already set on all filesystems to prevent reads generating writes.

Both units: every mount point already showing noatime — handled automatically by the openmediavault-flashmemory plugin. No action required.

+ +

5 Inspect SMART attributes per drive

sudo smartctl -A /dev/sda
+sudo smartctl -A /dev/sdb   # NAS08 data drives

Identify vendor-specific SMART attribute names before writing the monitoring script — attribute names differ significantly between PNY and Crucial.

NAS08 PNY CS900: uses SSD_Life_Left (attr 231), Bad_Blk_Ct_Lat/Erl (attr 170), no pending/uncorrectable attrs. NAS08 Crucial BX500: uses Percent_Lifetime_Remain (attr 202), Reallocate_NAND_Blk_Cnt (attr 5), Reported_Uncorrect. NAS16 drive: uses standard attribute names — no vendor mapping required.

+ +

+ + + +

What was already handled by OMV

+ +

noatime: Already active on all filesystems on both units via the openmediavault-flashmemory plugin. No manual configuration required.

+ +

Swap disabled, logs in tmpfs: Also handled automatically by the flashmemory plugin — swap off, /var/log and /tmp in RAM, reducing write amplification significantly.

+ +

⚠ Note

The systemd fstrim.timer was already running weekly on both units but was invisible in OMV's scheduler. It was replaced with an OMV Scheduled Job so all jobs are visible and manageable in one place.

+ + + +

TRIM report script — created and deployed

+ +

A custom script was written to replace the bare fstrim call with a full monitoring run that emails an HTML report matching the KingDezigns backup monitor style (dark header, badge/banner, section rows, cards, footer).

+ +

The email report includes: TRIM results per volume with total bytes freed, SSD SMART health per drive (health, temperature, wear, power-on hours, sector counts), filesystem usage with colour-coded warnings at 75% and 90%, and system info (OMV version, kernel, IP, uptime, load, memory, next run time).

+ +

Script location

/usr/scripts/omv/fstrim-report.sh

Log location

/var/log/fstrim/fstrim-YYYY-MM-DD.log   (90-day retention)

Deploy (preferred method — sudo nano)

sudo nano /usr/scripts/omv/fstrim-report.sh
+# paste script content, save with Ctrl+O, exit with Ctrl+X
+sudo chmod +x /usr/scripts/omv/fstrim-report.sh

Test run

sudo systemctl start fstrim.service
+sudo journalctl -u fstrim.service -f

Email delivery

sendmail -t   (via OMV's configured SMTP relay)

+ +

⚠ SMART attribute fixes required

Initial script used generic attribute names. Two rounds of fixes were needed after testing:

Round 1 — Crucial BX500 fix:
+  Reallocated: Reallocated_Sector → Reallocate_NAND_Blk_Cnt
+  Pending:     Current_Pending_Sector → Current_Pending_ECC_Cnt
+  Uncorrect:   Offline_Uncorrectable → Reported_Uncorrect
+  Temperature: $10 → $10+0  (strips "32 (Min/Max 25/56)" format)
+
+Round 2 — PNY CS900 fix:
+  Wear:        Wear_Leveling_Count → SSD_Life_Left
+  Reallocated: Reallocate_NAND_Blk_Cnt → Bad_Blk_Ct_Lat/Erl (first value before /)

Each pattern now tries vendor-specific names first, then falls back to generic names — script works correctly on any brand.

+ + + +

Migrating from systemd timer to OMV scheduler

+ +

The systemd fstrim.timer was disabled on both units and replaced with OMV Scheduled Jobs so all scheduled tasks are visible in one place.

+ +

1 Disable systemd timer (both units)

sudo systemctl disable fstrim.timer
+sudo systemctl stop fstrim.timer

Prevent the systemd timer from running alongside the new OMV job — avoids double execution.

Timer stopped and disabled on both NAS08 and NAS16.

+ +

2 Remove systemd override (both units)

sudo rm /etc/systemd/system/fstrim.service.d/override.conf
+sudo systemctl daemon-reload

Clean up the override that pointed fstrim.service to the script — no longer needed since OMV cron calls the script directly.

Override removed, systemd reloaded cleanly on both units.

+ +

3 Add OMV Scheduled Job (each unit)

System → Scheduled Jobs → Add
+Command: /usr/scripts/omv/fstrim-report.sh
+User:    root
+NAS08:   Wednesday at 1:00 AM
+NAS16:   Wednesday at 1:30 AM

Wednesday was chosen as it sits midweek, away from Sunday ZFS scrubs and the rolling every-3-day backup jobs.

Both units now show TRIM Report in OMV scheduler alongside all other scheduled jobs.

+ +

+ + + +

Final OMV scheduled jobs — both units

+ +

NAS08

+ + + + + + + + + + +

Schedule	Script	Tag
Every 3 days at 2:00 AM	`/usr/scripts/omv/nas08-backup.sh`	NAS08 Backup
Daily at 12:00 AM	`/usr/scripts/zfs/nas08_zfs_report.sh`	ZFS Status Report
Sunday at 3:00 AM	`/usr/scripts/zfs/nas08_zfs_scrub.sh`	ZFS Scrub
Wednesday at 1:00 AM	`/usr/scripts/omv/fstrim-report.sh`	TRIM Report

+ +

NAS16

+ + + + + + + + + + + + +

Schedule	Script	Tag
Every 3 days at 2:00 AM	`/usr/scripts/omv/nas16-backup.sh`	NAS16 Backup
Every 2 days at 2:30 AM	`bash /usr/scripts/nas/nas08_to_nas16_sync.sh`	NAS Drive Backup
Daily at 12:00 AM	`/usr/scripts/zfs/nas16_zfs_report.sh`	ZFS Status Report
Sunday at 3:00 AM	`/usr/scripts/zfs/nas16_zfs_scrub.sh`	ZFS Scrub
Every 15 minutes	`/usr/scripts/ddns/ddns_update.sh`	DDNS Update
Wednesday at 1:30 AM	`/usr/scripts/omv/fstrim-report.sh`	TRIM Report

+ + + +

Lessons learned

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Lesson	Detail
Always investigate before configuring	Both noatime and fstrim were already handled by OMV's flashmemory plugin — checking first avoided duplicating work.
SMART attribute names are vendor-specific	PNY and Crucial use completely different attribute names for wear, reallocated sectors, and temperature format. Always run `smartctl -A /dev/sdX` first to map the real names.
systemd timers are invisible in OMV scheduler	The default fstrim.timer runs correctly but doesn't appear in OMV's job list — migrate to OMV Scheduled Jobs for full visibility alongside ZFS scrubs and backups.
Never run script without sudo for testing	Running fstrim-report.sh without sudo causes permission denied on /var/log/fstrim/ and returns N/A for all SMART data — always test with sudo or via systemctl start.
Use sudo nano over cp for file edits	Preferred method for placing and editing scripts on the NAS — faster navigation to the correct path, avoids confusion about source file location.
Inline TRIM (discard mount option) is not recommended	Real-time per-delete TRIM adds latency under heavy load. Periodic batch TRIM via fstrim is the correct approach for NAS workloads.
PNY CS900 has no pending/uncorrectable attributes	N/A for those fields is correct and expected — not a script bug. The drive simply does not expose those attributes.

+ + + +

Documentation updated

+ + + + + + + + + + + + + + +

File	What changed
`server_nas08.md`	Added drive inventory table, full SSD Hardening & Maintenance section (TRIM, noatime, flashmemory plugin, SMART attribute mapping), and complete OMV Scheduled Jobs table.
`server_nas16.md`	Added drive inventory table with Power_Cycle_Count monitoring note, full SSD Hardening & Maintenance section, and complete OMV Scheduled Jobs table including NAS drive backup sync job.

+ + + + +

+ + diff --git a/omv/fstrim-override.conf b/omv/fstrim-override.conf new file mode 100644 index 0000000..ef054b6 --- /dev/null +++ b/omv/fstrim-override.conf @@ -0,0 +1,4 @@ +[Service] +# Clear the default ExecStart then replace with our script +ExecStart= +ExecStart=/usr/scripts/omv/fstrim-report.sh diff --git a/omv/fstrim-report.sh b/omv/fstrim-report.sh new file mode 100644 index 0000000..09c7aa4 --- /dev/null +++ b/omv/fstrim-report.sh @@ -0,0 +1,471 @@ +#!/bin/bash +# ============================================================================= +# NAS08 Weekly TRIM Report Script +# ============================================================================= +# Runs fstrim on all mounted filesystems, captures results, gathers SMART +# health data for each SSD, and sends an HTML email report matching the +# KingDezigns backup monitor style. +# +# Triggered by: systemd fstrim.service (via drop-in override) +# Schedule: Weekly (Monday ~12:33 AM — managed by fstrim.timer) +# +# Script location: /usr/scripts/omv/fstrim-report.sh +# Log location: /var/log/fstrim/ +# ============================================================================= + +set -uo pipefail + +# ----------------------------------------------------------------------------- +# CONFIGURATION +# ----------------------------------------------------------------------------- +HOSTNAME_LABEL=$(hostname -s 2>/dev/null || echo "NAS08") +REPORT_TO="rufus.king@kingdezigns.com" +REPORT_FROM="trim-monitor@nas08.local" # Display only — OMV uses your configured SMTP relay + +LOG_DIR="/var/log/fstrim" +KEEP_LOGS=90 # days to retain log files + +DATE=$(date +%Y-%m-%d) +REPORT_TIME=$(date "+%Y-%m-%d %H:%M:%S") +LOG_FILE="${LOG_DIR}/fstrim-${DATE}.log" + +# ----------------------------------------------------------------------------- +# STATUS TRACKING +# OVERALL_STATUS: SUCCESS | WARNING | FAILED +# ----------------------------------------------------------------------------- +OVERALL_STATUS="SUCCESS" +SECTION_ROWS_HTML="" +WARNING_COUNT=0 +ERROR_COUNT=0 + +# Append a result row to the section table +# Usage: add_section_row "Section name" "STATUS" "Detail message" +# STATUS: OK | WARNING | ERROR | SKIPPED +add_section_row() { + local section="$1" status="$2" detail="$3" + + case "$status" in + OK) + BADGE="✓ OK" + ;; + WARNING) + BADGE="⚠ WARNING" + WARNING_COUNT=$((WARNING_COUNT + 1)) + [[ "$OVERALL_STATUS" == "SUCCESS" ]] && OVERALL_STATUS="WARNING" + ;; + ERROR) + BADGE="✗ ERROR" + ERROR_COUNT=$((ERROR_COUNT + 1)) + OVERALL_STATUS="FAILED" + ;; + SKIPPED) + BADGE="— SKIPPED" + ;; + esac + + SECTION_ROWS_HTML+=" + + $section + $BADGE + $detail + " +} + +# ----------------------------------------------------------------------------- +# LOGGING +# ----------------------------------------------------------------------------- +mkdir -p "$LOG_DIR" + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE" +} + +log_section() { + echo "" | tee -a "$LOG_FILE" + echo "------------------------------------------------------------" | tee -a "$LOG_FILE" + log "$1" + echo "------------------------------------------------------------" | tee -a "$LOG_FILE" +} + +# ----------------------------------------------------------------------------- +# SECTION 1: RUN FSTRIM +# ----------------------------------------------------------------------------- +log_section "Running fstrim on all mounted filesystems" + +TRIM_RAW=$(fstrim -av 2>&1) +TRIM_EXIT=$? + +TRIM_ROWS_HTML="" +TOTAL_BYTES=0 +VOLUME_COUNT=0 + +while IFS= read -r line; do + if [[ "$line" =~ ^(/.+):[[:space:]]+(.+)[[:space:]]+$([0-9]+)[[:space:]]bytes$ ]]; then + MOUNT="${BASH_REMATCH[1]}" + HUMAN="${BASH_REMATCH[2]}" + BYTES="${BASH_REMATCH[3]}" + TOTAL_BYTES=$((TOTAL_BYTES + BYTES)) + VOLUME_COUNT=$((VOLUME_COUNT + 1)) + log "Trimmed ${MOUNT}: ${HUMAN}" + TRIM_ROWS_HTML+=" + + $MOUNT + $HUMAN freed + " + fi +done <<< "$TRIM_RAW" + +# Convert total bytes to human readable +if [ $TOTAL_BYTES -ge 1073741824 ]; then + TOTAL_HUMAN=$(awk "BEGIN {printf \"%.1f GiB\", $TOTAL_BYTES/1073741824}") +elif [ $TOTAL_BYTES -ge 1048576 ]; then + TOTAL_HUMAN=$(awk "BEGIN {printf \"%.1f MiB\", $TOTAL_BYTES/1048576}") +else + TOTAL_HUMAN="${TOTAL_BYTES} bytes" +fi + +if [ $TRIM_EXIT -eq 0 ]; then + log "fstrim completed — $VOLUME_COUNT volume(s) · $TOTAL_HUMAN total freed" + add_section_row "TRIM Execution" "OK" "$VOLUME_COUNT volume(s) processed · $TOTAL_HUMAN total freed" +else + log "ERROR: fstrim exited with code $TRIM_EXIT" + add_section_row "TRIM Execution" "ERROR" "fstrim exited with code $TRIM_EXIT" +fi + +# ----------------------------------------------------------------------------- +# SECTION 2: SSD SMART HEALTH +# ----------------------------------------------------------------------------- +log_section "Gathering SSD SMART health data" + +SMART_ROWS_HTML="" +SMART_WARN=0 +SMART_ERR=0 + +for DISK in $(lsblk -dpno NAME | grep -E '^/dev/sd|^/dev/nvme'); do + MODEL=$(smartctl -i "$DISK" 2>/dev/null | awk -F': ' '/Device Model|Model Number/{gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2; exit}') + SERIAL=$(smartctl -i "$DISK" 2>/dev/null | awk -F': ' '/Serial Number/{gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2; exit}') + CAPACITY=$(lsblk -dno SIZE "$DISK" 2>/dev/null) + HEALTH=$(smartctl -H "$DISK" 2>/dev/null | awk '/overall-health|test result/{print $NF}') + # Temperature: +0 forces numeric conversion, handling "32 (Min/Max 25/56)" Crucial format + TEMP=$(smartctl -A "$DISK" 2>/dev/null | awk '/Temperature_Celsius|Temperature_Internal/{print $10+0; exit}') + POWER_ON=$(smartctl -A "$DISK" 2>/dev/null | awk '/Power_On_Hours/{print $10; exit}') + # Wear: PNY=SSD_Life_Left, Crucial=Percent_Lifetime_Remain, Samsung/WD=Wear_Leveling_Count, Intel=Media_Wearout_Indicator + WEAR=$(smartctl -A "$DISK" 2>/dev/null | awk '/SSD_Life_Left|Percent_Lifetime_Remain|Wear_Leveling_Count|Media_Wearout_Indicator/{print $10; exit}') + # Reallocated: PNY=Bad_Blk_Ct_Lat/Erl (RAW_VALUE is "0/88" — take first number), Crucial=Reallocate_NAND_Blk_Cnt, others=Reallocated_Sector_Ct + REALLOCATED=$(smartctl -A "$DISK" 2>/dev/null | awk '/Bad_Blk_Ct_Lat\/Erl/{split($10,a,"/"); print a[1]; exit} /Reallocate_NAND_Blk_Cnt|Reallocated_Sector_Ct|Reallocated_Sector/{print $10; exit}') + # Pending: Crucial=Current_Pending_ECC_Cnt, others=Current_Pending_Sector (PNY has no equivalent — will show N/A) + PENDING=$(smartctl -A "$DISK" 2>/dev/null | awk '/Current_Pending_ECC_Cnt|Current_Pending_Sector/{print $10; exit}') + # Uncorrectable: Crucial=Reported_Uncorrect, others=Offline_Uncorrectable (PNY has no equivalent — will show N/A) + UNCORRECTABLE=$(smartctl -A "$DISK" 2>/dev/null | awk '/Reported_Uncorrect|Offline_Uncorrectable/{print $10; exit}') + + [ -z "$MODEL" ] && MODEL="Unknown" + [ -z "$SERIAL" ] && SERIAL="Unknown" + [ -z "$HEALTH" ] && HEALTH="Unknown" + [ -z "$TEMP" ] && TEMP="N/A" + [ -z "$POWER_ON" ] && POWER_ON="N/A" + [ -z "$WEAR" ] && WEAR="N/A" + [ -z "$REALLOCATED" ] && REALLOCATED="N/A" + [ -z "$PENDING" ] && PENDING="N/A" + [ -z "$UNCORRECTABLE" ] && UNCORRECTABLE="N/A" + + # Health colour + if [ "$HEALTH" = "PASSED" ]; then + HEALTH_COLOUR="#065f46" + HEALTH_BG="#d1fae5" + DISK_STATUS="OK" + else + HEALTH_COLOUR="#991b1b" + HEALTH_BG="#fee2e2" + DISK_STATUS="WARNING" + SMART_WARN=$((SMART_WARN + 1)) + fi + + # Flag non-zero reallocated/pending/uncorrectable sectors + SECTOR_NOTE="" + if [ "$REALLOCATED" != "N/A" ] && [ "$REALLOCATED" -gt 0 ] 2>/dev/null; then + SECTOR_NOTE+=" ⚠ Reallocated: $REALLOCATED" + SMART_WARN=$((SMART_WARN + 1)) + [[ "$OVERALL_STATUS" == "SUCCESS" ]] && OVERALL_STATUS="WARNING" + fi + if [ "$PENDING" != "N/A" ] && [ "$PENDING" -gt 0 ] 2>/dev/null; then + SECTOR_NOTE+=" ⚠ Pending: $PENDING" + SMART_WARN=$((SMART_WARN + 1)) + [[ "$OVERALL_STATUS" == "SUCCESS" ]] && OVERALL_STATUS="WARNING" + fi + if [ "$UNCORRECTABLE" != "N/A" ] && [ "$UNCORRECTABLE" -gt 0 ] 2>/dev/null; then + SECTOR_NOTE+=" 🚨 Uncorrectable: $UNCORRECTABLE" + SMART_ERR=$((SMART_ERR + 1)) + OVERALL_STATUS="FAILED" + fi + + log "Drive $DISK ($MODEL) — Health: $HEALTH | Temp: ${TEMP}°C | Wear: $WEAR | Power-on: ${POWER_ON}h${SECTOR_NOTE}" + + SMART_ROWS_HTML+=" + + + $DISK $MODEL · $CAPACITY · S/N: $SERIAL + + + + Overall health + $HEALTH${SECTOR_NOTE:+$SECTOR_NOTE} + + + Temperature + ${TEMP}°C + + + Power-on hours + ${POWER_ON} hrs + + + Wear leveling + $WEAR + + + Sectors (realloc / pending / uncorrectable) + $REALLOCATED / $PENDING / $UNCORRECTABLE + " +done + +if [ $SMART_ERR -gt 0 ]; then + add_section_row "SSD SMART Health" "ERROR" "$SMART_ERR drive(s) with uncorrectable sectors — immediate attention required" +elif [ $SMART_WARN -gt 0 ]; then + add_section_row "SSD SMART Health" "WARNING" "$SMART_WARN warning(s) — review drive details below" +else + add_section_row "SSD SMART Health" "OK" "All drives PASSED · no sector errors detected" +fi + +# ----------------------------------------------------------------------------- +# SECTION 3: FILESYSTEM USAGE +# ----------------------------------------------------------------------------- +log_section "Filesystem usage" + +FS_ROWS_HTML="" +FS_WARN=0 + +while IFS= read -r line; do + USE_PCT=$(echo "$line" | awk '{print $5}' | tr -d '%') + MOUNT_PT=$(echo "$line" | awk '{print $6}') + USED=$(echo "$line" | awk '{print $3}') + SIZE=$(echo "$line" | awk '{print $2}') + + if [ "$USE_PCT" -ge 90 ] 2>/dev/null; then + ROW_COLOUR="#fee2e2"; PCT_COLOUR="#991b1b" + FS_WARN=$((FS_WARN + 1)) + [[ "$OVERALL_STATUS" == "SUCCESS" ]] && OVERALL_STATUS="WARNING" + elif [ "$USE_PCT" -ge 75 ] 2>/dev/null; then + ROW_COLOUR="#fef3c7"; PCT_COLOUR="#92400e" + FS_WARN=$((FS_WARN + 1)) + [[ "$OVERALL_STATUS" == "SUCCESS" ]] && OVERALL_STATUS="WARNING" + else + ROW_COLOUR="transparent"; PCT_COLOUR="#065f46" + fi + + log "Filesystem $MOUNT_PT: ${USED} used of ${SIZE} (${USE_PCT}%)" + + FS_ROWS_HTML+=" + + $MOUNT_PT + $USED used of $SIZE + ${USE_PCT}% + " +done < <(df -h | grep -E '^/dev/') + +if [ $FS_WARN -gt 0 ]; then + add_section_row "Filesystem Usage" "WARNING" "$FS_WARN volume(s) above 75% — review recommended" +else + add_section_row "Filesystem Usage" "OK" "All volumes within normal range" +fi + +# ----------------------------------------------------------------------------- +# ROTATE OLD LOGS +# ----------------------------------------------------------------------------- +find "$LOG_DIR" -name "fstrim-*.log" -mtime +${KEEP_LOGS} -delete + +# ----------------------------------------------------------------------------- +# SYSTEM CONTEXT +# ----------------------------------------------------------------------------- +OMV_VER=$(dpkg -l openmediavault 2>/dev/null | awk '/^ii/{print $3}' || echo "Unknown") +KERNEL_VER=$(uname -r) +HOST_IP=$(hostname -I | awk '{print $1}') +UPTIME_STR=$(uptime -p) +LOAD_STR=$(uptime | awk -F'load average:' '{print $2}' | xargs) +MEM_USED=$(free -h | awk '/^Mem:/{print $3}') +MEM_TOTAL=$(free -h | awk '/^Mem:/{print $2}') +NEXT_RUN=$(systemctl show fstrim.timer --property=NextElapseUSecRealtime 2>/dev/null \ + | awk -F= '{print $2}' || echo "See: systemctl status fstrim.timer") + +log_section "Report complete — Overall: $OVERALL_STATUS" + +# ----------------------------------------------------------------------------- +# BUILD HTML EMAIL (style: KingDezigns monitor) +# ----------------------------------------------------------------------------- + +# Overall banner & badge +case "$OVERALL_STATUS" in + SUCCESS) + BADGE_HTML="

✅ SUCCESS

" + BANNER_HTML="✅ TRIM completed successfully — no action required." + ;; + WARNING) + BADGE_HTML="

⚠️ WARNING

" + BANNER_HTML="⚠️ TRIM completed with warnings — review recommended." + ;; + FAILED) + BADGE_HTML="

🚨 FAILED

" + BANNER_HTML="🚨 TRIM run failed — immediate attention required!" + ;; +esac + +HTML_BODY=$(cat < + + + + + +

+ + + + + + + $BANNER_HTML + + + + + + + + + + + + + + + + + + + +

+ + + + + +

SSD Maintenance Monitor

💽 $HOSTNAME_LABEL

$REPORT_TIME | Weekly TRIM · $WARNING_COUNT warning(s) · $ERROR_COUNT error(s)

$BADGE_HTML

+ 📋 Run Summary +

+ + + + + + + $SECTION_ROWS_HTML +

Section	Status	Detail

✂️ TRIM Results — $TOTAL_HUMAN Total Freed

+ + + + + + $TRIM_ROWS_HTML +

Mount point	Space freed

🔬 SSD SMART Health

+ + $SMART_ROWS_HTML +

💾 Filesystem Usage

+ + + + + + + $FS_ROWS_HTML +

Mount point	Usage	Used %

🖥️ System Info

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

OMV	${OMV_VER}	Kernel	${KERNEL_VER}

IP Address	${HOST_IP}	Uptime	${UPTIME_STR}

Load average	${LOAD_STR}	Memory	${MEM_USED} / ${MEM_TOTAL}

Log file	${LOG_FILE}

Next TRIM run	${NEXT_RUN}

+ SSD Maintenance Monitor | $HOSTNAME_LABEL | KingDezigns Infrastructure +

+ + +EOF +) + +# ----------------------------------------------------------------------------- +# SEND EMAIL +# ----------------------------------------------------------------------------- +SUBJECT="[TRIM] $HOSTNAME_LABEL — $OVERALL_STATUS | $REPORT_TIME" + +{ + echo "To: $REPORT_TO" + echo "From: $REPORT_FROM" + echo "Subject: $SUBJECT" + echo "MIME-Version: 1.0" + echo "Content-Type: text/html; charset=UTF-8" + echo "" + echo "$HTML_BODY" +} | sendmail -t + +log "Email sent to $REPORT_TO — Overall status: $OVERALL_STATUS" +log "===== fstrim-report.sh finished =====" diff --git a/omv/nas16-backup.sh b/omv/nas16-backup.sh new file mode 100644 index 0000000..e8be1ff --- /dev/null +++ b/omv/nas16-backup.sh @@ -0,0 +1,695 @@ +#!/bin/bash +# ============================================================================= +# NAS16 System Backup Script +# ============================================================================= +# Backs up OMV config, system /etc, web server configs (Apache/PHP), +# MariaDB/MySQL databases, Webmin config, and scheduled jobs. +# Writes a single dated .tar.gz archive to the NAS data drive. +# Retains the last 10 backups (~30 days), auto-removes older ones. +# Sends an HTML email report on completion (via OMV Postfix / Gmail relay). +# +# NOTE: Website files stored on NAS drives are NOT backed up here — +# they are covered separately via NAS-to-NAS redundancy. +# +# Schedule via OMV Scheduled Jobs (cron): +# 0 2 */3 * * (2:00 AM every 3 days) +# +# Backup destination: /export/kingdezignsnas-16/Backups/NAS16/ +# Script location: /usr/scripts/omv/nas16-backup.sh +# ============================================================================= + +set -uo pipefail + +# ----------------------------------------------------------------------------- +# CONFIGURATION +# ----------------------------------------------------------------------------- +HOSTNAME_LABEL=$(hostname -s 2>/dev/null || echo "NAS16") +REPORT_TO="rufus.king@kingdezigns.com" +REPORT_FROM="backup-monitor@nas16.local" # Display only — Postfix uses your Gmail relay + +BACKUP_ROOT="/export/kingdezignsnas-16/Backups/NAS16" +TMP_BASE="/export/kingdezignsnas-16/Backups/.tmp" +DATE=$(date +%Y-%m-%d) +TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S) +REPORT_TIME=$(date "+%Y-%m-%d %H:%M:%S") +ARCHIVE_NAME="nas16-backup-${DATE}.tar.gz" +TMP_DIR="${TMP_BASE}/nas16-backup-${TIMESTAMP}" +RETAIN_COUNT=10 +LOG_FILE="${BACKUP_ROOT}/backup.log" + +# MariaDB/MySQL connection — uses OS root auth (no password needed when run as root) +DB_BACKUP_DIR_NAME="databases" + +# ----------------------------------------------------------------------------- +# STATUS TRACKING +# Each section appends a row to SECTION_ROWS_HTML and sets OVERALL_STATUS. +# OVERALL_STATUS: SUCCESS | WARNING | FAILED +# ----------------------------------------------------------------------------- +OVERALL_STATUS="SUCCESS" +SECTION_ROWS_HTML="" +WARNING_COUNT=0 +ERROR_COUNT=0 + +# Append a result row to the section table +# Usage: add_section_row "Section name" "STATUS" "Detail message" +# STATUS: OK | WARNING | ERROR | SKIPPED +add_section_row() { + local section="$1" status="$2" detail="$3" + + case "$status" in + OK) + BADGE="✓ OK" + ;; + WARNING) + BADGE="⚠ WARNING" + WARNING_COUNT=$((WARNING_COUNT + 1)) + [[ "$OVERALL_STATUS" == "SUCCESS" ]] && OVERALL_STATUS="WARNING" + ;; + ERROR) + BADGE="✗ ERROR" + ERROR_COUNT=$((ERROR_COUNT + 1)) + OVERALL_STATUS="FAILED" + ;; + SKIPPED) + BADGE="— SKIPPED" + ;; + esac + + SECTION_ROWS_HTML+=" + + $section + $BADGE + $detail + " +} + +# ----------------------------------------------------------------------------- +# LOGGING +# ----------------------------------------------------------------------------- +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE" +} + +log_section() { + echo "" | tee -a "$LOG_FILE" + echo "------------------------------------------------------------" | tee -a "$LOG_FILE" + log "$1" + echo "------------------------------------------------------------" | tee -a "$LOG_FILE" +} + +# ----------------------------------------------------------------------------- +# PRE-FLIGHT CHECKS +# ----------------------------------------------------------------------------- +log_section "NAS16 Backup Starting" + +if [ ! -d "/export/kingdezignsnas-16" ]; then + log "ERROR: /export/kingdezignsnas-16 is not mounted or accessible. Aborting." + add_section_row "Pre-flight check" "ERROR" "/export/kingdezignsnas-16 not mounted or accessible" + + # Send failure email immediately and exit + OVERALL_STATUS="FAILED" + # (email block at bottom handles this via trap — see END section) + exit 1 +fi + +mkdir -p "$BACKUP_ROOT" +mkdir -p "$TMP_DIR" +log "Temporary working directory: $TMP_DIR" + +# ----------------------------------------------------------------------------- +# SECTION 1: OMV CONFIGURATION +# ----------------------------------------------------------------------------- +log_section "Backing up OMV configuration" + +OMV_DIR="${TMP_DIR}/omv-config" +mkdir -p "$OMV_DIR" +OMV_STATUS="OK" +OMV_DETAIL="" + +# omv-confbak is optional — if present, export the structured XML as a bonus. +# The raw config.xml copy below is the primary backup and is fully sufficient on its own. +if command -v omv-confbak &>/dev/null; then + if omv-confbak "${OMV_DIR}/omv-config.xml" 2>/dev/null; then + log "omv-confbak completed successfully" + OMV_DETAIL="omv-confbak exported + raw config.xml copied" + else + log "WARNING: omv-confbak failed or returned non-zero — raw config.xml still captured" + OMV_DETAIL="omv-confbak failed (non-fatal) + raw config.xml copied" + fi +else + log "omv-confbak not present — using raw config.xml (this is normal)" +fi + +if [ -f /etc/openmediavault/config.xml ]; then + cp /etc/openmediavault/config.xml "${OMV_DIR}/config.xml.raw" + log "Raw OMV config.xml copied" + [[ -z "$OMV_DETAIL" ]] && OMV_DETAIL="raw config.xml copied" +else + log "WARNING: /etc/openmediavault/config.xml not found — OMV config not backed up" + OMV_STATUS="WARNING" + OMV_DETAIL="config.xml not found at /etc/openmediavault/config.xml" +fi + +add_section_row "OMV Configuration" "$OMV_STATUS" "$OMV_DETAIL" + +# ----------------------------------------------------------------------------- +# SECTION 2: SYSTEM /etc +# ----------------------------------------------------------------------------- +log_section "Backing up /etc (system configuration)" + +ETC_DIR="${TMP_DIR}/etc" +mkdir -p "$ETC_DIR" + +if rsync -a --quiet \ + --exclude='/etc/mtab' \ + --exclude='/etc/fstab.d' \ + --exclude='/etc/blkid.tab' \ + /etc/ "${ETC_DIR}/" 2>/dev/null; then + log "/etc backed up successfully" + add_section_row "System /etc" "OK" "rsync completed successfully" +else + log "WARNING: /etc rsync encountered errors" + add_section_row "System /etc" "WARNING" "rsync completed with errors" +fi + +# ----------------------------------------------------------------------------- +# SECTION 3: APACHE CONFIGURATION +# ----------------------------------------------------------------------------- +log_section "Backing up Apache configuration" + +APACHE_DIR="${TMP_DIR}/apache" +mkdir -p "$APACHE_DIR" +APACHE_WARN=0 + +for DIR in /etc/apache2/sites-available /etc/apache2/sites-enabled \ + /etc/apache2/mods-enabled /etc/apache2/conf-available \ + /etc/apache2/conf-enabled /etc/apache2; do + if [ -d "$DIR" ]; then + rsync -a --quiet "$DIR/" "${APACHE_DIR}/$(basename "$DIR")/" \ + && log "Apache $(basename "$DIR") backed up" \ + || { log "WARNING: Apache $(basename "$DIR") rsync encountered errors"; APACHE_WARN=$((APACHE_WARN+1)); } + else + log "WARNING: $DIR not found — skipping" + APACHE_WARN=$((APACHE_WARN+1)) + fi +done + +if [ -f /etc/apache2/apache2.conf ]; then + cp /etc/apache2/apache2.conf "${APACHE_DIR}/apache2.conf" + log "apache2.conf copied" +fi + +if [ "$APACHE_WARN" -eq 0 ]; then + add_section_row "Apache Configuration" "OK" "All config directories backed up" +else + add_section_row "Apache Configuration" "WARNING" "$APACHE_WARN director(ies) missing or failed" +fi + +# ----------------------------------------------------------------------------- +# SECTION 4: PHP CONFIGURATION +# ----------------------------------------------------------------------------- +log_section "Backing up PHP configuration" + +PHP_DIR="${TMP_DIR}/php" +mkdir -p "$PHP_DIR" +PHP_FOUND=0 +PHP_VERSIONS="" +PHP_WARN=0 + +for PHP_VER_DIR in /etc/php/*/; do + if [ -d "$PHP_VER_DIR" ]; then + PHP_VER=$(basename "$PHP_VER_DIR") + rsync -a --quiet "$PHP_VER_DIR" "${PHP_DIR}/${PHP_VER}/" \ + && log "PHP ${PHP_VER} config backed up" \ + || { log "WARNING: PHP ${PHP_VER} rsync encountered errors"; PHP_WARN=$((PHP_WARN+1)); } + PHP_FOUND=1 + PHP_VERSIONS="${PHP_VERSIONS} ${PHP_VER}" + fi +done + +if [ "$PHP_FOUND" -eq 0 ]; then + log "WARNING: No PHP config directories found under /etc/php/ — skipping" + add_section_row "PHP Configuration" "SKIPPED" "No PHP installations found under /etc/php/" +elif [ "$PHP_WARN" -gt 0 ]; then + add_section_row "PHP Configuration" "WARNING" "Versions found:${PHP_VERSIONS} — $PHP_WARN version(s) had rsync errors" +else + add_section_row "PHP Configuration" "OK" "Versions backed up:${PHP_VERSIONS}" +fi + +# ----------------------------------------------------------------------------- +# SECTION 5: MARIADB / MYSQL DATABASES +# ----------------------------------------------------------------------------- +log_section "Backing up MariaDB/MySQL databases" + +DB_DIR="${TMP_DIR}/${DB_BACKUP_DIR_NAME}" +mkdir -p "$DB_DIR" +DB_COUNT=0 +DB_FAIL=0 + +if command -v mysqldump &>/dev/null; then + DB_CLIENT="mysqldump" +elif command -v mariadb-dump &>/dev/null; then + DB_CLIENT="mariadb-dump" +else + log "WARNING: mysqldump/mariadb-dump not found — skipping database backup" + DB_CLIENT="" +fi + +if [ -n "$DB_CLIENT" ]; then + DB_LIST=$(mysql --defaults-file=/etc/mysql/debian.cnf \ + -e "SHOW DATABASES;" 2>/dev/null \ + | grep -Ev "^(Database|information_schema|performance_schema|sys|mysql)$" \ + || true) + + if [ -z "$DB_LIST" ]; then + log "No user databases found — nothing to dump" + add_section_row "MariaDB / MySQL" "SKIPPED" "No user databases found" + else + while IFS= read -r DB_NAME; do + [ -z "$DB_NAME" ] && continue + $DB_CLIENT \ + --defaults-file=/etc/mysql/debian.cnf \ + --single-transaction \ + --routines \ + --triggers \ + --events \ + "$DB_NAME" > "${DB_DIR}/${DB_NAME}.sql" \ + && { log "Database dumped: ${DB_NAME}.sql"; DB_COUNT=$((DB_COUNT+1)); } \ + || { log "WARNING: dump failed for database: ${DB_NAME}"; DB_FAIL=$((DB_FAIL+1)); } + done <<< "$DB_LIST" + log "Total databases dumped: ${DB_COUNT}" + + gzip "${DB_DIR}"/*.sql 2>/dev/null && log "SQL dumps compressed" || true + + if [ "$DB_FAIL" -eq 0 ]; then + add_section_row "MariaDB / MySQL" "OK" "$DB_COUNT database(s) dumped and compressed" + else + add_section_row "MariaDB / MySQL" "WARNING" "$DB_COUNT dumped · $DB_FAIL failed" + fi + fi +else + add_section_row "MariaDB / MySQL" "SKIPPED" "mysqldump / mariadb-dump not found" +fi + +# ----------------------------------------------------------------------------- +# SECTION 6: WEBMIN CONFIGURATION +# ----------------------------------------------------------------------------- +log_section "Backing up Webmin configuration" + +WEBMIN_DIR="${TMP_DIR}/webmin" +mkdir -p "$WEBMIN_DIR" + +if [ -d /etc/webmin ]; then + if rsync -a --quiet /etc/webmin/ "${WEBMIN_DIR}/" 2>/dev/null; then + log "Webmin config backed up" + add_section_row "Webmin Configuration" "OK" "rsync from /etc/webmin completed" + else + log "WARNING: Webmin rsync encountered errors" + add_section_row "Webmin Configuration" "WARNING" "rsync completed with errors" + fi +else + log "WARNING: /etc/webmin not found — skipping (Webmin may not be installed)" + add_section_row "Webmin Configuration" "SKIPPED" "/etc/webmin not found — Webmin may not be installed" +fi + +# ----------------------------------------------------------------------------- +# SECTION 7: CRONTAB & SCHEDULED JOBS +# ----------------------------------------------------------------------------- +log_section "Backing up crontab and scheduled jobs" + +CRON_DIR="${TMP_DIR}/cron" +mkdir -p "$CRON_DIR" +CRON_WARN=0 + +crontab -l > "${CRON_DIR}/root-crontab.txt" 2>/dev/null \ + && log "Root crontab saved" \ + || { log "No root crontab found"; CRON_WARN=$((CRON_WARN+1)); } + +if [ -d /var/spool/cron/crontabs ]; then + cp -r /var/spool/cron/crontabs/ "${CRON_DIR}/crontabs/" \ + && log "All crontabs copied" \ + || { log "WARNING: crontabs copy failed"; CRON_WARN=$((CRON_WARN+1)); } +fi + +if [ -d /etc/cron.d ]; then + cp -r /etc/cron.d/ "${CRON_DIR}/cron.d/" \ + && log "cron.d copied" \ + || { log "WARNING: cron.d copy failed"; CRON_WARN=$((CRON_WARN+1)); } +fi + +if [ "$CRON_WARN" -eq 0 ]; then + add_section_row "Crontab / Scheduled Jobs" "OK" "root crontab, crontabs/, cron.d/ saved" +else + add_section_row "Crontab / Scheduled Jobs" "WARNING" "$CRON_WARN item(s) missing or failed to copy" +fi + +# ----------------------------------------------------------------------------- +# SECTION 8: BACKUP MANIFEST +# ----------------------------------------------------------------------------- +log_section "Generating backup manifest" + +APACHE_VER=$(apache2 -v 2>/dev/null | grep "Server version" | awk '{print $3}' || echo "Unknown") +PHP_VER_RUNTIME=$(php -r 'echo PHP_VERSION;' 2>/dev/null || echo "Unknown") +DB_VER=$(mysql --defaults-file=/etc/mysql/debian.cnf \ + -e "SELECT VERSION();" 2>/dev/null | tail -1 || echo "Unknown") +WEBMIN_VER=$(dpkg -l webmin 2>/dev/null | awk '/^ii/{print $3}' || echo "Unknown") +OMV_VER=$(dpkg -l openmediavault 2>/dev/null | awk '/^ii/{print $3}' || echo "Unknown") +KERNEL_VER=$(uname -r) +HOST_IP=$(hostname -I | awk '{print $1}') + +MANIFEST="${TMP_DIR}/MANIFEST.txt" +cat > "$MANIFEST" </dev/null; then + log "Archive created: ${BACKUP_ROOT}/${ARCHIVE_NAME}" + ARCHIVE_SIZE=$(du -sh "${BACKUP_ROOT}/${ARCHIVE_NAME}" | cut -f1) + log "Archive size: ${ARCHIVE_SIZE}" + add_section_row "Archive Creation" "OK" "${ARCHIVE_NAME} · ${ARCHIVE_SIZE}" +else + log "ERROR: Archive creation failed" + ARCHIVE_SIZE="N/A" + add_section_row "Archive Creation" "ERROR" "tar failed — archive not created" + rm -rf "$TMP_DIR" + OVERALL_STATUS="FAILED" +fi + +# ----------------------------------------------------------------------------- +# SECTION 10: CLEANUP TEMP FILES +# ----------------------------------------------------------------------------- +log_section "Cleaning up temporary files" + +if rm -rf "$TMP_DIR" 2>/dev/null; then + log "Temporary directory removed" +else + log "WARNING: Failed to remove temp directory: $TMP_DIR" +fi +rmdir "$TMP_BASE" 2>/dev/null && log ".tmp directory removed (empty)" || true + +# ----------------------------------------------------------------------------- +# SECTION 11: ROTATE OLD BACKUPS +# ----------------------------------------------------------------------------- +log_section "Rotating old backups (keeping last ${RETAIN_COUNT})" + +BACKUP_LIST=$(ls -1t "${BACKUP_ROOT}"/nas16-backup-*.tar.gz 2>/dev/null || true) +BACKUP_COUNT=$(echo "$BACKUP_LIST" | grep -c . || true) +DELETED_COUNT=0 + +if [ "$BACKUP_COUNT" -gt "$RETAIN_COUNT" ]; then + DELETE_LIST=$(echo "$BACKUP_LIST" | tail -n +$((RETAIN_COUNT + 1))) + while IFS= read -r OLD_BACKUP; do + rm -f "$OLD_BACKUP" + log "Removed old backup: $(basename "$OLD_BACKUP")" + DELETED_COUNT=$((DELETED_COUNT+1)) + done <<< "$DELETE_LIST" + add_section_row "Backup Rotation" "OK" "Kept $RETAIN_COUNT most recent · removed $DELETED_COUNT old archive(s)" +else + log "Backup count (${BACKUP_COUNT}) within retention limit — no rotation needed" + add_section_row "Backup Rotation" "OK" "$BACKUP_COUNT / $RETAIN_COUNT slots used — no rotation needed" +fi + +# ----------------------------------------------------------------------------- +# DONE — log summary +# ----------------------------------------------------------------------------- +log_section "Backup Complete" +log "Archive: ${BACKUP_ROOT}/${ARCHIVE_NAME}" +log "Size: ${ARCHIVE_SIZE:-N/A}" +log "Status: ${OVERALL_STATUS}" +log "" + +# ----------------------------------------------------------------------------- +# BUILD & SEND HTML EMAIL (pattern: nas08_zfs_scrub.sh) +# ----------------------------------------------------------------------------- + +# Overall banner & badge +case "$OVERALL_STATUS" in + SUCCESS) + BADGE_HTML="

✅ SUCCESS

" + BANNER_HTML="✅ Backup completed successfully — no action required." + ;; + WARNING) + BADGE_HTML="

⚠️ WARNING

" + BANNER_HTML="⚠️ Backup completed with warnings — review recommended." + ;; + FAILED) + BADGE_HTML="

🚨 FAILED

" + BANNER_HTML="🚨 Backup failed — immediate attention required!" + ;; +esac + +# Archive info card — shown only when archive was created +if [[ "${ARCHIVE_SIZE:-N/A}" != "N/A" ]]; then + ARCHIVE_CARD=" + +

🗜️ Archive

+ + + + + + + + +

Filename

$ARCHIVE_NAME

Size

$ARCHIVE_SIZE

Retained

$BACKUP_COUNT / $RETAIN_COUNT

+ " +else + ARCHIVE_CARD="" +fi + +# Software versions card +VERSIONS_CARD=" + +

🖥️ System Versions

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +

OMV	${OMV_VER}	Kernel	${KERNEL_VER}

Apache	${APACHE_VER}	PHP	${PHP_VER_RUNTIME}

MariaDB / MySQL	${DB_VER}	Webmin	${WEBMIN_VER}

IP Address	${HOST_IP}

+ " + +# Compose full HTML email +HTML_BODY=$(cat < + + + + + +

+ + + + + + + $BANNER_HTML + + + + + + $ARCHIVE_CARD + + + $VERSIONS_CARD + + + + + + + +

+ + + + + +

System Backup Monitor

💾 $HOSTNAME_LABEL

$REPORT_TIME | 7 sections · $WARNING_COUNT warning(s) · $ERROR_COUNT error(s)

$BADGE_HTML

+ 📋 Backup Section Results +

+ + + + + + + $SECTION_ROWS_HTML +

Section	Status	Detail

📊 Run Summary

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Host	$HOSTNAME_LABEL

Run Time	$REPORT_TIME

Archive	${ARCHIVE_NAME}

Archive Size	${ARCHIVE_SIZE:-N/A}

Backups on Disk	$BACKUP_COUNT of $RETAIN_COUNT max retained

Warnings	$WARNING_COUNT

Errors	$ERROR_COUNT

Overall Status	$OVERALL_STATUS

+ Automated Backup Monitor | $HOSTNAME_LABEL | KingDezigns Infrastructure +

+ + +EOF +) + +# Send email via OMV Postfix (Gmail relay) +SUBJECT="[Backup] $HOSTNAME_LABEL — $OVERALL_STATUS | $REPORT_TIME" + +{ + echo "To: $REPORT_TO" + echo "From: $REPORT_FROM" + echo "Subject: $SUBJECT" + echo "MIME-Version: 1.0" + echo "Content-Type: text/html; charset=UTF-8" + echo "" + echo "$HTML_BODY" +} | sendmail -t + +log "Email sent to $REPORT_TO — Overall status: $OVERALL_STATUS" +log "===== NAS16 Backup script finished =====" diff --git a/zfs/NAS16-zfs_pools.html b/zfs/NAS16-zfs_pools.html new file mode 100644 index 0000000..651f00e --- /dev/null +++ b/zfs/NAS16-zfs_pools.html @@ -0,0 +1,510 @@ + + + + + +KingDezigns — NAS16 ZFS Pool Maintenance Scripts + + + + + +

+ +

KingDezigns — NAS16 ZFS Pool Maintenance Scripts

+ 📅 2026-05-12 + 🖥 NAS16 / Raspberry Pi 5 + ⚙️ OpenMediaVault (OMV) + 🌐 Apache · PHP · MariaDB · Webmin + 💾 ZFS RAIDZ2 — Penta SATA HAT +

+ + + +

The goal

+ +

+ ZFS Scrubbing + ZFS Reporting +

+ +

The goal is to create regular maintenance to the ZFS pools for this NAS. This is done to protect and keep the pools healthy.

+ +

⚠ Critical — store this document off NAS16

Keep a copy of this document on your workstation, a USB drive, or in print. If NAS16 is down you cannot read files stored on it.

+ + + + + + +

Script to Create ZFS report

+ +

1 Create the folders on NAS16

This script gets the current ZFS pool status and reports it back to the administrator via email. Use these commands to setup the scripts on the NAS.

ssh rufusking@192.168.150.40
+sudo mkdir /usr/scripts/zfs
+sudo nano nas16_zfs_report.sh

+ +

2 Copy the content of the entire nas16_zfs_report.sh script and paste it in the terminal window. Save (CTRL + X) and commit (Y) the changes

nas16_zfs_report.sh

+ +

3 Make the script executable — NAS16

chmod +x /usr/scripts/zfs/nas16_zfs_report.sh

+ +

4 Run a test report — NAS16

sudo /usr/scripts/zfs/nas16_zfs_report.sh

Watch for the report in your inbox

+ + +

Script to perform a ZFS pool scrub

+ +

1 Create the folders on NAS16

This script initiates a ZFS pool scrub and reports back the status to the administrator via email. Use these commands to setup this script on the NAS.

ssh rufusking@192.168.150.40
+sudo mkdir /usr/scripts/zfs
+sudo nano nas16_zfs_scrub.sh

+ +

2 Copy the content of the entire nas16_zfs_scrub.sh script and paste it in the terminal window. Save (CTRL + X) and commit (Y) the changes

nas16_zfs_scrub.sh

+ +

3 Make the script executable — NAS16

chmod +x /usr/scripts/zfs/nas16_zfs_scrub.sh

+ +

4 Run a test report — NAS16

sudo /usr/scripts/zfs/nas16_zfs_report.sh

Watch for the report in your inbox

+ + + +

Key differences from NAS08

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Area	NAS08	NAS16
Primary role	Docker containers (Pi-hole, Plex, Vaultwarden, Nextcloud, NPM)	Web server (Apache/PHP), databases (MariaDB), Webmin admin
What's backed up beyond /etc + OMV	Docker Compose files, Pi-hole data, Plex config/metadata	Apache vhosts, PHP config, MariaDB dumps, Webmin config
Website files	N/A	Intentionally excluded — stored on NAS drives, covered by redundancy
Database backup	None (no databases)	Full mysqldump of all user databases, .sql.gz per database
Backup destination	/export/kingdezigns-all/Backups/NAS08/	/export/kingdezignsnas-16/Backups/NAS16/
ZFS pool name	kingdezignsnas	kingdezignsnas-16 (assumed — verify with `sudo zpool list`)
Recovery phases	9 phases, 34 steps	10 phases, 40 steps

+ + + +

Notes & assumptions to verify

+ +

⚠ Verify these before first run

The script was written based on information provided and follows Raspberry Pi OS / Debian conventions. Verify the following on NAS16 before treating any backup as production-ready.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Item	Assumed value	How to verify
ZFS pool name	`kingdezignsnas-16`	`sudo zpool list`
Backup destination path	`/export/kingdezignsnas-16/Backups/NAS16/`	`ls /export/` — confirm share name matches
MariaDB auth method	Uses `/etc/mysql/debian.cnf` (maintenance account, no password when run as root)	Run: `sudo mysql --defaults-file=/etc/mysql/debian.cnf -e "SHOW DATABASES;"` — should work without password
PHP version(s)	Auto-detected from `/etc/php/*/`	`php -v` and `ls /etc/php/`
Apache config location	`/etc/apache2/` (standard Debian)	`apache2 -V` — confirms config path
Webmin config location	`/etc/webmin/`	`ls /etc/webmin/`

+ + + +

Network map — items to update

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Item	Old value	New value
NAS16 — Backup status	`None`	`Automated — every 3 days — 30-day retention`
NAS16 — Backup destination	`None`	`/export/kingdezignsnas-16/Backups/NAS16/`
NAS16 — Backup script	`None`	`/usr/scripts/omv/nas16-backup.sh`
NAS16 — PCIe requirement	`not documented`	`dtparam=pciex1 + dtparam=pciex1_gen=3 required in /boot/firmware/config.txt on fresh OS`
NAS16 — Web stack	`not documented`	`Apache · PHP · MariaDB · Webmin · Adminer`
NAS16 — Recovery plan	`None`	`NAS16-Backup-Summary.html — 40 steps across 10 phases + ZFS troubleshooting`

+ + + + +

+ + diff --git a/zfs/nas16_zfs_report.sh b/zfs/nas16_zfs_report.sh new file mode 100644 index 0000000..f2b3d32 --- /dev/null +++ b/zfs/nas16_zfs_report.sh @@ -0,0 +1,351 @@ +#!/usr/bin/env bash +# ============================================================================= +# ZFS Pool Health Report — NAS16 +# Sends an HTML status email to rufus.king@kingdezigns.com +# Cron example: 0 7 * * * /opt/scripts/zfs_report_nas16.sh +# +# Commands executed per pool: +# zpool status -v — full status + verbose per-file error listing +# zpool list — pool-level size / capacity / health +# zfs list -r — per-dataset breakdown (used, avail, refer, mount) +# ============================================================================= + +set -euo pipefail + +# ── Configuration ───────────────────────────────────────────────────────────── +TARGET_HOST="NAS16" +REPORT_TO="rufus.king@kingdezigns.com" +REPORT_FROM="zfs-monitor@${TARGET_HOST,,}.local" +SCRUB_MAX_AGE_DAYS=8 # Warn if last scrub is older than this +CAP_WARN=80 # Capacity % that triggers WARNING +CAP_CRIT=90 # Capacity % that triggers CRITICAL +# ────────────────────────────────────────────────────────────────────────────── + +NOW=$(date '+%Y-%m-%d %H:%M:%S') +HOSTNAME_REAL=$(hostname -s 2>/dev/null || echo "$TARGET_HOST") + +# ── HTML escape helper ──────────────────────────────────────────────────────── +html_escape() { sed 's/&/\&/g; s//\>/g; s/"/\"/g'; } + +# ── Collect pool list ───────────────────────────────────────────────────────── +POOL_LIST=$(zpool list -H -o name 2>/dev/null) || { echo "ERROR: zpool not found"; exit 1; } + +OVERALL_STATUS="HEALTHY" +POOL_BLOCKS="" +POOL_COUNT=0 +PROBLEM_COUNT=0 + +# ── Per-pool loop ───────────────────────────────────────────────────────────── +while IFS= read -r POOL; do + [[ -z "$POOL" ]] && continue + POOL_COUNT=$((POOL_COUNT + 1)) + + # 1. zpool status -v ── verbose: includes per-file permanent error listing + STATUS_RAW=$(zpool status -v "$POOL" 2>&1) + + # 2. zpool list ── raw (-p) for math, human for display + ZPOOL_RAW=$(zpool list -H -p -o name,size,alloc,free,frag,cap,health "$POOL" 2>&1) + ZPOOL_HUM=$(zpool list -H -o name,size,alloc,free,frag,cap,health "$POOL" 2>&1) + + HEALTH=$(echo "$ZPOOL_RAW" | awk '{print $7}') + CAP_RAW=$(echo "$ZPOOL_RAW" | awk '{print $6}') + + SIZE_H=$(echo "$ZPOOL_HUM" | awk '{print $2}') + ALLOC_H=$(echo "$ZPOOL_HUM" | awk '{print $3}') + FREE_H=$(echo "$ZPOOL_HUM" | awk '{print $4}') + FRAG_H=$(echo "$ZPOOL_HUM" | awk '{print $5}') + CAP_H=$(echo "$ZPOOL_HUM" | awk '{print $6}') + + # 3. zfs list -r ── per-dataset: name, used, avail, refer, mountpoint + ZFS_LIST_RAW=$(zfs list -r -H -o name,used,avail,refer,mountpoint "$POOL" 2>&1) + + # Build dataset table rows + DATASET_ROWS="" + DATASET_COUNT=0 + while IFS=$'\t' read -r DS_NAME DS_USED DS_AVAIL DS_REFER DS_MOUNT; do + [[ -z "$DS_NAME" ]] && continue + DATASET_COUNT=$((DATASET_COUNT + 1)) + (( DATASET_COUNT % 2 == 0 )) && ROW_SHADE="#f9fafb" || ROW_SHADE="white" + DS_NAME_ESC=$(echo "$DS_NAME" | html_escape) + DS_MOUNT_ESC=$(echo "$DS_MOUNT" | html_escape) + DATASET_ROWS+=" + + ${DS_NAME_ESC} + ${DS_USED} + ${DS_AVAIL} + ${DS_REFER} + ${DS_MOUNT_ESC} + " + done <<< "$ZFS_LIST_RAW" + + DATASET_TABLE=" +

▶ Datasets / Filesystems — zfs list -r (${DATASET_COUNT} entries)

+ + + + + + + + + + + ${DATASET_ROWS} + +

Dataset	Used	Available	Referenced	Mountpoint

" + + # ── Scrub parse ──────────────────────────────────────────────────────────── + SCRUB_LINE=$(echo "$STATUS_RAW" | grep -E "scan:|scrub" | head -1 || true) + if echo "$SCRUB_LINE" | grep -q "scrub repaired"; then + LAST_SCRUB=$(echo "$SCRUB_LINE" | grep -oP '\w{3} \w{3} +\d+ \d+:\d+:\d+ \d{4}' | head -1 || echo "unknown") + ERRORS_SCRUB=$(echo "$SCRUB_LINE" | grep -oP '\d+ errors' | head -1 || echo "0 errors") + SCRUB_STATUS="Completed" + elif echo "$SCRUB_LINE" | grep -q "in progress"; then + LAST_SCRUB="In Progress"; SCRUB_STATUS="Running"; ERRORS_SCRUB="—" + elif echo "$SCRUB_LINE" | grep -q "none requested"; then + LAST_SCRUB="Never"; SCRUB_STATUS="Never Run"; ERRORS_SCRUB="—" + else + LAST_SCRUB=$(echo "$SCRUB_LINE" | sed 's/.*on //' | sed 's/ with.*//' || echo "unknown") + SCRUB_STATUS="Completed" + ERRORS_SCRUB=$(echo "$SCRUB_LINE" | grep -oP '\d+ errors' | head -1 || echo "0 errors") + fi + + # ── Severity logic ───────────────────────────────────────────────────────── + POOL_SEVERITY="ok" + POOL_LABEL="HEALTHY" + + if [[ "$HEALTH" != "ONLINE" ]]; then + POOL_SEVERITY="critical"; POOL_LABEL="$HEALTH" + OVERALL_STATUS="CRITICAL"; PROBLEM_COUNT=$((PROBLEM_COUNT + 1)) + fi + + CAP_NUM=${CAP_RAW//%/}; CAP_NUM=${CAP_NUM%%.*} + if [[ "$CAP_NUM" =~ ^[0-9]+$ ]]; then + if (( CAP_NUM >= CAP_CRIT )); then + POOL_SEVERITY="critical"; POOL_LABEL="CRITICAL — CAPACITY ${CAP_H}" + OVERALL_STATUS="CRITICAL" + [[ "$HEALTH" == "ONLINE" ]] && PROBLEM_COUNT=$((PROBLEM_COUNT + 1)) + elif (( CAP_NUM >= CAP_WARN )); then + [[ "$POOL_SEVERITY" == "ok" ]] && POOL_SEVERITY="warning" + [[ "$POOL_LABEL" == "HEALTHY" ]] && POOL_LABEL="WARNING — CAPACITY ${CAP_H}" + [[ "$OVERALL_STATUS" == "HEALTHY" ]] && OVERALL_STATUS="WARNING" + fi + fi + + if [[ "$LAST_SCRUB" == "Never" ]]; then + [[ "$POOL_SEVERITY" == "ok" ]] && POOL_SEVERITY="warning" + [[ "$POOL_LABEL" == "HEALTHY" ]] && POOL_LABEL="WARNING — NO SCRUB" + [[ "$OVERALL_STATUS" == "HEALTHY" ]] && OVERALL_STATUS="WARNING" + fi + + case "$POOL_SEVERITY" in + ok) BADGE_BG="#1a7f4b"; BADGE_TEXT="white"; ROW_BG="#f0fdf4" ;; + warning) BADGE_BG="#b45309"; BADGE_TEXT="white"; ROW_BG="#fffbeb" ;; + critical) BADGE_BG="#b91c1c"; BADGE_TEXT="white"; ROW_BG="#fff1f2" ;; + esac + + # ── Error blocks ─────────────────────────────────────────────────────────── + # a) Pool/vdev state errors + ERROR_LINES=$(echo "$STATUS_RAW" \ + | grep -E "(DEGRADED|FAULTED|OFFLINE|REMOVED|UNAVAIL|errors:)" \ + | grep -v "errors: No known data errors" || true) + + # b) Permanent per-file errors exposed by -v flag + VERBOSE_ERRORS=$(echo "$STATUS_RAW" \ + | awk '/Permanent errors have been detected/,0' \ + | grep -v "^$" || true) + + ERROR_HTML="" + if [[ -n "$ERROR_LINES" ]]; then + ERROR_LINES_ESC=$(echo "$ERROR_LINES" | head -30 | html_escape) + ERROR_HTML+=" +

⚠ Pool / vdev Errors

${ERROR_LINES_ESC}

" + fi + + if [[ -n "$VERBOSE_ERRORS" ]]; then + VERBOSE_ESC=$(echo "$VERBOSE_ERRORS" | head -50 | html_escape) + ERROR_HTML+=" +

📄 Affected Files — zpool status -v

${VERBOSE_ESC}

" + fi + + # ── vdev config tree ─────────────────────────────────────────────────────── + VDEV_TREE=$(echo "$STATUS_RAW" \ + | sed -n '/config:/,/errors:/p' \ + | grep -v "^$" | head -60 | html_escape || true) + + # ── Assemble pool card ───────────────────────────────────────────────────── + POOL_BLOCKS+=" +

+ +

+ + + +

📉 ${POOL} + pool

${POOL_LABEL}

+ +

+ + + + + + + + + + +

Total Size

${SIZE_H}

Used

${ALLOC_H}

Free

${FREE_H}

Capacity

${CAP_H}

Fragmentation

${FRAG_H}

Health

${HEALTH}

+ +

+ 🔍 Last Scrub + ${SCRUB_STATUS} | ${LAST_SCRUB} | Errors: ${ERRORS_SCRUB} +

+ + ${ERROR_HTML} + +

▶ vdev / Drive Configuration — zpool status -v

${VDEV_TREE}

+ + ${DATASET_TABLE} + +

" + +done <<< "$POOL_LIST" + +# ── Overall banner ───────────────────────────────────────────────────────────── +case "$OVERALL_STATUS" in + HEALTHY) BANNER_BG="#1a7f4b"; BANNER_ICON="✅"; BANNER_MSG="All pools are healthy — no action required." ;; + WARNING) BANNER_BG="#b45309"; BANNER_ICON="⚠️"; BANNER_MSG="One or more pools require your attention." ;; + CRITICAL) BANNER_BG="#b91c1c"; BANNER_ICON="🚨"; BANNER_MSG="CRITICAL issue detected — immediate action required!" ;; +esac + +SUBJECT="[ZFS] ${TARGET_HOST} — ${OVERALL_STATUS} — ${NOW}" + +# ── Build HTML email ─────────────────────────────────────────────────────────── +HTML=$(cat < + + + + + +

+ + + + + + + + + + + + +

+ + + +

ZFS Pool Health Monitor

🔌 ${TARGET_HOST}

${NOW} | ${POOL_COUNT} pool(s) monitored

${BANNER_ICON} ${OVERALL_STATUS}

+ ${BANNER_ICON} ${BANNER_MSG} +

${POOL_BLOCKS}

📋 Report Summary

+ + + + + + + + + + + + + + + +

Host	${TARGET_HOST} (${HOSTNAME_REAL})
Report Time	${NOW}
Pools Checked	${POOL_COUNT}
Pools with Issues	${PROBLEM_COUNT}
Capacity Warning Threshold	≥ ${CAP_WARN}%
Capacity Critical Threshold	≥ ${CAP_CRIT}%
Scrub Age Warning	> ${SCRUB_MAX_AGE_DAYS} days

Commands Executed


+            zpool status -v <pool>

+            zpool list -H -o name,size,alloc,free,frag,cap,health <pool>

+            zfs list -r -H -o name,used,avail,refer,mountpoint <pool>
+

+ Automated ZFS Monitor | ${TARGET_HOST} | KingDezigns Infrastructure +

+ + +EOF +) + +# ── Send email ───────────────────────────────────────────────────────────────── +send_email() { + if command -v sendmail &>/dev/null; then + { echo "To: ${REPORT_TO}"; echo "From: ${REPORT_FROM}"; echo "Subject: ${SUBJECT}" + echo "MIME-Version: 1.0"; echo "Content-Type: text/html; charset=UTF-8"; echo "" + echo "$HTML"; } | sendmail -t + echo "Report sent via sendmail to ${REPORT_TO}" + elif command -v mailx &>/dev/null; then + echo "$HTML" | mailx -a "Content-Type: text/html" -s "$SUBJECT" "$REPORT_TO" + echo "Report sent via mailx to ${REPORT_TO}" + elif command -v mail &>/dev/null; then + echo "$HTML" | mail -a "Content-Type: text/html; charset=UTF-8" -s "$SUBJECT" "$REPORT_TO" + echo "Report sent via mail to ${REPORT_TO}" + else + echo "ERROR: No mail transport found. Install sendmail, mailx, or msmtp." + FALLBACK="/tmp/zfs_report_${TARGET_HOST}_$(date +%Y%m%d_%H%M%S).html" + echo "$HTML" > "$FALLBACK" + echo "HTML report saved to: $FALLBACK" + exit 1 + fi +} + +send_email +echo "[$(date '+%Y-%m-%d %H:%M:%S')] ZFS report for ${TARGET_HOST} — Status: ${OVERALL_STATUS} — Pools: ${POOL_COUNT} — Problems: ${PROBLEM_COUNT}" diff --git a/zfs/nas16_zfs_scrub-1.sh b/zfs/nas16_zfs_scrub-1.sh new file mode 100644 index 0000000..917a1c8 --- /dev/null +++ b/zfs/nas16_zfs_scrub-1.sh @@ -0,0 +1,322 @@ +#!/bin/bash +# ============================================================================= +# ZFS Pool Scrub & Email Report — NAS16 +# Location: /usr/scripts/zfs/nas16_zfs_scrub.sh +# Schedule via OMV Scheduler (cron) +# ============================================================================= + +# ── Configuration ───────────────────────────────────────────────────────────── +HOSTNAME="NAS16" +REPORT_TO="rufus.king@kingdezigns.com" +REPORT_FROM="zfs-monitor@nas16.local" # Display only — Postfix uses your Gmail relay +SCRUB_WAIT_SECONDS=28800 # Max wait per pool scrub (8 hrs for large pools) +CAPACITY_WARN_THRESHOLD=80 # % capacity to flag as WARNING +SCRUB_AGE_WARN_DAYS=8 # Days since last scrub before flagging stale +LOG_FILE="/var/log/zfs_scrub_nas16.log" +# Note: Email is sent via OMV-configured Postfix (Gmail relay). No SMTP config needed here. +# ────────────────────────────────────────────────────────────────────────────── + +REPORT_TIME=$(date "+%Y-%m-%d %H:%M:%S") +POOLS=$(zpool list -H -o name) +POOL_COUNT=$(echo "$POOLS" | wc -l) + +log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; } + +log "===== ZFS Scrub started on $HOSTNAME =====" + +# ── Run scrubs sequentially ─────────────────────────────────────────────────── +for pool in $POOLS; do + log "Starting scrub on pool: $pool" + zpool scrub "$pool" + + ELAPSED=0 + while true; do + SCRUB_STATE=$(zpool status "$pool" | grep -E "scan:" | awk '{print $2}') + [[ "$SCRUB_STATE" == "scrub" ]] || break + sleep 30 + ELAPSED=$((ELAPSED + 30)) + if [[ $ELAPSED -ge $SCRUB_WAIT_SECONDS ]]; then + log "WARNING: Scrub on $pool exceeded wait time. Moving on." + break + fi + done + log "Scrub complete (or timed out) for pool: $pool" +done + +# ── Gather data & build pool cards ─────────────────────────────────────────── + +OVERALL_STATUS="HEALTHY" # HEALTHY | WARNING | CRITICAL +POOLS_WITH_ISSUES=0 +POOL_CARDS_HTML="" + +for pool in $POOLS; do + STATUS_RAW=$(zpool list -H -o health "$pool") + SIZE=$(zpool list -H -o size "$pool") + USED=$(zpool list -H -o alloc "$pool") + FREE=$(zpool list -H -o free "$pool") + CAP=$(zpool list -H -o cap "$pool" | tr -d '%') + FRAG=$(zpool list -H -o frag "$pool") + SCRUB_LINE=$(zpool status "$pool" | grep -E "scan:") + ERRORS_LINE=$(zpool status "$pool" | grep "errors:") + CONFIG_BLOCK=$(zpool status "$pool" | awk '/config:/,/errors:/' | head -n -1) + FAULTED_LINES=$(echo "$CONFIG_BLOCK" | grep -E "FAULTED|DEGRADED|UNAVAIL|REMOVED" | grep -v "^$") + + # Per-pool severity + POOL_SEVERITY="HEALTHY" + ISSUE_DETAIL="" + + if [[ "$STATUS_RAW" == "DEGRADED" || "$STATUS_RAW" == "FAULTED" || "$STATUS_RAW" == "UNAVAIL" ]]; then + POOL_SEVERITY="CRITICAL" + OVERALL_STATUS="CRITICAL" + ISSUE_DETAIL="$FAULTED_LINES" + elif [[ "$CAP" -ge "$CAPACITY_WARN_THRESHOLD" ]]; then + POOL_SEVERITY="WARNING" + [[ "$OVERALL_STATUS" != "CRITICAL" ]] && OVERALL_STATUS="WARNING" + fi + + # Scrub error count + SCRUB_ERRORS=$(echo "$SCRUB_LINE" | grep -oP '\d+ errors' | head -1) + [[ -z "$SCRUB_ERRORS" ]] && SCRUB_ERRORS="0 errors" + + # Scrub date + SCRUB_DATE=$(echo "$SCRUB_LINE" | grep -oP '[A-Z][a-z]{2} \d+ \d+:\d+:\d+ \d{4}' | head -1) + [[ -z "$SCRUB_DATE" ]] && SCRUB_DATE="No scrub data" + + # Scrub age warning + SCRUB_AGE_NOTE="" + if [[ -n "$SCRUB_DATE" && "$SCRUB_DATE" != "No scrub data" ]]; then + SCRUB_EPOCH=$(date -d "$SCRUB_DATE" +%s 2>/dev/null) + NOW_EPOCH=$(date +%s) + DIFF_DAYS=$(( (NOW_EPOCH - SCRUB_EPOCH) / 86400 )) + if [[ $DIFF_DAYS -ge $SCRUB_AGE_WARN_DAYS ]]; then + SCRUB_AGE_NOTE=" — ⚠️ Last scrub was ${DIFF_DAYS} days ago" + [[ "$POOL_SEVERITY" == "HEALTHY" ]] && POOL_SEVERITY="WARNING" + [[ "$OVERALL_STATUS" != "CRITICAL" ]] && OVERALL_STATUS="WARNING" + fi + fi + + [[ "$POOL_SEVERITY" != "HEALTHY" ]] && POOLS_WITH_ISSUES=$((POOLS_WITH_ISSUES + 1)) + + # Colors + case "$STATUS_RAW" in + ONLINE) HEALTH_COLOR="#1a7f4b" ;; + DEGRADED) HEALTH_COLOR="#b91c1c" ;; + FAULTED) HEALTH_COLOR="#b91c1c" ;; + *) HEALTH_COLOR="#b45309" ;; + esac + + case "$POOL_SEVERITY" in + CRITICAL) CARD_BORDER="#fca5a5"; CARD_HEADER_BG="#fff1f2"; POOL_BADGE_BG="#b91c1c"; POOL_BADGE_LABEL="$STATUS_RAW" ;; + WARNING) CARD_BORDER="#fcd34d"; CARD_HEADER_BG="#fffbeb"; POOL_BADGE_BG="#b45309"; POOL_BADGE_LABEL="WARNING" ;; + *) CARD_BORDER="#86efac"; CARD_HEADER_BG="#f0fdf4"; POOL_BADGE_BG="#1a7f4b"; POOL_BADGE_LABEL="HEALTHY" ;; + esac + + CAP_COLOR="#111827" + [[ "$CAP" -ge "$CAPACITY_WARN_THRESHOLD" ]] && CAP_COLOR="#b45309" + [[ "$CAP" -ge 90 ]] && CAP_COLOR="#b91c1c" + + # Issue block + ISSUE_BLOCK="" + if [[ -n "$ISSUE_DETAIL" ]]; then + ESCAPED_DETAIL=$(echo "$ISSUE_DETAIL" | sed 's//\>/g') + ISSUE_BLOCK=" + +

$ESCAPED_DETAIL

+ " + fi + + # Config block + ESCAPED_CONFIG=$(echo "$CONFIG_BLOCK" | sed 's//\>/g') + + POOL_CARDS_HTML+=" +

+ + +

+ + + + + +

+ 🗄️ $pool + pool +

+ $POOL_BADGE_LABEL +

+ + +

+ + + + + + + + + + + + + + + $ISSUE_BLOCK +

Total

$SIZE

Used

$USED

Free

$FREE

Capacity

${CAP}%

Frag

$FRAG

Health

$STATUS_RAW

+ + +

+ 🔍 Last Scrub + Completed | $SCRUB_DATE | $SCRUB_ERRORS${SCRUB_AGE_NOTE} +

+ + +

▶ vdev / drive configuration

$ESCAPED_CONFIG

" +done + +# ── Overall banner & badge ──────────────────────────────────────────────────── +case "$OVERALL_STATUS" in + CRITICAL) + BADGE_HTML="

🚨 CRITICAL

" + BANNER_HTML="🚨 CRITICAL issue detected — immediate action required!" + ISSUES_COLOR="#b91c1c" + ;; + WARNING) + BADGE_HTML="

⚠️ WARNING

" + BANNER_HTML="⚠️ Warning condition detected — review recommended." + ISSUES_COLOR="#b45309" + ;; + *) + BADGE_HTML="

✅ ALL HEALTHY

" + BANNER_HTML="✅ All pools are healthy — no action required." + ISSUES_COLOR="#1a7f4b" + ;; +esac + +# ── Compose full HTML email ─────────────────────────────────────────────────── +HTML_BODY=$(cat < + + + + + +

+ + + + + + + $BANNER_HTML + + + + + + + + + + +

+ + + + + +

ZFS Pool Health Monitor

🔌 $HOSTNAME

$REPORT_TIME | $POOL_COUNT pool(s) monitored

$BADGE_HTML

+ $POOL_CARDS_HTML +

📊 Report Summary

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Host	$HOSTNAME

Report Time	$REPORT_TIME

Pools Checked	$POOL_COUNT

Pools with Issues	$POOLS_WITH_ISSUES

Scrub Age Warning Threshold	$SCRUB_AGE_WARN_DAYS days

Capacity Warning Threshold	${CAPACITY_WARN_THRESHOLD}%

Overall Status	$OVERALL_STATUS

+ Automated ZFS Monitor | $HOSTNAME | KingDezigns Infrastructure +

+ + +EOF +) + +# ── Send email via OMV Postfix (Gmail relay) ────────────────────────────────── +SUBJECT="[ZFS] $HOSTNAME — $OVERALL_STATUS | $REPORT_TIME" + +{ + echo "To: $REPORT_TO" + echo "From: $REPORT_FROM" + echo "Subject: $SUBJECT" + echo "MIME-Version: 1.0" + echo "Content-Type: text/html; charset=UTF-8" + echo "" + echo "$HTML_BODY" +} | sendmail -t + +log "Email sent to $REPORT_TO — Overall status: $OVERALL_STATUS" +log "===== ZFS Scrub complete on $HOSTNAME =====" diff --git a/zfs/nas16_zfs_scrub.sh b/zfs/nas16_zfs_scrub.sh new file mode 100644 index 0000000..0101bba --- /dev/null +++ b/zfs/nas16_zfs_scrub.sh @@ -0,0 +1,295 @@ +#!/bin/bash +# ============================================================================= +# ZFS Pool Scrub & Email Report — NAS16 +# Location: /usr/scripts/zfs/nas16_zfs_scrub.sh +# Schedule via OMV Scheduler (cron) +# ============================================================================= + +# ── Configuration ───────────────────────────────────────────────────────────── +HOSTNAME="NAS16" +REPORT_TO="rufus.king@kingdezigns.com" +REPORT_FROM="zfs-monitor@nas16.local" # Display only — Postfix uses your Gmail relay +SCRUB_WAIT_SECONDS=28800 # Max wait per pool scrub (8 hrs for large pools) +CAPACITY_WARN_THRESHOLD=80 # % capacity to flag as WARNING +SCRUB_AGE_WARN_DAYS=8 # Days since last scrub before flagging stale +LOG_FILE="/var/log/zfs_scrub_nas16.log" +# Note: Email is sent via OMV-configured Postfix (Gmail relay). No SMTP config needed here. +# ────────────────────────────────────────────────────────────────────────────── + +REPORT_TIME=$(date "+%Y-%m-%d %H:%M:%S") +POOLS=$(zpool list -H -o name) +POOL_COUNT=$(echo "$POOLS" | wc -l) + +log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; } + +log "===== ZFS Scrub started on $HOSTNAME =====" + +# ── Run scrubs sequentially ─────────────────────────────────────────────────── +for pool in $POOLS; do + log "Starting scrub on pool: $pool" + zpool scrub "$pool" + + # Wait for scrub to finish + ELAPSED=0 + while true; do + SCRUB_STATE=$(zpool status "$pool" | grep -E "scan:" | awk '{print $2}') + [[ "$SCRUB_STATE" == "scrub" ]] || break + sleep 30 + ELAPSED=$((ELAPSED + 30)) + if [[ $ELAPSED -ge $SCRUB_WAIT_SECONDS ]]; then + log "WARNING: Scrub on $pool exceeded wait time. Moving on." + break + fi + done + log "Scrub complete (or timed out) for pool: $pool" +done + +# ── Gather data & build HTML report ────────────────────────────────────────── + +OVERALL_STATUS="HEALTHY" # HEALTHY | WARNING | CRITICAL +POOLS_WITH_ISSUES=0 +POOL_CARDS_HTML="" + +for pool in $POOLS; do + STATUS_RAW=$(zpool list -H -o health "$pool") + SIZE=$(zpool list -H -o size "$pool") + USED=$(zpool list -H -o alloc "$pool") + FREE=$(zpool list -H -o free "$pool") + CAP=$(zpool list -H -o cap "$pool" | tr -d '%') + FRAG=$(zpool list -H -o frag "$pool") + SCRUB_LINE=$(zpool status "$pool" | grep -E "scan:") + ERRORS_LINE=$(zpool status "$pool" | grep "errors:") + CONFIG_BLOCK=$(zpool status "$pool" | awk '/config:/,/errors:/' | head -n -1) + FAULTED_LINES=$(echo "$CONFIG_BLOCK" | grep -E "FAULTED|DEGRADED|UNAVAIL|REMOVED" | grep -v "^$") + + # Determine per-pool severity + POOL_SEVERITY="HEALTHY" + ISSUE_DETAIL="" + + if [[ "$STATUS_RAW" == "DEGRADED" || "$STATUS_RAW" == "FAULTED" || "$STATUS_RAW" == "UNAVAIL" ]]; then + POOL_SEVERITY="CRITICAL" + OVERALL_STATUS="CRITICAL" + ISSUE_DETAIL="$FAULTED_LINES" + elif [[ "$CAP" -ge "$CAPACITY_WARN_THRESHOLD" ]]; then + POOL_SEVERITY="WARNING" + [[ "$OVERALL_STATUS" != "CRITICAL" ]] && OVERALL_STATUS="WARNING" + fi + + # Scrub error count from status + SCRUB_ERRORS=$(echo "$SCRUB_LINE" | grep -oP '\d+ errors' | head -1) + [[ -z "$SCRUB_ERRORS" ]] && SCRUB_ERRORS="0 errors" + + # Scrub date + SCRUB_DATE=$(echo "$SCRUB_LINE" | grep -oP '[A-Z][a-z]{2} \d+ \d+:\d+:\d+ \d{4}' | head -1) + [[ -z "$SCRUB_DATE" ]] && SCRUB_DATE="No scrub data" + + # Scrub age warning + SCRUB_AGE_NOTE="" + if [[ -n "$SCRUB_DATE" && "$SCRUB_DATE" != "No scrub data" ]]; then + SCRUB_EPOCH=$(date -d "$SCRUB_DATE" +%s 2>/dev/null) + NOW_EPOCH=$(date +%s) + DIFF_DAYS=$(( (NOW_EPOCH - SCRUB_EPOCH) / 86400 )) + if [[ $DIFF_DAYS -ge $SCRUB_AGE_WARN_DAYS ]]; then + SCRUB_AGE_NOTE=" — ⚠️ Last scrub was ${DIFF_DAYS} days ago" + [[ "$POOL_SEVERITY" == "HEALTHY" ]] && POOL_SEVERITY="WARNING" + [[ "$OVERALL_STATUS" != "CRITICAL" ]] && OVERALL_STATUS="WARNING" + fi + fi + + [[ "$POOL_SEVERITY" != "HEALTHY" ]] && POOLS_WITH_ISSUES=$((POOLS_WITH_ISSUES + 1)) + + # Health value color + case "$STATUS_RAW" in + ONLINE) HEALTH_COLOR="#1a7f4b" ;; + DEGRADED) HEALTH_COLOR="#b91c1c" ;; + FAULTED) HEALTH_COLOR="#b91c1c" ;; + *) HEALTH_COLOR="#b45309" ;; + esac + + # Pool card header color + case "$POOL_SEVERITY" in + CRITICAL) CARD_BG="#fff1f2"; BADGE_BG="#b91c1c"; BADGE_LABEL="$STATUS_RAW" ;; + WARNING) CARD_BG="#fffbeb"; BADGE_BG="#b45309"; BADGE_LABEL="WARNING" ;; + *) CARD_BG="#f0fdf4"; BADGE_BG="#1a7f4b"; BADGE_LABEL="HEALTHY" ;; + esac + + # Capacity color + CAP_COLOR="#111827" + [[ "$CAP" -ge "$CAPACITY_WARN_THRESHOLD" ]] && CAP_COLOR="#b45309" + [[ "$CAP" -ge 90 ]] && CAP_COLOR="#b91c1c" + + # Issue block (only shown when there's a problem) + ISSUE_BLOCK="" + if [[ -n "$ISSUE_DETAIL" ]]; then + ESCAPED_DETAIL=$(echo "$ISSUE_DETAIL" | sed 's//\>/g') + ISSUE_BLOCK="

$ESCAPED_DETAIL

" + fi + + # Config block + ESCAPED_CONFIG=$(echo "$CONFIG_BLOCK" | sed 's//\>/g') + + POOL_CARDS_HTML+=" +

+ 🗄️ $pool + pool +

+ $BADGE_LABEL +

+ + + + + + + + + + + + + + +

Total Size

$SIZE

Used

$USED

Free

$FREE

Capacity

${CAP}%

Fragmentation

$FRAG

Health

$STATUS_RAW

+ 🔍 Last Scrub + Completed | $SCRUB_DATE | $SCRUB_ERRORS$SCRUB_AGE_NOTE +

+ $ISSUE_BLOCK +

▶ vdev / drive configuration

$ESCAPED_CONFIG

" +done + +# ── Overall banner ──────────────────────────────────────────────────────────── +case "$OVERALL_STATUS" in + CRITICAL) + BADGE_HTML="

🚨 CRITICAL

" + BANNER_HTML="🚨 CRITICAL issue detected — immediate action required!" + ;; + WARNING) + BADGE_HTML="

⚠️ WARNING

" + BANNER_HTML="⚠️ Warning condition detected — review recommended." + ;; + *) + BADGE_HTML="

✅ ALL HEALTHY

" + BANNER_HTML="✅ All pools are healthy — no action required." + ;; +esac + +# ── Compose full HTML email ─────────────────────────────────────────────────── +HTML_BODY=$(cat < + + + + + +

+ + + + + $BANNER_HTML + + + + + + + +

+ + + + + +

ZFS Pool Health Monitor

🔌 $HOSTNAME

$REPORT_TIME | $POOL_COUNT pool(s) monitored

$BADGE_HTML

+ $POOL_CARDS_HTML +

📋 Report Summary

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Host	$HOSTNAME

Report Time	$REPORT_TIME

Pools Checked	$POOL_COUNT

Pools with Issues	$POOLS_WITH_ISSUES

Scrub Age Warning Threshold	$SCRUB_AGE_WARN_DAYS days

Capacity Warning Threshold	${CAPACITY_WARN_THRESHOLD}%

+ Automated ZFS Monitor | $HOSTNAME | KingDezigns Infrastructure +

+ 📧 EMAIL REPORT PREVIEW — This is how the email will look in your inbox +

+ + + +

+ + + + + + + + + + + + + + + + + +

+ + + + + +

ZFS Pool Health Monitor

🔌 NAS08

2025-06-10 07:00:01 | 3 pool(s) monitored

+ 🚨 CRITICAL +

+ 🚨 CRITICAL issue detected — immediate action required! +

+ + +

+ 📉 tank0 + pool +

+ DEGRADED +

+ + + + + + + + + + + + + + +

Total Size

18.2T

Used

14.8T

Free

3.4T

Capacity

81%

Fragmentation

12%

Health

DEGRADED

+ 🔍 Last Scrub + Completed | Jun 3 07:01:44 2025 | Errors: 0 errors +

  sdb      DEGRADED     0     0     0
+    sdb1   FAULTED      5     8     0  too many errors

▶ vdev / drive configuration

config:
+        NAME        STATE     READ WRITE CKSUM
+        tank0       DEGRADED     0     0     0
+          raidz2-0  DEGRADED     0     0     0
+            sda     ONLINE       0     0     0
+            sdb     DEGRADED     5     8     0
+            sdc     ONLINE       0     0     0
+            sdd     ONLINE       0     0     0

+ + +

+ 📉 backup + pool +

+ WARNING — CAPACITY +

+ + + + + + + + + + + + + + +

Total Size

7.3T

Used

6.1T

Free

1.2T

Capacity

84%

Fragmentation

Health

ONLINE

+ 🔍 Last Scrub + Completed | Jun 3 07:01:44 2025 | Errors: 0 errors +

+ + +

+ 📉 media + pool +

+ HEALTHY +

+ + + + + + + + + + + + + + +

Total Size

21.8T

Used

9.2T

Free

12.6T

Capacity

42%

Fragmentation

Health

ONLINE

+ 🔍 Last Scrub + Completed | Jun 3 07:02:11 2025 | Errors: 0 errors +

+ +

📋 Report Summary

+ + + + + + + + + + + + + + + + + + + + + + + + + +

Host	NAS08

Report Time	2025-06-10 07:00:01

Pools Checked	3

Pools with Issues	2

Scrub Warning Threshold	8 days

+ Automated ZFS Monitor | NAS08 | KingDezigns Infrastructure +

+ +