The PLEX32 backup script ran at 10:00 PM instead of the scheduled 2:00 AM, indicating a scheduling problem. A separate gap was also identified: there was no way to remotely reboot PLEX32 when away from home if it became unresponsive.
The crontab entry on PLEX32 was correct (0 2 */3 * *), but the system timezone was set to Etc/UTC instead of America/New_York, causing all cron jobs to fire 4 hours early during EDT (UTC-4).
ls -lt /var/log/plex32-backup/
Confirm the 10 PM run actually happened and check if it was recurring.
Showed Jun 19 ran at 23:17, Jun 20 at 02:03, Jun 22 at 02:09 — pattern inconsistent, not a simple crontab typo.
sudo crontab -l
Rule out a bad cron expression as the cause.
Crontab was correct: 0 2 */3 * * — not the problem.
timedatectl
A 4-hour offset is the classic symptom of UTC vs EDT mismatch.
Time zone: Etc/UTC — confirmed root cause. Cron was firing at 2 AM UTC = 10 PM EDT.
ssh rufusking@192.168.150.35 "timedatectl | grep 'Time zone'" # NAS08 ssh rufusking@192.168.150.40 "timedatectl | grep 'Time zone'" # NAS16 ssh rufusking@192.168.150.30 "timedatectl | grep 'Time zone'" # HAS
If PLEX32 had the wrong timezone, other servers might too.
All other servers were already on America/New_York — PLEX32 was the only one affected.
PLEX32's system timezone was set to Etc/UTC (the Debian default). All cron jobs fired at the correct UTC time, which is 4 hours ahead of Eastern Daylight Time, causing the 2 AM backup to run at 10 PM EDT.
The backup log on Jun 19 showed 23:17 and Jun 20 showed 02:03 — the jump from late-night to early-morning suggested the timezone was corrected mid-cycle, but timedatectl showed it was still UTC. The log timestamps themselves are in UTC, so both runs were actually at 2 AM UTC.
-rw-r--r-- 1 root root 1559 Jun 22 02:09 backup-20260622.log -rw-rw-r-- 1 rufusking rufusking 3120 Jun 20 02:03 backup-20260620.log -rw-rw-r-- 1 rufusking rufusking 5879 Jun 19 23:17 backup-20260619.log
The Jun 19 timestamp difference was due to the backup running at a slightly different interval that cycle — not a timezone change.
Resolution: Set the timezone to America/New_York with sudo timedatectl set-timezone America/New_York. No crontab changes needed — cron reads system time automatically.
A remote reboot script was created on NAS08 to provide a way to reboot PLEX32 from outside the home network via OMV's Scheduled Jobs interface. The script was iteratively debugged through several test runs.
nc not installed on NAS08 — port check always failed
docker compose down clears restart policy — containers didn't auto-start
flat sleep 60 caused unnecessary delay in Plex detection
sudo ssh-copy-id -i /root/.ssh/id_ed25519.pub rufusking@192.168.150.45 sudo ssh -i /root/.ssh/id_ed25519 rufusking@192.168.150.45 "echo OK"
Key was already trusted — ssh-copy-id skipped it, echo OK confirmed passwordless access.
sudo visudo -f /etc/sudoers.d/rufusking-nopasswd # Appended: /sbin/reboot
The script runs sudo reboot over SSH — must be passwordless or it hangs.
Full sudoers line: apt-get update, fail2ban-client status, fail2ban-client status sshd, /sbin/reboot
docker ps # Result: empty — no containers running
The script ran docker compose down before rebooting, which clears the unless-stopped restart policy. Nothing brought the containers back up.
Plex was down after reboot — had to manually run docker compose up -d.
ssh ... "cd ~/docker/plex && docker compose up -d"
Containers started cleanly after subsequent reboots.
nc -z -w 5 192.168.150.45 32400; echo "nc exit: $?" # Result: nc exit: 127 (command not found)
nc (netcat) is not installed on NAS08 — the check always returned failure regardless of Plex state.
Switched to curl HTTP check — confirmed Plex returns HTTP 302 on port 32400.
while [ $PLEX_ELAPSED -lt 90 ]; do HTTP_CODE=$(curl ... http://192.168.150.45:32400/web) [ "$HTTP_CODE" = "302" ] && break sleep 5; PLEX_ELAPSED=$((PLEX_ELAPSED + 5)) done
Plex was up within ~10s of containers starting but the script was waiting a full 60s before checking.
Plex confirmed responding after 10s on final test run.
/usr/scripts/nas/plex32_reboot.sh
/var/log/plex32-reboot/reboot-YYYYMMDD-HHMMSS.log (90 day retention)
1. SSH to PLEX32 → docker compose down (graceful container stop) 2. SSH to PLEX32 → sudo reboot 3. Wait 30s then poll ping every 10s (max 3 min) until PLEX32 responds 4. SSH to PLEX32 → docker compose up -d 5. Poll curl http://192.168.150.45:32400/web every 5s (max 90s) 6. Log success (HTTP 302/200/301) or warning if timeout
Tag: PLEX32 Remote Reboot Command: /usr/scripts/nas/plex32_reboot.sh Enabled: OFF — enable manually, run once, disable again
[23:40:39] PLEX32 Remote Reboot — Initiated [23:40:45] Docker containers stopped cleanly. [23:40:45] Reboot command sent. PLEX32 is going down. [23:41:27] PLEX32 is back online (ping responded after ~40s). [23:41:30] Docker containers started. [23:41:40] Plex is responding (HTTP 302) after 10s — reboot successful.
| Lesson | Detail |
|---|---|
| Always set timezone on new installs | Debian/Ubuntu default is Etc/UTC. Run sudo timedatectl set-timezone America/New_York immediately after setup. Verify with timedatectl | grep "Time zone". A 4-hour cron offset is the telltale symptom. |
docker compose down clears restart policy |
Unlike a system reboot (which respects restart: unless-stopped), an explicit compose down removes containers entirely. Always follow a pre-reboot compose down with a post-boot compose up -d. |
nc is not installed on NAS08 |
Use curl -s -o /dev/null -w "%{http_code}" for HTTP service checks from NAS08. Plex returns HTTP 302 on port 32400 — treat 200, 301, and 302 all as success. |
| Flat sleep before health checks wastes time | Replace sleep N before a readiness check with a polling loop. Check every 5 seconds, break on success, cap with a max timeout. Services often come up well before the sleep expires. |
| Verify deployed scripts after editing | Use grep -n "key_string" /path/to/script to confirm the correct version is on disk before testing. Copy errors and stale files caused multiple confusing test failures in this session. |
| File | What changed |
|---|---|
server_plex32.md |
Added timezone section (was UTC, corrected to America/New_York). Updated sudoers entry to include /sbin/reboot. Updated AI summary with timezone note and NAS08→PLEX32 SSH key entry. |
server_nas08.md |
Added plex32_reboot.sh to OMV Scheduled Jobs table (disabled/manual). Added NAS08 root → PLEX32 SSH key note. Updated AI summary with remote reboot script details. |