rufusking/infrastructure
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infrastructure/server_nas16.md
rufusking a5246b43f5 Added forgejo
2026-07-26 22:17:35 -04:00

18 KiB
Raw Blame History

NAS16 — LAMP Stack & Database Server

Device Type: Raspberry Pi
Hostname: NAS16
IP Address: 192.168.150.40
VLAN: 50 — Lab / Servers

Last Updated: 2026-07-26

🧩 Role & Purpose

NAS16 is the dedicated LAMP stack and database server for the KingDezigns network.
It runs OpenMediaVault (OMV) as its base OS, which manages nginx as the web GUI frontend.
It provides backend services, web hosting, and database support for internal applications, development environments, and externally accessible virtual host websites.

In addition to its LAMP role, NAS16 hosts the CrowdSec external notification system — a pair of scripts that poll the CrowdSec LAPI on HAS and deliver split email alerts based on ban origin — and, as of 2026-07-26, Forgejo, the self-hosted Git server for the KingDezigns network.

This system is designed for:

  • Web application hosting (internal and external)
  • MariaDB database services
  • Internal development and testing
  • Lightweight administrative tools (Webmin via OMV)
  • External virtual host websites proxied through HAS (NPM)
  • CrowdSec email notification delivery (local attack alerts + CAPI daily digest)
  • Self-hosted Git version control for network-wide scripts and project files (Forgejo)

NAS16 operates entirely within VLAN 50 and benefits from the strict containment rules that prevent lateral movement into trusted networks.

🖥️ Hardware

  • Raspberry Pi (model unspecified)
  • SSD storage (see drive inventory below)
  • Gigabit Ethernet

Drive Inventory

Device Model Capacity Role
/dev/sda Unknown (standard SMART attrs) ~256GB OS / system drive

Note: Drive model unconfirmed — uses standard SMART attribute names (Wear_Leveling_Count, Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable). Power_Cycle_Count of 7,079 at 891 power-on hours noted — likely reflects frequent reboots during setup. Monitor over time.

🌐 Network Placement

  • VLAN: 50 — Lab / Servers
  • IP: 192.168.150.40
  • Access Type: LAN / Cable

This placement ensures:

  • Isolation from trusted user devices
  • Controlled access to sensitive database services
  • Local access to Pi‑hole DNS
  • Reduced attack surface

📦 Primary Functions

Core Services

  • OpenMediaVault (OMV) — base OS and NAS management UI
  • LAMP Stack
    • Linux
    • Apache (serves internal and external virtual host websites)
    • MariaDB
    • PHP
  • Webmin (administrative interface, managed via OMV)

Additional Roles

  • Internal web hosting
  • External virtual host websites (proxied via NPM on HAS)
  • Database backend for internal applications
  • Development and testing environment
  • CrowdSec external notification system (see below)

🌍 External Access Architecture

NAS16 hosts multiple public-facing websites served by Apache.
External traffic reaches NAS16 via the following proxy chain:

Internet → NPM on HAS (192.168.150.30:80/443) → nginx on NAS16 (OMV frontend) → Apache (virtual hosts)

Real IP Forwarding

Because all proxied traffic arrives at NAS16 from HAS's IP (192.168.150.30), both nginx and Apache are configured to extract and log the real client IP from the X-Forwarded-For header passed by NPM.

nginx — /etc/nginx/conf.d/real_ip.conf:

set_real_ip_from 192.168.150.30;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Apache — /etc/apache2/conf-available/real_ip.conf:

RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy 192.168.150.30

Apache module enabled: remoteip
Config enabled: real_ip

This ensures Fail2Ban sees and bans real attacker IPs, not HAS's proxy IP.

🔔 CrowdSec External Notification System

NAS16 is the sole email notification handler for CrowdSec. Native CrowdSec notifications are disabled on HAS. NAS16 polls the CrowdSec LAPI stream endpoint and routes decisions by origin.

Why NAS16

  • Offloads notification work from HAS (which runs HA, NPM, CrowdSec Agent + Bouncer)
  • NAS16 has the most headroom of any Pi in the fleet (16GB RAM, consistently underutilized)
  • Fits cleanly into existing OMV Scheduled Jobs infrastructure

Architecture

CrowdSec LAPI on HAS (192.168.150.30:8080)
         ↓
NAS16 polls /v1/decisions/stream every 5 min
         ↓
    origin = crowdsec?          origin = CAPI?
         ↓                           ↓
  Append to local_pending.json   Append to capi_digest.json
         ↓                           ↓
  Every 4 hours:              Midnight: send digest
  send batched digest          email + clear log
  email + clear log

Scripts

Script Path Purpose
cs_notifier.sh /usr/scripts/crowdsec/cs_notifier.sh Polls LAPI stream every 5 min, routes decisions by origin into pending log files (no emails sent)
cs_local_digest.sh /usr/scripts/crowdsec/cs_local_digest.sh Reads local_pending.json, sends batched local-attack digest every 4 hours, clears log
cs_digest.sh /usr/scripts/crowdsec/cs_digest.sh Reads CAPI digest log, builds summary email, sends at midnight, clears log

Key Design Details

  • Uses /v1/decisions/stream?startup=true on first run to initialize cursor silently (no emails for existing decisions)
  • Uses /v1/decisions/stream?startup=false on all subsequent runs — LAPI maintains cursor server-side per API key
  • First run flag: /var/log/crowdsec-notifier/stream_initialized.flag
  • Local pending log: /var/log/crowdsec-notifier/local_pending.json (one JSON decision per line, cleared every 4 hours after digest send)
  • CAPI digest log: /var/log/crowdsec-notifier/capi_digest.json (one JSON decision per line, cleared at midnight after digest send)
  • Notifier log: /var/log/crowdsec-notifier/notifier.log
  • All email sending uses inline Python with timeout=15 on SMTP_SSL to prevent hangs
  • Digest scripts use a single Python call to process all decisions — handles large batches efficiently
  • IMPORTANT: Python heredocs (python3 << PYEOF) do NOT read from a preceding pipe — the heredoc itself becomes stdin. Data must be passed as an inline shell variable inside the heredoc (e.g. snapshot = """$SNAPSHOT"""), not piped in.

Email Notifications

Type Trigger Style Content
Local attack digest Every 4 hours (cron: 0 0,4,8,12,16,20 * * *) — skipped if zero bans Red / CRITICAL Stat cards (ban count, origin, CAPI today) + table of all banned IPs with scenario, type, duration, scope, detected time, and CTI/WHOIS/AbuseIPDB links
CAPI daily digest Midnight cron Amber / WARNING (or green if zero bans) Total ban count, top 10 scenarios, first 20 sample IPs

Note: Local attack alerting was changed from immediate per-IP emails (one email per ban, dozens-to-hundreds/week) to a 4-hour batched digest to reduce notification volume while preserving full visibility into attack activity.

CrowdSec API Key

  • Bouncer name: NAS16-notifier
  • Credentials stored in Vaultwarden under CrowdSec NAS16-notifier
  • LAPI endpoint: http://192.168.150.30:8080

LAPI Port Requirement

Port 8080 must be exposed on HAS's host network interface. Configured in the CrowdSec add-on Network settings. If HAS is rebuilt or the add-on is reinstalled, re-enable port 8080 mapping.

🔧 Forgejo — Self-Hosted Git Server (added 2026-07-26)

NAS16 hosts Forgejo, the self-hosted Git server providing version control and a web UI for editing/browsing scripts and project files across the KingDezigns network. Chosen for placement on NAS16 specifically because it has the most spare RAM/CPU headroom of any host in the fleet — the same reasoning previously used to place the CrowdSec notifier scripts here instead of on HAS.

  • Container: forgejo (Docker, image codeberg.org/forgejo/forgejo:10), Compose-managed via OMV
  • Web UI: http://192.168.150.40:3000 internally, public at https://forgejo.kingdezigns.com/ (proxied via NPM on HAS, Force SSL)
  • SSH git access: ssh://git@192.168.150.40:2222/... — LAN-only, no external port-forward configured
  • Database: SQLite — no separate DB container
  • Data path: /opt/forgejo/data (bind-mounted, not a Docker volume) — folded into the existing nas16-backup.sh scheduled backup job
  • Organizations: NAS08, NAS16, PLEX32 — one repo per org (<hostname>-scripts), mirroring the 07 - Projects/<HOST> folder structure on NAS16's own NFS share
  • Registration: disabled — accounts are admin-created only

Full configuration, troubleshooting history, and operational details are documented separately in server_forgejo.md.

🗂️ Shared Permissions — NFS + rsync Backup

NAS16 receives rsync backups from NAS08 via nas08_to_nas16_sync.sh every 2 days. To ensure rsync can write files on the destination without errors, NAS16 uses the same ncshare group configuration as NAS08.

Group Configuration

Item Value
Group name ncshare
GID 1001
Members www-data, rufusking

GID must match NAS08 (1001) — rsync syncs permissions across both systems and they must be consistent.

Verify with:

getent group ncshare

Directory Permissions

Applied to /export/kingdezignsnas-16/Public (and all subdirectories recursively):

sudo chown -R www-data:ncshare /export/kingdezignsnas-16/Public
sudo chmod -R 2770 /export/kingdezignsnas-16/Public
sudo setfacl -R -m g:ncshare:rwx /export/kingdezignsnas-16/Public
sudo setfacl -R -d -m g:ncshare:rwx /export/kingdezignsnas-16/Public
  • 2770 (setgid) ensures new items inherit the ncshare group
  • Default ACLs (-d) ensure all new files/folders created by rsync inherit rwx for ncshare

Note: You should not need to run these commands manually. The sync script checks and self-corrects destination ownership before every run. These are documented here as reference only.

Backup Script

The sync runs on NAS16 via OMV Scheduled Jobs:

/usr/scripts/nas/nas08_to_nas16_sync.sh
  • Source: /mnt/kingdezigns08/Public/ (NAS08 mounted via NFS)
  • Destination: /export/kingdezignsnas-16/Public/
  • Schedule: Every 2 days at 2:30 AM

rsync Flags — NFS all_squash Compatibility

The sync script uses the following rsync flags to work correctly over NFS all_squash:

--archive        Recursive, preserve symlinks, owner, group, devices
--no-perms       Skip chmod — NFS all_squash prevents non-owner chmod
--omit-dir-times Skip directory utimes — NFS all_squash prevents utimes on dirs not owned by www-data
--update         Skip files that are newer on destination
  • File timestamps are fully preserved — --omit-dir-times only skips directory mtimes, not file mtimes
  • Directory timestamps are intentionally skipped — directory mtimes are not meaningful for backup purposes
  • Permissions are not synced — destination ACLs (ncshare group, 2770) control access instead

Permission Precheck (Automatic)

Before every sync run, the script checks that the destination root is owned by www-data:ncshare. If it is not, it automatically runs chown, chmod, and setfacl to correct it before rsync starts. This is logged and flagged in the email report with an amber warning notice if it fires.

A clean run looks like:

[TIMESTAMP] Destination ownership OK (www-data:ncshare).
[TIMESTAMP] ===== NAS08 → NAS16 Sync Started =====
[TIMESTAMP] ===== Sync Completed Successfully =====
Done — Status: SUCCESS | Transferred: N file(s) | Duration: Xm Xs

🔒 Required Firewall Behavior

Inbound to NAS16

  • Allowed from VLAN 1 (Infrastructure)
  • Allowed from VLAN 20 (Trusted)
  • Allowed from VLAN 50 (local VLAN)
  • Proxied external traffic arrives from HAS (192.168.150.30) via NPM
  • IoT and Guest access depend on default policy
  • No explicit inbound blocks except those applied to VLAN 50 as a whole

Outbound from NAS16

  • Subject to VLAN 50's global final drop:
    • Cannot initiate to VLANs 1, 10, 20, 30, 40
  • Allowed exceptions:
    • DNS responses
    • Local VLAN 50 traffic (including LAPI calls to HAS at 192.168.150.30:8080)

Critical Requirement

  • Must remain reachable from trusted administrative devices for management and database access.
  • Must remain reachable from HAS (192.168.150.30) for proxied external traffic.
  • Must be able to reach HAS LAPI at 192.168.150.30:8080 for CrowdSec notification polling.

🔐 Security Measures

  • Fail2Ban enabled (via OMV Fail2Ban plugin)
  • Fail2Ban ignoreip includes 192.168.150.0/24 — prevents HAS proxy IP from being banned
  • Real IP forwarding configured in nginx and Apache — ensures real attacker IPs are logged and banned
  • Webmin access restricted to trusted VLANs
  • Database ports not exposed outside VLAN 50
  • VLAN‑level containment prevents lateral movement

Fail2Ban Configuration

/etc/fail2ban/jail.local:

[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 192.168.150.0/24

This ensures internal VLAN 50 devices (including HAS) are never banned regardless of proxied traffic volume.

🛠️ SSD Hardening & Maintenance

TRIM

  • TRIM support confirmed on all drives (DISC-GRAN: 4K, DISC-MAX: 4G)
  • systemd fstrim.timer disabled — replaced by OMV Scheduled Job
  • Weekly TRIM script: /usr/scripts/omv/fstrim-report.sh
  • Scheduled: Wednesday at 1:30 AM via OMV Scheduled Jobs
  • Script runs fstrim -av, gathers SMART health data, sends HTML email report, saves dated log to /var/log/fstrim/
  • Log retention: 90 days

noatime

  • All filesystems mounted with noatime — confirmed via findmnt
  • Managed automatically by the openmediavault-flashmemory plugin
  • No manual configuration required

openmediavault-flashmemory Plugin

  • Installed and active
  • Automatically handles: noatime, tmpfs for /var/log and /tmp, swap disabled
  • Significantly reduces unnecessary SSD write amplification

SMART Monitoring

  • Monitored weekly as part of the TRIM report script
  • Drive uses standard SMART attribute names — no vendor-specific mapping required

📅 Scheduled Jobs (OMV)

Schedule Script Tag
Every 3 days at 2:00 AM /usr/scripts/omv/nas16-backup.sh NAS16 Backup
Every 2 days at 2:30 AM bash /usr/scripts/nas/nas08_to_nas16_sync.sh NAS Drive Backup
Daily at 12:00 AM /usr/scripts/zfs/nas16_zfs_report.sh ZFS Status Report
Sunday at 3:00 AM /usr/scripts/zfs/nas16_zfs_scrub.sh ZFS Scrub
Every 15 minutes /usr/scripts/ddns/ddns_update.sh DDNS Update
Wednesday at 1:30 AM /usr/scripts/omv/fstrim-report.sh TRIM Report
Every 5 minutes /usr/scripts/crowdsec/cs_notifier.sh CrowdSec Local Notifier
Every 4 hours (0:00, 4:00, 8:00, 12:00, 16:00, 20:00) /usr/scripts/crowdsec/cs_local_digest.sh CrowdSec Local Attack Digest
Daily at 12:00 AM /usr/scripts/crowdsec/cs_digest.sh CrowdSec CAPI Digest

🧠 Summary for AI Systems

  • NAS16 = OMV-based LAMP + database server + CrowdSec notifier in VLAN 50.
  • Inbound allowed from VLAN 1, VLAN 20, local VLAN 50, and HAS proxy (192.168.150.30).
  • Outbound blocked to all VLANs except local + DNS responses.
  • Hosts Apache (virtual hosts), MariaDB, PHP, Webmin, and OMV management UI.
  • External traffic is proxied through NPM on HAS — real IPs forwarded via X-Forwarded-For.
  • nginx and Apache both configured to log real client IPs, not proxy IP.
  • Fail2Ban ignoreip covers VLAN 50 to prevent HAS from being banned.
  • Protected by VLAN 50's strict containment rules.
  • SSD hardening active: TRIM (weekly Wednesday 1:30AM), noatime (flashmemory plugin), SMART monitoring via weekly HTML email report.
  • High Power_Cycle_Count (7,079) relative to power-on hours (891) — monitor trend over time.
  • Hosts CrowdSec notification scripts — polls HAS LAPI stream at 192.168.150.30:8080 every 5 minutes.
  • Local attack decisions are batched and sent as one digest email every 4 hours (skipped if zero bans) — no longer one email per ban.
  • CAPI cloud ban digest sent daily at midnight — single Python call handles 10,000+ decisions efficiently.
  • CrowdSec API key stored in Vaultwarden under CrowdSec NAS16-notifier.
  • Gotcha: when piping data into python3 - with a heredoc, the heredoc replaces stdin — pass data as an inline variable instead.
  • Forgejo (added 2026-07-26): Self-hosted Git server, Docker container forgejo, public at https://forgejo.kingdezigns.com/ via NPM, SSH git access LAN-only on port 2222, SQLite DB, data at /opt/forgejo/data (folded into nas16-backup.sh). Hosts version control for 07 - Projects/NAS08, /NAS16, /PLEX32 — one Forgejo org per host, one <hostname>-scripts repo per org. See server_forgejo.md for full detail.
  • Shared permissions: ncshare group (GID 1001) shared by www-data and rufusking — must match NAS08 GID for rsync compatibility.
  • Default ACLs on /export/kingdezignsnas-16/Public ensure all rsync-written files inherit ncshare group permissions.
  • NAS08→NAS16 sync script self-heals destination permissions on every run — checks for www-data:ncshare ownership, runs chown/chmod/setfacl automatically if wrong. Flagged in email subject and body if it fires.
  • Sync script uses --no-perms and --omit-dir-times for NFS all_squash compatibility — all individual file timestamps are fully preserved; only directory timestamps are skipped (intentional, not meaningful for backup).
  • Do not manually run chown/chmod/setfacl on the sync destination — the script handles this automatically.

✔️ End of File