NAS16 is the **dedicated LAMP stack and database server** for the KingDezigns network.
It runs **OpenMediaVault (OMV)** as its base OS, which manages nginx as the web GUI frontend.
It provides backend services, web hosting, and database support for internal applications, development environments, and externally accessible virtual host websites.
In addition to its LAMP role, NAS16 hosts the **CrowdSec external notification system** — a pair of scripts that poll the CrowdSec LAPI on HAS and deliver split email alerts based on ban origin — and, as of 2026-07-26, **Forgejo**, the self-hosted Git server for the KingDezigns network.
NAS16 operates entirely within VLAN 50 and benefits from the strict containment rules that prevent lateral movement into trusted networks.
---
## 🖥️ Hardware
- Raspberry Pi (model unspecified)
- SSD storage (see drive inventory below)
- Gigabit Ethernet
### **Drive Inventory**
| Device | Model | Capacity | Role |
|--------|-------|----------|------|
| /dev/sda | Unknown (standard SMART attrs) | ~256GB | OS / system drive |
> Note: Drive model unconfirmed — uses standard SMART attribute names (Wear_Leveling_Count, Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable). Power_Cycle_Count of 7,079 at 891 power-on hours noted — likely reflects frequent reboots during setup. Monitor over time.
---
## 🌐 Network Placement
- VLAN: **50 — Lab / Servers**
- IP: **192.168.150.40**
- Access Type: LAN / Cable
This placement ensures:
- Isolation from trusted user devices
- Controlled access to sensitive database services
- Local access to Pi‑hole DNS
- Reduced attack surface
---
## 📦 Primary Functions
### **Core Services**
- **OpenMediaVault (OMV)** — base OS and NAS management UI
- **LAMP Stack**
- Linux
- Apache (serves internal and external virtual host websites)
- MariaDB
- PHP
- **Webmin** (administrative interface, managed via OMV)
### **Additional Roles**
- Internal web hosting
- External virtual host websites (proxied via NPM on HAS)
- Database backend for internal applications
- Development and testing environment
- **CrowdSec external notification system** (see below)
---
## 🌍 External Access Architecture
NAS16 hosts multiple public-facing websites served by Apache.
External traffic reaches NAS16 via the following proxy chain:
```
Internet → NPM on HAS (192.168.150.30:80/443) → nginx on NAS16 (OMV frontend) → Apache (virtual hosts)
```
### **Real IP Forwarding**
Because all proxied traffic arrives at NAS16 from HAS's IP (192.168.150.30), both nginx and Apache are configured to extract and log the real client IP from the `X-Forwarded-For` header passed by NPM.
This ensures Fail2Ban sees and bans real attacker IPs, not HAS's proxy IP.
---
## 🔔 CrowdSec External Notification System
NAS16 is the sole email notification handler for CrowdSec. Native CrowdSec notifications are disabled on HAS. NAS16 polls the CrowdSec LAPI stream endpoint and routes decisions by origin.
### **Why NAS16**
- Offloads notification work from HAS (which runs HA, NPM, CrowdSec Agent + Bouncer)
- NAS16 has the most headroom of any Pi in the fleet (16GB RAM, consistently underutilized)
- Fits cleanly into existing OMV Scheduled Jobs infrastructure
### **Architecture**
```
CrowdSec LAPI on HAS (192.168.150.30:8080)
↓
NAS16 polls /v1/decisions/stream every 5 min
↓
origin = crowdsec? origin = CAPI?
↓ ↓
Append to local_pending.json Append to capi_digest.json
↓ ↓
Every 4 hours: Midnight: send digest
send batched digest email + clear log
email + clear log
```
### **Scripts**
| Script | Path | Purpose |
|--------|------|---------|
| cs_notifier.sh | `/usr/scripts/crowdsec/cs_notifier.sh` | Polls LAPI stream every 5 min, routes decisions by origin into pending log files (no emails sent) |
- All email sending uses inline Python with `timeout=15` on SMTP_SSL to prevent hangs
- Digest scripts use a single Python call to process all decisions — handles large batches efficiently
- **IMPORTANT:** Python heredocs (`python3 << PYEOF`) do NOT read from a preceding pipe — the heredoc itself becomes stdin. Data must be passed as an inline shell variable inside the heredoc (e.g. `snapshot = """$SNAPSHOT"""`), not piped in.
### **Email Notifications**
| Type | Trigger | Style | Content |
|------|---------|-------|---------|
| Local attack digest | Every 4 hours (cron: `0 0,4,8,12,16,20 * * *`) — skipped if zero bans | Red / CRITICAL | Stat cards (ban count, origin, CAPI today) + table of all banned IPs with scenario, type, duration, scope, detected time, and CTI/WHOIS/AbuseIPDB links |
| CAPI daily digest | Midnight cron | Amber / WARNING (or green if zero bans) | Total ban count, top 10 scenarios, first 20 sample IPs |
> **Note:** Local attack alerting was changed from immediate per-IP emails (one email per ban, dozens-to-hundreds/week) to a 4-hour batched digest to reduce notification volume while preserving full visibility into attack activity.
### **CrowdSec API Key**
- Bouncer name: `NAS16-notifier`
- Credentials stored in Vaultwarden under `CrowdSec NAS16-notifier`
- LAPI endpoint: `http://192.168.150.30:8080`
### **LAPI Port Requirement**
Port 8080 must be exposed on HAS's host network interface. Configured in the CrowdSec add-on Network settings. If HAS is rebuilt or the add-on is reinstalled, re-enable port 8080 mapping.
## 🔧 Forgejo — Self-Hosted Git Server (added 2026-07-26)
NAS16 hosts **Forgejo**, the self-hosted Git server providing version control and a web UI for editing/browsing scripts and project files across the KingDezigns network. Chosen for placement on NAS16 specifically because it has the most spare RAM/CPU headroom of any host in the fleet — the same reasoning previously used to place the CrowdSec notifier scripts here instead of on HAS.
- **Container:** `forgejo` (Docker, image `codeberg.org/forgejo/forgejo:10`), Compose-managed via OMV
- **Web UI:** `http://192.168.150.40:3000` internally, public at `https://forgejo.kingdezigns.com/` (proxied via NPM on HAS, Force SSL)
- **Data path:** `/opt/forgejo/data` (bind-mounted, not a Docker volume) — folded into the existing `nas16-backup.sh` scheduled backup job
- **Organizations:** `NAS08`, `NAS16`, `PLEX32` — one repo per org (`<hostname>-scripts`), mirroring the `07 - Projects/<HOST>` folder structure on NAS16's own NFS share
- **Registration:** disabled — accounts are admin-created only
Full configuration, troubleshooting history, and operational details are documented separately in `server_forgejo.md`.
NAS16 receives rsync backups from NAS08 via `nas08_to_nas16_sync.sh` every 2 days. To ensure rsync can write files on the destination without errors, NAS16 uses the same `ncshare` group configuration as NAS08.
### **Group Configuration**
| Item | Value |
|------|-------|
| Group name | `ncshare` |
| GID | `1001` |
| Members | `www-data`, `rufusking` |
> GID **must match NAS08** (1001) — rsync syncs permissions across both systems and they must be consistent.
Verify with:
```bash
getent group ncshare
```
### **Directory Permissions**
Applied to `/export/kingdezignsnas-16/Public` (and all subdirectories recursively):
-`2770` (setgid) ensures new items inherit the `ncshare` group
- Default ACLs (`-d`) ensure all new files/folders created by rsync inherit `rwx` for `ncshare`
> **Note:** You should not need to run these commands manually. The sync script checks and self-corrects destination ownership before every run. These are documented here as reference only.
### **Backup Script**
The sync runs on NAS16 via OMV Scheduled Jobs:
```bash
/usr/scripts/nas/nas08_to_nas16_sync.sh
```
- Source: `/mnt/kingdezigns08/Public/` (NAS08 mounted via NFS)
--omit-dir-times Skip directory utimes — NFS all_squash prevents utimes on dirs not owned by www-data
--update Skip files that are newer on destination
```
- **File timestamps are fully preserved** — `--omit-dir-times` only skips directory mtimes, not file mtimes
- **Directory timestamps are intentionally skipped** — directory mtimes are not meaningful for backup purposes
- **Permissions are not synced** — destination ACLs (`ncshare` group, `2770`) control access instead
### **Permission Precheck (Automatic)**
Before every sync run, the script checks that the destination root is owned by `www-data:ncshare`. If it is not, it automatically runs `chown`, `chmod`, and `setfacl` to correct it before rsync starts. This is logged and flagged in the email report with an amber warning notice if it fires.
A clean run looks like:
```
[TIMESTAMP] Destination ownership OK (www-data:ncshare).
[TIMESTAMP] ===== NAS08 → NAS16 Sync Started =====
- **Forgejo (added 2026-07-26):** Self-hosted Git server, Docker container `forgejo`, public at `https://forgejo.kingdezigns.com/` via NPM, SSH git access LAN-only on port 2222, SQLite DB, data at `/opt/forgejo/data` (folded into `nas16-backup.sh`). Hosts version control for `07 - Projects/NAS08`, `/NAS16`, `/PLEX32` — one Forgejo org per host, one `<hostname>-scripts` repo per org. See `server_forgejo.md` for full detail.
- **Shared permissions:** `ncshare` group (GID 1001) shared by `www-data` and `rufusking` — must match NAS08 GID for rsync compatibility.
- Default ACLs on `/export/kingdezignsnas-16/Public` ensure all rsync-written files inherit `ncshare` group permissions.
- **NAS08→NAS16 sync script self-heals destination permissions on every run** — checks for `www-data:ncshare` ownership, runs `chown`/`chmod`/`setfacl` automatically if wrong. Flagged in email subject and body if it fires.
- Sync script uses `--no-perms` and `--omit-dir-times` for NFS `all_squash` compatibility — all individual file timestamps are fully preserved; only directory timestamps are skipped (intentional, not meaningful for backup).
- **Do not manually run chown/chmod/setfacl on the sync destination** — the script handles this automatically.
