infrastructure/server_nas16.md

# NAS16 — LAMP Stack & Database Server
Device Type: **Raspberry Pi**  
Hostname: **NAS16**  
IP Address: **192.168.150.40**  
VLAN: **50 — Lab / Servers**

Last Updated: 2026-07-26

---

## 🧩 Role & Purpose
NAS16 is the **dedicated LAMP stack and database server** for the KingDezigns network.  
It runs **OpenMediaVault (OMV)** as its base OS, which manages nginx as the web GUI frontend.  
It provides backend services, web hosting, and database support for internal applications, development environments, and externally accessible virtual host websites.

In addition to its LAMP role, NAS16 hosts the **CrowdSec external notification system** — a pair of scripts that poll the CrowdSec LAPI on HAS and deliver split email alerts based on ban origin — and, as of 2026-07-26, **Forgejo**, the self-hosted Git server for the KingDezigns network.

This system is designed for:
- Web application hosting (internal and external)
- MariaDB database services
- Internal development and testing
- Lightweight administrative tools (Webmin via OMV)
- External virtual host websites proxied through HAS (NPM)
- CrowdSec email notification delivery (local attack alerts + CAPI daily digest)
- Self-hosted Git version control for network-wide scripts and project files (Forgejo)

NAS16 operates entirely within VLAN 50 and benefits from the strict containment rules that prevent lateral movement into trusted networks.

---

## 🖥️ Hardware
- Raspberry Pi (model unspecified)
- SSD storage (see drive inventory below)
- Gigabit Ethernet

### **Drive Inventory**
| Device | Model | Capacity | Role |
|--------|-------|----------|------|
| /dev/sda | Unknown (standard SMART attrs) | ~256GB | OS / system drive |

> Note: Drive model unconfirmed — uses standard SMART attribute names (Wear_Leveling_Count, Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable). Power_Cycle_Count of 7,079 at 891 power-on hours noted — likely reflects frequent reboots during setup. Monitor over time.

---

## 🌐 Network Placement
- VLAN: **50 — Lab / Servers**
- IP: **192.168.150.40**
- Access Type: LAN / Cable

This placement ensures:
- Isolation from trusted user devices
- Controlled access to sensitive database services
- Local access to Pi‑hole DNS
- Reduced attack surface

---

## 📦 Primary Functions

### **Core Services**
- **OpenMediaVault (OMV)** — base OS and NAS management UI
- **LAMP Stack**
  - Linux
  - Apache (serves internal and external virtual host websites)
  - MariaDB
  - PHP
- **Webmin** (administrative interface, managed via OMV)

### **Additional Roles**
- Internal web hosting
- External virtual host websites (proxied via NPM on HAS)
- Database backend for internal applications
- Development and testing environment
- **CrowdSec external notification system** (see below)

---

## 🌍 External Access Architecture

NAS16 hosts multiple public-facing websites served by Apache.  
External traffic reaches NAS16 via the following proxy chain:

```
Internet → NPM on HAS (192.168.150.30:80/443) → nginx on NAS16 (OMV frontend) → Apache (virtual hosts)
```

### **Real IP Forwarding**
Because all proxied traffic arrives at NAS16 from HAS's IP (192.168.150.30), both nginx and Apache are configured to extract and log the real client IP from the `X-Forwarded-For` header passed by NPM.

**nginx** — `/etc/nginx/conf.d/real_ip.conf`:
```nginx
set_real_ip_from 192.168.150.30;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
```

**Apache** — `/etc/apache2/conf-available/real_ip.conf`:
```apache
RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy 192.168.150.30
```

Apache module enabled: `remoteip`  
Config enabled: `real_ip`

This ensures Fail2Ban sees and bans real attacker IPs, not HAS's proxy IP.

---

## 🔔 CrowdSec External Notification System

NAS16 is the sole email notification handler for CrowdSec. Native CrowdSec notifications are disabled on HAS. NAS16 polls the CrowdSec LAPI stream endpoint and routes decisions by origin.

### **Why NAS16**
- Offloads notification work from HAS (which runs HA, NPM, CrowdSec Agent + Bouncer)
- NAS16 has the most headroom of any Pi in the fleet (16GB RAM, consistently underutilized)
- Fits cleanly into existing OMV Scheduled Jobs infrastructure

### **Architecture**
```
CrowdSec LAPI on HAS (192.168.150.30:8080)
         ↓
NAS16 polls /v1/decisions/stream every 5 min
         ↓
    origin = crowdsec?          origin = CAPI?
         ↓                           ↓
  Append to local_pending.json   Append to capi_digest.json
         ↓                           ↓
  Every 4 hours:              Midnight: send digest
  send batched digest          email + clear log
  email + clear log
```

### **Scripts**
| Script | Path | Purpose |
|--------|------|---------|
| cs_notifier.sh | `/usr/scripts/crowdsec/cs_notifier.sh` | Polls LAPI stream every 5 min, routes decisions by origin into pending log files (no emails sent) |
| cs_local_digest.sh | `/usr/scripts/crowdsec/cs_local_digest.sh` | Reads local_pending.json, sends batched local-attack digest every 4 hours, clears log |
| cs_digest.sh | `/usr/scripts/crowdsec/cs_digest.sh` | Reads CAPI digest log, builds summary email, sends at midnight, clears log |

### **Key Design Details**
- Uses `/v1/decisions/stream?startup=true` on first run to initialize cursor silently (no emails for existing decisions)
- Uses `/v1/decisions/stream?startup=false` on all subsequent runs — LAPI maintains cursor server-side per API key
- First run flag: `/var/log/crowdsec-notifier/stream_initialized.flag`
- Local pending log: `/var/log/crowdsec-notifier/local_pending.json` (one JSON decision per line, cleared every 4 hours after digest send)
- CAPI digest log: `/var/log/crowdsec-notifier/capi_digest.json` (one JSON decision per line, cleared at midnight after digest send)
- Notifier log: `/var/log/crowdsec-notifier/notifier.log`
- All email sending uses inline Python with `timeout=15` on SMTP_SSL to prevent hangs
- Digest scripts use a single Python call to process all decisions — handles large batches efficiently
- **IMPORTANT:** Python heredocs (`python3 << PYEOF`) do NOT read from a preceding pipe — the heredoc itself becomes stdin. Data must be passed as an inline shell variable inside the heredoc (e.g. `snapshot = """$SNAPSHOT"""`), not piped in.

### **Email Notifications**
| Type | Trigger | Style | Content |
|------|---------|-------|---------|
| Local attack digest | Every 4 hours (cron: `0 0,4,8,12,16,20 * * *`) — skipped if zero bans | Red / CRITICAL | Stat cards (ban count, origin, CAPI today) + table of all banned IPs with scenario, type, duration, scope, detected time, and CTI/WHOIS/AbuseIPDB links |
| CAPI daily digest | Midnight cron | Amber / WARNING (or green if zero bans) | Total ban count, top 10 scenarios, first 20 sample IPs |

> **Note:** Local attack alerting was changed from immediate per-IP emails (one email per ban, dozens-to-hundreds/week) to a 4-hour batched digest to reduce notification volume while preserving full visibility into attack activity.

### **CrowdSec API Key**
- Bouncer name: `NAS16-notifier`
- Credentials stored in Vaultwarden under `CrowdSec NAS16-notifier`
- LAPI endpoint: `http://192.168.150.30:8080`

### **LAPI Port Requirement**
Port 8080 must be exposed on HAS's host network interface. Configured in the CrowdSec add-on Network settings. If HAS is rebuilt or the add-on is reinstalled, re-enable port 8080 mapping.

---

## 🔧 Forgejo — Self-Hosted Git Server (added 2026-07-26)

NAS16 hosts **Forgejo**, the self-hosted Git server providing version control and a web UI for editing/browsing scripts and project files across the KingDezigns network. Chosen for placement on NAS16 specifically because it has the most spare RAM/CPU headroom of any host in the fleet — the same reasoning previously used to place the CrowdSec notifier scripts here instead of on HAS.

- **Container:** `forgejo` (Docker, image `codeberg.org/forgejo/forgejo:10`), Compose-managed via OMV
- **Web UI:** `http://192.168.150.40:3000` internally, public at `https://forgejo.kingdezigns.com/` (proxied via NPM on HAS, Force SSL)
- **SSH git access:** `ssh://git@192.168.150.40:2222/...` — LAN-only, no external port-forward configured
- **Database:** SQLite — no separate DB container
- **Data path:** `/opt/forgejo/data` (bind-mounted, not a Docker volume) — folded into the existing `nas16-backup.sh` scheduled backup job
- **Organizations:** `NAS08`, `NAS16`, `PLEX32` — one repo per org (`<hostname>-scripts`), mirroring the `07 - Projects/<HOST>` folder structure on NAS16's own NFS share
- **Registration:** disabled — accounts are admin-created only

Full configuration, troubleshooting history, and operational details are documented separately in `server_forgejo.md`.

---

## 🗂️ Shared Permissions — NFS + rsync Backup

NAS16 receives rsync backups from NAS08 via `nas08_to_nas16_sync.sh` every 2 days. To ensure rsync can write files on the destination without errors, NAS16 uses the same `ncshare` group configuration as NAS08.

### **Group Configuration**
| Item | Value |
|------|-------|
| Group name | `ncshare` |
| GID | `1001` |
| Members | `www-data`, `rufusking` |

> GID **must match NAS08** (1001) — rsync syncs permissions across both systems and they must be consistent.

Verify with:
```bash
getent group ncshare
```

### **Directory Permissions**
Applied to `/export/kingdezignsnas-16/Public` (and all subdirectories recursively):
```bash
sudo chown -R www-data:ncshare /export/kingdezignsnas-16/Public
sudo chmod -R 2770 /export/kingdezignsnas-16/Public
sudo setfacl -R -m g:ncshare:rwx /export/kingdezignsnas-16/Public
sudo setfacl -R -d -m g:ncshare:rwx /export/kingdezignsnas-16/Public
```

- `2770` (setgid) ensures new items inherit the `ncshare` group
- Default ACLs (`-d`) ensure all new files/folders created by rsync inherit `rwx` for `ncshare`

> **Note:** You should not need to run these commands manually. The sync script checks and self-corrects destination ownership before every run. These are documented here as reference only.

### **Backup Script**
The sync runs on NAS16 via OMV Scheduled Jobs:
```bash
/usr/scripts/nas/nas08_to_nas16_sync.sh
```
- Source: `/mnt/kingdezigns08/Public/` (NAS08 mounted via NFS)
- Destination: `/export/kingdezignsnas-16/Public/`
- Schedule: Every 2 days at 2:30 AM

### **rsync Flags — NFS all_squash Compatibility**
The sync script uses the following rsync flags to work correctly over NFS `all_squash`:

```
--archive        Recursive, preserve symlinks, owner, group, devices
--no-perms       Skip chmod — NFS all_squash prevents non-owner chmod
--omit-dir-times Skip directory utimes — NFS all_squash prevents utimes on dirs not owned by www-data
--update         Skip files that are newer on destination
```

- **File timestamps are fully preserved** — `--omit-dir-times` only skips directory mtimes, not file mtimes
- **Directory timestamps are intentionally skipped** — directory mtimes are not meaningful for backup purposes
- **Permissions are not synced** — destination ACLs (`ncshare` group, `2770`) control access instead

### **Permission Precheck (Automatic)**
Before every sync run, the script checks that the destination root is owned by `www-data:ncshare`. If it is not, it automatically runs `chown`, `chmod`, and `setfacl` to correct it before rsync starts. This is logged and flagged in the email report with an amber warning notice if it fires.

A clean run looks like:
```
[TIMESTAMP] Destination ownership OK (www-data:ncshare).
[TIMESTAMP] ===== NAS08 → NAS16 Sync Started =====
[TIMESTAMP] ===== Sync Completed Successfully =====
Done — Status: SUCCESS | Transferred: N file(s) | Duration: Xm Xs
```

---

## 🔒 Required Firewall Behavior

### **Inbound to NAS16**
- Allowed from VLAN 1 (Infrastructure)
- Allowed from VLAN 20 (Trusted)
- Allowed from VLAN 50 (local VLAN)
- Proxied external traffic arrives from HAS (192.168.150.30) via NPM
- IoT and Guest access depend on default policy
- No explicit inbound blocks except those applied to VLAN 50 as a whole

### **Outbound from NAS16**
- Subject to VLAN 50's global final drop:
  - Cannot initiate to VLANs 1, 10, 20, 30, 40
- Allowed exceptions:
  - DNS responses
  - Local VLAN 50 traffic (including LAPI calls to HAS at 192.168.150.30:8080)

### **Critical Requirement**
- Must remain reachable from trusted administrative devices for management and database access.
- Must remain reachable from HAS (192.168.150.30) for proxied external traffic.
- Must be able to reach HAS LAPI at 192.168.150.30:8080 for CrowdSec notification polling.

---

## 🔐 Security Measures
- **Fail2Ban enabled** (via OMV Fail2Ban plugin)
- **Fail2Ban ignoreip** includes `192.168.150.0/24` — prevents HAS proxy IP from being banned
- **Real IP forwarding** configured in nginx and Apache — ensures real attacker IPs are logged and banned
- Webmin access restricted to trusted VLANs
- Database ports not exposed outside VLAN 50
- VLAN‑level containment prevents lateral movement

### **Fail2Ban Configuration**
`/etc/fail2ban/jail.local`:
```ini
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 192.168.150.0/24
```

This ensures internal VLAN 50 devices (including HAS) are never banned regardless of proxied traffic volume.

---

## 🛠️ SSD Hardening & Maintenance

### **TRIM**
- TRIM support confirmed on all drives (`DISC-GRAN: 4K`, `DISC-MAX: 4G`)
- systemd `fstrim.timer` **disabled** — replaced by OMV Scheduled Job
- Weekly TRIM script: `/usr/scripts/omv/fstrim-report.sh`
- Scheduled: **Wednesday at 1:30 AM** via OMV Scheduled Jobs
- Script runs `fstrim -av`, gathers SMART health data, sends HTML email report, saves dated log to `/var/log/fstrim/`
- Log retention: 90 days

### **noatime**
- All filesystems mounted with `noatime` — confirmed via `findmnt`
- Managed automatically by the `openmediavault-flashmemory` plugin
- No manual configuration required

### **openmediavault-flashmemory Plugin**
- Installed and active
- Automatically handles: `noatime`, tmpfs for `/var/log` and `/tmp`, swap disabled
- Significantly reduces unnecessary SSD write amplification

### **SMART Monitoring**
- Monitored weekly as part of the TRIM report script
- Drive uses standard SMART attribute names — no vendor-specific mapping required

---

## 📅 Scheduled Jobs (OMV)

| Schedule | Script | Tag |
|----------|--------|-----|
| Every 3 days at 2:00 AM | `/usr/scripts/omv/nas16-backup.sh` | NAS16 Backup |
| Every 2 days at 2:30 AM | `bash /usr/scripts/nas/nas08_to_nas16_sync.sh` | NAS Drive Backup |
| Daily at 12:00 AM | `/usr/scripts/zfs/nas16_zfs_report.sh` | ZFS Status Report |
| Sunday at 3:00 AM | `/usr/scripts/zfs/nas16_zfs_scrub.sh` | ZFS Scrub |
| Every 15 minutes | `/usr/scripts/ddns/ddns_update.sh` | DDNS Update |
| Wednesday at 1:30 AM | `/usr/scripts/omv/fstrim-report.sh` | TRIM Report |
| Every 5 minutes | `/usr/scripts/crowdsec/cs_notifier.sh` | CrowdSec Local Notifier |
| Every 4 hours (0:00, 4:00, 8:00, 12:00, 16:00, 20:00) | `/usr/scripts/crowdsec/cs_local_digest.sh` | CrowdSec Local Attack Digest |
| Daily at 12:00 AM | `/usr/scripts/crowdsec/cs_digest.sh` | CrowdSec CAPI Digest |

---

## 🧠 Summary for AI Systems
- NAS16 = **OMV-based LAMP + database server + CrowdSec notifier** in VLAN 50.
- Inbound allowed from VLAN 1, VLAN 20, local VLAN 50, and HAS proxy (192.168.150.30).
- Outbound blocked to all VLANs except local + DNS responses.
- Hosts Apache (virtual hosts), MariaDB, PHP, Webmin, and OMV management UI.
- External traffic is proxied through NPM on HAS — real IPs forwarded via X-Forwarded-For.
- nginx and Apache both configured to log real client IPs, not proxy IP.
- Fail2Ban ignoreip covers VLAN 50 to prevent HAS from being banned.
- Protected by VLAN 50's strict containment rules.
- SSD hardening active: TRIM (weekly Wednesday 1:30AM), noatime (flashmemory plugin), SMART monitoring via weekly HTML email report.
- High Power_Cycle_Count (7,079) relative to power-on hours (891) — monitor trend over time.
- **Hosts CrowdSec notification scripts** — polls HAS LAPI stream at 192.168.150.30:8080 every 5 minutes.
- Local attack decisions are batched and sent as one digest email every 4 hours (skipped if zero bans) — no longer one email per ban.
- CAPI cloud ban digest sent daily at midnight — single Python call handles 10,000+ decisions efficiently.
- CrowdSec API key stored in Vaultwarden under `CrowdSec NAS16-notifier`.
- **Gotcha:** when piping data into `python3 -` with a heredoc, the heredoc replaces stdin — pass data as an inline variable instead.
- **Forgejo (added 2026-07-26):** Self-hosted Git server, Docker container `forgejo`, public at `https://forgejo.kingdezigns.com/` via NPM, SSH git access LAN-only on port 2222, SQLite DB, data at `/opt/forgejo/data` (folded into `nas16-backup.sh`). Hosts version control for `07 - Projects/NAS08`, `/NAS16`, `/PLEX32` — one Forgejo org per host, one `<hostname>-scripts` repo per org. See `server_forgejo.md` for full detail.
- **Shared permissions:** `ncshare` group (GID 1001) shared by `www-data` and `rufusking` — must match NAS08 GID for rsync compatibility.
- Default ACLs on `/export/kingdezignsnas-16/Public` ensure all rsync-written files inherit `ncshare` group permissions.
- **NAS08→NAS16 sync script self-heals destination permissions on every run** — checks for `www-data:ncshare` ownership, runs `chown`/`chmod`/`setfacl` automatically if wrong. Flagged in email subject and body if it fires.
- Sync script uses `--no-perms` and `--omit-dir-times` for NFS `all_squash` compatibility — all individual file timestamps are fully preserved; only directory timestamps are skipped (intentional, not meaningful for backup).
- **Do not manually run chown/chmod/setfacl on the sync destination** — the script handles this automatically.

---

# ✔️ End of File