346 lines
16 KiB
Markdown
346 lines
16 KiB
Markdown
# NAS16 — LAMP Stack & Database Server
|
||
Device Type: **Raspberry Pi**
|
||
Hostname: **NAS16**
|
||
IP Address: **192.168.150.40**
|
||
VLAN: **50 — Lab / Servers**
|
||
|
||
Last Updated: 2026-06-24
|
||
|
||
---
|
||
|
||
## 🧩 Role & Purpose
|
||
NAS16 is the **dedicated LAMP stack and database server** for the KingDezigns network.
|
||
It runs **OpenMediaVault (OMV)** as its base OS, which manages nginx as the web GUI frontend.
|
||
It provides backend services, web hosting, and database support for internal applications, development environments, and externally accessible virtual host websites.
|
||
|
||
In addition to its LAMP role, NAS16 hosts the **CrowdSec external notification system** — a pair of scripts that poll the CrowdSec LAPI on HAS and deliver split email alerts based on ban origin.
|
||
|
||
This system is designed for:
|
||
- Web application hosting (internal and external)
|
||
- MariaDB database services
|
||
- Internal development and testing
|
||
- Lightweight administrative tools (Webmin via OMV)
|
||
- External virtual host websites proxied through HAS (NPM)
|
||
- CrowdSec email notification delivery (local attack alerts + CAPI daily digest)
|
||
|
||
NAS16 operates entirely within VLAN 50 and benefits from the strict containment rules that prevent lateral movement into trusted networks.
|
||
|
||
---
|
||
|
||
## 🖥️ Hardware
|
||
- Raspberry Pi (model unspecified)
|
||
- SSD storage (see drive inventory below)
|
||
- Gigabit Ethernet
|
||
|
||
### **Drive Inventory**
|
||
| Device | Model | Capacity | Role |
|
||
|--------|-------|----------|------|
|
||
| /dev/sda | Unknown (standard SMART attrs) | ~256GB | OS / system drive |
|
||
|
||
> Note: Drive model unconfirmed — uses standard SMART attribute names (Wear_Leveling_Count, Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable). Power_Cycle_Count of 7,079 at 891 power-on hours noted — likely reflects frequent reboots during setup. Monitor over time.
|
||
|
||
---
|
||
|
||
## 🌐 Network Placement
|
||
- VLAN: **50 — Lab / Servers**
|
||
- IP: **192.168.150.40**
|
||
- Access Type: LAN / Cable
|
||
|
||
This placement ensures:
|
||
- Isolation from trusted user devices
|
||
- Controlled access to sensitive database services
|
||
- Local access to Pi‑hole DNS
|
||
- Reduced attack surface
|
||
|
||
---
|
||
|
||
## 📦 Primary Functions
|
||
|
||
### **Core Services**
|
||
- **OpenMediaVault (OMV)** — base OS and NAS management UI
|
||
- **LAMP Stack**
|
||
- Linux
|
||
- Apache (serves internal and external virtual host websites)
|
||
- MariaDB
|
||
- PHP
|
||
- **Webmin** (administrative interface, managed via OMV)
|
||
|
||
### **Additional Roles**
|
||
- Internal web hosting
|
||
- External virtual host websites (proxied via NPM on HAS)
|
||
- Database backend for internal applications
|
||
- Development and testing environment
|
||
- **CrowdSec external notification system** (see below)
|
||
|
||
---
|
||
|
||
## 🌍 External Access Architecture
|
||
|
||
NAS16 hosts multiple public-facing websites served by Apache.
|
||
External traffic reaches NAS16 via the following proxy chain:
|
||
|
||
```
|
||
Internet → NPM on HAS (192.168.150.30:80/443) → nginx on NAS16 (OMV frontend) → Apache (virtual hosts)
|
||
```
|
||
|
||
### **Real IP Forwarding**
|
||
Because all proxied traffic arrives at NAS16 from HAS's IP (192.168.150.30), both nginx and Apache are configured to extract and log the real client IP from the `X-Forwarded-For` header passed by NPM.
|
||
|
||
**nginx** — `/etc/nginx/conf.d/real_ip.conf`:
|
||
```nginx
|
||
set_real_ip_from 192.168.150.30;
|
||
real_ip_header X-Forwarded-For;
|
||
real_ip_recursive on;
|
||
```
|
||
|
||
**Apache** — `/etc/apache2/conf-available/real_ip.conf`:
|
||
```apache
|
||
RemoteIPHeader X-Forwarded-For
|
||
RemoteIPTrustedProxy 192.168.150.30
|
||
```
|
||
|
||
Apache module enabled: `remoteip`
|
||
Config enabled: `real_ip`
|
||
|
||
This ensures Fail2Ban sees and bans real attacker IPs, not HAS's proxy IP.
|
||
|
||
---
|
||
|
||
## 🔔 CrowdSec External Notification System
|
||
|
||
NAS16 is the sole email notification handler for CrowdSec. Native CrowdSec notifications are disabled on HAS. NAS16 polls the CrowdSec LAPI stream endpoint and routes decisions by origin.
|
||
|
||
### **Why NAS16**
|
||
- Offloads notification work from HAS (which runs HA, NPM, CrowdSec Agent + Bouncer)
|
||
- NAS16 has the most headroom of any Pi in the fleet (16GB RAM, consistently underutilized)
|
||
- Fits cleanly into existing OMV Scheduled Jobs infrastructure
|
||
|
||
### **Architecture**
|
||
```
|
||
CrowdSec LAPI on HAS (192.168.150.30:8080)
|
||
↓
|
||
NAS16 polls /v1/decisions/stream every 5 min
|
||
↓
|
||
origin = crowdsec? origin = CAPI?
|
||
↓ ↓
|
||
Append to local_pending.json Append to capi_digest.json
|
||
↓ ↓
|
||
Every 4 hours: Midnight: send digest
|
||
send batched digest email + clear log
|
||
email + clear log
|
||
```
|
||
|
||
### **Scripts**
|
||
| Script | Path | Purpose |
|
||
|--------|------|---------|
|
||
| cs_notifier.sh | `/usr/scripts/crowdsec/cs_notifier.sh` | Polls LAPI stream every 5 min, routes decisions by origin into pending log files (no emails sent) |
|
||
| cs_local_digest.sh | `/usr/scripts/crowdsec/cs_local_digest.sh` | Reads local_pending.json, sends batched local-attack digest every 4 hours, clears log |
|
||
| cs_digest.sh | `/usr/scripts/crowdsec/cs_digest.sh` | Reads CAPI digest log, builds summary email, sends at midnight, clears log |
|
||
|
||
### **Key Design Details**
|
||
- Uses `/v1/decisions/stream?startup=true` on first run to initialize cursor silently (no emails for existing decisions)
|
||
- Uses `/v1/decisions/stream?startup=false` on all subsequent runs — LAPI maintains cursor server-side per API key
|
||
- First run flag: `/var/log/crowdsec-notifier/stream_initialized.flag`
|
||
- Local pending log: `/var/log/crowdsec-notifier/local_pending.json` (one JSON decision per line, cleared every 4 hours after digest send)
|
||
- CAPI digest log: `/var/log/crowdsec-notifier/capi_digest.json` (one JSON decision per line, cleared at midnight after digest send)
|
||
- Notifier log: `/var/log/crowdsec-notifier/notifier.log`
|
||
- All email sending uses inline Python with `timeout=15` on SMTP_SSL to prevent hangs
|
||
- Digest scripts use a single Python call to process all decisions — handles large batches efficiently
|
||
- **IMPORTANT:** Python heredocs (`python3 << PYEOF`) do NOT read from a preceding pipe — the heredoc itself becomes stdin. Data must be passed as an inline shell variable inside the heredoc (e.g. `snapshot = """$SNAPSHOT"""`), not piped in.
|
||
|
||
### **Email Notifications**
|
||
| Type | Trigger | Style | Content |
|
||
|------|---------|-------|---------|
|
||
| Local attack digest | Every 4 hours (cron: `0 0,4,8,12,16,20 * * *`) — skipped if zero bans | Red / CRITICAL | Stat cards (ban count, origin, CAPI today) + table of all banned IPs with scenario, type, duration, scope, detected time, and CTI/WHOIS/AbuseIPDB links |
|
||
| CAPI daily digest | Midnight cron | Amber / WARNING (or green if zero bans) | Total ban count, top 10 scenarios, first 20 sample IPs |
|
||
|
||
> **Note:** Local attack alerting was changed from immediate per-IP emails (one email per ban, dozens-to-hundreds/week) to a 4-hour batched digest to reduce notification volume while preserving full visibility into attack activity.
|
||
|
||
### **CrowdSec API Key**
|
||
- Bouncer name: `NAS16-notifier`
|
||
- Credentials stored in Vaultwarden under `CrowdSec NAS16-notifier`
|
||
- LAPI endpoint: `http://192.168.150.30:8080`
|
||
|
||
### **LAPI Port Requirement**
|
||
Port 8080 must be exposed on HAS's host network interface. Configured in the CrowdSec add-on Network settings. If HAS is rebuilt or the add-on is reinstalled, re-enable port 8080 mapping.
|
||
|
||
---
|
||
|
||
## 🗂️ Shared Permissions — NFS + rsync Backup
|
||
|
||
NAS16 receives rsync backups from NAS08 via `nas08_to_nas16_sync.sh` every 2 days. To ensure rsync can write files on the destination without errors, NAS16 uses the same `ncshare` group configuration as NAS08.
|
||
|
||
### **Group Configuration**
|
||
| Item | Value |
|
||
|------|-------|
|
||
| Group name | `ncshare` |
|
||
| GID | `1001` |
|
||
| Members | `www-data`, `rufusking` |
|
||
|
||
> GID **must match NAS08** (1001) — rsync syncs permissions across both systems and they must be consistent.
|
||
|
||
Verify with:
|
||
```bash
|
||
getent group ncshare
|
||
```
|
||
|
||
### **Directory Permissions**
|
||
Applied to `/export/kingdezignsnas-16/Public` (and all subdirectories recursively):
|
||
```bash
|
||
sudo chown -R www-data:ncshare /export/kingdezignsnas-16/Public
|
||
sudo chmod -R 2770 /export/kingdezignsnas-16/Public
|
||
sudo setfacl -R -m g:ncshare:rwx /export/kingdezignsnas-16/Public
|
||
sudo setfacl -R -d -m g:ncshare:rwx /export/kingdezignsnas-16/Public
|
||
```
|
||
|
||
- `2770` (setgid) ensures new items inherit the `ncshare` group
|
||
- Default ACLs (`-d`) ensure all new files/folders created by rsync inherit `rwx` for `ncshare`
|
||
|
||
> **Note:** You should not need to run these commands manually. The sync script checks and self-corrects destination ownership before every run. These are documented here as reference only.
|
||
|
||
### **Backup Script**
|
||
The sync runs on NAS16 via OMV Scheduled Jobs:
|
||
```bash
|
||
/usr/scripts/nas/nas08_to_nas16_sync.sh
|
||
```
|
||
- Source: `/mnt/kingdezigns08/Public/` (NAS08 mounted via NFS)
|
||
- Destination: `/export/kingdezignsnas-16/Public/`
|
||
- Schedule: Every 2 days at 2:30 AM
|
||
|
||
### **rsync Flags — NFS all_squash Compatibility**
|
||
The sync script uses the following rsync flags to work correctly over NFS `all_squash`:
|
||
|
||
```
|
||
--archive Recursive, preserve symlinks, owner, group, devices
|
||
--no-perms Skip chmod — NFS all_squash prevents non-owner chmod
|
||
--omit-dir-times Skip directory utimes — NFS all_squash prevents utimes on dirs not owned by www-data
|
||
--update Skip files that are newer on destination
|
||
```
|
||
|
||
- **File timestamps are fully preserved** — `--omit-dir-times` only skips directory mtimes, not file mtimes
|
||
- **Directory timestamps are intentionally skipped** — directory mtimes are not meaningful for backup purposes
|
||
- **Permissions are not synced** — destination ACLs (`ncshare` group, `2770`) control access instead
|
||
|
||
### **Permission Precheck (Automatic)**
|
||
Before every sync run, the script checks that the destination root is owned by `www-data:ncshare`. If it is not, it automatically runs `chown`, `chmod`, and `setfacl` to correct it before rsync starts. This is logged and flagged in the email report with an amber warning notice if it fires.
|
||
|
||
A clean run looks like:
|
||
```
|
||
[TIMESTAMP] Destination ownership OK (www-data:ncshare).
|
||
[TIMESTAMP] ===== NAS08 → NAS16 Sync Started =====
|
||
[TIMESTAMP] ===== Sync Completed Successfully =====
|
||
Done — Status: SUCCESS | Transferred: N file(s) | Duration: Xm Xs
|
||
```
|
||
|
||
---
|
||
|
||
## 🔒 Required Firewall Behavior
|
||
|
||
### **Inbound to NAS16**
|
||
- Allowed from VLAN 1 (Infrastructure)
|
||
- Allowed from VLAN 20 (Trusted)
|
||
- Allowed from VLAN 50 (local VLAN)
|
||
- Proxied external traffic arrives from HAS (192.168.150.30) via NPM
|
||
- IoT and Guest access depend on default policy
|
||
- No explicit inbound blocks except those applied to VLAN 50 as a whole
|
||
|
||
### **Outbound from NAS16**
|
||
- Subject to VLAN 50's global final drop:
|
||
- Cannot initiate to VLANs 1, 10, 20, 30, 40
|
||
- Allowed exceptions:
|
||
- DNS responses
|
||
- Local VLAN 50 traffic (including LAPI calls to HAS at 192.168.150.30:8080)
|
||
|
||
### **Critical Requirement**
|
||
- Must remain reachable from trusted administrative devices for management and database access.
|
||
- Must remain reachable from HAS (192.168.150.30) for proxied external traffic.
|
||
- Must be able to reach HAS LAPI at 192.168.150.30:8080 for CrowdSec notification polling.
|
||
|
||
---
|
||
|
||
## 🔐 Security Measures
|
||
- **Fail2Ban enabled** (via OMV Fail2Ban plugin)
|
||
- **Fail2Ban ignoreip** includes `192.168.150.0/24` — prevents HAS proxy IP from being banned
|
||
- **Real IP forwarding** configured in nginx and Apache — ensures real attacker IPs are logged and banned
|
||
- Webmin access restricted to trusted VLANs
|
||
- Database ports not exposed outside VLAN 50
|
||
- VLAN‑level containment prevents lateral movement
|
||
|
||
### **Fail2Ban Configuration**
|
||
`/etc/fail2ban/jail.local`:
|
||
```ini
|
||
[DEFAULT]
|
||
ignoreip = 127.0.0.1/8 ::1 192.168.150.0/24
|
||
```
|
||
|
||
This ensures internal VLAN 50 devices (including HAS) are never banned regardless of proxied traffic volume.
|
||
|
||
---
|
||
|
||
## 🛠️ SSD Hardening & Maintenance
|
||
|
||
### **TRIM**
|
||
- TRIM support confirmed on all drives (`DISC-GRAN: 4K`, `DISC-MAX: 4G`)
|
||
- systemd `fstrim.timer` **disabled** — replaced by OMV Scheduled Job
|
||
- Weekly TRIM script: `/usr/scripts/omv/fstrim-report.sh`
|
||
- Scheduled: **Wednesday at 1:30 AM** via OMV Scheduled Jobs
|
||
- Script runs `fstrim -av`, gathers SMART health data, sends HTML email report, saves dated log to `/var/log/fstrim/`
|
||
- Log retention: 90 days
|
||
|
||
### **noatime**
|
||
- All filesystems mounted with `noatime` — confirmed via `findmnt`
|
||
- Managed automatically by the `openmediavault-flashmemory` plugin
|
||
- No manual configuration required
|
||
|
||
### **openmediavault-flashmemory Plugin**
|
||
- Installed and active
|
||
- Automatically handles: `noatime`, tmpfs for `/var/log` and `/tmp`, swap disabled
|
||
- Significantly reduces unnecessary SSD write amplification
|
||
|
||
### **SMART Monitoring**
|
||
- Monitored weekly as part of the TRIM report script
|
||
- Drive uses standard SMART attribute names — no vendor-specific mapping required
|
||
|
||
---
|
||
|
||
## 📅 Scheduled Jobs (OMV)
|
||
|
||
| Schedule | Script | Tag |
|
||
|----------|--------|-----|
|
||
| Every 3 days at 2:00 AM | `/usr/scripts/omv/nas16-backup.sh` | NAS16 Backup |
|
||
| Every 2 days at 2:30 AM | `bash /usr/scripts/nas/nas08_to_nas16_sync.sh` | NAS Drive Backup |
|
||
| Daily at 12:00 AM | `/usr/scripts/zfs/nas16_zfs_report.sh` | ZFS Status Report |
|
||
| Sunday at 3:00 AM | `/usr/scripts/zfs/nas16_zfs_scrub.sh` | ZFS Scrub |
|
||
| Every 15 minutes | `/usr/scripts/ddns/ddns_update.sh` | DDNS Update |
|
||
| Wednesday at 1:30 AM | `/usr/scripts/omv/fstrim-report.sh` | TRIM Report |
|
||
| Every 5 minutes | `/usr/scripts/crowdsec/cs_notifier.sh` | CrowdSec Local Notifier |
|
||
| Every 4 hours (0:00, 4:00, 8:00, 12:00, 16:00, 20:00) | `/usr/scripts/crowdsec/cs_local_digest.sh` | CrowdSec Local Attack Digest |
|
||
| Daily at 12:00 AM | `/usr/scripts/crowdsec/cs_digest.sh` | CrowdSec CAPI Digest |
|
||
|
||
---
|
||
|
||
## 🧠 Summary for AI Systems
|
||
- NAS16 = **OMV-based LAMP + database server + CrowdSec notifier** in VLAN 50.
|
||
- Inbound allowed from VLAN 1, VLAN 20, local VLAN 50, and HAS proxy (192.168.150.30).
|
||
- Outbound blocked to all VLANs except local + DNS responses.
|
||
- Hosts Apache (virtual hosts), MariaDB, PHP, Webmin, and OMV management UI.
|
||
- External traffic is proxied through NPM on HAS — real IPs forwarded via X-Forwarded-For.
|
||
- nginx and Apache both configured to log real client IPs, not proxy IP.
|
||
- Fail2Ban ignoreip covers VLAN 50 to prevent HAS from being banned.
|
||
- Protected by VLAN 50's strict containment rules.
|
||
- SSD hardening active: TRIM (weekly Wednesday 1:30AM), noatime (flashmemory plugin), SMART monitoring via weekly HTML email report.
|
||
- High Power_Cycle_Count (7,079) relative to power-on hours (891) — monitor trend over time.
|
||
- **Hosts CrowdSec notification scripts** — polls HAS LAPI stream at 192.168.150.30:8080 every 5 minutes.
|
||
- Local attack decisions are batched and sent as one digest email every 4 hours (skipped if zero bans) — no longer one email per ban.
|
||
- CAPI cloud ban digest sent daily at midnight — single Python call handles 10,000+ decisions efficiently.
|
||
- CrowdSec API key stored in Vaultwarden under `CrowdSec NAS16-notifier`.
|
||
- **Gotcha:** when piping data into `python3 -` with a heredoc, the heredoc replaces stdin — pass data as an inline variable instead.
|
||
- **Shared permissions:** `ncshare` group (GID 1001) shared by `www-data` and `rufusking` — must match NAS08 GID for rsync compatibility.
|
||
- Default ACLs on `/export/kingdezignsnas-16/Public` ensure all rsync-written files inherit `ncshare` group permissions.
|
||
- **NAS08→NAS16 sync script self-heals destination permissions on every run** — checks for `www-data:ncshare` ownership, runs `chown`/`chmod`/`setfacl` automatically if wrong. Flagged in email subject and body if it fires.
|
||
- Sync script uses `--no-perms` and `--omit-dir-times` for NFS `all_squash` compatibility — all individual file timestamps are fully preserved; only directory timestamps are skipped (intentional, not meaningful for backup).
|
||
- **Do not manually run chown/chmod/setfacl on the sync destination** — the script handles this automatically.
|
||
|
||
---
|
||
|
||
# ✔️ End of File
|