obsidian-vault/personal/tech/truenas-nfs4-acl-and-arrmultiuser.md

---
title: TrueNAS NFS4 ACL repair + cross-container media PUID
created: 2026-09-02
updated: 2026-09-22
type: tech
namespace: personal
tags: [truenas, nfs4, acl, midclt, arr, radarr, sonarr, jellyfin, transmission, docker, git]
related:
  - "[[truenas-zfs-panic-recovery]]"
  - "[[arr-stack-taiga]]"
  - "[[watchlist-disk-overflow-fix]]"
  - "[[2026-09-21-bare-repo-push-permission-denied]]"
---

# TrueNAS NFS4 ACL repair + cross-container media PUID

> Hard-won lessons from the 2026-08-31/09-01/09-02 TrueNAS restore-incident remediation. Mirror of skill `truenas-nfs4-acl-and-arrmultiuser`. Detailed incident log: `family/documents/vault-sync/2026-09-02-restore-privilege-scope.md`.
>
> **2026-09-21 addendum:** the same class of failure hit the **vault bare-git repo** (`/mnt/RED_2TB/storage/git/obsidian-vault.git`) — `git push` failed with `unable to write file ./objects/XX/…: Permission denied`. Root cause and the working midclt recipe are in the section «Bare git repo: push Permission denied» below.
>
> **2026-09-22 addendum (РЕЦИДИВ, закрыт):** дефект вернулся (35 root-каталогов). Починено:
> **236/236 каталогов `INHERIT` + `WRITE`**, push прошёл end-to-end, поставлен cron-сторож.
> 🔴 Ключевые уроки: `dacl` **с нуля собрать НЕЛЬЗЯ** (TrueNAS требует `id` в каждой ACE —
> только read-modify-write); флаги = `{"BASIC":"INHERIT"}`, НЕ булев dict; **`setacl` асинхронный —
> обязательно ждать job до `SUCCESS`**, иначе ложный провал. Рецепт и 4 питфолла — в секции
> «Durable fix with INHERITANCE» ниже. Инструменты: `/Users/admin/Developer/vault-tools/`.
> Хронология расследования: `family/documents/vault-sync/2026-09-21-bare-repo-push-permission-denied.md`.


## Trigger
Any "permission denied" / file not readable / git `loose object ... corrupt` / empty Jellyfin library on **TrueNAS SCALE** (acltype=nfsv4) after a zfs restore, OR wiring Prowlarr/Radarr/Sonarr/Jellyfin/Transmission so container-written files are readable by other containers.

## Core facts
- TrueNAS SCALE uses **NFSv4 ACL**, not plain POSIX. `ls -la` shows a POSIX mask that can mislead.
- Read real ACL: `midclt call filesystem.getacl <path>`; write: `midclt call filesystem.setacl`.
- **`truenas_admin` is FULL_ADMIN in midclt** → can chown/setacl without passwordless root.
- Web UI ACL editing only works on zfs **datasets**, not arbitrary subdirs under one dataset → use midclt/CLI.
- Post-restore a tree may be owned by wrong uid (e.g. 921 `transmission`) with files `mode 0000`.

## The two big gotchas
### 1. setacl takes ONE JSON with `path` INSIDE it
`midclt call filesystem.setacl /path {…}` → `[EFAULT] Too many arguments (expected 1, found 2)`.
Correct form:
```
midclt call filesystem.setacl '{"path":"/x","uid":950,"gid":950,"acltype":"NFS4","dacl":[...],"options":{...}}'
```
Jobs run async → returns a job id; verify `midclt call core.get_jobs` (state `SUCCESS`).

### 2. stripacl does NOT remove DENY; full dacl replacement does
`options.stripacl:true` reported `SUCCESS` but left the `owner@ DENY` ACE intact. Only passing a **complete dacl array** works — setacl treats it as a full REPLACE of the ACL.

## NFSv4 DENY overrides ALLOW (the corrupt-object trap)
A file owned by uid 950 carrying `owner@ DENY READ_DATA=True` cannot be read by uid 950 itself. Git then reports `loose object ... corrupt` — this is **ACL, not data damage** (data intact; root reads fine).
Diagnostic trap: `git fsck` under **root** is clean, but fetch/push under the owning uid fails → ACL, not corruption.
Broken files often show POSIX `mode 40` (`r--------`); healthy objects `750`.

## Repair / cross-container recipe (linuxserver media stack)
Goal: all containers run under ONE uid (950) so transmission→radarr/sonarr→jellyfin all read each other's files.
- transmission: add `PUID=950 PGID=950` to env in `docker-compose.yml`. It mounts `storage`→`/mnt/storage`, download-dir `/mnt/storage/Downloads`.
- radarr/sonarr/prowlarr/jellyfin compose: add `PUID=950 PGID=950` to each (retain `TZ`).
- linuxserver s6 images (Entrypoint `/init`, `/etc/s6-overlay`) honor PUID/PGID natively. Default (unset) = runs **root** → creates root-owned files that 950-services can't read → always set PUID/PGID.
- Recreate from proper folder: `cd /mnt/RED_2TB/docker/<svc> && docker compose up -d` (+`--force-recreate` if env changed). Verify daemon uid via `ps aux` inside container (process as `abc`/950 = applied). NOTE `docker exec id` ≠ daemon uid.
- Simple datasets: recursive POSIX `chown -R 950:950` + `chmod` under root SUFFICE (ACLs were trivial). Use midclt/ACL only when non-trivial ACEs present (check `filesystem.getacl` → `trivial:true`).
- Bare git repo with `owner@ DENY` on objects → full dacl replacement recursively (perms JSON like above).

### GOTCHA 3 — root dataset traverse (Permission denied despite clean file ACL)
After chown-ing all leaf dirs/files to 950, if a service still gets `Permission denied` reading `/storage/Movies/...` (or jellyfin FFmpeg exit 243), check the **root of the dataset** `/mnt/RED_2TB/storage` itself. In the restore it stayed `921:921` with `owner@/group@ ALLOW` but `everyone@ EXECUTE=False`. uid 950 (not owner, not in gid-921-group) falls under `everyone@` → **cannot traverse past the root** into the tree, even though every nested ACL is clean. Fix (root):
```bash
chown 950:950 /mnt/RED_2TB/storage
chmod 750 /mnt/RED_2TB/storage    # rwxr-x---: gid 950 = truenas_admin (containers' group) gets r-x traverse
```
Kernel needs EXEC (traverse) on EVERY path component. Diagnostic via `setpriv --reuid=950 ... ls` shows `Permission denied` on a dir whose `getacl` looks clean → suspect root/parent traverse. `stat` via root shows clean 770 but uid 950 can't ls.

### GOTCHA 4 — transmission "all torrents No Data Found" after data migration
If every transmission torrent shows `error 3: No Data Found` right after a container recreate/pool migration:
- Check: `docker exec -u` fine to read? downloadDir & files present? If files ARE there and readable as uid 950, the error is likely **stale from daemon startup while traverse was blocked** (see GOTCHA 3).
- `torrent-verify` on a single torrent does NOT clear it. Fix is trivial & non-destructive:
```bash
docker restart transmission
```
On restart the daemon re-validates → torrents clear (e.g. **254/255 instantly**). Leftover #1 → Verify Local Data in Web UI.

## Bare git repo: `push` → `Permission denied` (2026-09-21)

**Symptom:**
```
error: unable to write file ./objects/80/09a34e…: Permission denied
To ssh://…/obsidian-vault.git
 ! [remote rejected] main -> main (unable to migrate objects to permanent storage)
```

**Root cause — TWO independent defects in the repo's dirs:**

1. **13 dirs had `drw-r-x---`** (owner `rw-` **without `x`**): repo root, `objects`, `objects/info`, `objects/pack`, `refs{,/heads,/tags}`, `branches`, `hooks`, `info`, `logs{,/refs,/refs/heads}`. Without owner-`x` nothing can be created inside. Fixed with `chmod u+x` on dirs only (files were all fine: `0 files without u+r`).
2. **28 dirs `objects/XX/` were owned by `root`** with ACL `group@:r-x---a-R-c--s` (read-only for the group). Git runs as uid 950 (`truenas_admin`), not root → **cannot write**. This is the defect that actually blocked the push.

**Why it happened:** `mcpvault` runs **as root** on TrueNAS — `/opt/hermes/.venv/bin/python3 … mcp_stdio_watchdog.py … mcpvault /vault` (`ps aux` shows uid `root`). Every object it wrote landed root-owned, and `objects/XX/` dirs created by it were root-owned too.

> ⚠️ **Do not confuse this with the Mac's vault.** This `mcpvault` belongs to a **second Hermes instance living on TrueNAS itself** (`/opt/hermes`), whose vault is `/vault` — **not** Alex's `~/obsidian` on the Mac. It is not "the agent's own MCP server". Both instances merely push to the **same shared bare repo**, which is why the collision happens. PID refs seen 2026-09-21: 13727 (`mcp_stdio_watchdog.py`), 13738 (`node … mcpvault /vault`).

With the inheritance fix below, root-owned objects are no longer fatal (rights are inherited from the now-writable parent), but **the root instance is still a latent hazard** — restarting it as uid 950 removes the cause. Left as an open item.

**What does NOT work:**
- `chown -R 950:950` and `chmod -R g+w` via SSH as `truenas_admin` → **`Operation not permitted`** (uid 950 is not the owner of root-owned objects). `sudo` on TrueNAS **requires a password** (`sudo -n true` fails; unlike Kraken where `sudo -n` is passwordless).
- `midclt call filesystem.setperm '{...}'` → `[EINVAL] Non-trivial ACL present … Option "stripacl" required to change permission`. **Don't reach for `stripacl`** — it tears the ACL down to POSIX.

**What DOES work — `filesystem.setacl` via midclt (no root needed):**
uid 950 can read and write ACLs through midclt. Read the current ACL, add the group perms, write it back. Reference implementation (`/tmp/fix_vault_acl.py` on TrueNAS, ~120 dirs, ~90 s):

```python
FOR each dir in [repo] + listdir(f"{repo}/objects"):
    cur = midclt("filesystem.getacl", path)          # path as BARE STRING
    for ace in cur["acl"]:
        if ace["tag"] == "group@" and ace["type"] == "ALLOW":
            ace["perms"]["WRITE_DATA"]    = True
            ace["perms"]["APPEND_DATA"]   = True
            ace["perms"]["EXECUTE"]       = True
            ace["perms"]["WRITE_ATTRIBUTES"] = True
        if ace["tag"] == "owner@" and ace["type"] == "ALLOW":
            ace["perms"]["EXECUTE"] = True           # root-owned dirs had this off
    midclt("filesystem.setacl", {
        "path": path, "dacl": cur["acl"],
        "options": {"recursive": False, "traverse": False, "stripacl": False},
    })
    # setacl returns a JOB ID → poll core.get_jobs until SUCCESS/FAILED
```

**⚠️ midclt argument-form pitfalls (cost real time):**
- 🔴🔴 **Каждая ACE требует поле `id`.** Собирать dacl «с нуля» как `{tag, type, perms, flags}` **НЕЛЬЗЯ** —
  TrueNAS отклонит: `[EINVAL] filesystem_acl.dacl.0: ACL entry is missing required keys(s): {'id'}`
  (проверено 2026-09-22: 245 каталогов, 0 успешных). Работает только **read-modify-write**:
  взять ACL через `getacl` (там `id` уже проставлен), править в нём **только `perms`/`flags`**,
  передать `cur["acl"]` обратно в `setacl`. Именно так был сделан успешный прогон 2026-09-21 (291 каталог).
- `filesystem.getacl` и `filesystem.listdir` take a **BARE STRING path**, NOT `{"path": …}`. Passing an object gives `[EINVAL] path: Not a string`.
- `filesystem.setacl` takes **ONE JSON object with `path` INSIDE it** (see gotcha #1 above) — the opposite convention from getacl on the same service. Easy to invert.
- `filesystem.setacl` / `setperm` are **async**: they return a job id. Polling immediately returns state `WAITING` → looks like failure. Must poll until `SUCCESS`/`FAILED`. With ~120 dirs over SSH the loop outlives the SSH timeout — **run it detached** (`nohup … &`) and read the log later; the first SSH-timed-out run had in fact already applied the ACLs.
- Passwordless `sudo` is available on **Kraken** (`sudo -n true` → ok) but **NOT** on TrueNAS `truenas_admin`.

**Verification (cheap, do it before re-trying push):**
```bash
touch /mnt/RED_2TB/storage/git/obsidian-vault.git/objects/80/.wrtest && echo WRITE_OK
```
`WRITE OK` → push will succeed. Confirmed: `f9a5c2b..21505ec main -> main`, then a clean `git pull` on Kraken.

### Durable fix with INHERITANCE (2026-09-21 — 291 dirs, 2026-09-22 — 236 dirs)

The minimal recipe above (add perms to existing `group@` ACE) restores write for *current* dirs, **but the bug returns**: the next push by the root-owned `mcpvault` creates a NEW `objects/XX/` dir whose ACL is inherited from the *parent*, and if the parent's `everyone@`/`group@` lacks `WRITE_DATA` + inherit flags, the new dir is root-owned and read-only again → push fails again.

🔴 **RE-CUR 2026-09-22.** Дефект вернулся (35 root-каталогов). Итог фикса: **236/236 каталогов
с `INHERIT` + `WRITE`**, push прошёл end-to-end. Порядок работ и уроки — в
`family/documents/vault-sync/2026-09-21-bare-repo-push-permission-denied.md` §«✅ ФИНАЛЬНЫЙ ФИКС».

**Working fix = read-modify-write, НЕ сборка dacl с нуля:**

```python
# perms granted to owner@ AND group@ AND everyone@:
#   READ_DATA WRITE_DATA APPEND_DATA EXECUTE DELETE_CHILD
#   READ_ATTRIBUTES WRITE_ATTRIBUTES READ_NAMED_ATTRS WRITE_NAMED_ATTRS
#   READ_ACL (and WRITE_ACL WRITE_OWNER DELETE for owner@)
# flags задаются КАК СТРОКА, не булев dict:
#   ace["flags"] = {"BASIC": "INHERIT"}      ← FILE_INHERIT + DIRECTORY_INHERIT
# call:  midclt call filesystem.setacl '<one json with path + dacl + options>'
#        options = {"recursive": False, "traverse": True, "stripacl": False}
```

🔴 **Четыре питфолла, стоившие времени:**

1. **Собирать `dacl` «с нуля» НЕЛЬЗЯ.** TrueNAS требует поле `id` в каждой ACE:
   `[EINVAL] filesystem_acl.dacl.0: ACL entry is missing required keys(s): {'id'}`.
   Работает только **read-modify-write**: `getacl` → править `perms`/`flags` → `setacl`.
2. **Флаги — `{"BASIC":"INHERIT"}`, НЕ `{"FILE_INHERIT":True,"DIRECTORY_INHERIT":True}`.**
   Булев dict отклоняется той же ошибкой про `id` (при неверном формате TrueNAS
   пересобирает ACE и теряет поле).
3. **`setacl` АСИНХРОННЫЙ — надо ЖДАТЬ job до `SUCCESS`.** Проверка `getacl` сразу после
   вызова видит СТАРЫЕ флаги → ложный `fail`. Признак гонки: `fixed=0, already=N, fail=M`.
   ```python
   jid = int(midclt("filesystem.setacl", arg))
   while deadline:
       j = json.loads(midclt("core.get_jobs", f'[["id","=",{jid}]]'))
       if j[0]["state"] in ("SUCCESS","FAILED","ABORTED"): break
       time.sleep(1)
   ```
4. **Владелец `root` сам по себе НЕ блокирует запись.** `getacl` даёт `uid:0, gid:950`,
   запись от uid 950 идёт через `group@` (gid 950). `write-тест` в 35 root-каталогов:
   OK=35 FAIL=0.

**Верификация (обязательна, делай до retry push):**
```bash
touch <repo>/objects/XX/.wrtest && echo WRITE_OK     # у КАЖДОГО каталога
# + тест наследования: mkdir objects/zzz → getacl → должен быть INHERITED:True → touch внутрь
# + end-to-end: git clone <repo> && cd && echo x > f && git add f && git commit && git push
```

**Сторож (поставлен 2026-09-22):**
```
~/scripts/vault-acl-watchdog.py     (TrueNAS, от truenas_admin)
cron: 0 */2 * * * python3 ~/scripts/vault-acl-watchdog.py >> ~/scripts/vault-acl-watchdog.log 2>&1
```
SILENT если всё ок; чинит + докладывает если сломано.

⚠️ **Корень НЕ устранён:** `mcpvault` контейнера `hermes-taiga`
(`/mnt/RED_2TB/docker/hermes/docker-compose.yml`, `HERMES_ALLOW_ROOT_GATEWAY=1`, `User:` пусто)
работает от root. Перманентный фикс — добавить `user: "950:950"` в `taiga` service.

Working fix = **full dacl replacement** (not per-ACE edit) with **inheritance flags ON**, applied to every dir:

```python
# perms granted to owner@ AND group@ AND everyone@:
#   READ_DATA WRITE_DATA APPEND_DATA EXECUTE DELETE_CHILD
#   READ_ATTRIBUTES WRITE_ATTRIBUTES READ_NAMED_ATTRS WRITE_NAMED_ATTRS
#   READ_ACL (and WRITE_ACL WRITE_OWNER DELETE for owner@)
# flags on each ACE:  {"FILE_INHERIT": True, "DIRECTORY_INHERIT": True,
#                      "NO_PROPAGATE_INHERIT": False, "INHERIT_ONLY": False,
#                      "INHERITED": False}
# call:  midclt call filesystem.setacl '<one json with path + dacl + options>'
#        options = {"recursive": False, "traverse": True, "stripacl": False}
```

**Result:** 291 dirs processed, 0 failures. Note `filesystem.listdir <BARE STRING>` to enumerate (same convention as `getacl`), and each `setacl` is async — but 291 sequential dirs finished inside one SSH call (~2 min), no `nohup` needed this time.

⚠️ **Caveat — the ACL was replaced, not proven functional yet.** The write-test `touch objects/80/.wrtest` was NOT executed (approval timed out). Verification was indirect: `git push nas main` → `Everything up-to-date`, and `git rev-parse HEAD == git rev-parse nas/main` (`177da1e`). That proves the *current* tree is in sync, **not** that a fresh root-owned object can be written. A real `touch` in `objects/80/` as uid 950 is still the definitive test.

**Vault topology (for future sessions):** one bare repo on TrueNAS is the single hub. Mac pushes/pulls remote **`nas`**; Kraken uses remote **`origin`** — both point at `ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git`. Mac and Kraken never talk directly. Host `truenas` does **not** resolve as an SSH alias from the Mac — use `truenas_admin@mallexxx.duckdns.org`.

## Docker network persistency (per-service compose)
- `docker network connect <net> <container>` is LOST on recreate → declare `<net>` in the service's compose `networks:`.
- Caddy must co-own a network with its reverse-proxy target. transmission Web UI works because caddy is ALSO in `transmission_default`; radarr↔transmission works because both in `media_net`.
- `external: true` networks (media_net/caddy_default) aren't created by compose — if they vanish (pool rebuild), `docker network create media_net`.

## Containers / uid summary (as of 2026-09-02, TrueNAS "Taiga")
- prowlarr/radarr/sonarr/jellyfin: all running, all daemons under **uid 950**, project `arr`.
- transmission: PUID/PGID=950, in networks `media_net` + `transmission_default` (both external).
- Radarr/sonarr/jellyfin mount `/mnt/RED_2TB/storage` as `/storage` (NOT `/media` as on Kraken) → set radarr root-folder accordingly.