13 KiB
title, created, updated, type, namespace, tags, related
| title | created | updated | type | namespace | tags | related | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TrueNAS NFS4 ACL repair + cross-container media PUID | 2026-09-02 | 2026-09-21 | tech | personal |
|
|
TrueNAS NFS4 ACL repair + cross-container media PUID
Hard-won lessons from the 2026-08-31/09-01/09-02 TrueNAS restore-incident remediation. Mirror of skill
truenas-nfs4-acl-and-arrmultiuser. Detailed incident log:family/documents/vault-sync/2026-09-02-restore-privilege-scope.md.2026-09-21 addendum: the same class of failure hit the vault bare-git repo (
/mnt/RED_2TB/storage/git/obsidian-vault.git) —git pushfailed withunable to write file ./objects/XX/…: Permission denied. Root cause and the working midclt recipe are in the section «Bare git repo: push Permission denied» below.
Trigger
Any "permission denied" / file not readable / git loose object ... corrupt / empty Jellyfin library on TrueNAS SCALE (acltype=nfsv4) after a zfs restore, OR wiring Prowlarr/Radarr/Sonarr/Jellyfin/Transmission so container-written files are readable by other containers.
Core facts
- TrueNAS SCALE uses NFSv4 ACL, not plain POSIX.
ls -lashows a POSIX mask that can mislead. - Read real ACL:
midclt call filesystem.getacl <path>; write:midclt call filesystem.setacl. truenas_adminis FULL_ADMIN in midclt → can chown/setacl without passwordless root.- Web UI ACL editing only works on zfs datasets, not arbitrary subdirs under one dataset → use midclt/CLI.
- Post-restore a tree may be owned by wrong uid (e.g. 921
transmission) with filesmode 0000.
The two big gotchas
1. setacl takes ONE JSON with path INSIDE it
midclt call filesystem.setacl /path {…} → [EFAULT] Too many arguments (expected 1, found 2).
Correct form:
midclt call filesystem.setacl '{"path":"/x","uid":950,"gid":950,"acltype":"NFS4","dacl":[...],"options":{...}}'
Jobs run async → returns a job id; verify midclt call core.get_jobs (state SUCCESS).
2. stripacl does NOT remove DENY; full dacl replacement does
options.stripacl:true reported SUCCESS but left the owner@ DENY ACE intact. Only passing a complete dacl array works — setacl treats it as a full REPLACE of the ACL.
NFSv4 DENY overrides ALLOW (the corrupt-object trap)
A file owned by uid 950 carrying owner@ DENY READ_DATA=True cannot be read by uid 950 itself. Git then reports loose object ... corrupt — this is ACL, not data damage (data intact; root reads fine).
Diagnostic trap: git fsck under root is clean, but fetch/push under the owning uid fails → ACL, not corruption.
Broken files often show POSIX mode 40 (r--------); healthy objects 750.
Repair / cross-container recipe (linuxserver media stack)
Goal: all containers run under ONE uid (950) so transmission→radarr/sonarr→jellyfin all read each other's files.
- transmission: add
PUID=950 PGID=950to env indocker-compose.yml. It mountsstorage→/mnt/storage, download-dir/mnt/storage/Downloads. - radarr/sonarr/prowlarr/jellyfin compose: add
PUID=950 PGID=950to each (retainTZ). - linuxserver s6 images (Entrypoint
/init,/etc/s6-overlay) honor PUID/PGID natively. Default (unset) = runs root → creates root-owned files that 950-services can't read → always set PUID/PGID. - Recreate from proper folder:
cd /mnt/RED_2TB/docker/<svc> && docker compose up -d(+--force-recreateif env changed). Verify daemon uid viaps auxinside container (process asabc/950 = applied). NOTEdocker exec id≠ daemon uid. - Simple datasets: recursive POSIX
chown -R 950:950+chmodunder root SUFFICE (ACLs were trivial). Use midclt/ACL only when non-trivial ACEs present (checkfilesystem.getacl→trivial:true). - Bare git repo with
owner@ DENYon objects → full dacl replacement recursively (perms JSON like above).
GOTCHA 3 — root dataset traverse (Permission denied despite clean file ACL)
After chown-ing all leaf dirs/files to 950, if a service still gets Permission denied reading /storage/Movies/... (or jellyfin FFmpeg exit 243), check the root of the dataset /mnt/RED_2TB/storage itself. In the restore it stayed 921:921 with owner@/group@ ALLOW but everyone@ EXECUTE=False. uid 950 (not owner, not in gid-921-group) falls under everyone@ → cannot traverse past the root into the tree, even though every nested ACL is clean. Fix (root):
chown 950:950 /mnt/RED_2TB/storage
chmod 750 /mnt/RED_2TB/storage # rwxr-x---: gid 950 = truenas_admin (containers' group) gets r-x traverse
Kernel needs EXEC (traverse) on EVERY path component. Diagnostic via setpriv --reuid=950 ... ls shows Permission denied on a dir whose getacl looks clean → suspect root/parent traverse. stat via root shows clean 770 but uid 950 can't ls.
GOTCHA 4 — transmission "all torrents No Data Found" after data migration
If every transmission torrent shows error 3: No Data Found right after a container recreate/pool migration:
- Check:
docker exec -ufine to read? downloadDir & files present? If files ARE there and readable as uid 950, the error is likely stale from daemon startup while traverse was blocked (see GOTCHA 3). torrent-verifyon a single torrent does NOT clear it. Fix is trivial & non-destructive:
docker restart transmission
On restart the daemon re-validates → torrents clear (e.g. 254/255 instantly). Leftover #1 → Verify Local Data in Web UI.
Bare git repo: push → Permission denied (2026-09-21)
Symptom:
error: unable to write file ./objects/80/09a34e…: Permission denied
To ssh://…/obsidian-vault.git
! [remote rejected] main -> main (unable to migrate objects to permanent storage)
Root cause — TWO independent defects in the repo's dirs:
- 13 dirs had
drw-r-x---(ownerrw-withoutx): repo root,objects,objects/info,objects/pack,refs{,/heads,/tags},branches,hooks,info,logs{,/refs,/refs/heads}. Without owner-xnothing can be created inside. Fixed withchmod u+xon dirs only (files were all fine:0 files without u+r). - 28 dirs
objects/XX/were owned byrootwith ACLgroup@:r-x---a-R-c--s(read-only for the group). Git runs as uid 950 (truenas_admin), not root → cannot write. This is the defect that actually blocked the push.
Why it happened: mcpvault (the Obsidian MCP server) runs as root — /opt/hermes/.venv/bin/python3 … mcp_stdio_watchdog.py … mcpvault /vault (ps aux shows uid root). Every object it wrote landed root-owned, and objects/XX/ dirs created by it were root-owned too. The problem WILL recur unless mcpvault is restarted as uid 950. Left as an open item.
What does NOT work:
chown -R 950:950andchmod -R g+wvia SSH astruenas_admin→Operation not permitted(uid 950 is not the owner of root-owned objects).sudoon TrueNAS requires a password (sudo -n truefails; unlike Kraken wheresudo -nis passwordless).midclt call filesystem.setperm '{...}'→[EINVAL] Non-trivial ACL present … Option "stripacl" required to change permission. Don't reach forstripacl— it tears the ACL down to POSIX.
What DOES work — filesystem.setacl via midclt (no root needed):
uid 950 can read and write ACLs through midclt. Read the current ACL, add the group perms, write it back. Reference implementation (/tmp/fix_vault_acl.py on TrueNAS, ~120 dirs, ~90 s):
FOR each dir in [repo] + listdir(f"{repo}/objects"):
cur = midclt("filesystem.getacl", path) # path as BARE STRING
for ace in cur["acl"]:
if ace["tag"] == "group@" and ace["type"] == "ALLOW":
ace["perms"]["WRITE_DATA"] = True
ace["perms"]["APPEND_DATA"] = True
ace["perms"]["EXECUTE"] = True
ace["perms"]["WRITE_ATTRIBUTES"] = True
if ace["tag"] == "owner@" and ace["type"] == "ALLOW":
ace["perms"]["EXECUTE"] = True # root-owned dirs had this off
midclt("filesystem.setacl", {
"path": path, "dacl": cur["acl"],
"options": {"recursive": False, "traverse": False, "stripacl": False},
})
# setacl returns a JOB ID → poll core.get_jobs until SUCCESS/FAILED
⚠️ midclt argument-form pitfalls (cost real time):
filesystem.getaclandfilesystem.listdirtake a BARE STRING path, NOT{"path": …}. Passing an object gives[EINVAL] path: Not a string.filesystem.setacltakes ONE JSON object withpathINSIDE it (see gotcha #1 above) — the opposite convention from getacl on the same service. Easy to invert.filesystem.setacl/setpermare async: they return a job id. Polling immediately returns stateWAITING→ looks like failure. Must poll untilSUCCESS/FAILED. With ~120 dirs over SSH the loop outlives the SSH timeout — run it detached (nohup … &) and read the log later; the first SSH-timed-out run had in fact already applied the ACLs.- Passwordless
sudois available on Kraken (sudo -n true→ ok) but NOT on TrueNAStruenas_admin.
Verification (cheap, do it before re-trying push):
touch /mnt/RED_2TB/storage/git/obsidian-vault.git/objects/80/.wrtest && echo WRITE_OK
WRITE OK → push will succeed. Confirmed: f9a5c2b..21505ec main -> main, then a clean git pull on Kraken.
Durable fix with INHERITANCE (2026-09-21, applied — 291 dirs)
The minimal recipe above (add perms to existing group@ ACE) restores write for current dirs, but the bug returns: the next push by the root-owned mcpvault creates a NEW objects/XX/ dir whose ACL is inherited from the parent, and if the parent's everyone@/group@ lacks WRITE_DATA + inherit flags, the new dir is root-owned and read-only again → push fails again.
Working fix = full dacl replacement (not per-ACE edit) with inheritance flags ON, applied to every dir:
# perms granted to owner@ AND group@ AND everyone@:
# READ_DATA WRITE_DATA APPEND_DATA EXECUTE DELETE_CHILD
# READ_ATTRIBUTES WRITE_ATTRIBUTES READ_NAMED_ATTRS WRITE_NAMED_ATTRS
# READ_ACL (and WRITE_ACL WRITE_OWNER DELETE for owner@)
# flags on each ACE: {"FILE_INHERIT": True, "DIRECTORY_INHERIT": True,
# "NO_PROPAGATE_INHERIT": False, "INHERIT_ONLY": False,
# "INHERITED": False}
# call: midclt call filesystem.setacl '<one json with path + dacl + options>'
# options = {"recursive": False, "traverse": True, "stripacl": False}
Result: 291 dirs processed, 0 failures. Note filesystem.listdir <BARE STRING> to enumerate (same convention as getacl), and each setacl is async — but 291 sequential dirs finished inside one SSH call (~2 min), no nohup needed this time.
⚠️ Caveat — the ACL was replaced, not proven functional yet. The write-test touch objects/80/.wrtest was NOT executed (approval timed out). Verification was indirect: git push nas main → Everything up-to-date, and git rev-parse HEAD == git rev-parse nas/main (177da1e). That proves the current tree is in sync, not that a fresh root-owned object can be written. A real touch in objects/80/ as uid 950 is still the definitive test.
Vault topology (for future sessions): one bare repo on TrueNAS is the single hub. Mac pushes/pulls remote nas; Kraken uses remote origin — both point at ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git. Mac and Kraken never talk directly. Host truenas does not resolve as an SSH alias from the Mac — use truenas_admin@mallexxx.duckdns.org.
Docker network persistency (per-service compose)
docker network connect <net> <container>is LOST on recreate → declare<net>in the service's composenetworks:.- Caddy must co-own a network with its reverse-proxy target. transmission Web UI works because caddy is ALSO in
transmission_default; radarr↔transmission works because both inmedia_net. external: truenetworks (media_net/caddy_default) aren't created by compose — if they vanish (pool rebuild),docker network create media_net.
Containers / uid summary (as of 2026-09-02, TrueNAS "Taiga")
- prowlarr/radarr/sonarr/jellyfin: all running, all daemons under uid 950, project
arr. - transmission: PUID/PGID=950, in networks
media_net+transmission_default(both external). - Radarr/sonarr/jellyfin mount
/mnt/RED_2TB/storageas/storage(NOT/mediaas on Kraken) → set radarr root-folder accordingly.