0 Fork 0

Files

T

Alexey Martemyanov 8b0b614a76 [2026-09-21] eagle: family/documents/vault-sync/2026-09-21-bare-repo-push-permission-denied.md family/how-to/vault-git-sync.md personal/tech/truenas-nfs4-acl-and-arrmultiuser.md

2026-09-21 19:32:02 +06:00

13 KiB

Raw Blame History

title, created, updated, type, namespace, tags, related

title

created

updated

type

namespace

TrueNAS NFS4 ACL repair + cross-container media PUID

Hard-won lessons from the 2026-08-31/09-01/09-02 TrueNAS restore-incident remediation. Mirror of skill truenas-nfs4-acl-and-arrmultiuser. Detailed incident log: family/documents/vault-sync/2026-09-02-restore-privilege-scope.md.

2026-09-21 addendum: the same class of failure hit the vault bare-git repo (/mnt/RED_2TB/storage/git/obsidian-vault.git) — git push failed with unable to write file ./objects/XX/…: Permission denied. Root cause and the working midclt recipe are in the section «Bare git repo: push Permission denied» below.

Trigger

Any "permission denied" / file not readable / git loose object ... corrupt / empty Jellyfin library on TrueNAS SCALE (acltype=nfsv4) after a zfs restore, OR wiring Prowlarr/Radarr/Sonarr/Jellyfin/Transmission so container-written files are readable by other containers.

Core facts

TrueNAS SCALE uses NFSv4 ACL, not plain POSIX. ls -la shows a POSIX mask that can mislead.
Read real ACL: midclt call filesystem.getacl <path>; write: midclt call filesystem.setacl.
truenas_admin is FULL_ADMIN in midclt → can chown/setacl without passwordless root.
Web UI ACL editing only works on zfs datasets, not arbitrary subdirs under one dataset → use midclt/CLI.
Post-restore a tree may be owned by wrong uid (e.g. 921 transmission) with files mode 0000.

The two big gotchas

1. setacl takes ONE JSON with `path` INSIDE it

midclt call filesystem.setacl /path {…} → [EFAULT] Too many arguments (expected 1, found 2). Correct form:

midclt call filesystem.setacl '{"path":"/x","uid":950,"gid":950,"acltype":"NFS4","dacl":[...],"options":{...}}'

Jobs run async → returns a job id; verify midclt call core.get_jobs (state SUCCESS).

2. stripacl does NOT remove DENY; full dacl replacement does

options.stripacl:true reported SUCCESS but left the owner@ DENY ACE intact. Only passing a complete dacl array works — setacl treats it as a full REPLACE of the ACL.

NFSv4 DENY overrides ALLOW (the corrupt-object trap)

A file owned by uid 950 carrying owner@ DENY READ_DATA=True cannot be read by uid 950 itself. Git then reports loose object ... corrupt — this is ACL, not data damage (data intact; root reads fine). Diagnostic trap: git fsck under root is clean, but fetch/push under the owning uid fails → ACL, not corruption. Broken files often show POSIX mode 40 (r--------); healthy objects 750.

Repair / cross-container recipe (linuxserver media stack)

Goal: all containers run under ONE uid (950) so transmission→radarr/sonarr→jellyfin all read each other's files.

transmission: add PUID=950 PGID=950 to env in docker-compose.yml. It mounts storage→/mnt/storage, download-dir /mnt/storage/Downloads.
radarr/sonarr/prowlarr/jellyfin compose: add PUID=950 PGID=950 to each (retain TZ).
linuxserver s6 images (Entrypoint /init, /etc/s6-overlay) honor PUID/PGID natively. Default (unset) = runs root → creates root-owned files that 950-services can't read → always set PUID/PGID.
Recreate from proper folder: cd /mnt/RED_2TB/docker/<svc> && docker compose up -d (+--force-recreate if env changed). Verify daemon uid via ps aux inside container (process as abc/950 = applied). NOTE docker exec id ≠ daemon uid.
Simple datasets: recursive POSIX chown -R 950:950 + chmod under root SUFFICE (ACLs were trivial). Use midclt/ACL only when non-trivial ACEs present (check filesystem.getacl → trivial:true).
Bare git repo with owner@ DENY on objects → full dacl replacement recursively (perms JSON like above).

GOTCHA 3 — root dataset traverse (Permission denied despite clean file ACL)

After chown-ing all leaf dirs/files to 950, if a service still gets Permission denied reading /storage/Movies/... (or jellyfin FFmpeg exit 243), check the root of the dataset /mnt/RED_2TB/storage itself. In the restore it stayed 921:921 with owner@/group@ ALLOW but everyone@ EXECUTE=False. uid 950 (not owner, not in gid-921-group) falls under everyone@ → cannot traverse past the root into the tree, even though every nested ACL is clean. Fix (root):

chown 950:950 /mnt/RED_2TB/storage
chmod 750 /mnt/RED_2TB/storage    # rwxr-x---: gid 950 = truenas_admin (containers' group) gets r-x traverse

Kernel needs EXEC (traverse) on EVERY path component. Diagnostic via setpriv --reuid=950 ... ls shows Permission denied on a dir whose getacl looks clean → suspect root/parent traverse. stat via root shows clean 770 but uid 950 can't ls.

GOTCHA 4 — transmission "all torrents No Data Found" after data migration

If every transmission torrent shows error 3: No Data Found right after a container recreate/pool migration:

Check: docker exec -u fine to read? downloadDir & files present? If files ARE there and readable as uid 950, the error is likely stale from daemon startup while traverse was blocked (see GOTCHA 3).
torrent-verify on a single torrent does NOT clear it. Fix is trivial & non-destructive:

docker restart transmission

On restart the daemon re-validates → torrents clear (e.g. 254/255 instantly). Leftover #1 → Verify Local Data in Web UI.

Bare git repo: `push` → `Permission denied` (2026-09-21)

Symptom:

error: unable to write file ./objects/80/09a34e…: Permission denied
To ssh://…/obsidian-vault.git
 ! [remote rejected] main -> main (unable to migrate objects to permanent storage)

Root cause — TWO independent defects in the repo's dirs:

13 dirs had drw-r-x--- (owner rw- without x): repo root, objects, objects/info, objects/pack, refs{,/heads,/tags}, branches, hooks, info, logs{,/refs,/refs/heads}. Without owner-x nothing can be created inside. Fixed with chmod u+x on dirs only (files were all fine: 0 files without u+r).
28 dirs objects/XX/ were owned by root with ACL group@:r-x---a-R-c--s (read-only for the group). Git runs as uid 950 (truenas_admin), not root → cannot write. This is the defect that actually blocked the push.

Why it happened: mcpvault runs as root on TrueNAS — /opt/hermes/.venv/bin/python3 … mcp_stdio_watchdog.py … mcpvault /vault (ps aux shows uid root). Every object it wrote landed root-owned, and objects/XX/ dirs created by it were root-owned too.

⚠️ Do not confuse this with the Mac's vault. This mcpvault belongs to a second Hermes instance living on TrueNAS itself (/opt/hermes), whose vault is /vault — not Alex's ~/obsidian on the Mac. It is not "the agent's own MCP server". Both instances merely push to the same shared bare repo, which is why the collision happens. PID refs seen 2026-09-21: 13727 (mcp_stdio_watchdog.py), 13738 (node … mcpvault /vault).

With the inheritance fix below, root-owned objects are no longer fatal (rights are inherited from the now-writable parent), but the root instance is still a latent hazard — restarting it as uid 950 removes the cause. Left as an open item.

What does NOT work:

chown -R 950:950 and chmod -R g+w via SSH as truenas_admin → Operation not permitted (uid 950 is not the owner of root-owned objects). sudo on TrueNAS requires a password (sudo -n true fails; unlike Kraken where sudo -n is passwordless).
midclt call filesystem.setperm '{...}' → [EINVAL] Non-trivial ACL present … Option "stripacl" required to change permission. Don't reach for stripacl — it tears the ACL down to POSIX.

What DOES work — filesystem.setacl via midclt (no root needed): uid 950 can read and write ACLs through midclt. Read the current ACL, add the group perms, write it back. Reference implementation (/tmp/fix_vault_acl.py on TrueNAS, ~120 dirs, ~90 s):

FOR each dir in [repo] + listdir(f"{repo}/objects"):
    cur = midclt("filesystem.getacl", path)          # path as BARE STRING
    for ace in cur["acl"]:
        if ace["tag"] == "group@" and ace["type"] == "ALLOW":
            ace["perms"]["WRITE_DATA"]    = True
            ace["perms"]["APPEND_DATA"]   = True
            ace["perms"]["EXECUTE"]       = True
            ace["perms"]["WRITE_ATTRIBUTES"] = True
        if ace["tag"] == "owner@" and ace["type"] == "ALLOW":
            ace["perms"]["EXECUTE"] = True           # root-owned dirs had this off
    midclt("filesystem.setacl", {
        "path": path, "dacl": cur["acl"],
        "options": {"recursive": False, "traverse": False, "stripacl": False},
    })
    # setacl returns a JOB ID → poll core.get_jobs until SUCCESS/FAILED

⚠️ midclt argument-form pitfalls (cost real time):

filesystem.getacl and filesystem.listdir take a BARE STRING path, NOT {"path": …}. Passing an object gives [EINVAL] path: Not a string.
filesystem.setacl takes ONE JSON object with path INSIDE it (see gotcha #1 above) — the opposite convention from getacl on the same service. Easy to invert.
filesystem.setacl / setperm are async: they return a job id. Polling immediately returns state WAITING → looks like failure. Must poll until SUCCESS/FAILED. With ~120 dirs over SSH the loop outlives the SSH timeout — run it detached (nohup … &) and read the log later; the first SSH-timed-out run had in fact already applied the ACLs.
Passwordless sudo is available on Kraken (sudo -n true → ok) but NOT on TrueNAS truenas_admin.

Verification (cheap, do it before re-trying push):

touch /mnt/RED_2TB/storage/git/obsidian-vault.git/objects/80/.wrtest && echo WRITE_OK

WRITE OK → push will succeed. Confirmed: f9a5c2b..21505ec main -> main, then a clean git pull on Kraken.

Durable fix with INHERITANCE (2026-09-21, applied — 291 dirs)

The minimal recipe above (add perms to existing group@ ACE) restores write for current dirs, but the bug returns: the next push by the root-owned mcpvault creates a NEW objects/XX/ dir whose ACL is inherited from the parent, and if the parent's everyone@/group@ lacks WRITE_DATA + inherit flags, the new dir is root-owned and read-only again → push fails again.

Working fix = full dacl replacement (not per-ACE edit) with inheritance flags ON, applied to every dir:

# perms granted to owner@ AND group@ AND everyone@:
#   READ_DATA WRITE_DATA APPEND_DATA EXECUTE DELETE_CHILD
#   READ_ATTRIBUTES WRITE_ATTRIBUTES READ_NAMED_ATTRS WRITE_NAMED_ATTRS
#   READ_ACL (and WRITE_ACL WRITE_OWNER DELETE for owner@)
# flags on each ACE:  {"FILE_INHERIT": True, "DIRECTORY_INHERIT": True,
#                      "NO_PROPAGATE_INHERIT": False, "INHERIT_ONLY": False,
#                      "INHERITED": False}
# call:  midclt call filesystem.setacl '<one json with path + dacl + options>'
#        options = {"recursive": False, "traverse": True, "stripacl": False}

Result: 291 dirs processed, 0 failures. Note filesystem.listdir <BARE STRING> to enumerate (same convention as getacl), and each setacl is async — but 291 sequential dirs finished inside one SSH call (~2 min), no nohup needed this time.

⚠️ Caveat — the ACL was replaced, not proven functional yet. The write-test touch objects/80/.wrtest was NOT executed (approval timed out). Verification was indirect: git push nas main → Everything up-to-date, and git rev-parse HEAD == git rev-parse nas/main (177da1e). That proves the current tree is in sync, not that a fresh root-owned object can be written. A real touch in objects/80/ as uid 950 is still the definitive test.

Vault topology (for future sessions): one bare repo on TrueNAS is the single hub. Mac pushes/pulls remote nas; Kraken uses remote origin — both point at ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git. Mac and Kraken never talk directly. Host truenas does not resolve as an SSH alias from the Mac — use truenas_admin@mallexxx.duckdns.org.

Docker network persistency (per-service compose)

docker network connect <net> <container> is LOST on recreate → declare <net> in the service's compose networks:.
Caddy must co-own a network with its reverse-proxy target. transmission Web UI works because caddy is ALSO in transmission_default; radarr↔transmission works because both in media_net.
external: true networks (media_net/caddy_default) aren't created by compose — if they vanish (pool rebuild), docker network create media_net.

Containers / uid summary (as of 2026-09-02, TrueNAS "Taiga")

prowlarr/radarr/sonarr/jellyfin: all running, all daemons under uid 950, project arr.
transmission: PUID/PGID=950, in networks media_net + transmission_default (both external).
Radarr/sonarr/jellyfin mount /mnt/RED_2TB/storage as /storage (NOT /media as on Kraken) → set radarr root-folder accordingly.

13 KiB Raw Blame History