0 Fork 0

Files

T

Alexey Martemyanov f482778257 [2026-09-26] eagle: family/documents/vault-sync/2026-09-26-syncthing-clone-root-objects-acl-confusion.md family/how-to/vault-git-sync.md personal/plans/balda-mm-phase4-agents.md personal/tech/docker_colima_setup.md personal/tech/macos-docker-virtiofs-mount-limits.md personal/tech/truenas-nfs4-acl-and-arrmultiuser.md

2026-09-26 15:52:55 +06:00

24 KiB

Executable File

Raw Blame History

title, created, updated, type, namespace, tags, related

title

created

updated

type

namespace

TrueNAS NFS4 ACL repair + cross-container media PUID

Hard-won lessons from the 2026-08-31/09-01/09-02 TrueNAS restore-incident remediation. Mirror of skill truenas-nfs4-acl-and-arrmultiuser. Detailed incident log: family/documents/vault-sync/2026-09-02-restore-privilege-scope.md.

2026-09-21 addendum: the same class of failure hit the vault bare-git repo (/mnt/RED_2TB/storage/git/obsidian-vault.git) — git push failed with unable to write file ./objects/XX/…: Permission denied. Root cause and the working midclt recipe are in the section «Bare git repo: push Permission denied» below.

2026-09-22 addendum (РЕЦИДИВ, закрыт): дефект вернулся (35 root-каталогов). Починено: 236/236 каталогов INHERIT + WRITE, push прошёл end-to-end, поставлен cron-сторож. 🔴 Ключевые уроки: dacl с нуля собрать НЕЛЬЗЯ (TrueNAS требует id в каждой ACE — только read-modify-write); флаги = {"BASIC":"INHERIT"}, НЕ булев dict; setacl асинхронный — обязательно ждать job до SUCCESS, иначе ложный провал. Рецепт и 4 питфолла — в секции «Durable fix with INHERITANCE» ниже. Инструменты: /Users/admin/Developer/vault-tools/. Хронология расследования: family/documents/vault-sync/2026-09-21-bare-repo-push-permission-denied.md.

Trigger

Any "permission denied" / file not readable / git loose object ... corrupt / empty Jellyfin library on TrueNAS SCALE (acltype=nfsv4) after a zfs restore, OR wiring Prowlarr/Radarr/Sonarr/Jellyfin/Transmission so container-written files are readable by other containers.

Core facts

TrueNAS SCALE uses NFSv4 ACL, not plain POSIX. ls -la shows a POSIX mask that can mislead.
Read real ACL: midclt call filesystem.getacl <path>; write: midclt call filesystem.setacl.
🔴 getfacl (POSIX view) HIDES the absence of a custom ACL — use nfs4xdr_getfacl (2026-09-26). getfacl prints user::rwx / group::rwx / other::--- whether or not named ACEs exist. nfs4xdr_getfacl <path> prints trivial_acl: true (only owner@/group@/everyone@) — the definitive "no custom ACL here". Start every permission diagnosis with it, not getfacl.
🔴 "We already fixed this with ACL" ≠ "the ACL is set on THIS path" (2026-09-26). A recursive group ACL had been applied to /mnt/RED_2TB/Edu and was remembered as "done" — but the failing path was a different folder with trivial_acl: true (never touched). Verify by fact on the target object before repeating or ruling out a fix. Find past sessions: sqlite3 ~/.hermes/hermes-whale/state.db "SELECT session_id, substr(content,1,300) FROM messages WHERE content LIKE '%setfacl%' ORDER BY id DESC LIMIT 40;"
🔴 A one-off chown is symptom-treatment (2026-09-26). If a root process keeps writing into the folder, root-owned files/objects reappear on the very next run. Permanent fixes only: ACL + inherit, OR run the writer not as root (user: "950:950" in compose).
truenas_admin is FULL_ADMIN in midclt → can chown/setacl without passwordless root.
Web UI ACL editing only works on zfs datasets, not arbitrary subdirs under one dataset → use midclt/CLI.
Post-restore a tree may be owned by wrong uid (e.g. 921 transmission) with files mode 0000.

The two big gotchas

1. setacl takes ONE JSON with `path` INSIDE it

midclt call filesystem.setacl /path {…} → [EFAULT] Too many arguments (expected 1, found 2). Correct form:

midclt call filesystem.setacl '{"path":"/x","uid":950,"gid":950,"acltype":"NFS4","dacl":[...],"options":{...}}'

Jobs run async → returns a job id; verify midclt call core.get_jobs (state SUCCESS).

2. stripacl does NOT remove DENY; full dacl replacement does

options.stripacl:true reported SUCCESS but left the owner@ DENY ACE intact. Only passing a complete dacl array works — setacl treats it as a full REPLACE of the ACL.

NFSv4 DENY overrides ALLOW (the corrupt-object trap)

A file owned by uid 950 carrying owner@ DENY READ_DATA=True cannot be read by uid 950 itself. Git then reports loose object ... corrupt — this is ACL, not data damage (data intact; root reads fine). Diagnostic trap: git fsck under root is clean, but fetch/push under the owning uid fails → ACL, not corruption. Broken files often show POSIX mode 40 (r--------); healthy objects 750.

Repair / cross-container recipe (linuxserver media stack)

Goal: all containers run under ONE uid (950) so transmission→radarr/sonarr→jellyfin all read each other's files.

transmission: add PUID=950 PGID=950 to env in docker-compose.yml. It mounts storage→/mnt/storage, download-dir /mnt/storage/Downloads.
radarr/sonarr/prowlarr/jellyfin compose: add PUID=950 PGID=950 to each (retain TZ).
linuxserver s6 images (Entrypoint /init, /etc/s6-overlay) honor PUID/PGID natively. Default (unset) = runs root → creates root-owned files that 950-services can't read → always set PUID/PGID.
Recreate from proper folder: cd /mnt/RED_2TB/docker/<svc> && docker compose up -d (+--force-recreate if env changed). Verify daemon uid via ps aux inside container (process as abc/950 = applied). NOTE docker exec id ≠ daemon uid.
Simple datasets: recursive POSIX chown -R 950:950 + chmod under root SUFFICE (ACLs were trivial). Use midclt/ACL only when non-trivial ACEs present (check filesystem.getacl → trivial:true).
Bare git repo with owner@ DENY on objects → full dacl replacement recursively (perms JSON like above).

GOTCHA 3 — root dataset traverse (Permission denied despite clean file ACL)

After chown-ing all leaf dirs/files to 950, if a service still gets Permission denied reading /storage/Movies/... (or jellyfin FFmpeg exit 243), check the root of the dataset /mnt/RED_2TB/storage itself. In the restore it stayed 921:921 with owner@/group@ ALLOW but everyone@ EXECUTE=False. uid 950 (not owner, not in gid-921-group) falls under everyone@ → cannot traverse past the root into the tree, even though every nested ACL is clean. Fix (root):

chown 950:950 /mnt/RED_2TB/storage
chmod 750 /mnt/RED_2TB/storage    # rwxr-x---: gid 950 = truenas_admin (containers' group) gets r-x traverse

Kernel needs EXEC (traverse) on EVERY path component. Diagnostic via setpriv --reuid=950 ... ls shows Permission denied on a dir whose getacl looks clean → suspect root/parent traverse. stat via root shows clean 770 but uid 950 can't ls.

GOTCHA 4 — transmission "all torrents No Data Found" after data migration

If every transmission torrent shows error 3: No Data Found right after a container recreate/pool migration:

Check: docker exec -u fine to read? downloadDir & files present? If files ARE there and readable as uid 950, the error is likely stale from daemon startup while traverse was blocked (see GOTCHA 3).
torrent-verify on a single torrent does NOT clear it. Fix is trivial & non-destructive:

docker restart transmission

On restart the daemon re-validates → torrents clear (e.g. 254/255 instantly). Leftover #1 → Verify Local Data in Web UI.

Bare git repo: `push` → `Permission denied` (2026-09-21)

Symptom:

error: unable to write file ./objects/80/09a34e…: Permission denied
To ssh://…/obsidian-vault.git
 ! [remote rejected] main -> main (unable to migrate objects to permanent storage)

Root cause — TWO independent defects in the repo's dirs:

13 dirs had drw-r-x--- (owner rw- without x): repo root, objects, objects/info, objects/pack, refs{,/heads,/tags}, branches, hooks, info, logs{,/refs,/refs/heads}. Without owner-x nothing can be created inside. Fixed with chmod u+x on dirs only (files were all fine: 0 files without u+r).
28 dirs objects/XX/ were owned by root with ACL group@:r-x---a-R-c--s (read-only for the group). Git runs as uid 950 (truenas_admin), not root → cannot write. This is the defect that actually blocked the push.

Why it happened: mcpvault runs as root on TrueNAS — /opt/hermes/.venv/bin/python3 … mcp_stdio_watchdog.py … mcpvault /vault (ps aux shows uid root). Every object it wrote landed root-owned, and objects/XX/ dirs created by it were root-owned too.

⚠️ Do not confuse this with the Mac's vault. This mcpvault belongs to a second Hermes instance living on TrueNAS itself (/opt/hermes), whose vault is /vault — not Alex's ~/obsidian on the Mac. It is not "the agent's own MCP server". Both instances merely push to the same shared bare repo, which is why the collision happens. PID refs seen 2026-09-21: 13727 (mcp_stdio_watchdog.py), 13738 (node … mcpvault /vault).

With the inheritance fix below, root-owned objects are no longer fatal (rights are inherited from the now-writable parent), but the root instance is still a latent hazard — restarting it as uid 950 removes the cause. Left as an open item.

What does NOT work:

chown -R 950:950 and chmod -R g+w via SSH as truenas_admin → Operation not permitted (uid 950 is not the owner of root-owned objects). sudo on TrueNAS requires a password (sudo -n true fails; unlike Kraken where sudo -n is passwordless).
midclt call filesystem.setperm '{...}' → [EINVAL] Non-trivial ACL present … Option "stripacl" required to change permission. Don't reach for stripacl — it tears the ACL down to POSIX.

What DOES work — filesystem.setacl via midclt (no root needed): uid 950 can read and write ACLs through midclt. Read the current ACL, add the group perms, write it back. Reference implementation (/tmp/fix_vault_acl.py on TrueNAS, ~120 dirs, ~90 s):

FOR each dir in [repo] + listdir(f"{repo}/objects"):
    cur = midclt("filesystem.getacl", path)          # path as BARE STRING
    for ace in cur["acl"]:
        if ace["tag"] == "group@" and ace["type"] == "ALLOW":
            ace["perms"]["WRITE_DATA"]    = True
            ace["perms"]["APPEND_DATA"]   = True
            ace["perms"]["EXECUTE"]       = True
            ace["perms"]["WRITE_ATTRIBUTES"] = True
        if ace["tag"] == "owner@" and ace["type"] == "ALLOW":
            ace["perms"]["EXECUTE"] = True           # root-owned dirs had this off
    midclt("filesystem.setacl", {
        "path": path, "dacl": cur["acl"],
        "options": {"recursive": False, "traverse": False, "stripacl": False},
    })
    # setacl returns a JOB ID → poll core.get_jobs until SUCCESS/FAILED

⚠️ midclt argument-form pitfalls (cost real time):

🔴🔴 Каждая ACE требует поле id. Собирать dacl «с нуля» как {tag, type, perms, flags} НЕЛЬЗЯ — TrueNAS отклонит: [EINVAL] filesystem_acl.dacl.0: ACL entry is missing required keys(s): {'id'} (проверено 2026-09-22: 245 каталогов, 0 успешных). Работает только read-modify-write: взять ACL через getacl (там id уже проставлен), править в нём только perms/flags, передать cur["acl"] обратно в setacl. Именно так был сделан успешный прогон 2026-09-21 (291 каталог).
filesystem.getacl и filesystem.listdir take a BARE STRING path, NOT {"path": …}. Passing an object gives [EINVAL] path: Not a string.
filesystem.setacl takes ONE JSON object with path INSIDE it (see gotcha #1 above) — the opposite convention from getacl on the same service. Easy to invert.
filesystem.setacl / setperm are async: they return a job id. Polling immediately returns state WAITING → looks like failure. Must poll until SUCCESS/FAILED. With ~120 dirs over SSH the loop outlives the SSH timeout — run it detached (nohup … &) and read the log later; the first SSH-timed-out run had in fact already applied the ACLs.
Passwordless sudo is available on Kraken (sudo -n true → ok) but NOT on TrueNAS truenas_admin.

Verification (cheap, do it before re-trying push):

touch /mnt/RED_2TB/storage/git/obsidian-vault.git/objects/80/.wrtest && echo WRITE_OK

WRITE OK → push will succeed. Confirmed: f9a5c2b..21505ec main -> main, then a clean git pull on Kraken.

Durable fix with INHERITANCE (2026-09-21 — 291 dirs, 2026-09-22 — 236 dirs)

The minimal recipe above (add perms to existing group@ ACE) restores write for current dirs, but the bug returns: the next push by the root-owned mcpvault creates a NEW objects/XX/ dir whose ACL is inherited from the parent, and if the parent's everyone@/group@ lacks WRITE_DATA + inherit flags, the new dir is root-owned and read-only again → push fails again.

🔴 RE-CUR 2026-09-22. Дефект вернулся (35 root-каталогов). Итог фикса: 236/236 каталогов с INHERIT + WRITE, push прошёл end-to-end. Порядок работ и уроки — в family/documents/vault-sync/2026-09-21-bare-repo-push-permission-denied.md §«✅ ФИНАЛЬНЫЙ ФИКС».

Working fix = read-modify-write, НЕ сборка dacl с нуля:

# perms granted to owner@ AND group@ AND everyone@:
#   READ_DATA WRITE_DATA APPEND_DATA EXECUTE DELETE_CHILD
#   READ_ATTRIBUTES WRITE_ATTRIBUTES READ_NAMED_ATTRS WRITE_NAMED_ATTRS
#   READ_ACL (and WRITE_ACL WRITE_OWNER DELETE for owner@)
# flags задаются КАК СТРОКА, не булев dict:
#   ace["flags"] = {"BASIC": "INHERIT"}      ← FILE_INHERIT + DIRECTORY_INHERIT
# call:  midclt call filesystem.setacl '<one json with path + dacl + options>'
#        options = {"recursive": False, "traverse": True, "stripacl": False}

🔴 Четыре питфолла, стоившие времени:

Собирать dacl «с нуля» НЕЛЬЗЯ. TrueNAS требует поле id в каждой ACE: [EINVAL] filesystem_acl.dacl.0: ACL entry is missing required keys(s): {'id'}. Работает только read-modify-write: getacl → править perms/flags → setacl.
Флаги — {"BASIC":"INHERIT"}, НЕ {"FILE_INHERIT":True,"DIRECTORY_INHERIT":True}. Булев dict отклоняется той же ошибкой про id (при неверном формате TrueNAS пересобирает ACE и теряет поле).
setacl АСИНХРОННЫЙ — надо ЖДАТЬ job до SUCCESS. Проверка getacl сразу после вызова видит СТАРЫЕ флаги → ложный fail. Признак гонки: fixed=0, already=N, fail=M.
```
jid = int(midclt("filesystem.setacl", arg))
while deadline:
    j = json.loads(midclt("core.get_jobs", f'[["id","=",{jid}]]'))
    if j[0]["state"] in ("SUCCESS","FAILED","ABORTED"): break
    time.sleep(1)
```
Владелец root сам по себе НЕ блокирует запись. getacl даёт uid:0, gid:950, запись от uid 950 идёт через group@ (gid 950). write-тест в 35 root-каталогов: OK=35 FAIL=0.

Верификация (обязательна, делай до retry push):

touch <repo>/objects/XX/.wrtest && echo WRITE_OK     # у КАЖДОГО каталога
# + тест наследования: mkdir objects/zzz → getacl → должен быть INHERITED:True → touch внутрь
# + end-to-end: git clone <repo> && cd && echo x > f && git add f && git commit && git push

Сторож (поставлен 2026-09-22):

~/scripts/vault-acl-watchdog.py     (TrueNAS, от truenas_admin)
cron: 0 */2 * * * python3 ~/scripts/vault-acl-watchdog.py >> ~/scripts/vault-acl-watchdog.log 2>&1

SILENT если всё ок; чинит + докладывает если сломано.

⚠️ Корень НЕ устранён: mcpvault контейнера hermes-taiga (/mnt/RED_2TB/docker/hermes/docker-compose.yml, HERMES_ALLOW_ROOT_GATEWAY=1, User: пусто) работает от root. Перманентный фикс — добавить user: "950:950" в taiga service.

Working fix = full dacl replacement (not per-ACE edit) with inheritance flags ON, applied to every dir:

# perms granted to owner@ AND group@ AND everyone@:
#   READ_DATA WRITE_DATA APPEND_DATA EXECUTE DELETE_CHILD
#   READ_ATTRIBUTES WRITE_ATTRIBUTES READ_NAMED_ATTRS WRITE_NAMED_ATTRS
#   READ_ACL (and WRITE_ACL WRITE_OWNER DELETE for owner@)
# flags on each ACE:  {"FILE_INHERIT": True, "DIRECTORY_INHERIT": True,
#                      "NO_PROPAGATE_INHERIT": False, "INHERIT_ONLY": False,
#                      "INHERITED": False}
# call:  midclt call filesystem.setacl '<one json with path + dacl + options>'
#        options = {"recursive": False, "traverse": True, "stripacl": False}

Result: 291 dirs processed, 0 failures. Note filesystem.listdir <BARE STRING> to enumerate (same convention as getacl), and each setacl is async — but 291 sequential dirs finished inside one SSH call (~2 min), no nohup needed this time.

⚠️ Caveat — the ACL was replaced, not proven functional yet. The write-test touch objects/80/.wrtest was NOT executed (approval timed out). Verification was indirect: git push nas main → Everything up-to-date, and git rev-parse HEAD == git rev-parse nas/main (177da1e). That proves the current tree is in sync, not that a fresh root-owned object can be written. A real touch in objects/80/ as uid 950 is still the definitive test.

Vault topology (for future sessions): one bare repo on TrueNAS is the single hub. Mac pushes/pulls remote nas; Kraken uses remote origin — both point at ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git. Mac and Kraken never talk directly. Host truenas does not resolve as an SSH alias from the Mac — use truenas_admin@mallexxx.duckdns.org.

Sharing a whole tree by GROUP (`nas_users`) — the simplest fix (2026-09-23)

Trigger: a directory tree inside a dataset is unreachable — e.g. /mnt/RED_2TB/Edu and /mnt/RED_2TB/Edu/Маскелиаде gave "no access" both via WebDAV and via SSH as truenas_admin, although getacl showed a plain ALLOW.

When to prefer this over per-file/uid-950 recipes above: the requirement is "make the whole tree accessible to a set of humans/services", not "make one container read another's files". One dacl write covers everything — no per-uid chown of thousands of files.

🔴 THE key insight — group@ in NFSv4 is the file's POSIX gid, not a standalone ACE. Setting a recursive setacl with group@ ALLOW + INHERIT does not grant anything by itself: group@ resolves to each object's own gid. If the gid is still 0 (root) or 921, the ACE grants the wrong group. The gid must be changed too.

Second trap: chgrp -R as truenas_admin FAILS with Operation not permitted on every file (uid 950 is neither the owner nor in gid 0 — same limitation as the 2026-09-21 repo fix). So the gid change must go through midclt (filesystem.setacl carries uid/gid and runs the recursive job as root). Plan for two passes: pass 1 = ACL with INHERIT, pass 2 = setacl with the new gid (and uid passed through unchanged).

Choose the group by membership, not by guess:

id truenas_admin
# uid=950(truenas_admin) gid=950(truenas_admin)
#   groups=950(truenas_admin),544(builtin_administrators),921(transmission),
#          999(docker),1000(nas),3000(nas_users)

nas_users (gid 3000) contains root, truenas_admin, mallexxx, nas, transmission — one ACE reaches all of them. Backup-target trees on this NAS already use nas/nas_users.

Working recipe (VM-verified 2026-09-23 on /mnt/RED_2TB/Edu):

# pass 1 — ACL: set INHERIT on ALL ACEs of the tree root (read-modify-write, keep id)
cur = getacl(PATH)                      # id is already populated — never build dacl from scratch
for ace in cur["acl"]:
    ace["type"]  = "ALLOW"              # group@ must be ALLOW
    if ace["tag"] == "group@":
        ace["perms"] = { …, "READ_DATA":True,"WRITE_DATA":True,"EXECUTE":True,
                         "APPEND_DATA":True,"DELETE_CHILD":True,"DELETE":True,
                         "WRITE_ACL":True,"WRITE_OWNER":True }   # full grant
    ace["flags"] = {"BASIC": "INHERIT"} # STRING, not a boolean dict
setacl({"path":PATH,"dacl":cur["acl"],
        "options":{"recursive":True,"traverse":True,"stripacl":False}})   # poll job → SUCCESS

# pass 2 — gid: without this the group@ ACE matches the OLD group and grants nothing
setacl({"path":PATH,"uid":cur["uid"],"gid":3000,"acltype":"NFS4","dacl":cur["acl"],
        "options":{"recursive":True,"traverse":True,"stripacl":False}})

Verification (all three):

ls -ld /mnt/RED_2TB/Edu        # drwxrwxr-x root nas_users  ← gid showed up
ls /mnt/RED_2TB/Edu/           # contents now list as truenas_admin
touch /mnt/RED_2TB/Edu/.wrtest && echo WRITE_OK && rm -f /mnt/RED_2TB/Edu/.wrtest
midclt call filesystem.getacl "<child dir>"   # every ACE must show INHERITED: True + gid 3000

Observed: Edu → drwxrwxr-x root:nas_users, gid=3000; children Маскелиаде, Книги, учебники → gid=3000 with INHERITED: True on all three ACEs; WRITE_OK. Pre-change ACL backed up to /mnt/RED_2TB/storage/acl-edu-before-<ts>.json.

📌 Cyrillic directory names inside shell commands. .../Edu/Маскелиаде tripped the security scanner as confusable_text (HIGH) twice, each time blocking the command. Wrap such paths in double quotes and never let a Cyrillic word touch an ASCII word; where possible use getacl with the path as a bare string or find -iname patterns instead of typing the name by hand.

Mac-side corollary — WebDAV was never the blocker. The webdav container runs as root (docker top webdav → root … webdav --config /config.yml), and owner@ was ALLOW, so the ACL never denied it. Container ls/id are unavailable (hacdias/webdav is a distroless image — exec: "ls": executable file not found), so use docker top to learn its uid.

Docker network persistency (per-service compose)

docker network connect <net> <container> is LOST on recreate → declare <net> in the service's compose networks:.
Caddy must co-own a network with its reverse-proxy target. transmission Web UI works because caddy is ALSO in transmission_default; radarr↔transmission works because both in media_net.
external: true networks (media_net/caddy_default) aren't created by compose — if they vanish (pool rebuild), docker network create media_net.

Containers / uid summary (as of 2026-09-02, TrueNAS "Taiga")

prowlarr/radarr/sonarr/jellyfin: all running, all daemons under uid 950, project arr.
transmission: PUID/PGID=950, in networks media_net + transmission_default (both external).
Radarr/sonarr/jellyfin mount /mnt/RED_2TB/storage as /storage (NOT /media as on Kraken) → set radarr root-folder accordingly.

24 KiB Executable File Raw Blame History