[2026-09-21] eagle: family/how-to/arr-stack-kraken.md family/plans/watchlist-disk-overflow-fix.md family/projects/watchlist-automation.md personal/tech/truenas-nfs4-acl-and-arrmultiuser.md

personal/tech/truenas-nfs4-acl-and-arrmultiuser.md

@@ -1,18 +1,21 @@
 ---
 title: TrueNAS NFS4 ACL repair + cross-container media PUID
 created: 2026-09-02
-updated: 2026-09-02
+updated: 2026-09-21
 type: tech
 namespace: personal
-tags: [truenas, nfs4, acl, midclt, arr, radarr, sonarr, jellyfin, transmission, docker]
+tags: [truenas, nfs4, acl, midclt, arr, radarr, sonarr, jellyfin, transmission, docker, git]
 related:
   - "[[truenas-zfs-panic-recovery]]"
   - "[[arr-stack-taiga]]"
+  - "[[watchlist-disk-overflow-fix]]"
 ---
 
 # TrueNAS NFS4 ACL repair + cross-container media PUID
 
 > Hard-won lessons from the 2026-08-31/09-01/09-02 TrueNAS restore-incident remediation. Mirror of skill `truenas-nfs4-acl-and-arrmultiuser`. Detailed incident log: `family/documents/vault-sync/2026-09-02-restore-privilege-scope.md`.
+>
+> **2026-09-21 addendum:** the same class of failure hit the **vault bare-git repo** (`/mnt/RED_2TB/storage/git/obsidian-vault.git`) — `git push` failed with `unable to write file ./objects/XX/…: Permission denied`. Root cause and the working midclt recipe are in the section «Bare git repo: push Permission denied» below.
 
 ## Trigger
 Any "permission denied" / file not readable / git `loose object ... corrupt` / empty Jellyfin library on **TrueNAS SCALE** (acltype=nfsv4) after a zfs restore, OR wiring Prowlarr/Radarr/Sonarr/Jellyfin/Transmission so container-written files are readable by other containers.
@@ -67,6 +70,61 @@ docker restart transmission
 ```
 On restart the daemon re-validates → torrents clear (e.g. **254/255 instantly**). Leftover #1 → Verify Local Data in Web UI.
 
+## Bare git repo: `push` → `Permission denied` (2026-09-21)
+
+**Symptom:**
+```
+error: unable to write file ./objects/80/09a34e…: Permission denied
+To ssh://…/obsidian-vault.git
+ ! [remote rejected] main -> main (unable to migrate objects to permanent storage)
+```
+
+**Root cause — TWO independent defects in the repo's dirs:**
+
+1. **13 dirs had `drw-r-x---`** (owner `rw-` **without `x`**): repo root, `objects`, `objects/info`, `objects/pack`, `refs{,/heads,/tags}`, `branches`, `hooks`, `info`, `logs{,/refs,/refs/heads}`. Without owner-`x` nothing can be created inside. Fixed with `chmod u+x` on dirs only (files were all fine: `0 files without u+r`).
+2. **28 dirs `objects/XX/` were owned by `root`** with ACL `group@:r-x---a-R-c--s` (read-only for the group). Git runs as uid 950 (`truenas_admin`), not root → **cannot write**. This is the defect that actually blocked the push.
+
+**Why it happened:** `mcpvault` (the Obsidian MCP server) runs **as root** — `/opt/hermes/.venv/bin/python3 … mcp_stdio_watchdog.py … mcpvault /vault` (`ps aux` shows uid `root`). Every object it wrote landed root-owned, and `objects/XX/` dirs created by it were root-owned too. **The problem WILL recur** unless mcpvault is restarted as uid 950. Left as an open item.
+
+**What does NOT work:**
+- `chown -R 950:950` and `chmod -R g+w` via SSH as `truenas_admin` → **`Operation not permitted`** (uid 950 is not the owner of root-owned objects). `sudo` on TrueNAS **requires a password** (`sudo -n true` fails; unlike Kraken where `sudo -n` is passwordless).
+- `midclt call filesystem.setperm '{...}'` → `[EINVAL] Non-trivial ACL present … Option "stripacl" required to change permission`. **Don't reach for `stripacl`** — it tears the ACL down to POSIX.
+
+**What DOES work — `filesystem.setacl` via midclt (no root needed):**
+uid 950 can read and write ACLs through midclt. Read the current ACL, add the group perms, write it back. Reference implementation (`/tmp/fix_vault_acl.py` on TrueNAS, ~120 dirs, ~90 s):
+
+```python
+FOR each dir in [repo] + listdir(f"{repo}/objects"):
+    cur = midclt("filesystem.getacl", path)          # path as BARE STRING
+    for ace in cur["acl"]:
+        if ace["tag"] == "group@" and ace["type"] == "ALLOW":
+            ace["perms"]["WRITE_DATA"]    = True
+            ace["perms"]["APPEND_DATA"]   = True
+            ace["perms"]["EXECUTE"]       = True
+            ace["perms"]["WRITE_ATTRIBUTES"] = True
+        if ace["tag"] == "owner@" and ace["type"] == "ALLOW":
+            ace["perms"]["EXECUTE"] = True           # root-owned dirs had this off
+    midclt("filesystem.setacl", {
+        "path": path, "dacl": cur["acl"],
+        "options": {"recursive": False, "traverse": False, "stripacl": False},
+    })
+    # setacl returns a JOB ID → poll core.get_jobs until SUCCESS/FAILED
+```
+
+**⚠️ midclt argument-form pitfalls (cost real time):**
+- `filesystem.getacl` and `filesystem.listdir` take a **BARE STRING path**, NOT `{"path": …}`. Passing an object gives `[EINVAL] path: Not a string`.
+- `filesystem.setacl` takes **ONE JSON object with `path` INSIDE it** (see gotcha #1 above) — the opposite convention from getacl on the same service. Easy to invert.
+- `filesystem.setacl` / `setperm` are **async**: they return a job id. Polling immediately returns state `WAITING` → looks like failure. Must poll until `SUCCESS`/`FAILED`. With ~120 dirs over SSH the loop outlives the SSH timeout — **run it detached** (`nohup … &`) and read the log later; the first SSH-timed-out run had in fact already applied the ACLs.
+- Passwordless `sudo` is available on **Kraken** (`sudo -n true` → ok) but **NOT** on TrueNAS `truenas_admin`.
+
+**Verification (cheap, do it before re-trying push):**
+```bash
+touch /mnt/RED_2TB/storage/git/obsidian-vault.git/objects/80/.wrtest && echo WRITE_OK
+```
+`WRITE OK` → push will succeed. Confirmed: `f9a5c2b..21505ec main -> main`, then a clean `git pull` on Kraken.
+
+**Vault topology (for future sessions):** one bare repo on TrueNAS is the single hub. Mac pushes/pulls remote **`nas`**; Kraken uses remote **`origin`** — both point at `ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git`. Mac and Kraken never talk directly. Host `truenas` does **not** resolve as an SSH alias from the Mac — use `truenas_admin@mallexxx.duckdns.org`.
+
 ## Docker network persistency (per-service compose)
 - `docker network connect <net> <container>` is LOST on recreate → declare `<net>` in the service's compose `networks:`.
 - Caddy must co-own a network with its reverse-proxy target. transmission Web UI works because caddy is ALSO in `transmission_default`; radarr↔transmission works because both in `media_net`.