obsidian-vault/work/tech-design/opaque-url-fragment-detection.md

title, created, tags, status, chosen-approach, asana

title	created	tags	status	chosen-approach	asana
Tech Design: Opaque URL Fragment Detection for Same-Document Navigation	2026-06-24	tech-design apple navigation webkit url crash	in-progress	_web_originalDataAsString via URLComponents(webKitUrl:)	https://app.asana.com/1/137249556945/project/1202406491309510/task/1215403820548743

Author: Apple team Project: fix-fragment-crash (macOS / iOS browser) Sentry: APPLE-IOS-DY5A · DZ58 · DZ5C

---

Background

Every navigation through WKWebView goes through DistributedNavigationDelegate, which classifies each WKNavigationAction as a same-document navigation (fragment-only change, e.g. #section) via isSameDocumentNavigation. Misclassification corrupts the navigation state machine.

isSameDocumentNavigation relies on hasFragment and .sameDocument URL equality, both of which must correctly extract the fragment — which is where the bug lives.

Two URL parsers in WebKit

Parser	Used where	Fragment in opaque URLs (data:, about:, blob:)
WTF::URLParser	WebKit internals, navigation decisions	✅ Correctly split at #
NSURL / CFURL	Foundation APIs, Swift URL	❌ Encodes # as %23; .fragment returns nil

When WebKit bridges a WTF::URL to Objective-C it calls:

CFURLCreateAbsoluteURLWithBytes(alloc, rawBytes, len, kCFStringEncodingUTF8, nil, true)

The raw bytes contain the literal #, but CFURL's old-RFC parser does not recognise # as a fragment delimiter in opaque-scheme URLs — it encodes it as %23 and stores the whole thing as the opaque path. Result for data:text/html,hello#anchor:

Property	WTF::URL	NSURL / Swift URL
.fragment	"anchor"	nil
.absoluteString	data:text/html,hello#anchor	data:text/html,hello%23anchor

---

Problem

A multi-MB data: URL reaches isSameDocumentNavigation on the main thread. The old hashedSuffix implementation called String.firstIndex(of: "#"), iterating grapheme-by-grapheme — O(n) per navigation decision on the main thread. For a large enough URL this blocks the thread >10 s → RunningBoard watchdog SIGKILL (0x8BADF00D).

The fix must:

1. Make fragment detection fast (no O(n) grapheme scan).
2. Correctly detect fragments in opaque URLs where NSURL has already encoded # as %23.

---

Information-Loss Chain

Fragment information is irreversibly lost between WebKit and Swift:

WTF::URL  (m_string = "data:…#anchor")
    ↓  createCFURL()
    ↓  CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor")  ← '#' visible here
    ↓  CFURL re-parses; '#' → '%23'
NSURL  (.absoluteString = "…%23anchor",  .fragment = nil)
    ↓
WKNavigationAction.request.URL          ← what Swift sees

Every downstream Foundation API (URL.fragment, URLComponents, NSURLComponents) operates on the already-broken NSURL — they all return nil for the fragment. No public or private WebKit API exposes the raw WTF::URL string … except one (see Chosen Approach).

---

Approaches Considered

1. Scan absoluteString for %23 ❌

Since NSURL encodes # as %23, scanning for it recovers the fragment in many cases. Partly working, but fundamentally ambiguous: %23 is also valid payload in data: URLs (SVG fill colors, anchor hrefs, CSS). The two cases produce identical absoluteString and cannot be distinguished from the string alone. A size guard avoids scanning large payloads but causes incorrect equality results.

2. CFURLCreateAbsoluteURLWithBytes GOT-patch hook

WebKit calls CFURLCreateAbsoluteURLWithBytes() at the exact moment it converts a WTF::URL to a CFURL. The raw byte buffer at that point still contains the literal #. The hook intercepts this call using runtime GOT patching (same technique as CxaThrowSwapper / fishhook) — not a link-time __DATA,__interpose section, which would not reach WebKit's internal call from WebKit.framework into CoreFoundation.framework.

Mechanism:

1. _dyld_register_func_for_add_image registers a callback that fires for every Mach-O image loaded into the process (including already-loaded images at registration time).
2. The callback walks the image's indirect symbol table (ImageMap.rebindSymbol) targeting only the WebKit dylib, finds its GOT entry for CFURLCreateAbsoluteURLWithBytes, and replaces it with the hook pointer. ARM64e pointer authentication is handled via BSKStripFunctionPointer / BSKSignFunctionPointer.
3. The hook calls the original via the saved pointer, then scans the raw byte buffer with memchr (SIMD-vectorised on ARM64/x86) for ':' (opaque check) then '#', and stores the resulting NSRange as an associated object on the CFURL result.
4. URL.opaqueFragmentAnnotation reads/writes that associated object; URL.isOpaqueFragmentScanned guards against double-scanning the same instance.
5. uninstallCFURLSwapper() walks all loaded images and restores the saved original pointer.

// install: register dyld callback → patch WebKit's GOT entry at runtime
_dyld_register_func_for_add_image(_cfURLProcessMachHeader)

// hook: memchr scan + associated-object annotation
if let hashRaw = memchr(searchBase, Int32(UInt8(ascii: "#")), searchLen) {
    url.opaqueFragmentAnnotation = NSRange(location: hashOff, length: len - hashOff)
}

Downside: couples to WebKit's dylib name and a private CFURL symbol; requires bypassing page protection to write into a read-only GOT page; depends on dyld image-load ordering; massive and fragile.

2a. __DATA,__interpose (static link-time interpose) ⚠️ (option, not chosen)

A related but distinct mechanism: a __DATA,__interpose section in the binary redirects calls to the named C function to a replacement at load time, before any code runs. Simpler than GOT patching (no dyld callback, no image walk), and dyld applies it process-wide so it does intercept calls from other frameworks.

Downside: the __DATA,__interpose section is a well-known jailbreak / hooking fingerprint; App Store review tooling flags it, making approval unlikely.

---

Chosen Approach: Private WebKit URL String (_web_originalDataAsString)

NSURL instances that originate from WebKit (via createNSURL()) carry a private property _web_originalDataAsString which holds the original WTF::URL byte string — before CFURL re-encoded # as %23. Parsing that string with Swift's native URLComponents(string:) gives correct path/query/fragment decomposition for all opaque URLs.

// URLComponentsExtension.swift
extension URLComponents {
    /// Parses a WebKit-origin URL using the original WTF::URL string.
    /// Falls back to URLComponents(url:) for non-WebKit or hierarchical URLs.
    init?(webKitUrl: URL) {
        guard webKitUrl.isOpaque,
              let originalString = (webKitUrl as NSURL).value(forKey: "_web_originalDataAsString") as? String,
              let swiftNativeURLComponents = URLComponents(string: originalString) else {
            self.init(url: webKitUrl, resolvingAgainstBaseURL: false)
            return
        }
        self = swiftNativeURLComponents
    }
}

// URLExtension.swift
public var originalWebKitString: String? {
    (self as NSURL).value(forKey: "_web_originalDataAsString") as? String
}

public var hasFragment: Bool {
    guard let components = URLComponents(webKitUrl: self),
          let fragment = components.fragment else { return false }
    return !fragment.isEmpty
}

public func equals(_ other: URL, by components: EqualityComponents) -> Bool {
    guard let selfParsed  = URLComponents(webKitUrl: self),
          let otherParsed = URLComponents(webKitUrl: other) else { return false }
    // component-by-component comparison …
}

Why this is correct

• _web_originalDataAsString holds the WTF::URL's m_string verbatim — # is still a literal character, not %23.
• Swift's URLComponents(string:) is WHATWG-aligned and correctly splits at #, so .fragment returns the real fragment.
• No ambiguity with %23 payload: we parse the original string, not the NSURL's re-encoded absoluteString.
• The property is absent on non-WebKit URLs (URL(string:)-created values) — the initialiser falls back to URLComponents(url:), which is correct for hierarchical URLs.
• Pure Swift/Obj-C KVC access; no interpose, no associated objects, no extra files.

Call flow

WTF::URL  (m_string = "data:…#anchor")
    ↓  createNSURL()
NSURL  (.absoluteString = "…%23anchor",  ._web_originalDataAsString = "data:…#anchor")
    ↓
URLComponents(webKitUrl: url)
    → value(forKey: "_web_originalDataAsString") → "data:…#anchor"
    → URLComponents(string: "data:…#anchor")
    → .fragment = "anchor"  ✅

---

Testing

• Unit tests (URLExtensionTests.swift):
  • hasFragment and equals(by: .sameDocument / .fuzzyIdentity) on about:blank#section, data:text/html,…#anchor, opaque URLs with query AND fragment.
  • Large (20 MB) data: URL: comparison must complete in < 0.1 s.
  • Diagnostic timing test (tempOpaqueNSURLComponentTimings) for all Foundation URL APIs on a 20 MB data: URL.
• Regression: navigation tests in DistributedNavigationDelegateTests.swift updated to use .equals(by: .fuzzyIdentity) in place of the removed .matches() API.

---

Notes

• WebKit bug: bugs.webkit.org/68089. Chromium fixed # handling in M72 (2019). WTF::URLParser is correct; only the CFURL bridge is broken.
• Primary safety fix is the O(n) crash: isSameDocumentNavigation now uses URLComponents(webKitUrl:) which is constant-time relative to URL length (KVC + a single string parse), replacing the old grapheme-scan loop.
• %23 ambiguity only affects data: / about:. For blob: and javascript: the opaque path never legitimately contains %23 as encoded payload.
• No user data collected or transmitted; all processing is local and in-process.