obsidian-vault/work/tech-design/opaque-url-fragment-detection.md

title, created, tags, status, asana

title	created	tags	status	asana
Tech Design: Opaque URL Fragment Detection for Same-Document Navigation	2026-06-24	tech-design apple navigation webkit url crash	draft	https://app.asana.com/1/137249556945/project/1202406491309510/task/1215403820548743

Author: Apple team Project: fix-fragment-crash (macOS / iOS browser) Sentry: APPLE-IOS-DY5A · DZ58 · DZ5C

---

Background & Requirements

The DuckDuckGo browser uses WKWebView for all page rendering. Every navigation goes through the decidePolicyFor delegate chain managed by DistributedNavigationDelegate (BrowserServicesKit). A central piece of that chain is classifying each WKNavigationAction as a same-document navigation (fragment-only change within the current page, e.g. #section) via:

// WKNavigationActionExtension.swift:184
public var isSameDocumentNavigation: Bool {
    switch navigationType {
    case .linkActivated, .other:
        return !isRedirect && newURL.hasFragment && currentURL.equals(newURL, by: .sameDocument)
    case .backForward:
        return (newURL.hasFragment || currentURL.hasFragment) && currentURL.equals(newURL, by: .sameDocument)
    ...
    }
}

isSameDocumentNavigation drives the DistributedNavigationDelegate state machine: same-document navigations reuse the existing Navigation object and trigger client-redirect handling rather than starting a fresh navigation, while non-same-document navigations allocate a new Navigation. Misclassification corrupts navigation state.

hasFragment and the .sameDocument equality both depend on extracting the URL fragment—which is where the bug lives.

Two URL parsers coexist in WebKit on Apple platforms

Parser	Used where	Standard	Fragment in opaque URLs (data:, about:, blob:)
WTF::URLParser	WebKit rendering, WKWebView navigation decisions	WHATWG URL	✅ Correctly split at #
NSURL / CFURL	Foundation APIs, Swift URL	Old RFC 1808/2396	❌ Encodes # as %23, returns nil for .fragment

When WebKit bridges its internal WTF::URL to Objective-C it calls:

CFURLCreateAbsoluteURLWithBytes(alloc, rawBytes, len, kCFStringEncodingUTF8, nil, true)

The raw bytes at this point still contain the literal #. CFURL re-parses them with its old-RFC parser, which does not recognise # as a fragment delimiter in opaque-scheme URLs. It encodes # → %23 and stores the whole thing as the opaque path.

Result for data:text/html,hello#anchor:

Property	WTF::URL	NSURL / Swift URL
.path	text/html,hello	"" (empty)
.query	nil	nil
.fragment	"anchor"	nil
.absoluteString	data:text/html,hello#anchor	data:text/html,hello%23anchor

---

Problem Statement

A pathological URL—a multi-MB data: URL or attacker-crafted href—reaches WKNavigationAction.isSameDocumentNavigation on the main thread. The call chain:

1. WebKit::WebPageProxy::decidePolicyForNavigationAction dispatches to the main-thread delegate.
2. DistributedNavigationDelegate reads wkNavigationAction.isSameDocumentNavigation.
3. isSameDocumentNavigation calls newURL.absoluteString.hashedSuffix (and for .backForward, also currentURL.absoluteString.hashedSuffix).
4. hashedSuffix → hashedSuffixRange → String.firstIndex(of: "#") (StringExtension.swift:67) iterates grapheme-by-grapheme over absoluteString.
5. For a multi-MB data: URL this is O(n) UTF-8 grapheme decoding on the main thread, per navigation decision.
6. Main thread blocked >10 s under thermal-state serious at 100% CPU → RunningBoard scene-update watchdog kills the app: SIGKILL, terminationReason 0x8BADF00D.

The fix must:

1. Make hashedSuffixRange / isSameDocumentNavigation fast for arbitrarily large URLs (no O(n) grapheme scan).
2. Correctly detect fragments in opaque URLs where NSURL has already encoded # as %23.

---

Why Standard APIs Can't Help: The Information-Loss Chain

Fragment information is irreversibly lost between WebKit's internal representation and what Swift code receives in decidePolicyFor:

WTF::URL  (WHATWG-compliant; m_string = "data:…#anchor", m_queryEnd before '#')
    ↓  URL::createNSURL() → createCFURL()
    ↓  CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor")   ← '#' still visible here
    ↓  CFURL re-parses with old-RFC parser; encodes '#' → '%23'
NSURL  (.absoluteString = "…%23anchor",  .fragment = nil,  .path = "")
    ↓  [NSMutableURLRequest initWithURL: url().createNSURL()]   (ResourceRequestCocoa.mm)
NSURLRequest  (same NSURL)
    ↓
WKNavigationAction.request.URL          ← what Swift sees
WKNavigationAction._originalURL         ← same path, same loss (createNSURL() again)

Every Foundation/Swift API is downstream of the encoding step:

• URL.fragment / .path / .query — return nil / "" for all opaque URLs.
• URLComponents(url:) / (string:) and NSURLComponents equivalents — either use NSURL's already-broken cached parse tree, or re-parse absoluteString where # is already %23. RFC 3986 treats %23 as a literal character, not a delimiter. Fragment stays nil in all four variants.
• WKNavigationActionPrivate.h (_originalURL, _isRedirect, _hitTestResult, …) — all URL-returning properties go through WTF::URL::createNSURL() → same createCFURL() path. No private API exposes the raw WTF string.

There is no Foundation or WebKit API, public or private, that recovers the original # once the NSURL has been constructed.

---

Approaches Tried

1. Scan absoluteString for %23

Since NSURL encodes the original # as %23 in absoluteString, scanning for %23 as a fragment delimiter recovers the fragment in many cases. Implemented and partly working.

Fundamental problem: %23 is also valid payload data in data: URLs:

• SVG fill colors: data:image/svg+xml,<rect fill="%23ff0000"/> — %23 is # in the SVG
• HTML anchor hrefs: data:text/html,<a href="%23section">
• CSS: data:text/css,color:%23333

Both cases produce identical absoluteString. They are indistinguishable from the string alone. A 4 KB size guard was added to avoid scanning large data: URLs, but that makes two different large data: URLs compare as equal — semantically wrong.

2. Private API / WKNavigationAction._originalURL

See information-loss chain above. No private API bypasses it; all paths go through createCFURL().

---

Recommended Approach: CFURLCreateAbsoluteURLWithBytes Interpose

Key Insight

WebKit calls CFURLCreateAbsoluteURLWithBytes() at the exact moment it converts a WTF::URL to a CFURL/NSURL. The raw byte buffer at that call site still contains the literal #, before CFURL encodes it. Intercepting this C function gives unambiguous access to the original bytes.

Mechanism: DYLD_INTERPOSE

The __DATA,__interpose Mach-O section replaces any C function in the same dylib at load time, without method swizzling or private SPI:

#define DYLD_INTERPOSE(_replacement, _replacee) \
    __attribute__((used)) \
    static struct { const void *replacement; const void *replacee; } \
    _interpose_##_replacee \
    __attribute__((section("__DATA,__interpose"))) = \
    { (const void *)&(_replacement), (const void *)&(_replacee) }

Implementation

Files:

• CFURLCreateAbsoluteURLWithBytesInterpose.m — DEBUG-only (#ifdef DEBUG)
• NSURL+FragmentByteRange.h / .m — category exposing the stored range

Interpose logic:

1. Call the real CFURLCreateAbsoluteURLWithBytes to get the CFURLRef.
2. For opaque URLs (scheme present, no // authority), scan the raw byte buffer for '#' (byte 0x23) with a simple C loop—O(n) but done once at NSURL construction, not per navigation decision.
3. If found, store the byte range as an NSValue associated object on the bridged NSURL via objc_setAssociatedObject.
4. Callers read NSURL.ddg_fragmentByteRange for the unambiguous # position.

if (relativeURLBytes[i] == '#') {
    NSRange fragmentRange = NSMakeRange(i, length - i);
    NSURL *nsurl = (__bridge NSURL *)result;
    objc_setAssociatedObject(nsurl, &kFragmentByteRangeKey,
                             [NSValue valueWithRange:fragmentRange],
                             OBJC_ASSOCIATION_RETAIN_NONATOMIC);
}

Why This Is Correct

• Bytes are scanned before CFURL encodes # → %23 — no ambiguity with payload %23.
• Associated object is set on the exact NSURL instance for that WTF::URL.
• ddg_fragmentByteRange.location matches WTF::URL's internal m_queryEnd + 1.
• Works for all opaque schemes: data:, about:, blob:, javascript:.
• DEBUG-only: zero production overhead; interpose section absent from release builds.

Complete Call Flow with Interpose

WTF::URL  (m_string = "data:…#anchor",  m_queryEnd before '#')
    ↓  URL::createCFURL()
    ↓  CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor")
         ↑↑ INTERPOSED: scan rawBytes → store NSRange on result NSURL
    ↓  NSURL  (.absoluteString = "…%23anchor",  .fragment = nil)
         ↑ .ddg_fragmentByteRange → {location=N}  — unambiguous '#' position
Swift URLExtension.opaqueComponents:
    DEBUG   → use ddg_fragmentByteRange (exact)
    release → NSString.range(of: "#") fallback (unambiguous for non-data: schemes)

---

Notes

[1] The %23 ambiguity only affects data: or about: URLs. For blob:, javascript: the opaque path never legitimately contains %23 as encoded payload.

[2] The primary safety fix is the O(n) crash: replace String.firstIndex(of: "#") with a byte-level scan and short-circuit on data: scheme or length threshold. The interpose is the accuracy improvement layered on top for DEBUG validation.

[3] WebKit bug for NSURL not handling # in opaque URLs: bugs.webkit.org/68089. Chromium fixed this in M72 (2019). WebKit's WTF::URLParser is correct; only the Foundation/CFURL bridge is broken.

---

Testing

• Unit tests (URLExtensionTests.swift):

  • equals(by: .sameDocument) and equals(by: .fuzzyIdentity) on opaque URLs with fragment: about:blank#section, data:text/html,…#anchor
  • Opaque URLs with query AND fragment: data:text/html,…?key=value#anchor
  • Large (20 MB) data: URL performance test: comparison must complete in < 0.1 s
  • Diagnostic timing test (tempOpaqueNSURLComponentTimings) printing per-operation durations for all Foundation URL APIs on a 20 MB data: URL with query and fragment

• Manual test (DEBUG build): verify ddg_fragmentByteRange is set on WKNavigationAction.request.URL when navigating to a data: URL with a # fragment via WKWebView.

• Regression: navigation tests in DistributedNavigationDelegateTests.swift and NavigationTestHelpers.swift updated to use .equals(by: .fuzzyIdentity) in place of removed .matches() API.

---

Additional Considerations

Privacy

• No user data collected or transmitted. All processing is local and in-process.

Security

• No extra data is stored or passed anywhere
• The approach injects into the existing API and stores a helper pointer

Site Breakage

• The O(n) fix prevents crashes on large URLs. Fragment-detection correctness improves navigation state machine accuracy but does not change whether a navigation is allowed or blocked.

Operational

• No new infrastructure, network calls, or persisted state. The crash was a SIGKILL watchdog kill visible in Sentry; the fix eliminates the 0x8BADF00D terminations for this call chain.