discord_bot/specs/SPEC-011-url-reading.md

SPEC-011 — URL reading

A fetch_url tool alongside IGDB: the model decides when to read a link (user pastes a URL + question; a news item links an article Luma wants details on). Web pages are the number-one injection vector, so everything fetched is sanitized (SAF-03) and the fetch itself is SSRF-guarded — the bot runs on shared hosting. Active only when enable-url-reading = true.

URL-01 — fetch_url is offered as a tool (coverage: test)

When enable-url-reading is true, the chat call's tools list includes a fetch_url function (url string param) next to any IGDB tools. When false, it is absent.

URL-02 — Only http/https are fetched (coverage: test)

file:, ftp:, data:, gopher: and schemeless inputs are refused before any network call, with an error result the model can relay.

URL-03 — SSRF guard blocks non-public addresses (coverage: test)

Before fetching, the host is resolved and every resulting IP is checked; the fetch is refused when any is private, loopback, link-local, or otherwise non-global (RFC1918, 127/8, 169.254/16, ::1, fc00::/7, etc.). A URL literal that is already such an IP is refused without DNS.

URL-04 — Redirects are re-validated (coverage: test)

Redirects are followed manually; each hop's target passes URL-02 and URL-03 again. A public URL that 302-redirects to localhost or an internal IP is refused at the redirect, not fetched. HTML meta-refresh redirects (link shorteners, the old getnews stubs) are also followed — the target is SSRF-re-guarded and fetched, so the reader returns the real article, not the "Redirecting…" stub.

URL-05 — Fetched text is bounded and sanitized (coverage: test)

Responses are capped at url-max-bytes (default 2 MB) with a download timeout; HTML is reduced to readable text (script/style dropped, tags stripped, whitespace collapsed) and passed through sanitize_external_text before it reaches the model, truncated to url-max-chars (default 6000).

URL-06 — Page images feed the cache (coverage: test)

Up to url-max-images (default 2) prominent images (og:image, then large <img>) are ingested into the ImageCache for the requesting channel (SSRF-guarded like the page), so the model can see them and picture_edit can remix them. Ingestion failures are skipped, never fatal to the text result.

URL-07 — Fetches are metered and capped (coverage: test)

Each fetch increments a per-user daily counter; over url-daily-per-user (default 20) fetch_url refuses with an error result. The budget gate (SAF-04) still applies to the surrounding model calls.