discord_bot/specs/SPEC-011-url-reading.md

SPEC-011 — URL reading

A fetch_url tool alongside IGDB: the model decides when to read a link (user pastes a URL + question; a news item links an article Luma wants details on). Web pages are the number-one injection vector, so everything fetched is sanitized (SAF-03) and the fetch itself is SSRF-guarded — the bot runs on shared hosting. Active only when enable-url-reading = true.

URL-01 — fetch_url is offered as a tool (coverage: test)

When enable-url-reading is true, the chat call's tools list includes a fetch_url function (url string param) next to any IGDB tools. When false, it is absent.

URL-02 — Only http/https are fetched (coverage: test)

file:, ftp:, data:, gopher: and schemeless inputs are refused before any network call, with an error result the model can relay.

URL-03 — SSRF guard blocks non-public addresses (coverage: test)

Before fetching, the host is resolved and every resulting IP is checked; the fetch is refused when any is private, loopback, link-local, or otherwise non-global (RFC1918, 127/8, 169.254/16, ::1, fc00::/7, etc.). A URL literal that is already such an IP is refused without DNS.

URL-04 — Redirects are re-validated (coverage: test)

Redirects are followed manually; each hop's target passes URL-02 and URL-03 again. A public URL that 302-redirects to localhost or an internal IP is refused at the redirect, not fetched. HTML meta-refresh redirects (link shorteners, the old getnews stubs) are also followed — the target is SSRF-re-guarded and fetched, so the reader returns the real article, not the "Redirecting…" stub.

URL-05 — Fetched text is bounded and sanitized (coverage: test)

Responses are capped at url-max-bytes (default 2 MB) with a download timeout; HTML is reduced to readable text (script/style dropped, tags stripped, whitespace collapsed) and passed through sanitize_external_text before it reaches the model, truncated to url-max-chars (default 6000).

URL-06 — Page images feed the cache (coverage: test)

Up to url-max-images (default 2) prominent images (og:image, then large <img>) are ingested into the ImageCache for the requesting channel (SSRF-guarded like the page), so the model can see them and picture_edit can remix them. Ingestion failures are skipped, never fatal to the text result.

URL-07 — Fetches are metered and capped (coverage: test)

Each fetch increments a per-user daily counter; over url-daily-per-user (default 20) fetch_url refuses with an error result. The budget gate (SAF-04) still applies to the surrounding model calls.

URL-08 — Main-content extraction (coverage: test)

fetch_url text drops page chrome: content inside nav/header/footer/aside/form/select/button is skipped like scripts, and text blocks dominated by link text (over 60 % of a block's characters inside <a> and the block shorter than 200 chars — menus, related-article lists, tag clouds) are treated as boilerplate and removed. Body paragraphs with inline links survive. The default url-max-chars cap rises to 8000 now that the budget is spent on content, not chrome.

URL-09 — Fetched images become vision input (coverage: test)

The images the URL reader already caches from a fetched page (og:image first, then body images, url-max-images cap, every candidate SSRF-guarded and magic-byte-sniffed by the image cache) now travel to the model as image input alongside the tool result — the model sees the picture, not just an images_cached count. A URL whose response is itself an image (content-type image/*) is ingested directly and returns text (image); a body at the byte cap is treated as possibly truncated and not ingested. The data URLs ride in a vision key that the responder detaches before the JSON tool text is built (a base64 data URL would blow the 8000-char sanitizer cap): they are appended as input_image items on the Responses path and as image_url parts on the legacy path. Vision is per-turn — nothing extra is historised; the file stays in the image cache for later picture_edit (IMG-15).