Oleksandr Kozachuk
89 lines
4.0 KiB
Markdown
89 lines
4.0 KiB
Markdown
# SPEC-011 — URL reading
|
|
|
|
A `fetch_url` tool alongside IGDB: the model decides when to read a
|
|
link (user pastes a URL + question; a news item links an article
|
|
Luma wants details on). Web pages are the number-one injection
|
|
vector, so everything fetched is sanitized (SAF-03) and the fetch
|
|
itself is SSRF-guarded — the bot runs on shared hosting. Active only
|
|
when `enable-url-reading = true`.
|
|
|
|
### URL-01 — fetch_url is offered as a tool (coverage: test)
|
|
|
|
When `enable-url-reading` is true, the chat call's `tools` list
|
|
includes a `fetch_url` function (url string param) next to any IGDB
|
|
tools. When false, it is absent.
|
|
|
|
### URL-02 — Only http/https are fetched (coverage: test)
|
|
|
|
`file:`, `ftp:`, `data:`, `gopher:` and schemeless inputs are
|
|
refused before any network call, with an error result the model can
|
|
relay.
|
|
|
|
### URL-03 — SSRF guard blocks non-public addresses (coverage: test)
|
|
|
|
Before fetching, the host is resolved and every resulting IP is
|
|
checked; the fetch is refused when any is private, loopback,
|
|
link-local, or otherwise non-global (RFC1918, 127/8, 169.254/16,
|
|
::1, fc00::/7, etc.). A URL literal that is already such an IP is
|
|
refused without DNS.
|
|
|
|
### URL-04 — Redirects are re-validated (coverage: test)
|
|
|
|
Redirects are followed manually; each hop's target passes URL-02 and
|
|
URL-03 again. A public URL that 302-redirects to `localhost` or an
|
|
internal IP is refused at the redirect, not fetched. **HTML
|
|
meta-refresh** redirects (link shorteners, the old getnews stubs) are
|
|
also followed — the target is SSRF-re-guarded and fetched, so the
|
|
reader returns the real article, not the "Redirecting…" stub.
|
|
|
|
### URL-05 — Fetched text is bounded and sanitized (coverage: test)
|
|
|
|
Responses are capped at `url-max-bytes` (default 2 MB) with a
|
|
download timeout; HTML is reduced to readable text (script/style
|
|
dropped, tags stripped, whitespace collapsed) and passed through
|
|
`sanitize_external_text` before it reaches the model, truncated to
|
|
`url-max-chars` (default 6000).
|
|
|
|
### URL-06 — Page images feed the cache (coverage: test)
|
|
|
|
Up to `url-max-images` (default 2) prominent images (og:image, then
|
|
large `<img>`) are ingested into the ImageCache for the requesting
|
|
channel (SSRF-guarded like the page), so the model can see them and
|
|
`picture_edit` can remix them. Ingestion failures are skipped, never
|
|
fatal to the text result.
|
|
|
|
### URL-07 — Fetches are metered and capped (coverage: test)
|
|
|
|
Each fetch increments a per-user daily counter; over
|
|
`url-daily-per-user` (default 20) `fetch_url` refuses with an error
|
|
result. The budget gate (SAF-04) still applies to the surrounding
|
|
model calls.
|
|
|
|
### URL-08 — Main-content extraction (coverage: test)
|
|
|
|
`fetch_url` text drops page chrome: content inside
|
|
`nav`/`header`/`footer`/`aside`/`form`/`select`/`button` is skipped
|
|
like scripts, and text blocks dominated by link text (over 60 % of a
|
|
block's characters inside `<a>` and the block shorter than 200 chars
|
|
— menus, related-article lists, tag clouds) are treated as
|
|
boilerplate and removed. Body paragraphs with inline links survive.
|
|
The default `url-max-chars` cap rises to 8000 now that the budget is
|
|
spent on content, not chrome.
|
|
|
|
### URL-09 — Fetched images become vision input (coverage: test)
|
|
|
|
The images the URL reader already caches from a fetched page
|
|
(`og:image` first, then body images, `url-max-images` cap, every
|
|
candidate SSRF-guarded and magic-byte-sniffed by the image cache) now
|
|
travel to the model as image input alongside the tool result — the
|
|
model sees the picture, not just an `images_cached` count. A URL
|
|
whose response is itself an image (content-type `image/*`) is
|
|
ingested directly and returns text `(image)`; a body at the byte cap
|
|
is treated as possibly truncated and not ingested. The data URLs
|
|
ride in a `vision` key that the responder detaches before the JSON
|
|
tool text is built (a base64 data URL would blow the 8000-char
|
|
sanitizer cap): they are appended as `input_image` items on the
|
|
Responses path and as `image_url` parts on the legacy path. Vision
|
|
is per-turn — nothing extra is historised; the file stays in the
|
|
image cache for later `picture_edit` (IMG-15).
|