Oleksandr Kozachuk
61 lines
2.6 KiB
Markdown
61 lines
2.6 KiB
Markdown
# SPEC-011 — URL reading
|
|
|
|
A `fetch_url` tool alongside IGDB: the model decides when to read a
|
|
link (user pastes a URL + question; a news item links an article
|
|
Luma wants details on). Web pages are the number-one injection
|
|
vector, so everything fetched is sanitized (SAF-03) and the fetch
|
|
itself is SSRF-guarded — the bot runs on shared hosting. Active only
|
|
when `enable-url-reading = true`.
|
|
|
|
### URL-01 — fetch_url is offered as a tool (coverage: test)
|
|
|
|
When `enable-url-reading` is true, the chat call's `tools` list
|
|
includes a `fetch_url` function (url string param) next to any IGDB
|
|
tools. When false, it is absent.
|
|
|
|
### URL-02 — Only http/https are fetched (coverage: test)
|
|
|
|
`file:`, `ftp:`, `data:`, `gopher:` and schemeless inputs are
|
|
refused before any network call, with an error result the model can
|
|
relay.
|
|
|
|
### URL-03 — SSRF guard blocks non-public addresses (coverage: test)
|
|
|
|
Before fetching, the host is resolved and every resulting IP is
|
|
checked; the fetch is refused when any is private, loopback,
|
|
link-local, or otherwise non-global (RFC1918, 127/8, 169.254/16,
|
|
::1, fc00::/7, etc.). A URL literal that is already such an IP is
|
|
refused without DNS.
|
|
|
|
### URL-04 — Redirects are re-validated (coverage: test)
|
|
|
|
Redirects are followed manually; each hop's target passes URL-02 and
|
|
URL-03 again. A public URL that 302-redirects to `localhost` or an
|
|
internal IP is refused at the redirect, not fetched. **HTML
|
|
meta-refresh** redirects (link shorteners, the old getnews stubs) are
|
|
also followed — the target is SSRF-re-guarded and fetched, so the
|
|
reader returns the real article, not the "Redirecting…" stub.
|
|
|
|
### URL-05 — Fetched text is bounded and sanitized (coverage: test)
|
|
|
|
Responses are capped at `url-max-bytes` (default 2 MB) with a
|
|
download timeout; HTML is reduced to readable text (script/style
|
|
dropped, tags stripped, whitespace collapsed) and passed through
|
|
`sanitize_external_text` before it reaches the model, truncated to
|
|
`url-max-chars` (default 6000).
|
|
|
|
### URL-06 — Page images feed the cache (coverage: test)
|
|
|
|
Up to `url-max-images` (default 2) prominent images (og:image, then
|
|
large `<img>`) are ingested into the ImageCache for the requesting
|
|
channel (SSRF-guarded like the page), so the model can see them and
|
|
`picture_edit` can remix them. Ingestion failures are skipped, never
|
|
fatal to the text result.
|
|
|
|
### URL-07 — Fetches are metered and capped (coverage: test)
|
|
|
|
Each fetch increments a per-user daily counter; over
|
|
`url-daily-per-user` (default 20) `fetch_url` refuses with an error
|
|
result. The budget gate (SAF-04) still applies to the surrounding
|
|
model calls.
|