Oleksandr Kozachuk
112 lines
5.0 KiB
Markdown
112 lines
5.0 KiB
Markdown
# SPEC-013 — News digest
|
|
|
|
Replaces the broken pre-1.0-openai `news_feed.py`. A CLI
|
|
(`python -m fjerkroa_bot.news --config <cfg>`) fetches the
|
|
`news-feeds` and writes a compact digest to the `news` file that
|
|
`AIResponder.message` injects into the `{news}` slot. Feeds are
|
|
external input and operator-configured.
|
|
|
|
### NEWS-01 — RSS (2.0 and 1.0/RDF) and Atom parse to items (coverage: test)
|
|
|
|
`parse_feed(bytes, label)` extracts `{title, link, source}` from both
|
|
RSS (`<item>`) and Atom (`<entry>`) documents, tolerates malformed
|
|
XML (returns an empty list, logs), and never raises.
|
|
|
|
### NEWS-02 — Digest is sanitized and bounded (coverage: test)
|
|
|
|
`render_digest` caps at `news-max-items`, and every headline passes
|
|
`sanitize_external_text` (SAF-03) — a feed cannot inject `@everyone`
|
|
or control characters into the prompt via a headline.
|
|
|
|
### NEWS-03 — Feeds are SSRF-guarded and deduped (coverage: test)
|
|
|
|
`NewsFetcher.collect` skips any feed URL the SSRF guard rejects,
|
|
skips feeds that fail to fetch (one bad feed never sinks the run),
|
|
and drops duplicate headlines across feeds.
|
|
|
|
## Webhook posting (ggg model)
|
|
|
|
`--post` mode fetches feeds mapped to channels and posts NEW items to
|
|
the channel's Discord webhook — replacing the py3.8 `getnews.py`
|
|
(dead play3 feed, 35 MB substring-scan state file, HTML-redirect
|
|
cruft). Config: `news-post-feeds = [[url, label, channel], …]`,
|
|
`news-post-webhooks = {channel = url}`, `news-post-state`.
|
|
|
|
### NEWS-04 — Only unseen items post, then are marked seen (coverage: test)
|
|
|
|
`NewsPoster.run_post` posts each item whose key (link, else title) is
|
|
not in the seen-set, adds it to the set, and posts to the mapped
|
|
channel's webhook. Re-runs over the same feed post nothing new.
|
|
|
|
### NEWS-05 — First run seeds without flooding (coverage: test)
|
|
|
|
With no prior state file (`seed_only`), every current item is marked
|
|
seen but nothing is posted — migrating off getnews.py never dumps a
|
|
backlog into the channels. `news-post-max-per-run` caps steady-state
|
|
posts per run.
|
|
|
|
### NEWS-06 — Post failures and bad channels are survived (coverage: test)
|
|
|
|
A feed the SSRF guard rejects, a feed that fails to fetch, an item
|
|
whose channel has no configured webhook, and a webhook POST that
|
|
raises are each logged and skipped — one failure never sinks the
|
|
run, and the seen-set still advances for successfully-processed
|
|
items.
|
|
|
|
### NEWS-07 — Item summaries are extracted (coverage: test)
|
|
|
|
`parse_feed` also captures each item's short description — RSS
|
|
`<description>`, Atom `<summary>` or `<content>` — with HTML stripped,
|
|
entities unescaped, and whitespace collapsed, so an item carries what
|
|
it is about, not only a headline. Missing descriptions yield an empty
|
|
summary, never an error.
|
|
|
|
### NEWS-08 — The digest carries summaries (coverage: test)
|
|
|
|
`render_digest` appends the sanitized, length-capped
|
|
(`news-summary-chars`, default 200) summary after each headline, so
|
|
the bot's ambient `{news}` context knows the gist of each story, not
|
|
just its title. A zero cap restores the title-only digest.
|
|
|
|
### NEWS-09 — Fetched news is stored, deduped, and rolled over (coverage: test)
|
|
|
|
Both the digest run (kroa) and the posting run (ggg) upsert every
|
|
fetched item into a `news` table keyed by link (or title), so the same
|
|
story is stored once. After each run the store is pruned to the newest
|
|
`news-keep` rows (default 400), a rolling window that bounds growth
|
|
while keeping recent history searchable.
|
|
|
|
### NEWS-10 — get_news is offered as a tool (coverage: test)
|
|
|
|
When `enable-news-tool` is true and a store is configured, the chat
|
|
call's `tools` list includes a `get_news` function (optional `topic`,
|
|
`source`, `limit`) next to the other tools. Without a store or the
|
|
flag it is absent.
|
|
|
|
### NEWS-11 — get_news retrieves filtered, sanitized items (coverage: test)
|
|
|
|
`get_news` returns recent stored items, newest first, optionally
|
|
narrowed by `topic` (every keyword must appear in the title, summary,
|
|
or source label — so `topic: "Nordland"` finds items from that source)
|
|
and/or an exact `source`; `limit` is clamped to 1..30. Each result's title and
|
|
summary are passed through `sanitize_external_text`. The bot can then
|
|
`fetch_url` a returned link for the full article.
|
|
|
|
### NEWS-12 — get_news is metered per user (coverage: test)
|
|
|
|
Each `get_news` call increments a per-user daily counter; over
|
|
`news-daily-per-user` (default 30) the tool refuses with an error
|
|
result without touching the store. The budget gate (SAF-04) still
|
|
applies to the surrounding model calls.
|
|
|
|
### NEWS-13 — Topic misses degrade softly, never empty-handed (coverage: test)
|
|
|
|
A `topic` whose AND-match (NEWS-11) finds nothing falls back to an
|
|
any-term match, ranked by how many keywords hit (ties: newest first);
|
|
if that too is empty, the newest stored items are returned instead.
|
|
Both fallbacks set a `note` field naming the degradation so the model
|
|
can answer honestly ("nothing on that exactly, but…"). A model
|
|
passing a multi-word or wrong-language topic (the live
|
|
`"Nordland road accident"` → `[]` case) thus still gets usable
|
|
context. Exact matches return no `note`.
|