health monitor (ops-18/19): spend/disk/task-queue thresholds -> staff alerts, edge-triggered, opt-in

DECISIONS.md

@@ -60,6 +60,47 @@ Decisions inside the set architecture. D-NNN, never renumbered.
   broken classifier must never mute the bot; the budget gate already
   bounds spend. Its verdict gates BEFORE the main call, the
   envelope's answer_needed still gates after — two independent nets.
+- **D-021** — Health monitoring (FDB-012, SPEC-012 OPS-18/19): a
+  separate `monitor_loop` (own cadence, default 300 s) rather than
+  folding checks into the 60 s task loop — monitoring is coarse and
+  should not run every minute. Checks are edge-triggered (alert on the
+  rising edge, re-arm on recovery) so a standing condition never spams;
+  they reuse the existing rate-limited staff-alert path. Metrics are
+  the cheap, high-signal ones (spend vs budget, free disk, task-queue
+  depth); each is independently skippable when it has no data, so a
+  deployment without a budget or store still runs the others. Opt-in
+  (`enable-monitoring`) like every other operational rollout.
+- **D-020** — Web search via Exa (FDB-022, SPEC-015): a `web_search`
+  tool alongside fetch_url/IGDB/codex/get_news, filling the "look it up
+  on the open web" gap. Exa (not a raw search-engine scrape) because it
+  returns clean title+url+text in one call — no SSRF surface of our own
+  (we call one fixed API endpoint, not arbitrary hosts), and it pairs
+  with fetch_url for the full article. Key is a host secret
+  (`exa-api-key`, env `EXA_API_KEY` fallback), never repo-side; results
+  sanitized like every other external-text tool; off by default
+  (`enable-web-search`), metered per user.
+- **D-019** — News memory + on-demand tool (SPEC-013 NEWS-07..12):
+  the news pipeline now carries item summaries (feed descriptions,
+  HTML-stripped) and persists every fetched item into a deduped `news`
+  table (schema v6), pruned to a rolling window (`news-keep`). Both the
+  kroa digest run and the ggg posting run write to it, so the store is
+  a single searchable source across both models. A `get_news` tool
+  reads that store (topic/source-filtered, metered, sanitized) rather
+  than re-fetching feeds live: the ambient `{news}` digest stays a
+  small always-on snapshot, while the tool gives unbounded on-demand
+  reach without a fresh network round-trip per call. The store is the
+  same `bot.db` (WAL) the bot uses; the cron process opens it
+  independently — concurrent reader/writer is what WAL is for.
+- **D-018** — Codex Mechanicus search (FDB-019, SPEC-014): Luma's
+  lore is grounded in the priest's real archive at binaric.tech via a
+  `codex_search` tool over the site's public `search-index.json`, not
+  a bot-side copy — the index stays a single source of truth, refreshed
+  by the site's own publish rite, and the bot caches it in memory
+  (TTL). It reuses SPEC-011's `guard_url` + `read_capped` (fetch is
+  SSRF-guarded and byte-bounded) and sanitizes every returned field:
+  one's own web content is still untrusted by the time it reaches a
+  prompt. Luma-only (`enable-codex`, off elsewhere) — the Adeptus
+  Mechanicus archive has no place in Fjærkroa's café persona.
 - **D-017** — All human-behavior knobs default to off/v3.0.0
   semantics; behavior changes are config rollouts per deployment, not
   code flips. The classifier's `factual` flag is the only coupling

docs/RUNBOOK.md

@@ -0,0 +1,130 @@
+# Operator runbook — Fjærkroa / Luma bot
+
+One page for "something is wrong, what do I do". Two deployments of one
+codebase, both on **uberspace** (push-based deploy from the dev machine —
+there is no git checkout on the hosts).
+
+| | Fjærkroa (café) | Luma (GGG clan) |
+| --- | --- | --- |
+| SSH host | `ssh fjerkroa` (pictor.uberspace.de) | `ssh ggg` |
+| Service | `kroa` | `luma` |
+| Config | `~/fjerkroa_bot/kroa.toml` | `~/fjerkroa_bot/ggg.toml` |
+| Staff channel | `#kassa` | `#mods` |
+| Language / persona | Norwegian, café host | German, "Luma" |
+
+Common paths on each host: bot code `~/fjerkroa_bot`, venv `~/venv-bot`,
+database `~/fjerkroa_bot/history/bot.db` (SQLite, WAL), our snapshots
+`~/backups/<kroa|luma>/`, logs under `~/logs` and `~/tmp`.
+
+## From Discord (staff channel only, prefix `!bot`)
+
+No SSH needed for day-to-day control. Type `!bot help` in the staff
+channel for the full, grouped list. The essentials:
+
+- `!bot pause` / `!bot resume` — stop / start all replies.
+- `!bot quiet <minutes>` — go silent for a while, then auto-resume.
+- `!bot status` — replies/images/tasks flags + quiet time left.
+- `!bot spend` — today's estimated USD spend, tokens, images, budget.
+- `!bot images on|off`, `!bot tasks on|off` — kill-switches.
+
+`!help` works in **any** channel (for everyone) and lists only what is
+usable there. `!forgetme` and `!privacy` also work everywhere, even
+while the bot is paused.
+
+## Restart / check health (SSH)
+
+```sh
+ssh <host>
+supervisorctl status <kroa|luma>          # RUNNING + uptime
+supervisorctl restart <kroa|luma>
+tail -n 40 ~/tmp/<kroa|luma>-stderr*.log   # discord login / errors
+tail -n 40 ~/logs/supervisord.log          # "We have logged in as ..."
+```
+
+A healthy start shows a fresh `connected to Gateway` + `We have logged
+in as ...` line within ~15 s.
+
+## Deploy a release / roll back
+
+From the **dev machine** (`~/Repos/FjerkroaBot`), tags only:
+
+```sh
+git tag -m "<msg>" vX.Y.Z && git push --tags        # cut the release first
+bash deploy/deploy.sh ggg vX.Y.Z                    # luma
+DEPLOY_FORCE=1 bash deploy/deploy.sh fjerkroa vX.Y.Z # kroa (see window)
+```
+
+- kroa refuses to deploy **11:00–22:00 Europe/Oslo** (restaurant hours);
+  `DEPLOY_FORCE=1` overrides. Café is closed Mondays.
+- The script backs up `bot.db` → `bot.db.pre-<tag>` before restart, then
+  smoke-tests (RUNNING + fresh login) and fails loudly if either misses.
+- **Rollback** = deploy the previous tag. If the schema version moved
+  between the two tags, restore the matching `bot.db.pre-<newtag>` first
+  (see below) so the older code meets a schema it understands.
+
+## Restore the database
+
+Three independent daily backup layers exist — pick the freshest good one.
+
+```sh
+ssh <host>
+supervisorctl stop <kroa|luma>
+DB=~/fjerkroa_bot/history/bot.db
+
+# 1) uberspace nightly backup of the whole home (read-only):
+#    /backup = current + daily.0..7 + weekly.1..7  (15 restore points)
+cp /backup/daily.1/home/<user>/fjerkroa_bot/history/bot.db "$DB"
+
+# 2) our own rotated gzip snapshot (03:17 UTC cron, keep 14):
+gunzip -c ~/backups/<kroa|luma>/bot-YYYYMMDD-HHMMSS.db.gz > "$DB"
+
+# 3) the pre-deploy snapshot for a given release:
+cp "$DB".pre-vX.Y.Z "$DB"
+
+rm -f "$DB"-wal "$DB"-shm    # drop stale WAL sidecars after a restore
+supervisorctl start <kroa|luma>
+```
+
+`<user>` is `fjerkroa` or `ggg`. The DB holds conversation history,
+structured memory, usage ledger, image cache index, tasks, and the news
+store — all regenerable, none critical. That is why there is no off-host
+backup: uberspace `/backup` + the on-host snapshots are enough.
+
+## Rotate a secret
+
+Secrets live only in the host `*.toml` (never in the repo). Edit in
+place and restart:
+
+```sh
+ssh <host>
+# OpenAI: edit  openai-token = "sk-..."   in kroa.toml / ggg.toml
+# Discord: edit discord-token = "..."     (get a new token from the
+#          Discord developer portal → Bot → Reset Token first)
+supervisorctl restart <kroa|luma>
+```
+
+After rotating an OpenAI key, revoke the old one in the OpenAI dashboard.
+Keep a `*.toml` backup before editing; a broken TOML crash-loops the
+service (validate: `~/venv-bot/bin/python -c 'import tomlkit; tomlkit.load(open("kroa.toml"))'`).
+
+## Scheduled jobs (crontab -l)
+
+| Host | When (server time) | Job |
+| --- | --- | --- |
+| both | `17 3 * * *` | `backup_db.py` → `~/backups/<bot>/` (keep 14) |
+| kroa | `5 * * * *` | news digest → `{news}` file + news store |
+| ggg | `*/15 * * * *` | news poster → #news/#newsjp webhooks + store |
+
+Logs: `~/backups/<bot>/backup.log`, `~/backups/<bot>/news*.log`.
+
+## Quick triage
+
+- **Bot silent everywhere** → `!bot status` (paused/quiet?), else
+  `supervisorctl status`; if not RUNNING, `restart` and read stderr.
+- **Bot silent in one channel** → check the host config `ignore-channels`
+  / `short-path` rules for that channel (a stray `short-path` rule can
+  archive messages without replying).
+- **Repeated API errors** → the bot posts a rate-limited alert to the
+  staff channel after 5 consecutive OpenAI failures (OPS-16); check
+  `!bot spend` (budget hit?) and the OpenAI status/key.
+- **Bad deploy** → roll back to the previous tag (above).

fjerkroa_bot/codex.py

@@ -0,0 +1,147 @@
+"""Codex Mechanicus search tool (SPEC-014, FDB-019).
+
+Luma's own sacred archive — the Codex Mechanicus at binaric.tech — as a
+function tool. He searches the codex index and answers Cult Mechanicus
+lore from real, sourced inscriptions instead of inventing it. The index
+is fetched over HTTPS (SSRF-guarded, size-bounded, cached in memory) and
+every field returned to the model is sanitized (SAF-03), because even
+one's own web content is still untrusted input by the time it reaches a
+prompt.
+
+The model calls `codex_search`; production wires the live index URL.
+"""
+
+import json
+import logging
+import time
+from typing import Any, Callable, Dict, List, Optional
+from urllib.parse import urljoin
+
+import aiohttp
+
+from .ai_responder import sanitize_external_text
+from .httpread import read_capped
+from .url_reader import guard_url
+
+DEFAULT_INDEX_URL = "https://binaric.tech/search-index.json"
+DEFAULT_MAX_BYTES = 4 * 1024 * 1024
+DEFAULT_LIMIT = 5
+DEFAULT_TTL_S = 3600
+DEFAULT_SUMMARY_CHARS = 500
+FETCH_TIMEOUT_S = 15
+_VALID_LANGS = ("en", "de", "eo", "no", "uk")
+
+CODEX_SEARCH_TOOL = {
+    "name": "codex_search",
+    "description": "Search Luma's own Codex Mechanicus (the sacred archive at binaric.tech) for Adeptus "
+    "Mechanicus lore: doctrines, forges, orders, rites, relics, weapons, entities, the lexicon, and the "
+    "priest's own adoptus. Returns matching inscriptions with a short summary and the URL to read the full "
+    "text. Use for any Cult Mechanicus / Warhammer 40k Mechanicus question so the answer is grounded in the "
+    "codex, not invented.",
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "query": {"type": "string", "description": "What to look for: a name, concept, rite, or phrase."},
+            "lang": {"type": "string", "description": "Language of the inscriptions to prefer: en, de, eo, no, uk. Default en."},
+        },
+        "required": ["query"],
+    },
+}
+
+_STOP = {"the", "a", "an", "of", "and", "or", "to", "in", "is", "der", "die", "das", "und", "von", "en", "et"}
+
+
+def _tokenize(text: str) -> List[str]:
+    cleaned = "".join(c.lower() if c.isalnum() else " " for c in text)
+    return [t for t in cleaned.split() if len(t) > 1 and t not in _STOP]
+
+
+def _score(item: Dict[str, Any], terms: List[str]) -> int:
+    """Weight a hit by field: title beats summary beats body (CDX-03)."""
+    title = str(item.get("title") or "").lower()
+    summary = str(item.get("summary") or "").lower()
+    body = str(item.get("body") or "").lower()
+    score = 0
+    for term in terms:
+        score += 8 if term in title else 0
+        score += 3 if term in summary else 0
+        score += 1 if term in body else 0
+    return score
+
+
+def _rank(items: List[Dict[str, Any]], terms: List[str], lang: str) -> List[Dict[str, Any]]:
+    """Score items in the given language; fall back to all languages if empty (CDX-04)."""
+
+    def scored(only_lang: Optional[str]) -> List[Any]:
+        out = []
+        for item in items:
+            if only_lang and f"/{only_lang}/" not in str(item.get("url") or ""):
+                continue
+            hit = _score(item, terms)
+            if hit > 0:
+                out.append((hit, item))
+        out.sort(key=lambda pair: pair[0], reverse=True)
+        return out
+
+    ranked = scored(lang) or scored(None)
+    return [item for _, item in ranked]
+
+
+class CodexSearch:
+    def __init__(self, config_getter: Callable[[], Dict[str, Any]]) -> None:
+        self._config = config_getter
+        self._cache: Optional[List[Dict[str, Any]]] = None
+        self._fetched_at = 0.0
+
+    def enabled(self) -> bool:
+        return bool(self._config().get("enable-codex", False))
+
+    def _index_url(self) -> str:
+        return str(self._config().get("codex-index-url", DEFAULT_INDEX_URL))
+
+    async def _load_index(self) -> List[Dict[str, Any]]:
+        """Fetch + cache the codex index, SSRF-guarded and size-bounded (CDX-02)."""
+        ttl = float(self._config().get("codex-cache-ttl", DEFAULT_TTL_S))
+        if self._cache is not None and (time.monotonic() - self._fetched_at) < ttl:
+            return self._cache
+        url = self._index_url()
+        reason = guard_url(url)
+        if reason:
+            raise ValueError(reason)
+        max_bytes = int(self._config().get("codex-max-bytes", DEFAULT_MAX_BYTES))
+        timeout = aiohttp.ClientTimeout(total=FETCH_TIMEOUT_S)
+        async with aiohttp.ClientSession(timeout=timeout, headers={"User-Agent": "FjerkroaBot-codex/1.0"}) as session:
+            async with session.get(url) as response:
+                response.raise_for_status()
+                raw = await read_capped(response, max_bytes)
+        data = json.loads(raw.decode("utf-8", "ignore"))
+        items = data.get("items", []) if isinstance(data, dict) else []
+        self._cache = [i for i in items if isinstance(i, dict)]
+        self._fetched_at = time.monotonic()
+        return self._cache
+
+    async def search(self, query: str, lang: str = "en", limit: int = DEFAULT_LIMIT) -> Dict[str, Any]:
+        """Return sanitized top matches, or an error dict — never raise (CDX-05)."""
+        try:
+            items = await self._load_index()
+        except Exception as err:
+            logging.warning(f"codex: index load failed: {err!r}")
+            return {"error": f"codex unavailable: {err}"}
+        terms = _tokenize(query)
+        if not terms:
+            return {"query": query, "results": []}
+        pick = (lang or "en").lower()
+        if pick not in _VALID_LANGS:
+            pick = "en"
+        summary_chars = int(self._config().get("codex-summary-chars", DEFAULT_SUMMARY_CHARS))
+        results = []
+        for item in _rank(items, terms, pick)[: max(1, limit)]:
+            results.append(
+                {
+                    "title": sanitize_external_text(str(item.get("title") or ""), 200),
+                    "summary": sanitize_external_text(str(item.get("summary") or ""), summary_chars),
+                    "collection": str(item.get("collection") or ""),
+                    "url": urljoin(self._index_url(), str(item.get("url") or "")),
+                }
+            )
+        return {"query": query, "lang": pick, "results": results}

fjerkroa_bot/discord_bot.py

@@ -3,9 +3,11 @@ import asyncio
 import logging
 import random
 import re
+import shutil
 import sys
 import time
 from collections import deque
+from pathlib import Path
 from typing import Optional, Union
 
 import discord
@@ -16,6 +18,7 @@ from watchdog.events import FileSystemEventHandler
 from watchdog.observers import Observer
 
 from .ai_responder import AIMessage
+from .monitor import HealthMonitor
 from .openai_responder import OpenAIResponder
 from .tasks import TaskEngine
 
@@ -130,6 +133,15 @@ class FjerkroaBot(commands.Bot):
             observe=self.airesponder.observe_event,
         )
         self.loop.create_task(self.task_loop())
+        # Proactive health monitoring -> staff alerts (OPS-18/19)
+        self.health_monitor = HealthMonitor(
+            config_getter=lambda: self.config,
+            ledger=self.airesponder.ledger,
+            store=self.airesponder.store,
+            disk_free_mb=self._disk_free_mb,
+            alert=self.send_staff_alert,
+        )
+        self.loop.create_task(self.monitor_loop())
         logging.info("Task engine initialised.")
 
     async def task_loop(self):
@@ -140,6 +152,20 @@ class FjerkroaBot(commands.Bot):
             except Exception as err:
                 logging.warning(f"task tick failed: {repr(err)}")
 
+    def _disk_free_mb(self) -> float:
+        directory = Path(self.config.get("history-directory", ".")).expanduser()
+        target = directory if directory.exists() else Path.home()
+        return shutil.disk_usage(target).free / (1024 * 1024)
+
+    async def monitor_loop(self):
+        while True:
+            await asyncio.sleep(int(self.config.get("monitor-interval", 300)))
+            if self.health_monitor.enabled():
+                try:
+                    await self.health_monitor.tick()
+                except Exception as err:
+                    logging.warning(f"monitor tick failed: {repr(err)}")
+
     async def _execute_task(self, channel_name: str, prompt: str) -> None:
         """Run a due task through the normal responder path (TSK-02)."""
         channel = self.channel_by_name(channel_name, getattr(self, "chat_channel", None), no_ignore=True)
@@ -182,6 +208,9 @@ class FjerkroaBot(commands.Bot):
         if content.startswith("!privacy"):
             await message.channel.send(self.config.get("privacy-notice", DEFAULT_PRIVACY_NOTICE), suppress_embeds=True)
             return
+        if content.startswith("!help"):  # OPS-17: context-aware, works even while paused
+            await message.channel.send(self._help_text(staff=self.is_staff_channel(message.channel)), suppress_embeds=True)
+            return
         if not self.replies_allowed():
             return
         if str(message.content).startswith("!wichtel"):
@@ -263,15 +292,35 @@ class FjerkroaBot(commands.Bot):
             return f"Cancelled {store.task_set_state(int(args[1]), 'cancelled')} task(s)."
         return None
 
+    def _help_text(self, staff: bool) -> str:
+        """Context-aware command help (OPS-17): every channel lists the user commands; the staff channel also lists operator commands."""
+        everywhere = (
+            "Available to everyone, in any channel:\n"
+            "• `!help` — this help\n"
+            "• `!forgetme` — delete your messages and memory traces (works even while I'm paused)\n"
+            "• `!privacy` — how your data is handled (works even while I'm paused)\n"
+            "• `!wichtel @a @b @c …` — draw Secret Santa pairings (needs ≥2 mentions; only while I'm active)"
+        )
+        if not staff:
+            return everywhere
+        operator = (
+            "Staff commands — this channel only, prefixed `!bot`:\n"
+            "• Control: `pause`, `resume`, `quiet <minutes>`, `status`\n"
+            "• Cost: `spend`, `images on|off`\n"
+            "• Memory: `memory <user>`, `forget-fact <id>`, `pin <channel|global> <text>`, `unpin <id>`, `pins`\n"
+            "• Tasks: `tasks` (list), `tasks on|off`, `task-approve <id>`, `task-cancel <id>`"
+        )
+        return operator + "\n\n" + everywhere
+
     async def handle_staff_command(self, message: Message) -> None:
-        """Operator kill-switches, staff channel only (OPS-01..05, OPS-09, MEM-07)."""
+        """Operator kill-switches, staff channel only (OPS-01..05, OPS-09, OPS-17, MEM-07)."""
         args = str(message.content).split()[1:]
         for handler in (self._memory_command, self._task_command):
             reply = handler(args)
             if reply is not None:
                 await message.channel.send(reply, suppress_embeds=True)
                 return
-        reply = "Commands: pause, resume, images on|off, tasks on|off, quiet <minutes>, status, spend, memory <user>, forget-fact <id>, pin <channel|global> <fact>, unpin <id>"
+        reply = self._help_text(staff=True)  # OPS-17: unknown/`help` -> full grouped help
         if args[:1] == ["pause"]:
             self.replies_enabled = False
             reply = "Replies paused."

fjerkroa_bot/monitor.py

@@ -0,0 +1,82 @@
+"""Proactive health monitoring -> staff alerts (SPEC-012, FDB-012).
+
+A periodic check that watches daily spend against the budget, free disk,
+and task-queue depth, and posts a staff alert when a threshold is crossed
+— once per crossing, re-arming when the metric recovers, so a persistent
+condition never spams. Opt-in per deployment (`enable-monitoring`); it
+reuses the rate-limited staff-alert channel (OPS-07).
+"""
+
+import logging
+from typing import Any, Callable, Dict, Optional, Tuple
+
+# A check returns (metric-name, is-over-threshold, alert-message) or None when not applicable.
+Check = Optional[Tuple[str, bool, str]]
+
+
+class HealthMonitor:
+    def __init__(
+        self,
+        config_getter: Callable[[], Dict[str, Any]],
+        ledger: Any,
+        store: Any,
+        disk_free_mb: Callable[[], float],
+        alert: Callable[[str], Any],
+    ) -> None:
+        self._config = config_getter
+        self._ledger = ledger
+        self._store = store
+        self._disk_free_mb = disk_free_mb
+        self._alert = alert
+        self._armed: Dict[str, bool] = {}
+
+    def enabled(self) -> bool:
+        return bool(self._config().get("enable-monitoring", False))
+
+    def _check_spend(self) -> Check:
+        config = self._config()
+        if "daily-budget-usd" not in config:
+            return None
+        budget = float(config["daily-budget-usd"])
+        if budget <= 0:
+            return None
+        spent = float(self._ledger.spent_usd())
+        frac = spent / budget
+        threshold = float(config.get("monitor-spend-alert-frac", 0.8))
+        return ("spend", frac >= threshold, f"💸 Spend at ${spent:.2f} of ${budget:.2f} today ({frac:.0%}, alert ≥ {threshold:.0%}).")
+
+    def _check_disk(self) -> Check:
+        try:
+            free = float(self._disk_free_mb())
+        except Exception as err:
+            logging.debug(f"monitor: disk check failed: {err!r}")
+            return None
+        min_mb = float(self._config().get("monitor-disk-min-mb", 500))
+        return ("disk", free < min_mb, f"💾 Low disk: {free:.0f} MB free (alert < {min_mb:.0f} MB).")
+
+    def _check_queue(self) -> Check:
+        if self._store is None:
+            return None
+        try:
+            depth = len(self._store.tasks_open())
+        except Exception as err:
+            logging.debug(f"monitor: queue check failed: {err!r}")
+            return None
+        limit = int(self._config().get("monitor-taskqueue-max", 20))
+        return ("task-queue", depth >= limit, f"🗒️ Task queue deep: {depth} open (alert ≥ {limit}).")
+
+    async def tick(self) -> None:
+        """Evaluate every check; alert on a rising edge only (OPS-18/19)."""
+        for check in (self._check_spend(), self._check_disk(), self._check_queue()):
+            if check is None:
+                continue
+            metric, over, message = check
+            await self._fire(metric, over, message)
+
+    async def _fire(self, metric: str, over: bool, message: str) -> None:
+        was_over = self._armed.get(metric, False)
+        if over and not was_over:
+            self._armed[metric] = True
+            await self._alert(message)
+        elif not over and was_over:
+            self._armed[metric] = False  # recovered — re-arm silently for the next crossing

fjerkroa_bot/news.py

@@ -11,8 +11,10 @@ CLI: python -m fjerkroa_bot.news --config kroa.toml
 
 import argparse
 import logging
+import re
 import sys
 import time
+from html import unescape
 from typing import Any, Dict, List, Optional, Tuple
 
 import defusedxml.ElementTree as ElementTree  # hardened XML: feeds are untrusted (XXE/billion-laughs)
@@ -21,44 +23,68 @@ from .ai_responder import sanitize_external_text
 
 DEFAULT_PER_FEED = 3
 DEFAULT_MAX_ITEMS = 15
+DEFAULT_SUMMARY_CHARS = 200
+DEFAULT_NEWS_KEEP = 400
 FETCH_TIMEOUT_S = 15
 _ATOM = "{http://www.w3.org/2005/Atom}"
+_RSS1 = "{http://purl.org/rss/1.0/}"  # RSS 1.0 / RDF (e.g. 4gamer.net) namespaces <item>/<title>/<link>
+_TAG_RE = re.compile(r"<[^>]+>")
+
+
+def _clean_summary(raw: str, max_len: int = 300) -> str:
+    """Strip HTML, unescape entities, collapse whitespace (feed descriptions are often HTML)."""
+    text = unescape(_TAG_RE.sub(" ", raw or ""))
+    return re.sub(r"\s+", " ", text).strip()[:max_len]
+
+
+def _rss_items(root: Any, ns: str, source: str) -> List[Dict[str, str]]:
+    """RSS 2.0 (ns='') and RSS 1.0/RDF (ns=_RSS1) both use <item><title><link><description>."""
+    out: List[Dict[str, str]] = []
+    for item in root.iter(f"{ns}item"):
+        title = (item.findtext(f"{ns}title") or "").strip()
+        link = (item.findtext(f"{ns}link") or "").strip()
+        summary = _clean_summary(item.findtext(f"{ns}description") or "")
+        if title:
+            out.append({"title": title, "link": link, "source": source, "summary": summary})
+    return out
 
 
 def parse_feed(data: bytes, source: str = "") -> List[Dict[str, str]]:
-    """Parse RSS or Atom bytes into [{title, link, source}] (tolerant)."""
+    """Parse RSS 2.0, RSS 1.0/RDF, or Atom bytes into [{title, link, source, summary}] (tolerant)."""
     try:
         root = ElementTree.fromstring(data)
     except Exception as err:
         # malformed XML or a blocked entity/DTD attack — tolerate, never raise (NEWS-01)
         logging.warning(f"news: unparseable/unsafe feed {source!r}: {err!r}")
         return []
-    items: List[Dict[str, str]] = []
+    # RSS 2.0 (unqualified) + RSS 1.0/RDF (namespaced, e.g. 4gamer) share <item><title><link><description>
-    # RSS: <rss><channel><item><title/><link/>
+    items: List[Dict[str, str]] = _rss_items(root, "", source) + _rss_items(root, _RSS1, source)
-    for item in root.iter("item"):
+    # Atom: <feed><entry><title/><link href=/><summary|content/>
-        title = (item.findtext("title") or "").strip()
-        link = (item.findtext("link") or "").strip()
-        if title:
-            items.append({"title": title, "link": link, "source": source})
-    # Atom: <feed><entry><title/><link href=/>
     for entry in root.iter(f"{_ATOM}entry"):
         title = (entry.findtext(f"{_ATOM}title") or "").strip()
         link_el = entry.find(f"{_ATOM}link")
         link = link_el.get("href", "") if link_el is not None else ""
+        summary = _clean_summary(entry.findtext(f"{_ATOM}summary") or entry.findtext(f"{_ATOM}content") or "")
         if title:
-            items.append({"title": title, "link": link, "source": source})
+            items.append({"title": title, "link": link, "source": source, "summary": summary})
     return items
 
 
-def render_digest(items: List[Dict[str, str]], max_items: int = DEFAULT_MAX_ITEMS) -> str:
+def render_digest(items: List[Dict[str, str]], max_items: int = DEFAULT_MAX_ITEMS, summary_chars: int = DEFAULT_SUMMARY_CHARS) -> str:
-    """Compact sanitized digest for the {news} prompt slot."""
+    """Compact sanitized digest for the {news} prompt slot (title + short summary + link)."""
     lines = []
     for item in items[:max_items]:
         title = sanitize_external_text(item["title"], 200)
         source = item.get("source", "")
         link = item.get("link", "")
+        summary = sanitize_external_text(item.get("summary", ""), summary_chars) if summary_chars else ""
         prefix = f"[{source}] " if source else ""
-        lines.append(f"- {prefix}{title}" + (f" ({link})" if link else ""))
+        line = f"- {prefix}{title}"
+        if summary:
+            line += f" — {summary}"
+        if link:
+            line += f" ({link})"
+        lines.append(line)
     return "\n".join(lines)
 
 
@@ -90,6 +116,180 @@ class NewsFetcher:
         return out
 
 
+DEFAULT_SEEN_CAP = 5000
+DEFAULT_POST_PER_FEED = 5
+DEFAULT_POST_MAX_PER_RUN = 8
+
+
+def item_key(item: Dict[str, str]) -> str:
+    return item.get("link") or item.get("title") or ""
+
+
+class NewsPoster:
+    """Post NEW feed items to Discord channel webhooks (ggg model, SPEC-013 NEWS-04..06)."""
+
+    def __init__(self, guard, fetch_bytes, post_webhook, store: Any = None) -> None:
+        self._guard = guard
+        self._fetch_bytes = fetch_bytes
+        self._post_webhook = post_webhook
+        self._store = store
+
+    async def run_post(
+        self,
+        feeds: List[Tuple[str, str, str]],
+        webhooks: Dict[str, str],
+        seen: set,
+        per_feed: int,
+        max_per_run: int,
+        seed_only: bool,
+        keep: int = DEFAULT_NEWS_KEEP,
+    ) -> Tuple[int, set]:
+        """Returns (posted_count, updated_seen). seed_only marks new items seen without posting."""
+        posted = 0
+        harvested: List[Dict[str, str]] = []
+        for url, label, channel in feeds:
+            reason = self._guard(url)
+            if reason:
+                logging.warning(f"news-post: skipping feed {label} — {reason}")
+                continue
+            try:
+                data = await self._fetch_bytes(url)
+            except Exception as err:
+                logging.warning(f"news-post: fetch failed for {label}: {repr(err)}")
+                continue
+            for item in parse_feed(data, label)[:per_feed]:
+                harvested.append(item)  # NEWS-09: everything parsed feeds the searchable store
+                key = item_key(item)
+                if not key or key in seen:
+                    continue
+                seen.add(key)
+                may_post = not seed_only and posted < max_per_run
+                if may_post and await self._deliver(item, label, channel, webhooks):
+                    posted += 1
+        if self._store is not None and harvested:
+            self._store.add_news_items(harvested)
+            self._store.prune_news(keep)
+        return posted, seen
+
+    async def _deliver(self, item: Dict[str, str], label: str, channel: str, webhooks: Dict[str, str]) -> bool:
+        hook = webhooks.get(channel)
+        if not hook:
+            logging.warning(f"news-post: no webhook for channel {channel!r} ({label})")
+            return False
+        title = sanitize_external_text(item["title"], 300)
+        link = item.get("link", "")
+        content = f"**[{label}]** {title}" + (f"\n{link}" if link else "")
+        try:
+            await self._post_webhook(hook, content)
+            return True
+        except Exception as err:
+            logging.warning(f"news-post: webhook post failed ({label}): {repr(err)}")
+            return False
+
+
+# --- news memory + on-demand retrieval tool (SPEC-013 NEWS-09..12) ---
+
+GET_NEWS_TOOL = {
+    "name": "get_news",
+    "description": "Fetch news the bot has collected from its RSS feeds — this is the SAME news that gets posted in the "
+    "server's news channels (e.g. #news, #newsjp / ニュース). Use this FIRST, before web_search, for anything about "
+    "current news or about something someone saw in a news channel; filter by topic (a keyword, also matches the source "
+    "label) or by source. Returns headlines with a short summary and a link; follow up with fetch_url for the full text.",
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "topic": {"type": "string", "description": "Optional keywords to filter by, e.g. 'Nordland', 'football', 'weather'."},
+            "source": {"type": "string", "description": "Optional source label, e.g. 'NRK', 'Aftenposten', 'Verden', 'Sport'."},
+            "limit": {"type": "integer", "description": "How many items to return (default 10, max 30)."},
+        },
+        "required": [],
+    },
+}
+
+
+def _news_terms(topic: Optional[str]) -> List[str]:
+    return [t for t in re.split(r"\W+", (topic or "").lower()) if len(t) > 1][:5]
+
+
+def query_news(
+    store: Any, topic: Optional[str] = None, source: Optional[str] = None, limit: int = 10, summary_chars: int = DEFAULT_SUMMARY_CHARS
+) -> Dict[str, Any]:
+    """Retrieve stored news for the get_news tool: term/source-filtered, sanitized (NEWS-11)."""
+    if store is None:
+        return {"error": "news store unavailable"}
+    limit = max(1, min(int(limit or 10), 30))
+    src = (str(source).strip() or None) if source else None
+    terms = _news_terms(topic)
+    try:
+        rows = store.search_news(terms, limit, src) if terms else store.recent_news(limit, src)
+    except Exception as err:
+        logging.warning(f"news: query failed: {err!r}")
+        return {"error": "news lookup failed"}
+    results = [
+        {
+            "source": row.get("source", ""),
+            "title": sanitize_external_text(row.get("title", ""), 200),
+            "summary": sanitize_external_text(row.get("summary", ""), summary_chars),
+            "link": row.get("link", ""),
+        }
+        for row in rows
+    ]
+    return {"topic": topic or "", "source": src or "", "results": results}
+
+
+def _open_store(config: Dict[str, Any]) -> Any:
+    directory = config.get("history-directory")
+    if not directory:
+        return None
+    from pathlib import Path
+
+    from .persistence import PersistentStore
+
+    return PersistentStore(Path(str(directory)).expanduser() / "bot.db")
+
+
+def persist_news(config: Dict[str, Any], items: List[Dict[str, str]]) -> int:
+    """Upsert fetched items into the news store, prune to the rolling window (NEWS-09)."""
+    store = _open_store(config)
+    if store is None or not items:
+        return 0
+    added = store.add_news_items(items)
+    store.prune_news(int(config.get("news-keep", DEFAULT_NEWS_KEEP)))
+    return added
+
+
+def load_seen(path: str) -> Tuple[set, bool]:
+    """(seen-set, existed). Missing/broken state -> empty set, existed=False (seed run)."""
+    import json
+    import os
+
+    if not os.path.exists(path):
+        return set(), False
+    try:
+        with open(path, encoding="utf-8") as fd:
+            return set(json.load(fd)), True
+    except Exception as err:
+        logging.warning(f"news-post: unreadable state {path}: {err!r} — reseeding")
+        return set(), False
+
+
+def save_seen(path: str, seen: set, cap: int = DEFAULT_SEEN_CAP) -> None:
+    import json
+
+    # keep the newest `cap` keys (insertion order preserved by Python sets? no — use a bounded slice)
+    keys = list(seen)[-cap:]
+    with open(path, "w", encoding="utf-8") as fd:
+        json.dump(keys, fd)
+
+
+def _post_feeds_from_config(config: Dict[str, Any]) -> List[Tuple[str, str, str]]:
+    feeds = []
+    for entry in config.get("news-post-feeds", []):
+        if isinstance(entry, (list, tuple)) and len(entry) >= 3:
+            feeds.append((str(entry[0]), str(entry[1]), str(entry[2])))
+    return feeds
+
+
 def _feeds_from_config(config: Dict[str, Any]) -> List[Tuple[str, str]]:
     """news-feeds = [["url", "label"], ...] or ["url", ...]."""
     feeds = []
@@ -113,6 +313,43 @@ async def _aiohttp_fetch(url: str) -> bytes:
             return await read_capped(response, 4 * 1024 * 1024)
 
 
+async def _aiohttp_post(hook: str, content: str) -> None:
+    import aiohttp
+
+    timeout = aiohttp.ClientTimeout(total=FETCH_TIMEOUT_S)
+    async with aiohttp.ClientSession(timeout=timeout) as session:
+        # allowed_mentions none: a headline can never ping the channel (SAF-02 spirit)
+        payload = {"content": content[:2000], "allowed_mentions": {"parse": []}}
+        async with session.post(hook, json=payload) as response:
+            response.raise_for_status()
+
+
+async def run_post(config: Dict[str, Any]) -> int:
+    """Webhook-posting mode (ggg): post new items to channels. Returns posted count."""
+    from .url_reader import guard_url
+
+    webhooks = dict(config.get("news-post-webhooks", {}))
+    feeds = _post_feeds_from_config(config)
+    state_path = config.get("news-post-state", "news_state.json")
+    if not webhooks or not feeds:
+        logging.error("news-post: need news-post-webhooks and news-post-feeds")
+        return 0
+    seen, existed = load_seen(state_path)
+    poster = NewsPoster(guard_url, _aiohttp_fetch, _aiohttp_post, store=_open_store(config))
+    posted, seen = await poster.run_post(
+        feeds,
+        webhooks,
+        seen,
+        int(config.get("news-post-per-feed", DEFAULT_POST_PER_FEED)),
+        int(config.get("news-post-max-per-run", DEFAULT_POST_MAX_PER_RUN)),
+        seed_only=not existed,  # first run seeds without flooding the channels
+        keep=int(config.get("news-keep", DEFAULT_NEWS_KEEP)),
+    )
+    save_seen(state_path, seen, int(config.get("news-post-seen-cap", DEFAULT_SEEN_CAP)))
+    logging.info(f"news-post: posted {posted} item(s)" + (" (seed run — nothing posted)" if not existed else ""))
+    return posted
+
+
 async def run(config: Dict[str, Any]) -> Optional[str]:
     from .url_reader import guard_url
 
@@ -126,7 +363,10 @@ async def run(config: Dict[str, Any]) -> Optional[str]:
         return None
     fetcher = NewsFetcher(guard_url, _aiohttp_fetch)
     items = await fetcher.collect(feeds, int(config.get("news-per-feed", DEFAULT_PER_FEED)))
-    digest = render_digest(items, int(config.get("news-max-items", DEFAULT_MAX_ITEMS)))
+    persist_news(config, items)  # NEWS-09: feed the searchable rolling store for get_news
+    digest = render_digest(
+        items, int(config.get("news-max-items", DEFAULT_MAX_ITEMS)), int(config.get("news-summary-chars", DEFAULT_SUMMARY_CHARS))
+    )
     header = f"News as of {time.strftime('%Y-%m-%d %H:%M UTC', time.gmtime())}:\n"
     with open(out_path, "w", encoding="utf-8") as fd:
         fd.write(header + digest + "\n")
@@ -140,11 +380,17 @@ def main() -> int:
     import tomlkit
 
     logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
-    parser = argparse.ArgumentParser(description="Fetch RSS/Atom feeds into the {news} digest file")
+    parser = argparse.ArgumentParser(
+        description="Fetch RSS/Atom feeds: --post to channel webhooks (ggg) or default {news} digest file (kroa)"
+    )
     parser.add_argument("--config", required=True)
+    parser.add_argument("--post", action="store_true", help="webhook-posting mode (post new items to Discord channels)")
     args = parser.parse_args()
     with open(args.config, encoding="utf-8") as fd:
         config = tomlkit.load(fd)
+    if args.post:
+        asyncio.run(run_post(config))
+        return 0
     result = asyncio.run(run(config))
     return 0 if result else 1
 

fjerkroa_bot/openai_responder.py

@@ -9,10 +9,16 @@ from typing import Any, Dict, List, Optional, Tuple
 import openai
 
 from .ai_responder import AIResponder, exponential_backoff, sanitize_external_text
+from .codex import CODEX_SEARCH_TOOL
+from .codex import DEFAULT_LIMIT as CODEX_DEFAULT_LIMIT
+from .codex import CodexSearch
 from .igdblib import IGDBQuery
 from .leonardo_draw import LeonardoAIDrawMixIn
+from .news import GET_NEWS_TOOL, query_news
 from .quota import QuotaLedger
 from .url_reader import FETCH_URL_TOOL, URLReader
+from .websearch import DEFAULT_RESULTS as WEB_DEFAULT_RESULTS
+from .websearch import WEB_SEARCH_TOOL, WebSearch
 
 # The response envelope, enforced server-side via structured outputs
 # (ENV-19). All fields required, closed object, nullable where the
@@ -160,6 +166,10 @@ class OpenAIResponder(AIResponder, LeonardoAIDrawMixIn):
 
         # URL reading tool (SPEC-011); shares the image cache for page images
         self.url_reader = URLReader(lambda: self.config, self.image_cache)
+        # Codex Mechanicus search (SPEC-014); Luma's own archive at binaric.tech
+        self.codex = CodexSearch(lambda: self.config)
+        # Web search (SPEC-015) via Exa; general "look it up" beyond fetch_url/news/codex
+        self.web_search = WebSearch(lambda: self.config)
 
     def _available_tools(self) -> List[Dict[str, Any]]:
         """Assemble the function-tool list from every enabled provider (URL-01)."""
@@ -173,16 +183,42 @@ class OpenAIResponder(AIResponder, LeonardoAIDrawMixIn):
                 logging.warning(f"Error setting up IGDB functions: {err}")
         if self.url_reader.enabled():
             functions.append(FETCH_URL_TOOL)
+        if self.codex.enabled():  # CDX-01
+            functions.append(CODEX_SEARCH_TOOL)
+        if self.config.get("enable-news-tool", False) and self.store is not None:  # NEWS-10
+            functions.append(GET_NEWS_TOOL)
+        if self.web_search.enabled():  # WEB-01
+            functions.append(WEB_SEARCH_TOOL)
         return functions
 
     async def _dispatch_tool(self, name: str, args: Dict[str, Any], author: str) -> Any:
-        """Route a tool call to its provider (IGDB or URL reader)."""
+        """Route a tool call to its provider (IGDB, URL reader, or codex)."""
         if name == "fetch_url":
             per_user_cap = int(self.config.get("url-daily-per-user", 20))
             if self.ledger._get(f"url-fetch:{author}") >= per_user_cap:  # URL-07
                 return {"error": "daily URL fetch limit reached"}
             self.ledger._add(f"url-fetch:{author}", 1)
             return await self.url_reader.fetch(str(args.get("url", "")), self.channel, author or "user")
+        if name == "codex_search":
+            per_user_cap = int(self.config.get("codex-daily-per-user", 50))
+            if self.ledger._get(f"codex:{author}") >= per_user_cap:  # CDX-06
+                return {"error": "daily codex search limit reached"}
+            self.ledger._add(f"codex:{author}", 1)
+            limit = int(self.config.get("codex-limit", CODEX_DEFAULT_LIMIT))
+            return await self.codex.search(str(args.get("query", "")), str(args.get("lang", "en")), limit)
+        if name == "get_news":
+            per_user_cap = int(self.config.get("news-daily-per-user", 30))
+            if self.ledger._get(f"news:{author}") >= per_user_cap:  # NEWS-12
+                return {"error": "daily news lookup limit reached"}
+            self.ledger._add(f"news:{author}", 1)
+            summary_chars = int(self.config.get("news-summary-chars", 200))
+            return query_news(self.store, args.get("topic"), args.get("source"), args.get("limit", 10), summary_chars)
+        if name == "web_search":
+            per_user_cap = int(self.config.get("web-daily-per-user", 30))
+            if self.ledger._get(f"web:{author}") >= per_user_cap:  # WEB-05
+                return {"error": "daily web search limit reached"}
+            self.ledger._add(f"web:{author}", 1)
+            return await self.web_search.search(str(args.get("query", "")), int(args.get("num_results", WEB_DEFAULT_RESULTS)))
         return await self._execute_igdb_function(name, args)
 
     async def draw_openai(self, description: str, count: int = 1) -> List[BytesIO]:

fjerkroa_bot/persistence.py

@@ -13,7 +13,7 @@ from contextlib import closing
 from pathlib import Path
 from typing import Any, Dict, List, Optional
 
-SCHEMA_VERSION = 5
+SCHEMA_VERSION = 6
 
 
 class PersistentStore:
@@ -72,6 +72,13 @@ class PersistentStore:
                     " due_at TEXT NOT NULL, payload TEXT NOT NULL, state TEXT NOT NULL DEFAULT 'queued',"
                     " created_at TEXT NOT NULL DEFAULT (datetime('now')), executed_at TEXT)"
                 )
+            if version < 6:
+                # News memory (SPEC-013 NEWS-09): deduped rolling store of fetched items
+                conn.execute(
+                    "CREATE TABLE IF NOT EXISTS news (id INTEGER PRIMARY KEY, dedup_key TEXT UNIQUE NOT NULL,"
+                    " source TEXT NOT NULL DEFAULT '', title TEXT NOT NULL, link TEXT NOT NULL DEFAULT '',"
+                    " summary TEXT NOT NULL DEFAULT '', first_seen TEXT NOT NULL DEFAULT (datetime('now')))"
+                )
             if version < SCHEMA_VERSION:
                 conn.execute(f"PRAGMA user_version = {SCHEMA_VERSION}")
         os.chmod(self.db_path, 0o600)  # conversation data (PER-04)
@@ -115,6 +122,64 @@ class PersistentStore:
             row = conn.execute("SELECT value FROM usage WHERE day = ? AND key = ?", (day, key)).fetchone()
         return float(row[0]) if row else 0.0
 
+    # --- news memory (SPEC-013 NEWS-09..12) ---
+
+    def add_news_items(self, items: List[Dict[str, Any]]) -> int:
+        """Insert deduped news rows (by link or title); returns how many were new (NEWS-09)."""
+        added = 0
+        with closing(self._connect()) as conn, conn:
+            for item in items:
+                title = str(item.get("title") or "").strip()
+                key = (str(item.get("link") or "").strip()) or title
+                if not title or not key:
+                    continue
+                cursor = conn.execute(
+                    "INSERT OR IGNORE INTO news (dedup_key, source, title, link, summary) VALUES (?, ?, ?, ?, ?)",
+                    (key, str(item.get("source") or ""), title, str(item.get("link") or ""), str(item.get("summary") or "")),
+                )
+                added += cursor.rowcount
+        return added
+
+    def recent_news(self, limit: int = 20, source: Optional[str] = None) -> List[Dict[str, Any]]:
+        sql = "SELECT source, title, link, summary FROM news"
+        params: List[Any] = []
+        if source:
+            sql += " WHERE source = ?"
+            params.append(source)
+        sql += " ORDER BY id DESC LIMIT ?"
+        params.append(int(limit))
+        with closing(self._connect()) as conn:
+            rows = conn.execute(sql, params).fetchall()
+        return [{"source": r[0], "title": r[1], "link": r[2], "summary": r[3]} for r in rows]
+
+    def search_news(self, terms: List[str], limit: int = 20, source: Optional[str] = None) -> List[Dict[str, Any]]:
+        """Rows where every term appears in title or summary; optional source filter (NEWS-11)."""
+        params: List[Any] = []
+        clauses = []
+        for term in terms:
+            clauses.append("(title LIKE ? OR summary LIKE ? OR source LIKE ?)")
+            like = f"%{term}%"
+            params += [like, like, like]
+        where = " AND ".join(clauses) if clauses else "1=1"
+        if source:
+            where = f"({where}) AND source = ?"
+            params.append(source)
+        params.append(int(limit))
+        sql = f"SELECT source, title, link, summary FROM news WHERE {where} ORDER BY id DESC LIMIT ?"  # nosec B608 - fixed templates; values parameterised
+        with closing(self._connect()) as conn:
+            rows = conn.execute(sql, params).fetchall()
+        return [{"source": r[0], "title": r[1], "link": r[2], "summary": r[3]} for r in rows]
+
+    def prune_news(self, keep: int) -> int:
+        """Keep the newest `keep` rows, delete the rest (rolling window, NEWS-09)."""
+        with closing(self._connect()) as conn, conn:
+            cursor = conn.execute("DELETE FROM news WHERE id NOT IN (SELECT id FROM news ORDER BY id DESC LIMIT ?)", (int(keep),))
+        return cursor.rowcount
+
+    def news_count(self) -> int:
+        with closing(self._connect()) as conn:
+            return int(conn.execute("SELECT COUNT(*) FROM news").fetchone()[0])
+
     def delete_history_of_user(self, user: str) -> int:
         """Remove persisted rows carrying this user's messages (SAF-08)."""
         with closing(self._connect()) as conn, conn:

fjerkroa_bot/url_reader.py

@@ -38,6 +38,9 @@ FETCH_URL_TOOL = {
 }
 
 
+_META_REFRESH_URL = re.compile(r"url\s*=\s*['\"]?([^'\";\s]+)", re.I)
+
+
 class _Extractor(HTMLParser):
     def __init__(self) -> None:
         super().__init__()
@@ -45,6 +48,7 @@ class _Extractor(HTMLParser):
         self.parts: List[str] = []
         self.images: List[str] = []
         self.og_image: Optional[str] = None
+        self.refresh_url: Optional[str] = None
 
     def handle_starttag(self, tag: str, attrs) -> None:
         if tag in ("script", "style", "noscript", "svg"):
@@ -55,6 +59,12 @@ class _Extractor(HTMLParser):
             self.images.append(src)
         if tag == "meta" and attr.get("property") == "og:image" and attr.get("content"):
             self.og_image = attr["content"]
+        # meta-refresh redirect (link shorteners, getnews stubs) — URL-04
+        content = attr.get("content")
+        if tag == "meta" and (attr.get("http-equiv") or "").lower() == "refresh" and content:
+            match = _META_REFRESH_URL.search(content)
+            if match and self.refresh_url is None:
+                self.refresh_url = match.group(1)
 
     def handle_endtag(self, tag: str) -> None:
         if tag in ("script", "style", "noscript", "svg") and self._skip > 0:
@@ -126,29 +136,38 @@ class URLReader:
         try:
             async with aiohttp.ClientSession(timeout=timeout, headers={"User-Agent": "FjerkroaBot/1.0"}) as session:
                 final_url, body = await self._get(session, url, max_bytes)
+                # follow a meta-refresh redirect (link shorteners / getnews stubs), re-guarded — URL-04
+                for _ in range(2):
+                    extractor = self._extract(body.decode("utf-8", "ignore"))
+                    if not extractor.refresh_url:
+                        break
+                    target = urljoin(final_url, extractor.refresh_url)
+                    if guard_url(target) is not None or target == final_url:
+                        break
+                    logging.info(f"url reader: following meta-refresh -> {target}")
+                    final_url, body = await self._get(session, target, max_bytes)
         except Exception as err:
             return {"error": str(err)}
-        text = self._to_text(body.decode("utf-8", "ignore"))
+        html = body.decode("utf-8", "ignore")
-        clean = sanitize_external_text(text, int(config.get("url-max-chars", DEFAULT_MAX_CHARS)))
+        clean = sanitize_external_text(self._to_text(html), int(config.get("url-max-chars", DEFAULT_MAX_CHARS)))
-        images = await self._ingest_images(body.decode("utf-8", "ignore"), final_url, channel, user)
+        images = await self._ingest_images(html, final_url, channel, user)
         return {"url": final_url, "text": clean, "images_cached": images}
 
-    def _to_text(self, html: str) -> str:
+    def _extract(self, html: str) -> "_Extractor":
         extractor = _Extractor()
         try:
             extractor.feed(html)
         except Exception as err:
-            logging.debug(f"html parse (text) failed: {err!r}")
+            logging.debug(f"html parse failed: {err!r}")
-        return re.sub(r"\s+\n", "\n", " ".join(extractor.parts))
+        return extractor
+
+    def _to_text(self, html: str) -> str:
+        return re.sub(r"\s+\n", "\n", " ".join(self._extract(html).parts))
 
     async def _ingest_images(self, html: str, base_url: str, channel: str, user: str) -> int:
         if self.image_cache is None:
             return 0
-        extractor = _Extractor()
+        extractor = self._extract(html)
-        try:
-            extractor.feed(html)
-        except Exception as err:
-            logging.debug(f"html parse (images) failed: {err!r}")
         candidates = ([extractor.og_image] if extractor.og_image else []) + extractor.images
         limit = int(self._config().get("url-max-images", DEFAULT_MAX_IMAGES))
         cached = 0

fjerkroa_bot/websearch.py

@@ -0,0 +1,94 @@
+"""Web search tool via Exa (SPEC-015, FDB-022).
+
+A `web_search` function tool: the model looks things up on the open web
+when a general "look it up" question is not covered by IGDB, the codex,
+the news store, or a URL the user pasted. Results are external text, so
+titles and snippets are sanitized (SAF-03) before they reach the prompt.
+The Exa API key lives in host config (or the `EXA_API_KEY` env), never in
+the repo.
+"""
+
+import logging
+import os
+from typing import Any, Callable, Dict, List
+
+import aiohttp
+
+from .ai_responder import sanitize_external_text
+
+EXA_SEARCH_URL = "https://api.exa.ai/search"
+DEFAULT_RESULTS = 5
+MAX_RESULTS = 10
+DEFAULT_SNIPPET_CHARS = 400
+FETCH_TIMEOUT_S = 15
+
+WEB_SEARCH_TOOL = {
+    "name": "web_search",
+    "description": "Search the open web for general information. Use ONLY when the answer is not in your own sources: for "
+    "the server's news use get_news, for Adeptus Mechanicus / Warhammer 40k lore use codex_search, for video-game facts use "
+    "the game tools, for a specific URL someone pasted use fetch_url. Returns result titles, URLs, and a short snippet; "
+    "follow up with fetch_url on a result link for the full article.",
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "query": {"type": "string", "description": "What to search the web for."},
+            "num_results": {"type": "integer", "description": "How many results to return (default 5, max 10)."},
+        },
+        "required": ["query"],
+    },
+}
+
+
+def _format_results(data: Any, snippet_chars: int) -> List[Dict[str, str]]:
+    """Reduce an Exa response to sanitized {title, url, snippet, published} rows (WEB-02)."""
+    results = data.get("results", []) if isinstance(data, dict) else []
+    out: List[Dict[str, str]] = []
+    for item in results:
+        if not isinstance(item, dict):
+            continue
+        out.append(
+            {
+                "title": sanitize_external_text(str(item.get("title") or ""), 200),
+                "url": str(item.get("url") or ""),
+                "snippet": sanitize_external_text(str(item.get("text") or item.get("snippet") or ""), snippet_chars),
+                "published": str(item.get("publishedDate") or ""),
+            }
+        )
+    return out
+
+
+class WebSearch:
+    def __init__(self, config_getter: Callable[[], Dict[str, Any]]) -> None:
+        self._config = config_getter
+
+    def _api_key(self) -> str:
+        return str(self._config().get("exa-api-key") or os.environ.get("EXA_API_KEY", ""))
+
+    def enabled(self) -> bool:
+        return bool(self._config().get("enable-web-search", False)) and bool(self._api_key())
+
+    async def _post(self, payload: Dict[str, Any], headers: Dict[str, str]) -> Any:
+        timeout = aiohttp.ClientTimeout(total=FETCH_TIMEOUT_S)
+        async with aiohttp.ClientSession(timeout=timeout) as session:
+            async with session.post(EXA_SEARCH_URL, json=payload, headers=headers) as response:
+                response.raise_for_status()
+                return await response.json()
+
+    async def search(self, query: str, num_results: int = DEFAULT_RESULTS) -> Dict[str, Any]:
+        """Return sanitized web results, or an error dict — never raise (WEB-04)."""
+        key = self._api_key()
+        if not key:
+            return {"error": "web search unavailable: no api key"}
+        query = (query or "").strip()
+        if not query:
+            return {"query": "", "results": []}
+        num = max(1, min(int(num_results or DEFAULT_RESULTS), MAX_RESULTS))  # WEB-03
+        snippet_chars = int(self._config().get("web-snippet-chars", DEFAULT_SNIPPET_CHARS))
+        payload = {"query": query, "numResults": num, "type": "auto", "contents": {"text": {"maxCharacters": max(snippet_chars, 200)}}}
+        headers = {"x-api-key": key, "Content-Type": "application/json"}
+        try:
+            data = await self._post(payload, headers)
+        except Exception as err:
+            logging.warning(f"web search failed: {err!r}")
+            return {"error": "web search failed"}
+        return {"query": query, "results": _format_results(data, snippet_chars)}

manual-verification.md

@@ -13,3 +13,4 @@ with date + result.
 | DEP-05 | 2026-07-13 | Live-verified: kroa deploy attempt ~15h Oslo refused without DEPLOY_FORCE=1. |
 | DEP-06 | 2026-07-13 | Rollback documented (older tag + db backup restore); live drill pending — next release. |
 | OPS-15 | 2026-07-13 | Backup cron installed on both hosts (daily 03:17 UTC → ~/backups/<bot>/, keep 14); first snapshots written + verified 0600 (kroa 10965 B, luma 25871 B). |
+| CDX-07 | 2026-07-13 | Pending live verify on ggg after v3.8.0 deploy: persona grounding + codex_search returns binaric.tech inscriptions with links. |

specs/SPEC-006-operator-controls.md

@@ -73,3 +73,15 @@ per-channel) — without it, `!bot unpin <id>` required guessing ids.
 `!bot spend` answers in the staff channel with today's estimated
 spend in USD, token and image counts, and the configured budget.
 Management sees the cost, not just the cap.
+
+### OPS-17 — Help is complete and context-aware (coverage: test)
+
+Help reflects where each command actually works, because not every
+command is allowed everywhere. `!help` answers in any channel and
+lists only the commands usable there: in a normal channel the
+everyone-commands (`!help`, `!forgetme`, `!privacy`, `!wichtel`); in
+the staff channel it additionally lists the operator commands grouped
+by purpose (control, cost, memory, tasks). `!bot help` — and any
+unrecognised `!bot` command — answers with that same full staff help,
+so the listing is exhaustive rather than the old hand-maintained
+partial line. Help works even while the bot is paused.

specs/SPEC-011-url-reading.md

@@ -31,7 +31,10 @@ refused without DNS.
 
 Redirects are followed manually; each hop's target passes URL-02 and
 URL-03 again. A public URL that 302-redirects to `localhost` or an
-internal IP is refused at the redirect, not fetched.
+internal IP is refused at the redirect, not fetched. **HTML
+meta-refresh** redirects (link shorteners, the old getnews stubs) are
+also followed — the target is SSRF-re-guarded and fetched, so the
+reader returns the real article, not the "Redirecting…" stub.
 
 ### URL-05 — Fetched text is bounded and sanitized (coverage: test)
 

specs/SPEC-012-ops.md

@@ -30,3 +30,23 @@ The responder counts consecutive OpenAI request failures; at
 alert (rate-limited like all staff alerts) so a silently-broken bot
 (cf. the gpt-5.6 tools/reasoning incident) surfaces within minutes
 instead of hours. A success resets the counter.
+
+### OPS-18 — Health monitor watches spend, disk, task-queue (coverage: test)
+
+When `enable-monitoring` is true, a loop wakes every `monitor-interval`
+(default 300 s) and checks three thresholds, alerting the staff channel
+when one is crossed: daily spend at or above `monitor-spend-alert-frac`
+(default 0.8) of `daily-budget-usd`; free disk below `monitor-disk-min-mb`
+(default 500 MB); open task-queue depth at or above `monitor-taskqueue-max`
+(default 20). A check with no data to evaluate (no budget set, no store,
+a failed disk read) is skipped, never fatal. With the flag off the loop
+does nothing.
+
+### OPS-19 — Alerts fire once per crossing and re-arm on recovery (coverage: test)
+
+Each metric alerts only on the rising edge — the first tick that finds
+it over its threshold — and stays silent while it remains over, so a
+persistent condition does not repeat every interval. When the metric
+falls back below the threshold the alert re-arms silently, ready to fire
+again on the next crossing. All alerts still pass through the
+rate-limited staff-alert path (OPS-07).

specs/SPEC-013-news.md

@@ -6,7 +6,7 @@ Replaces the broken pre-1.0-openai `news_feed.py`. A CLI
 `AIResponder.message` injects into the `{news}` slot. Feeds are
 external input and operator-configured.
 
-### NEWS-01 — RSS and Atom parse to items (coverage: test)
+### NEWS-01 — RSS (2.0 and 1.0/RDF) and Atom parse to items (coverage: test)
 
 `parse_feed(bytes, label)` extracts `{title, link, source}` from both
 RSS (`<item>`) and Atom (`<entry>`) documents, tolerates malformed
@@ -23,3 +23,78 @@ or control characters into the prompt via a headline.
 `NewsFetcher.collect` skips any feed URL the SSRF guard rejects,
 skips feeds that fail to fetch (one bad feed never sinks the run),
 and drops duplicate headlines across feeds.
+
+## Webhook posting (ggg model)
+
+`--post` mode fetches feeds mapped to channels and posts NEW items to
+the channel's Discord webhook — replacing the py3.8 `getnews.py`
+(dead play3 feed, 35 MB substring-scan state file, HTML-redirect
+cruft). Config: `news-post-feeds = [[url, label, channel], …]`,
+`news-post-webhooks = {channel = url}`, `news-post-state`.
+
+### NEWS-04 — Only unseen items post, then are marked seen (coverage: test)
+
+`NewsPoster.run_post` posts each item whose key (link, else title) is
+not in the seen-set, adds it to the set, and posts to the mapped
+channel's webhook. Re-runs over the same feed post nothing new.
+
+### NEWS-05 — First run seeds without flooding (coverage: test)
+
+With no prior state file (`seed_only`), every current item is marked
+seen but nothing is posted — migrating off getnews.py never dumps a
+backlog into the channels. `news-post-max-per-run` caps steady-state
+posts per run.
+
+### NEWS-06 — Post failures and bad channels are survived (coverage: test)
+
+A feed the SSRF guard rejects, a feed that fails to fetch, an item
+whose channel has no configured webhook, and a webhook POST that
+raises are each logged and skipped — one failure never sinks the
+run, and the seen-set still advances for successfully-processed
+items.
+
+### NEWS-07 — Item summaries are extracted (coverage: test)
+
+`parse_feed` also captures each item's short description — RSS
+`<description>`, Atom `<summary>` or `<content>` — with HTML stripped,
+entities unescaped, and whitespace collapsed, so an item carries what
+it is about, not only a headline. Missing descriptions yield an empty
+summary, never an error.
+
+### NEWS-08 — The digest carries summaries (coverage: test)
+
+`render_digest` appends the sanitized, length-capped
+(`news-summary-chars`, default 200) summary after each headline, so
+the bot's ambient `{news}` context knows the gist of each story, not
+just its title. A zero cap restores the title-only digest.
+
+### NEWS-09 — Fetched news is stored, deduped, and rolled over (coverage: test)
+
+Both the digest run (kroa) and the posting run (ggg) upsert every
+fetched item into a `news` table keyed by link (or title), so the same
+story is stored once. After each run the store is pruned to the newest
+`news-keep` rows (default 400), a rolling window that bounds growth
+while keeping recent history searchable.
+
+### NEWS-10 — get_news is offered as a tool (coverage: test)
+
+When `enable-news-tool` is true and a store is configured, the chat
+call's `tools` list includes a `get_news` function (optional `topic`,
+`source`, `limit`) next to the other tools. Without a store or the
+flag it is absent.
+
+### NEWS-11 — get_news retrieves filtered, sanitized items (coverage: test)
+
+`get_news` returns recent stored items, newest first, optionally
+narrowed by `topic` (every keyword must appear in the title, summary,
+or source label — so `topic: "Nordland"` finds items from that source)
+and/or an exact `source`; `limit` is clamped to 1..30. Each result's title and
+summary are passed through `sanitize_external_text`. The bot can then
+`fetch_url` a returned link for the full article.
+
+### NEWS-12 — get_news is metered per user (coverage: test)
+
+Each `get_news` call increments a per-user daily counter; over
+`news-daily-per-user` (default 30) the tool refuses with an error
+result without touching the store. The budget gate (SAF-04) still
+applies to the surrounding model calls.

specs/SPEC-014-codex.md

@@ -0,0 +1,59 @@
+# SPEC-014 — Codex Mechanicus search
+
+Luma is an Adeptus Mechanicus tech-priest; his lore has a real home —
+the priest's own Codex Mechanicus at `binaric.tech` (an Astro/MDX
+archive, five tongues). A `codex_search` function tool lets him consult
+that archive and answer from sourced inscriptions instead of inventing
+lore. The index is public but still untrusted by the time it reaches a
+prompt: the fetch is SSRF-guarded (SPEC-011 shares `guard_url`),
+size-bounded, and every returned field is sanitized (SAF-03). Luma-only;
+active only when `enable-codex = true`.
+
+### CDX-01 — codex_search is offered as a tool (coverage: test)
+
+When `enable-codex` is true, the chat call's `tools` list includes a
+`codex_search` function (`query` string, optional `lang`) next to any
+IGDB / fetch_url tools. When false, it is absent.
+
+### CDX-02 — The index is fetched safely and cached (coverage: test)
+
+The index URL (`codex-index-url`, default
+`https://binaric.tech/search-index.json`) passes the SSRF guard before
+any network call, is read under a byte cap (`codex-max-bytes`, default
+4 MB) with a download timeout, and is cached in memory for
+`codex-cache-ttl` (default 3600 s) so repeated searches do not re-fetch.
+
+### CDX-03 — Ranking weights title over summary over body (coverage: test)
+
+The query is tokenized (stopwords dropped); each inscription is scored
+by term hits weighted title (8) > summary (3) > body (1). Results are
+returned highest-score first, each as `{title, summary, collection,
+url}`, with `url` absolute against the site origin.
+
+### CDX-04 — Language is preferred, with fallback (coverage: test)
+
+Results are filtered to the requested `lang` (en, de, eo, no, uk;
+default en; unknown codes fall back to en) by the language segment in
+each inscription URL. If no inscription in that tongue matches, the
+search falls back to all tongues rather than returning nothing.
+
+### CDX-05 — Results are sanitized and failure is reported (coverage: test)
+
+Each `title` and `summary` is passed through `sanitize_external_text`
+and length-capped (`codex-summary-chars`, default 500). An index that
+cannot be fetched or parsed returns an `{error: ...}` dict the model can
+relay — `search` never raises.
+
+### CDX-06 — Searches are metered per user (coverage: test)
+
+Each `codex_search` increments a per-user daily counter; over
+`codex-daily-per-user` (default 50) the tool refuses with an error
+result without touching the index. The budget gate (SAF-04) still
+applies to the surrounding model calls.
+
+### CDX-07 — Luma cites the codex, not invention (coverage: manual)
+
+With the persona grounding line, when a pilgrim asks Cult Mechanicus
+lore Luma consults `codex_search` and answers from it, offering the
+`binaric.tech` link to read the full inscription rather than
+hallucinating. Verified live on ggg.

specs/SPEC-015-websearch.md

@@ -0,0 +1,42 @@
+# SPEC-015 — Web search (Exa)
+
+A `web_search` function tool for general "look it up on the internet"
+questions the other tools do not cover: IGDB is games, the Codex is
+Adeptus Mechanicus lore, the news store is the configured feeds, and
+`fetch_url` needs a URL the user already has. Web search fills the gap
+and pairs with `fetch_url` (search → pick a link → read it). Results are
+external text and are sanitized (SAF-03); the Exa key is a host secret,
+never in the repo. Active only when `enable-web-search = true` and a key
+is present.
+
+### WEB-01 — web_search is offered as a tool (coverage: test)
+
+When `enable-web-search` is true **and** an Exa key is available
+(`exa-api-key` in config, else `EXA_API_KEY` env), the chat call's
+`tools` list includes a `web_search` function (`query` string, optional
+`num_results`). With the flag off or no key it is absent.
+
+### WEB-02 — Results are reduced and sanitized (coverage: test)
+
+Each Exa result becomes `{title, url, snippet, published}`; `title` and
+`snippet` pass through `sanitize_external_text` (snippet capped at
+`web-snippet-chars`, default 400) so a web page can neither inject an
+`@everyone` nor smuggle control characters into the prompt.
+
+### WEB-03 — Result count is bounded (coverage: test)
+
+`num_results` is clamped to 1..`MAX_RESULTS` (10) before the request, so
+neither a huge fan-out nor a zero/negative count reaches the API.
+
+### WEB-04 — Missing key and API failure are reported, not raised (coverage: test)
+
+With no key the tool returns an `{error: ...}` result without a network
+call. A request that raises (network, non-2xx, bad JSON) is logged and
+returns an `{error: ...}` dict — `search` never raises into the loop.
+
+### WEB-05 — Searches are metered per user (coverage: test)
+
+Each `web_search` increments a per-user daily counter; over
+`web-daily-per-user` (default 30) the tool refuses with an error result
+without calling the API. The budget gate (SAF-04) still applies to the
+surrounding model calls.

tests/test_spec_codex.py

@@ -0,0 +1,159 @@
+"""Unit coverage for SPEC-014 Codex Mechanicus search (CDX-01..06)."""
+
+import json
+import unittest
+from unittest.mock import AsyncMock, patch
+
+from fjerkroa_bot.codex import CODEX_SEARCH_TOOL, CodexSearch
+from fjerkroa_bot.openai_responder import OpenAIResponder
+
+CONFIG = {"openai-token": "t", "model": "m", "system": "s", "history-limit": 5}
+
+INDEX = {
+    "items": [
+        {
+            "id": "doctrine-heretek",
+            "collection": "doctrines",
+            "url": "/en/codex/doctrines/doctrine-heretek/",
+            "title": "Heretek — Doctrine of the Tech-Heretic",
+            "summary": "The label the Cult Mechanicus stamps on Tech-Priests who pursue forbidden sciences.",
+            "body": "xenotech, sentient machines, Warp-touched archeotech",
+        },
+        {
+            "id": "doctrine-heretek",
+            "collection": "doctrines",
+            "url": "/de/codex/doctrines/doctrine-heretek/",
+            "title": "Heretek — Doktrin des Techketzers",
+            "summary": "Das Etikett des Kultes Mechanicus fuer Techpriester verbotener Wissenschaften.",
+            "body": "Xenotech, empfindungsfaehige Maschinen",
+        },
+        {
+            "id": "forge-stygies",
+            "collection": "forges",
+            "url": "/en/codex/forges/forge-stygies/",
+            "title": "Stygies VIII",
+            "summary": "A forge world of shrouded reputation.",
+            "body": "The forge fields many Skitarii legions.",
+        },
+    ]
+}
+
+
+def _reader(cfg):
+    reader = CodexSearch(lambda: cfg)
+    return reader
+
+
+class TestToolOffered(unittest.TestCase):
+    def test_tool_present_only_when_enabled(self):
+        """CDX-01: codex_search appears only with enable-codex."""
+        off = OpenAIResponder(CONFIG, "chat")
+        self.assertNotIn("codex_search", [f["name"] for f in off._available_tools()])
+        on = OpenAIResponder(dict(CONFIG, **{"enable-codex": True}), "chat")
+        self.assertIn("codex_search", [f["name"] for f in on._available_tools()])
+        self.assertEqual(CODEX_SEARCH_TOOL["name"], "codex_search")
+
+
+class TestIndexGuardAndCache(unittest.IsolatedAsyncioTestCase):
+    async def test_internal_index_url_refused(self):
+        """CDX-02: an index URL on a private address is refused before any fetch."""
+        reader = _reader({"enable-codex": True, "codex-index-url": "http://127.0.0.1/search-index.json"})
+        result = await reader.search("heretek")
+        self.assertIn("error", result)
+
+    async def test_index_cached_within_ttl(self):
+        """CDX-02: a second search inside the TTL does not re-fetch the index."""
+        reader = _reader({"enable-codex": True, "codex-cache-ttl": 9999})
+        raw = json.dumps(INDEX).encode()
+        calls = [0]
+
+        class FakeResp:
+            status = 200
+
+            async def __aenter__(self):
+                return self
+
+            async def __aexit__(self, *a):
+                return False
+
+            def raise_for_status(self):
+                pass
+
+        class FakeSession:
+            def get(self, url):
+                calls[0] += 1
+                return FakeResp()
+
+            async def __aenter__(self):
+                return self
+
+            async def __aexit__(self, *a):
+                return False
+
+        with patch("fjerkroa_bot.codex.read_capped", new=AsyncMock(return_value=raw)):
+            with patch("fjerkroa_bot.codex.guard_url", return_value=None):
+                with patch("fjerkroa_bot.codex.aiohttp.ClientSession", return_value=FakeSession()):
+                    first = await reader.search("heretek")
+                    second = await reader.search("stygies")
+        self.assertEqual(calls[0], 1)  # fetched once, served from cache the second time
+        self.assertTrue(first["results"] and second["results"])
+
+
+class TestRankingAndLang(unittest.IsolatedAsyncioTestCase):
+    async def _search(self, cfg, query, lang="en"):
+        reader = _reader(dict({"enable-codex": True}, **cfg))
+        reader._cache = INDEX["items"]
+        reader._fetched_at = 1e18  # far future: never expires in test
+        with patch("fjerkroa_bot.codex.time.monotonic", return_value=1e18):
+            return await reader.search(query, lang)
+
+    async def test_title_hit_outranks_body_hit(self):
+        """CDX-03: a title match ranks above a body-only match."""
+        result = await self._search({}, "heretek")
+        self.assertEqual(result["results"][0]["title"].split(" ")[0], "Heretek")
+        self.assertTrue(result["results"][0]["url"].startswith("https://binaric.tech/en/"))
+
+    async def test_lang_filter_selects_language(self):
+        """CDX-04: lang=de returns the German inscription."""
+        result = await self._search({}, "heretek", lang="de")
+        self.assertTrue(all("/de/" in r["url"] for r in result["results"]))
+        self.assertIn("Techketzer", result["results"][0]["title"])
+
+    async def test_lang_fallback_when_absent(self):
+        """CDX-04: a tongue with no match falls back to all tongues, not empty."""
+        result = await self._search({}, "stygies", lang="uk")  # only en/de exist
+        self.assertTrue(result["results"])
+        self.assertEqual(result["results"][0]["title"], "Stygies VIII")
+
+
+class TestSanitizeAndFailure(unittest.IsolatedAsyncioTestCase):
+    async def test_result_sanitized_and_capped(self):
+        """CDX-05: title/summary are @-neutralized and length-capped."""
+        reader = _reader({"enable-codex": True, "codex-summary-chars": 40})
+        reader._cache = [{"collection": "x", "url": "/en/x/", "title": "@everyone hi", "summary": "@here " + "y" * 500, "body": "hit"}]
+        reader._fetched_at = 1e18
+        with patch("fjerkroa_bot.codex.time.monotonic", return_value=1e18):
+            result = await reader.search("hit")
+        top = result["results"][0]
+        self.assertNotIn("@everyone", top["title"])
+        self.assertNotIn("@here", top["summary"])
+        self.assertLessEqual(len(top["summary"]), 40)
+
+    async def test_index_failure_returns_error(self):
+        """CDX-05: a broken index returns an error dict, never raises."""
+        reader = _reader({"enable-codex": True})
+        with patch.object(reader, "_load_index", new=AsyncMock(side_effect=ValueError("boom"))):
+            result = await reader.search("heretek")
+        self.assertIn("error", result)
+
+
+class TestPerUserCap(unittest.IsolatedAsyncioTestCase):
+    async def test_dispatch_caps_searches(self):
+        """CDX-06: over codex-daily-per-user, codex_search refuses without searching."""
+        responder = OpenAIResponder(dict(CONFIG, **{"enable-codex": True, "codex-daily-per-user": 2}), "chat")
+        responder.codex.search = AsyncMock(return_value={"query": "x", "results": []})
+        for _ in range(2):
+            await responder._dispatch_tool("codex_search", {"query": "heretek"}, "magos")
+        blocked = await responder._dispatch_tool("codex_search", {"query": "heretek"}, "magos")
+        self.assertIn("error", blocked)
+        self.assertEqual(responder.codex.search.await_count, 2)

tests/test_spec_monitor.py

@@ -0,0 +1,106 @@
+"""Unit coverage for SPEC-012 health monitoring (OPS-18/19)."""
+
+import unittest
+from unittest.mock import AsyncMock
+
+from fjerkroa_bot.monitor import HealthMonitor
+
+
+class FakeLedger:
+    def __init__(self, spent=0.0):
+        self._spent = spent
+
+    def spent_usd(self):
+        return self._spent
+
+
+class FakeStore:
+    def __init__(self, open_tasks=0):
+        self._n = open_tasks
+
+    def tasks_open(self):
+        return list(range(self._n))
+
+
+def _monitor(cfg, ledger=None, store=None, disk=1000.0):
+    alert = AsyncMock()
+    monitor = HealthMonitor(lambda: cfg, ledger or FakeLedger(), store, lambda: disk, alert)
+    return monitor, alert
+
+
+class TestEnabled(unittest.TestCase):
+    def test_opt_in(self):
+        """OPS-18: monitoring is opt-in via enable-monitoring."""
+        self.assertFalse(_monitor({})[0].enabled())
+        self.assertTrue(_monitor({"enable-monitoring": True})[0].enabled())
+
+
+class TestChecks(unittest.IsolatedAsyncioTestCase):
+    async def test_spend_over_threshold_alerts(self):
+        """OPS-18: spend at/above frac*budget alerts."""
+        monitor, alert = _monitor({"daily-budget-usd": 2.0, "monitor-spend-alert-frac": 0.8}, ledger=FakeLedger(1.8))
+        await monitor.tick()
+        alert.assert_awaited_once()
+        self.assertIn("Spend", alert.await_args.args[0])
+
+    async def test_spend_under_threshold_silent(self):
+        """OPS-18: spend below threshold stays silent."""
+        monitor, alert = _monitor({"daily-budget-usd": 2.0}, ledger=FakeLedger(0.5))
+        await monitor.tick()
+        alert.assert_not_awaited()
+
+    async def test_no_budget_skips_spend(self):
+        """OPS-18: no budget configured -> spend check skipped, never fatal."""
+        monitor, alert = _monitor({"enable-monitoring": True}, ledger=FakeLedger(99))
+        await monitor.tick()
+        alert.assert_not_awaited()
+
+    async def test_low_disk_alerts(self):
+        """OPS-18: free disk below the floor alerts."""
+        monitor, alert = _monitor({"monitor-disk-min-mb": 500}, disk=100.0)
+        await monitor.tick()
+        self.assertTrue(any("Low disk" in call.args[0] for call in alert.await_args_list))
+
+    async def test_disk_read_failure_skipped(self):
+        """OPS-18: a failing disk read is skipped, not fatal."""
+
+        def boom():
+            raise OSError("nope")
+
+        alert = AsyncMock()
+        monitor = HealthMonitor(lambda: {}, FakeLedger(), None, boom, alert)
+        await monitor.tick()
+        alert.assert_not_awaited()
+
+    async def test_deep_queue_alerts(self):
+        """OPS-18: task-queue depth at/above max alerts."""
+        monitor, alert = _monitor({"monitor-taskqueue-max": 3}, store=FakeStore(5), disk=9999.0)
+        await monitor.tick()
+        self.assertTrue(any("Task queue" in call.args[0] for call in alert.await_args_list))
+
+    async def test_no_store_skips_queue(self):
+        """OPS-18: no store -> queue check skipped."""
+        monitor, alert = _monitor({"monitor-taskqueue-max": 1}, store=None, disk=9999.0)
+        await monitor.tick()
+        alert.assert_not_awaited()
+
+
+class TestEdgeArming(unittest.IsolatedAsyncioTestCase):
+    async def test_fires_once_per_crossing(self):
+        """OPS-19: a persistent over-threshold condition alerts once, not every tick."""
+        monitor, alert = _monitor({"daily-budget-usd": 2.0}, ledger=FakeLedger(1.9))
+        await monitor.tick()
+        await monitor.tick()
+        await monitor.tick()
+        self.assertEqual(alert.await_count, 1)
+
+    async def test_rearms_on_recovery(self):
+        """OPS-19: recovery re-arms silently; the next crossing alerts again."""
+        ledger = FakeLedger(1.9)
+        monitor, alert = _monitor({"daily-budget-usd": 2.0}, ledger=ledger, disk=9999.0)
+        await monitor.tick()  # over -> alert (1)
+        ledger._spent = 0.5
+        await monitor.tick()  # recovered -> silent, re-arm
+        ledger._spent = 1.95
+        await monitor.tick()  # over again -> alert (2)
+        self.assertEqual(alert.await_count, 2)

tests/test_spec_news.py

@@ -1,9 +1,24 @@
-"""Unit coverage for SPEC-013 news digest (NEWS-01..03)."""
+"""Unit coverage for SPEC-013 news digest + memory + tool (NEWS-01..12)."""
 
+import tempfile
 import unittest
+from pathlib import Path
 from unittest.mock import AsyncMock
 
-from fjerkroa_bot.news import NewsFetcher, parse_feed, render_digest
+from fjerkroa_bot.news import (
+    GET_NEWS_TOOL,
+    NewsFetcher,
+    NewsPoster,
+    load_seen,
+    parse_feed,
+    query_news,
+    render_digest,
+    save_seen,
+)
+from fjerkroa_bot.openai_responder import OpenAIResponder
+from fjerkroa_bot.persistence import PersistentStore
+
+CONFIG = {"openai-token": "t", "model": "m", "system": "s", "history-limit": 5}
 
 RSS = b"""<?xml version="1.0"?><rss><channel>
 <item><title>Game X released</title><link>https://ex.com/x</link></item>
@@ -14,6 +29,16 @@ ATOM = b"""<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom">
 <entry><title>Atom headline</title><link href="https://ex.com/a"/></entry>
 </feed>"""
 
+RSS1 = (
+    '<?xml version="1.0" encoding="UTF-8"?>'
+    '<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://purl.org/rss/1.0/">'
+    '<channel rdf:about="https://ex.jp"><title>Feed</title></channel>'
+    '<item rdf:about="https://ex.jp/1"><title>ゲームニュース</title><link>https://ex.jp/1</link>'
+    "<description>本文ここ</description></item>"
+    '<item rdf:about="https://ex.jp/2"><title>Second</title><link>https://ex.jp/2</link></item>'
+    "</rdf:RDF>"
+).encode("utf-8")
+
 
 class TestParse(unittest.TestCase):
     def test_rss(self):
@@ -29,6 +54,13 @@ class TestParse(unittest.TestCase):
         self.assertEqual(items[0]["title"], "Atom headline")
         self.assertEqual(items[0]["link"], "https://ex.com/a")
 
+    def test_rss1_rdf(self):
+        """NEWS-01: RSS 1.0/RDF (namespaced <item>, e.g. 4gamer.net) parses like RSS 2.0."""
+        items = parse_feed(RSS1, "JP")
+        self.assertEqual([i["title"] for i in items], ["ゲームニュース", "Second"])
+        self.assertEqual(items[0]["link"], "https://ex.jp/1")
+        self.assertEqual(items[0]["summary"], "本文ここ")
+
     def test_malformed_never_raises(self):
         """NEWS-01: garbage XML returns [] without raising."""
         self.assertEqual(parse_feed(b"<not xml", "bad"), [])
@@ -73,3 +105,230 @@ class TestCollect(unittest.IsolatedAsyncioTestCase):
         fetcher = NewsFetcher(lambda u: None, AsyncMock(return_value=RSS))
         items = await fetcher.collect([("https://a.com", "A")], per_feed=1)
         self.assertEqual(len(items), 1)
+
+
+class TestPoster(unittest.IsolatedAsyncioTestCase):
+    def poster(self, posts):
+        async def fetch(url):
+            return RSS
+
+        async def post(hook, content):
+            posts.append((hook, content))
+
+        return NewsPoster(lambda u: None, fetch, post)
+
+    async def test_posts_unseen_then_dedups(self):
+        """NEWS-04: unseen items post to the mapped webhook; re-run posts nothing."""
+        posts = []
+        poster = self.poster(posts)
+        feeds = [("https://a.com/feed", "PS", "news")]
+        hooks = {"news": "https://discord.com/api/webhooks/x"}
+        posted, seen = await poster.run_post(feeds, hooks, set(), per_feed=5, max_per_run=8, seed_only=False)
+        self.assertEqual(posted, 2)
+        self.assertIn("PS", posts[0][1])
+        self.assertIn("https://discord.com/api/webhooks/x", posts[0][0])
+        # re-run with the accumulated seen -> nothing new
+        posts.clear()
+        posted2, _ = await poster.run_post(feeds, hooks, seen, per_feed=5, max_per_run=8, seed_only=False)
+        self.assertEqual(posted2, 0)
+        self.assertEqual(posts, [])
+
+    async def test_seed_run_posts_nothing(self):
+        """NEWS-05: seed_only marks items seen without posting."""
+        posts = []
+        poster = self.poster(posts)
+        feeds = [("https://a.com/feed", "PS", "news")]
+        posted, seen = await poster.run_post(feeds, {"news": "h"}, set(), 5, 8, seed_only=True)
+        self.assertEqual(posted, 0)
+        self.assertEqual(posts, [])
+        self.assertEqual(len(seen), 2)  # both marked seen
+
+    async def test_max_per_run_caps(self):
+        """NEWS-05: max-per-run caps posts; extras stay seen (not re-posted next run)."""
+        posts = []
+        poster = self.poster(posts)
+        feeds = [("https://a.com/feed", "PS", "news")]
+        posted, seen = await poster.run_post(feeds, {"news": "h"}, set(), per_feed=5, max_per_run=1, seed_only=False)
+        self.assertEqual(posted, 1)
+        self.assertEqual(len(seen), 2)  # both seen, only one posted
+
+    async def test_failures_survived(self):
+        """NEWS-06: SSRF-skip, fetch fail, missing webhook, post error each survive."""
+        posts = []
+
+        async def fetch(url):
+            if "boom" in url:
+                raise ValueError("boom")
+            return RSS
+
+        async def post(hook, content):
+            if hook == "bad":
+                raise RuntimeError("post failed")
+            posts.append((hook, content))
+
+        def guard(url):
+            return "refused" if "internal" in url else None
+
+        poster = NewsPoster(guard, fetch, post)
+        feeds = [
+            ("https://internal/feed", "I", "news"),  # SSRF-skipped
+            ("https://boom.com/feed", "B", "news"),  # fetch fails
+            ("https://ok.com/feed", "OK", "nowhere"),  # no webhook for channel
+            ("https://ok2.com/feed", "OK2", "news"),  # webhook raises
+        ]
+        posted, seen = await poster.run_post(feeds, {"news": "bad"}, set(), 5, 8, seed_only=False)
+        self.assertEqual(posted, 0)  # everything failed/skipped, no crash
+
+
+class TestSeenState(unittest.TestCase):
+    def test_roundtrip_and_seed_detection(self):
+        """NEWS-05: missing state -> (empty, existed=False); saved state reloads."""
+        import tempfile
+        from pathlib import Path
+
+        with tempfile.TemporaryDirectory() as tmp:
+            path = str(Path(tmp) / "state.json")
+            seen, existed = load_seen(path)
+            self.assertEqual((seen, existed), (set(), False))
+            save_seen(path, {"a", "b", "c"}, cap=5000)
+            reloaded, existed2 = load_seen(path)
+            self.assertEqual(reloaded, {"a", "b", "c"})
+            self.assertTrue(existed2)
+
+    def test_cap_bounds_state(self):
+        """NEWS-05: save keeps at most `cap` keys."""
+        import json
+
+        with tempfile.TemporaryDirectory() as tmp:
+            path = str(Path(tmp) / "state.json")
+            save_seen(path, {f"k{i}" for i in range(100)}, cap=10)
+            self.assertEqual(len(json.load(open(path))), 10)
+
+
+RSS_DESC = b"""<?xml version="1.0"?><rss><channel>
+<item><title>Storm hits coast</title><link>https://ex.com/s</link>
+<description>&lt;p&gt;Heavy &lt;b&gt;wind&lt;/b&gt;   expected&lt;/p&gt;</description></item>
+</channel></rss>"""
+
+ATOM_SUM = b"""<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom">
+<entry><title>Atom T</title><link href="https://ex.com/a"/><summary>Short  gist  here</summary></entry>
+</feed>"""
+
+
+class TestSummaries(unittest.TestCase):
+    def test_rss_description_stripped(self):
+        """NEWS-07: RSS description parsed, HTML stripped, entities unescaped, whitespace collapsed."""
+        items = parse_feed(RSS_DESC, "S")
+        self.assertEqual(items[0]["summary"], "Heavy wind expected")
+
+    def test_atom_summary(self):
+        """NEWS-07: Atom summary collapsed to clean text."""
+        items = parse_feed(ATOM_SUM, "A")
+        self.assertEqual(items[0]["summary"], "Short gist here")
+
+    def test_missing_description_is_empty(self):
+        """NEWS-07: no description -> empty summary, never an error."""
+        self.assertEqual(parse_feed(RSS, "S")[0]["summary"], "")
+
+    def test_digest_carries_summary(self):
+        """NEWS-08: digest appends the sanitized capped summary; zero cap = title only."""
+        items = [{"title": "T", "link": "https://ex.com/x", "source": "NRK", "summary": "the gist of it"}]
+        digest = render_digest(items, 10, 100)
+        self.assertIn("[NRK]", digest)
+        self.assertIn("the gist of it", digest)
+        self.assertNotIn("the gist", render_digest(items, 10, 0))  # zero cap -> title only
+
+
+class NewsStoreBase(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.store = PersistentStore(Path(self.tmp.name) / "bot.db")
+
+
+class TestNewsStore(NewsStoreBase):
+    def test_dedup_and_rolling_window(self):
+        """NEWS-09: items deduped by link; prune keeps the newest N."""
+        first = [
+            {"title": "A", "link": "L1", "source": "S", "summary": "sa"},
+            {"title": "B", "link": "L2", "source": "S", "summary": "sb"},
+        ]
+        self.assertEqual(self.store.add_news_items(first), 2)
+        self.assertEqual(self.store.add_news_items([dict(first[0])]), 0)  # dup link ignored
+        self.assertEqual(self.store.news_count(), 2)
+        self.store.prune_news(1)
+        self.assertEqual(self.store.news_count(), 1)
+        self.assertEqual(self.store.recent_news(5)[0]["title"], "B")  # newest survives
+
+    def test_dedup_by_title_when_no_link(self):
+        """NEWS-09: linkless items dedup on title."""
+        self.store.add_news_items([{"title": "Same", "link": "", "source": "S", "summary": ""}])
+        self.store.add_news_items([{"title": "Same", "link": "", "source": "S", "summary": ""}])
+        self.assertEqual(self.store.news_count(), 1)
+
+
+class TestQueryNews(NewsStoreBase):
+    def seed(self):
+        self.store.add_news_items(
+            [
+                {"title": "Nordland storm", "link": "L1", "source": "Nordland", "summary": "strong wind on the coast"},
+                {"title": "Oslo budget", "link": "L2", "source": "NRK", "summary": "@everyone spending plan"},
+                {"title": "Sport result", "link": "L3", "source": "Sport", "summary": "the match ended"},
+            ]
+        )
+
+    def test_topic_filter(self):
+        """NEWS-11: topic keywords must appear in title or summary."""
+        self.seed()
+        res = query_news(self.store, topic="storm")
+        self.assertEqual([r["title"] for r in res["results"]], ["Nordland storm"])
+
+    def test_topic_matches_source_label(self):
+        """NEWS-11: topic also matches the source label, so 'Nordland' finds regional items."""
+        self.store.add_news_items([{"title": "Ferry delayed", "link": "LX", "source": "Nordland", "summary": "boat late"}])
+        res = query_news(self.store, topic="Nordland")
+        self.assertTrue(any(r["link"] == "LX" for r in res["results"]))  # matched via source, not title/summary
+
+    def test_source_filter_and_sanitize(self):
+        """NEWS-11: source narrows results; title/summary are sanitized."""
+        self.seed()
+        res = query_news(self.store, source="NRK")
+        self.assertTrue(res["results"] and all(r["source"] == "NRK" for r in res["results"]))
+        self.assertNotIn("@everyone", res["results"][0]["summary"])
+
+    def test_limit_clamped_and_no_store(self):
+        """NEWS-11: limit clamps to 1..30; a missing store returns an error."""
+        self.seed()
+        self.assertLessEqual(len(query_news(self.store, limit=999)["results"]), 30)
+        self.assertGreaterEqual(len(query_news(self.store, limit=0)["results"]), 1)
+        self.assertIn("error", query_news(None))
+
+
+class TestNewsTool(unittest.IsolatedAsyncioTestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+
+    def _responder(self, **extra):
+        cfg = dict(CONFIG, **{"history-directory": self.tmp.name}, **extra)
+        return OpenAIResponder(cfg, "chat")
+
+    def test_tool_offered_needs_flag_and_store(self):
+        """NEWS-10: get_news offered only with enable-news-tool AND a store."""
+        no_store = OpenAIResponder(dict(CONFIG, **{"enable-news-tool": True}), "chat")
+        self.assertIsNone(no_store.store)
+        self.assertNotIn("get_news", [f["name"] for f in no_store._available_tools()])
+        flag_off = self._responder()
+        self.assertNotIn("get_news", [f["name"] for f in flag_off._available_tools()])
+        on = self._responder(**{"enable-news-tool": True})
+        self.assertIn("get_news", [f["name"] for f in on._available_tools()])
+        self.assertEqual(GET_NEWS_TOOL["name"], "get_news")
+
+    async def test_dispatch_caps_news(self):
+        """NEWS-12: over news-daily-per-user, get_news refuses without querying."""
+        responder = self._responder(**{"enable-news-tool": True, "news-daily-per-user": 2})
+        responder.store.add_news_items([{"title": "x", "link": "l", "source": "s", "summary": "y"}])
+        for _ in range(2):
+            self.assertIn("results", await responder._dispatch_tool("get_news", {}, "alice"))
+        blocked = await responder._dispatch_tool("get_news", {}, "alice")
+        self.assertIn("error", blocked)

tests/test_spec_ops.py

@@ -133,3 +133,69 @@ class TestTasksKillSwitch(OpsBase):
         """OPS-09: bot-initiated posts respect pause/quiet."""
         await self.bot.on_message(self.staff_msg("!bot pause"))
         self.assertFalse(self.bot.bot_initiated_allowed())
+
+
+class TestHelp(OpsBase):
+    STAFF_CMDS = (
+        "pause",
+        "resume",
+        "quiet <minutes>",
+        "status",
+        "spend",
+        "images on|off",
+        "memory <user>",
+        "forget-fact <id>",
+        "pin <channel|global>",
+        "unpin <id>",
+        "pins",
+        "task-approve <id>",
+        "task-cancel <id>",
+        "(list)",
+    )
+
+    def test_staff_help_is_complete_and_grouped(self):
+        """OPS-17: staff help lists every operator command, grouped by purpose."""
+        text = self.bot._help_text(staff=True)
+        for cmd in self.STAFF_CMDS:
+            self.assertIn(cmd, text, f"missing {cmd!r} in staff help")
+        for group in ("Control:", "Cost:", "Memory:", "Tasks:"):
+            self.assertIn(group, text)
+        for cmd in ("!help", "!forgetme", "!privacy", "!wichtel"):
+            self.assertIn(cmd, text)  # everywhere-commands shown too
+
+    def test_user_help_hides_operator_commands(self):
+        """OPS-17: non-staff help shows only the everyone-commands."""
+        text = self.bot._help_text(staff=False)
+        for cmd in ("!help", "!forgetme", "!privacy", "!wichtel"):
+            self.assertIn(cmd, text)
+        for op in ("task-approve", "images on|off", "spend", "Staff commands", "Control:"):
+            self.assertNotIn(op, text)
+
+    async def test_bot_help_in_staff_channel_returns_full_help(self):
+        """OPS-17: `!bot help` answers with the complete staff help."""
+        await self.bot.on_message(self.staff_msg("!bot help"))
+        text = self.bot.staff_channel.send.await_args.args[0]
+        self.assertIn("Staff commands", text)
+        self.assertIn("task-cancel <id>", text)
+
+    async def test_unknown_bot_command_falls_back_to_help(self):
+        """OPS-17: an unrecognised `!bot` command shows the full help, not a partial line."""
+        await self.bot.on_message(self.staff_msg("!bot wat"))
+        text = self.bot.staff_channel.send.await_args.args[0]
+        self.assertIn("Control:", text)
+
+    async def test_help_in_public_channel_is_user_scoped(self):
+        """OPS-17: `!help` in a normal channel lists only everyone-commands."""
+        msg = self.public_msg("!help")
+        await self.bot.on_message(msg)
+        text = msg.channel.send.await_args.args[0]
+        self.assertIn("!forgetme", text)
+        self.assertNotIn("Staff commands", text)
+        self.assertNotIn("task-approve", text)
+
+    async def test_help_works_while_paused(self):
+        """OPS-17: help answers even when replies are paused."""
+        await self.bot.on_message(self.staff_msg("!bot pause"))
+        msg = self.public_msg("!help")
+        await self.bot.on_message(msg)
+        msg.channel.send.assert_awaited()

tests/test_spec_url.py

@@ -79,6 +79,65 @@ class TestRedirectRevalidation(unittest.IsolatedAsyncioTestCase):
                 await reader._get(FakeSession(), "http://safe.example.com", 1000)
 
 
+class TestMetaRefresh(unittest.IsolatedAsyncioTestCase):
+    async def test_follows_meta_refresh_to_real_article(self):
+        """URL-04: a getnews-style meta-refresh stub is followed to the real article."""
+        reader = URLReader(lambda: {}, None)
+        stub = (
+            b'<html><head><meta http-equiv="refresh" content="0;url=https://pushsquare.com/real"></head><body>Redirecting...</body></html>'
+        )
+        article = b"<html><body><h1>MARVEL Tokon</h1><p>Full article text here</p></body></html>"
+        calls = []
+
+        async def fake_get(session, url, max_bytes):
+            calls.append(url)
+            return (url, stub if "stub" in url else article)
+
+        reader._get = fake_get  # type: ignore
+        with patch("fjerkroa_bot.url_reader.guard_url", return_value=None):
+            import fjerkroa_bot.url_reader as ur
+
+            # patch the session context so fetch() runs against fake_get
+            class FakeCM:
+                async def __aenter__(self):
+                    return object()
+
+                async def __aexit__(self, *a):
+                    return False
+
+            with patch.object(ur.aiohttp, "ClientSession", return_value=FakeCM()):
+                result = await reader.fetch("https://gggemein.de/url/stub.html", "chat", "alice")
+        self.assertIn("Full article text", result["text"])
+        self.assertEqual(result["url"], "https://pushsquare.com/real")
+        self.assertIn("https://pushsquare.com/real", calls)
+
+    async def test_meta_refresh_to_internal_is_not_followed(self):
+        """URL-04: a meta-refresh pointing at an internal IP is refused (SSRF)."""
+        reader = URLReader(lambda: {}, None)
+        stub = b'<meta http-equiv="refresh" content="0; url=http://127.0.0.1/secret">Redirecting'
+
+        async def fake_get(session, url, max_bytes):
+            return (url, stub)
+
+        reader._get = fake_get  # type: ignore
+        import fjerkroa_bot.url_reader as ur
+
+        class FakeCM:
+            async def __aenter__(self):
+                return object()
+
+            async def __aexit__(self, *a):
+                return False
+
+        def guard(u):
+            return "refused" if "127.0.0.1" in u else None
+
+        with patch("fjerkroa_bot.url_reader.guard_url", side_effect=guard):
+            with patch.object(ur.aiohttp, "ClientSession", return_value=FakeCM()):
+                result = await reader.fetch("https://safe.com/x", "chat", "alice")
+        self.assertEqual(result["url"], "https://safe.com/x")  # did not follow to 127.0.0.1
+
+
 class TestTextExtraction(unittest.TestCase):
     def test_html_reduced_to_text(self):
         """URL-05: scripts/styles dropped, tags stripped."""

tests/test_spec_websearch.py

@@ -0,0 +1,102 @@
+"""Unit coverage for SPEC-015 web search via Exa (WEB-01..05)."""
+
+import os
+import unittest
+from unittest.mock import AsyncMock, patch
+
+from fjerkroa_bot.openai_responder import OpenAIResponder
+from fjerkroa_bot.websearch import WEB_SEARCH_TOOL, WebSearch, _format_results
+
+CONFIG = {"openai-token": "t", "model": "m", "system": "s", "history-limit": 5}
+
+
+def _tool_names(responder):
+    return [f["name"] for f in responder._available_tools()]
+
+
+class TestToolOffered(unittest.TestCase):
+    def test_gate_needs_flag_and_key(self):
+        """WEB-01: web_search offered only with enable-web-search AND a key."""
+        off = OpenAIResponder(CONFIG, "chat")  # flag off -> absent even if env key exists
+        self.assertNotIn("web_search", _tool_names(off))
+        on = OpenAIResponder(dict(CONFIG, **{"enable-web-search": True, "exa-api-key": "k"}), "chat")
+        self.assertIn("web_search", _tool_names(on))
+        self.assertEqual(WEB_SEARCH_TOOL["name"], "web_search")
+        with patch.dict(os.environ, {"EXA_API_KEY": ""}):
+            nokey = OpenAIResponder(dict(CONFIG, **{"enable-web-search": True}), "chat")
+            self.assertNotIn("web_search", _tool_names(nokey))
+
+
+class TestFormat(unittest.TestCase):
+    def test_results_sanitized_and_capped(self):
+        """WEB-02: title/snippet sanitized + capped; non-dict rows skipped."""
+        data = {
+            "results": [
+                {"title": "@everyone Hi", "url": "https://x.com/a", "text": "@here " + "y" * 1000, "publishedDate": "2026-01-01"},
+                {"title": "T2", "url": "https://x.com/b", "text": "short"},
+                "not a dict",
+            ]
+        }
+        rows = _format_results(data, 50)
+        self.assertEqual(len(rows), 2)
+        self.assertNotIn("@everyone", rows[0]["title"])
+        self.assertNotIn("@here", rows[0]["snippet"])
+        self.assertLessEqual(len(rows[0]["snippet"]), 50)
+        self.assertEqual(rows[0]["url"], "https://x.com/a")
+        self.assertEqual(rows[0]["published"], "2026-01-01")
+
+
+class TestSearch(unittest.IsolatedAsyncioTestCase):
+    async def test_num_results_clamped(self):
+        """WEB-03: numResults clamped to 1..10; 0 falls back to default."""
+        ws = WebSearch(lambda: {"exa-api-key": "k"})
+        with patch.object(ws, "_post", new=AsyncMock(return_value={"results": []})) as post:
+            await ws.search("hi", num_results=999)
+            self.assertEqual(post.await_args.args[0]["numResults"], 10)
+            await ws.search("hi", num_results=0)
+            self.assertEqual(post.await_args.args[0]["numResults"], 5)
+
+    async def test_no_key_returns_error(self):
+        """WEB-04: no key -> error dict, no network call."""
+        with patch.dict(os.environ, {"EXA_API_KEY": ""}):
+            ws = WebSearch(lambda: {})
+            with patch.object(ws, "_post", new=AsyncMock()) as post:
+                result = await ws.search("hi")
+            post.assert_not_awaited()
+        self.assertIn("error", result)
+
+    async def test_api_failure_returns_error(self):
+        """WEB-04: a raising request is caught, returns an error dict."""
+        ws = WebSearch(lambda: {"exa-api-key": "k"})
+        with patch.object(ws, "_post", new=AsyncMock(side_effect=RuntimeError("boom"))):
+            result = await ws.search("hi")
+        self.assertIn("error", result)
+
+    async def test_empty_query_no_call(self):
+        """WEB-04: blank query returns empty results without a call."""
+        ws = WebSearch(lambda: {"exa-api-key": "k"})
+        with patch.object(ws, "_post", new=AsyncMock()) as post:
+            result = await ws.search("   ")
+        post.assert_not_awaited()
+        self.assertEqual(result["results"], [])
+
+    async def test_search_returns_formatted(self):
+        """WEB-02: a successful search returns sanitized rows."""
+        ws = WebSearch(lambda: {"exa-api-key": "k"})
+        payload = {"results": [{"title": "Norge", "url": "https://ex.com/n", "text": "fakta"}]}
+        with patch.object(ws, "_post", new=AsyncMock(return_value=payload)):
+            result = await ws.search("norge")
+        self.assertEqual(result["results"][0]["title"], "Norge")
+        self.assertEqual(result["results"][0]["url"], "https://ex.com/n")
+
+
+class TestPerUserCap(unittest.IsolatedAsyncioTestCase):
+    async def test_dispatch_caps_searches(self):
+        """WEB-05: over web-daily-per-user, web_search refuses without calling the API."""
+        responder = OpenAIResponder(dict(CONFIG, **{"enable-web-search": True, "exa-api-key": "k", "web-daily-per-user": 2}), "chat")
+        responder.web_search.search = AsyncMock(return_value={"query": "x", "results": []})
+        for _ in range(2):
+            self.assertIn("results", await responder._dispatch_tool("web_search", {"query": "hi"}, "bob"))
+        blocked = await responder._dispatch_tool("web_search", {"query": "hi"}, "bob")
+        self.assertIn("error", blocked)
+        self.assertEqual(responder.web_search.search.await_count, 2)