igdb search + capped http reads: native fulltext, full-page fetch
search_games used `where name ~ "<q>"*`: prefix-only, diacritic- and word-order-sensitive -- "MARVEL Tōkon" found nothing though IGDB has it. Switch to IGDB's native `search` clause (relevance-ranked, diacritic-insensitive). fetch_url and image downloads read bodies with content.read(n), which returns only the first buffered chunk (~7 KB): pages collapsed to their <title>. httpread.read_capped collects chunks up to the byte cap; image downloads read limit+1 so over-limit files are still rejected instead of cached truncated (IMG-10). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
"""Bounded HTTP body read (leaf module, no intra-package imports).
|
||||
|
||||
`response.content.read(n)` returns whatever is buffered, not n bytes,
|
||||
so it silently truncates large or chunked bodies (and web feeds/pages
|
||||
parse to garbage). This accumulates decompressed chunks up to a hard
|
||||
cap instead.
|
||||
"""
|
||||
|
||||
CHUNK = 65536
|
||||
|
||||
|
||||
async def read_capped(response, max_bytes: int) -> bytes:
|
||||
buf = bytearray()
|
||||
async for chunk in response.content.iter_chunked(CHUNK):
|
||||
buf.extend(chunk)
|
||||
if len(buf) > max_bytes:
|
||||
break
|
||||
return bytes(buf[:max_bytes])
|
||||
Reference in New Issue
Block a user