When the simple HTTP fetch returns empty (JS-rendered pages, Cloudflare
Turnstile protection), fall back to agent-browser headless Chromium to
extract paragraph text. Tested on TechCrunch, The Verge, VentureBeat —
all return full article content where the simple fetcher got nothing.
Paywall domains (Bloomberg, NYT, WSJ, FT) remain in SKIP_DOMAINS and
are bypassed entirely — bot detection blocks even real browsers there.
Fetch tiers:
1. CF Markdown / simple HTTP (parallel, fast)
2. agent-browser Chromium (sequential, last resort for empty results)
3. Skip (paywalled/non-article domains)
When the dedup pre-filter removes all valid candidates (e.g. a
force-run on a candidates file that was already processed), the LLM
was receiving an empty article list and hallucinating picks from the
recent_posts context injected into the prompt. This caused exact
duplicate stories to be re-sent.
Now llm_editor.py exits immediately with 0 picks when
total_candidates == 0 after pre-filtering, preventing the LLM from
ever being called with no real candidates.
Major dedup overhaul — persistent SQLite database (dedup_db.py) stores
normalized URLs and titles across all scans, replacing text-file-only
matching that missed query-param variants and same-event rewrites.
LLM editor now uses a 3-tier failover chain (Gemini Flash Lite → Grok
via OpenRouter → Gemini Flash Preview) instead of single-provider with
raw-fallback. Providers alternate to avoid double failure on outages.
Inline AI keyword filter in the orchestrator blocks non-AI articles at
the RSS extraction stage. Editorial rules updated from "must produce 5"
to "up to 7, quality over quantity."
Includes 68 unit tests, updated README, changelog, and migration guide.
kriptoburak