mirror of
https://github.com/jacob-bd/openclaw-newsroom.git
synced 2026-08-14 00:47:57 +00:00
Initial release: OpenClaw automated news scanning pipeline
11 scripts + editorial profile template for a complete AI-powered news scanning workflow. 5 data sources (RSS, Reddit, Twitter, GitHub, Tavily), quality scoring, article enrichment, and Gemini Flash editorial curation. ~$5/month to run.
This commit is contained in:
jacob-bd
+23
@@ -0,0 +1,23 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.egg-info/
|
||||
|
||||
# Runtime state (created by the pipeline)
|
||||
github_trending_state.json
|
||||
last_scan_candidates.txt
|
||||
last_scan_github.txt
|
||||
scanner_presented.md
|
||||
editorial_decisions.md
|
||||
news_log.md
|
||||
|
||||
# Environment / secrets
|
||||
.env
|
||||
*.key
|
||||
|
||||
# macOS
|
||||
.DS_Store
|
||||
|
||||
# Temp files
|
||||
/tmp/
|
||||
*.log
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Jacob Ben David
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,424 @@
|
||||
# OpenClaw Automated News Scanner
|
||||
|
||||
A complete, automated AI news scanning pipeline for [OpenClaw](https://github.com/openclaw/openclaw). Scans 5 data sources every 2 hours, scores and deduplicates results, enriches top articles with full text, and uses Gemini Flash as an AI editor to curate the best stories for your channel.
|
||||
|
||||
**Pipeline cost:** ~$5/month (Gemini Flash API + Tavily free tier)
|
||||
|
||||
---
|
||||
|
||||
## How This Fits Into OpenClaw
|
||||
|
||||
This pipeline is designed to run as an **OpenClaw cron job**. Here's how it integrates:
|
||||
|
||||
```
|
||||
OpenClaw Gateway
|
||||
├── Cron scheduler fires every 2 hours
|
||||
│ └── Runs news_scan_deduped.sh (the orchestrator)
|
||||
│ ├── Calls 5 data source scripts (RSS, Reddit, Twitter, GitHub, Tavily)
|
||||
│ ├── Scores + deduplicates via quality_score.py
|
||||
│ ├── Enriches top articles via enrich_top_articles.py
|
||||
│ └── Curates via llm_editor.py (Gemini Flash API)
|
||||
│
|
||||
├── Agent receives the pipeline output
|
||||
│ └── Formats and delivers to your channel (Telegram, Slack, etc.)
|
||||
│
|
||||
├── Nightly cron (optional)
|
||||
│ └── Runs update_editorial_profile.py to learn from your approvals/rejections
|
||||
│
|
||||
└── memory/ directory
|
||||
├── editorial_profile.md ← LLM editor reads this for guidance
|
||||
├── editorial_decisions.md ← Your approval/rejection log
|
||||
├── scanner_presented.md ← Auto-logged: what was presented
|
||||
├── news_log.md ← Your posted stories (for dedup)
|
||||
├── last_scan_candidates.txt ← Persistent for "next 10" requests
|
||||
└── github_trending_state.json ← Star velocity tracking
|
||||
```
|
||||
|
||||
**Key integration points:**
|
||||
|
||||
1. **Scripts live in** `~/.openclaw/workspace/scripts/` — OpenClaw's standard location for agent-callable scripts
|
||||
2. **Memory files live in** `~/.openclaw/workspace/memory/` — persistent across sessions
|
||||
3. **The cron job** uses `sessionTarget: "isolated"` so each scan gets a clean session (no context contamination)
|
||||
4. **The agent model** (e.g., Kimi K2.5) orchestrates the pipeline. The actual AI curation uses Gemini Flash directly via API — so your cron model doesn't need to be expensive
|
||||
5. **Delivery** is handled by OpenClaw's channel system (Telegram, Slack, etc.)
|
||||
|
||||
**Not using OpenClaw?** The scripts work standalone too — just run `./news_scan_deduped.sh` from a regular cron job or shell. The only OpenClaw-specific parts are the cron job setup and channel delivery.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ news_scan_deduped.sh │
|
||||
│ (Main Orchestrator) │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ [1] RSS Feeds ──→ filter_ai_news.sh (25 feeds) │
|
||||
│ [2] Reddit JSON API ──→ fetch_reddit_news.py (13 subs) │
|
||||
│ [3] Twitter/X ──→ scan_twitter_ai.sh (bird CLI) │
|
||||
│ ──→ fetch_twitter_api.py (API search) │
|
||||
│ [4] GitHub ──→ github_trending.py (trending+rel) │
|
||||
│ [5] Tavily Web Search ──→ fetch_web_news.py (5 queries) │
|
||||
│ │
|
||||
│ All sources are best-effort — failures don't kill the pipeline │
|
||||
│ │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ quality_score.py → Score + dedup (80% title similarity) │
|
||||
│ Output: top 50 scored candidates │
|
||||
│ │
|
||||
│ enrich_top_articles.py → Fetch full text for top 8 articles │
|
||||
│ CF Markdown preferred, HTML fallback │
|
||||
│ │
|
||||
│ llm_editor.py → Gemini Flash editorial curation │
|
||||
│ Reads editorial_profile.md for guidance │
|
||||
│ Checks news_log.md to avoid repeats │
|
||||
│ Output: top 7 ranked picks (JSON) │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Required
|
||||
- **OpenClaw** (v2026.2.23+) — the AI agent platform that runs the cron job
|
||||
- **Python 3.9+** — all scripts use stdlib only (no pip packages)
|
||||
- **blogwatcher** — RSS feed scanner (`brew install blogwatcher` or equivalent)
|
||||
|
||||
### API Keys (set as environment variables)
|
||||
| Key | Required? | Purpose | Free Tier |
|
||||
|-----|-----------|---------|-----------|
|
||||
| `GEMINI_API_KEY` | Yes | Gemini Flash for LLM editorial curation | Google AI Studio — generous free tier |
|
||||
| `GH_TOKEN` | Recommended | GitHub API (5000 req/h vs 60/h unauthenticated) | GitHub personal access token (free) |
|
||||
| `TAVILY_API_KEY` | Optional | Tavily web search for breaking news | 1000 queries/month free |
|
||||
| `TWITTERAPI_IO_KEY` | Optional | twitterapi.io keyword search supplement | Paid (small monthly fee) |
|
||||
|
||||
### Optional Tools
|
||||
- **bird** — Twitter/X CLI tool (for `scan_twitter_ai.sh`). If not installed, the Twitter bird CLI source is skipped gracefully.
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Step 1: Copy Scripts
|
||||
|
||||
Copy all scripts from the `scripts/` directory to your OpenClaw workspace:
|
||||
|
||||
```bash
|
||||
cp scripts/*.sh scripts/*.py ~/.openclaw/workspace/scripts/
|
||||
chmod +x ~/.openclaw/workspace/scripts/news_scan_deduped.sh
|
||||
chmod +x ~/.openclaw/workspace/scripts/filter_ai_news.sh
|
||||
chmod +x ~/.openclaw/workspace/scripts/scan_twitter_ai.sh
|
||||
```
|
||||
|
||||
### Step 2: Set Up RSS Feeds (blogwatcher)
|
||||
|
||||
Install blogwatcher and add your RSS feeds. Here's a recommended starter set:
|
||||
|
||||
```bash
|
||||
# Wire services (Tier 1 — highest trust)
|
||||
blogwatcher add "Reuters Tech" "https://www.reuters.com/technology/rss"
|
||||
blogwatcher add "Axios AI" "https://api.axios.com/feed/top/technology"
|
||||
|
||||
# Tech press (Tier 2)
|
||||
blogwatcher add "TechCrunch AI" "https://techcrunch.com/category/artificial-intelligence/feed/"
|
||||
blogwatcher add "The Verge" "https://www.theverge.com/rss/ai-artificial-intelligence/index.xml"
|
||||
blogwatcher add "THE DECODER" "https://the-decoder.com/feed/"
|
||||
blogwatcher add "Ars Technica" "https://feeds.arstechnica.com/arstechnica/technology-lab"
|
||||
blogwatcher add "VentureBeat AI" "https://venturebeat.com/category/ai/feed/"
|
||||
blogwatcher add "Wired AI" "https://www.wired.com/feed/tag/ai/latest/rss"
|
||||
blogwatcher add "MIT Tech Review" "https://www.technologyreview.com/feed/"
|
||||
|
||||
# AI company blogs (Tier 1-2)
|
||||
blogwatcher add "OpenAI Blog" "https://openai.com/blog/rss.xml"
|
||||
blogwatcher add "Google AI Blog" "https://blog.google/technology/ai/rss/"
|
||||
blogwatcher add "Hugging Face Blog" "https://huggingface.co/blog/feed.xml"
|
||||
|
||||
# Bloggers & newsletters (Tier 2-3)
|
||||
blogwatcher add "Simon Willison" "https://simonwillison.net/atom/everything/"
|
||||
blogwatcher add "Bens Bites" "https://www.bensbites.com/feed"
|
||||
```
|
||||
|
||||
Adjust the `SOURCE_TIERS` dictionary in `filter_ai_news.sh` to match your feed names exactly.
|
||||
|
||||
### Step 3: Set Up Editorial Profile
|
||||
|
||||
Copy and customize the editorial profile template:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.openclaw/workspace/memory
|
||||
cp config/editorial_profile_template.md ~/.openclaw/workspace/memory/editorial_profile.md
|
||||
```
|
||||
|
||||
Edit `~/.openclaw/workspace/memory/editorial_profile.md` to reflect your channel's editorial voice:
|
||||
- What topics you always pick
|
||||
- What you usually skip
|
||||
- Your source trust ranking
|
||||
- Story selection rules
|
||||
|
||||
This profile is read by the LLM editor on every scan and directly influences story selection.
|
||||
|
||||
### Step 4: Set Environment Variables
|
||||
|
||||
Add API keys to your OpenClaw LaunchAgent plist (macOS):
|
||||
|
||||
```bash
|
||||
# Add to ~/Library/LaunchAgents/ai.openclaw.gateway.plist under EnvironmentVariables:
|
||||
# <key>GEMINI_API_KEY</key>
|
||||
# <string>your-gemini-api-key</string>
|
||||
# <key>GH_TOKEN</key>
|
||||
# <string>your-github-token</string>
|
||||
# <key>TAVILY_API_KEY</key>
|
||||
# <string>your-tavily-key</string>
|
||||
# <key>TWITTERAPI_IO_KEY</key>
|
||||
# <string>your-twitterapi-key</string>
|
||||
|
||||
# Then restart the gateway:
|
||||
launchctl kickstart -k gui/$(id -u)/ai.openclaw.gateway
|
||||
```
|
||||
|
||||
Or export them in your shell for testing:
|
||||
|
||||
```bash
|
||||
export GEMINI_API_KEY="your-key"
|
||||
export GH_TOKEN="your-token"
|
||||
export TAVILY_API_KEY="your-key"
|
||||
```
|
||||
|
||||
### Step 5: Create the Cron Job
|
||||
|
||||
Add the news scan as an OpenClaw cron job:
|
||||
|
||||
```bash
|
||||
openclaw cron add \
|
||||
--name "Bi-Hourly News Scan" \
|
||||
--cron "40 9,11,13,15,17,19,21 * * *" \
|
||||
--message "Run the Gen AI news scanner: bash ~/.openclaw/workspace/scripts/news_scan_deduped.sh" \
|
||||
--agent main \
|
||||
--model "kimi-coding/k2p5" \
|
||||
--announce \
|
||||
--channel telegram \
|
||||
--tz "America/New_York"
|
||||
```
|
||||
|
||||
**Schedule breakdown:** Runs at :40 past the hour at 9am, 11am, 1pm, 3pm, 5pm, 7pm, 9pm. Adjust the hours and timezone to match your audience.
|
||||
|
||||
**Model choice:** The cron job uses a cheap/mid-tier model (like Kimi K2.5) to orchestrate the pipeline. The actual AI curation happens via Gemini Flash API directly (called by `llm_editor.py`), so the cron model doesn't need to be expensive.
|
||||
|
||||
### Step 6: Test the Pipeline
|
||||
|
||||
Run a manual test:
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/scripts
|
||||
./news_scan_deduped.sh --top 5
|
||||
```
|
||||
|
||||
You should see output like:
|
||||
```
|
||||
═══════════════════════════════════════════════════════════
|
||||
📡 [YOUR_CHANNEL_NAME] — News Scanner v2 (top 5)
|
||||
═══════════════════════════════════════════════════════════
|
||||
|
||||
📰 [1/5] Scanning RSS feeds...
|
||||
✅ Extracted 12 new RSS articles
|
||||
🔴 [2/5] Scanning Reddit (JSON API)...
|
||||
✅ Found 45 Reddit posts (score-filtered)
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How Each Script Works
|
||||
|
||||
### 1. `news_scan_deduped.sh` — Main Orchestrator
|
||||
The master script that calls everything else in sequence. Collects articles from all 5 sources, pipes through scoring/enrichment/LLM, and formats output. All sources are best-effort — if one fails, the pipeline continues with what it has.
|
||||
|
||||
### 2. `filter_ai_news.sh` — RSS Keyword Filter
|
||||
Reads articles from blogwatcher, filters by AI-related keywords (with word-boundary matching for short keywords like "AI" to avoid false positives), assigns source tiers, and filters out Reddit noise (questions, rants, memes).
|
||||
|
||||
### 3. `fetch_reddit_news.py` — Reddit JSON API Scanner
|
||||
Fetches posts from 13 AI-related subreddits using Reddit's public JSON API (no auth needed). Features:
|
||||
- Per-subreddit score thresholds (30-50 upvotes minimum)
|
||||
- Flair filtering for noisy subs (e.g., only "News" flair from r/technology)
|
||||
- Noise filter (skips questions, rants, short titles)
|
||||
- Concurrent fetching (3 workers)
|
||||
|
||||
### 4. `scan_twitter_ai.sh` — Twitter/X bird CLI Scanner
|
||||
Scans official AI company accounts, tech reporters/leakers, and CEO accounts using the `bird` CLI tool. Three-tier account system:
|
||||
- Tier 1: Official accounts (OpenAI, Anthropic, Google, etc.)
|
||||
- Tier 2: Reporters and leakers (break news first)
|
||||
- Tier 3: CEOs (context, not breaking news)
|
||||
|
||||
### 5. `fetch_twitter_api.py` — twitterapi.io Keyword Search
|
||||
Supplements bird CLI with keyword-based search. Uses engagement filtering (50+ likes or 5000+ followers) to cut noise. Properly tags tweet-only stories (no external article URL).
|
||||
|
||||
### 6. `github_trending.py` — GitHub Trending + Releases
|
||||
Three strategies:
|
||||
- **Emerging:** Repos created in the last 7 days with 50+ stars
|
||||
- **Velocity:** Established repos (1000+ stars) gaining traction fast
|
||||
- **Releases:** New releases from 16 key AI repos (Anthropic SDK, OpenAI SDK, Ollama, etc.)
|
||||
|
||||
Maintains state between runs to calculate star velocity.
|
||||
|
||||
### 7. `fetch_web_news.py` — Tavily Web Search
|
||||
Catches breaking news that RSS feeds miss. 5 focused queries, 2-day freshness filter. Skips domains already covered by RSS (Reddit, Twitter, GitHub, YouTube, arxiv). Filters out homepage URLs.
|
||||
|
||||
### 8. `quality_score.py` — Scoring + Deduplication
|
||||
Scores every article based on:
|
||||
- Source tier (wire services get +5, tech press +3, etc.)
|
||||
- High-value keywords (acquisitions, billion, launch, security, etc.)
|
||||
- Breaking news signals (exclusive, confirmed, first look, etc.)
|
||||
- Title quality (length heuristic)
|
||||
|
||||
Deduplicates by title similarity (80% threshold using SequenceMatcher). Outputs top 50.
|
||||
|
||||
### 9. `enrich_top_articles.py` — Full Text Fetcher
|
||||
Fetches full article text for the top 8 scored articles. Tries Cloudflare Markdown for Agents first (clean markdown), falls back to HTML extraction. Skips paywalled sites. 1200 character cap per article.
|
||||
|
||||
### 10. `llm_editor.py` — LLM Editorial Curation
|
||||
The AI brain of the pipeline. Sends all scored candidates + editorial profile + recent post history to Gemini Flash. The LLM selects the top N stories, ranks them, assigns categories, and writes 1-sentence summaries.
|
||||
|
||||
Features:
|
||||
- Deterministic URL pre-filter (skips already-posted URLs before calling the LLM)
|
||||
- Editorial profile integration (learns your preferences over time)
|
||||
- Structured JSON output with validation
|
||||
- Graceful fallback to raw scoring if LLM fails
|
||||
- Logs all presented stories to `scanner_presented.md`
|
||||
|
||||
### 11. `update_editorial_profile.py` — Profile Updater
|
||||
Runs nightly. Analyzes your approval/rejection patterns and updates the editorial profile's stats section. Also identifies "blind spots" — topics you manually seek out but the scanner doesn't catch.
|
||||
|
||||
---
|
||||
|
||||
## Customization Guide
|
||||
|
||||
### Adding RSS Feeds
|
||||
1. Add the feed to blogwatcher: `blogwatcher add "Feed Name" "https://feed-url/rss"`
|
||||
2. Add the feed name to `SOURCE_TIERS` in `filter_ai_news.sh` with the appropriate tier (1-3)
|
||||
3. Add any new keywords to the `LONG_KEYWORDS` list if needed
|
||||
|
||||
### Adding Reddit Subreddits
|
||||
Edit the `SUBREDDITS` list in `fetch_reddit_news.py`:
|
||||
```python
|
||||
{"sub": "YourSubreddit", "sort": "hot", "limit": 25, "min_score": 30,
|
||||
"flairs": ["News", "Discussion"]}, # flairs are optional
|
||||
```
|
||||
|
||||
### Adding Twitter Accounts to Monitor
|
||||
Edit the account arrays in `scan_twitter_ai.sh`:
|
||||
- `OFFICIAL_ACCOUNTS` — for company accounts
|
||||
- `REPORTER_ACCOUNTS` — for journalists and leakers
|
||||
- `CEO_ACCOUNTS` — for thought leaders
|
||||
|
||||
### Adding GitHub Release Repos
|
||||
Add to the `RELEASE_REPOS` list in `github_trending.py`:
|
||||
```python
|
||||
"owner/repo-name",
|
||||
```
|
||||
|
||||
### Changing the LLM Model
|
||||
Edit `GEMINI_MODEL` in `llm_editor.py`. Any Gemini model works. Flash is recommended for cost.
|
||||
|
||||
### Adjusting Scan Frequency
|
||||
Edit the cron expression:
|
||||
```bash
|
||||
openclaw cron edit <job-id> --cron "0 */3 * * *" # every 3 hours
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
openclaw-news-scan/
|
||||
├── README.md # This file
|
||||
├── scripts/
|
||||
│ ├── news_scan_deduped.sh # Main orchestrator
|
||||
│ ├── filter_ai_news.sh # RSS keyword filter
|
||||
│ ├── fetch_reddit_news.py # Reddit JSON API
|
||||
│ ├── scan_twitter_ai.sh # Twitter bird CLI
|
||||
│ ├── fetch_twitter_api.py # twitterapi.io search
|
||||
│ ├── github_trending.py # GitHub trending + releases
|
||||
│ ├── fetch_web_news.py # Tavily web search
|
||||
│ ├── quality_score.py # Scoring + dedup
|
||||
│ ├── enrich_top_articles.py # Full text fetcher
|
||||
│ ├── llm_editor.py # LLM editorial curation
|
||||
│ └── update_editorial_profile.py # Editorial profile updater
|
||||
└── config/
|
||||
└── editorial_profile_template.md # Template — customize for your channel
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Flow Summary
|
||||
|
||||
```
|
||||
RSS (25 feeds) ─────────┐
|
||||
Reddit (13 subs) ───────┤
|
||||
Twitter (bird + API) ───┤──→ quality_score.py ──→ enrich_top_articles.py ──→ llm_editor.py ──→ Output
|
||||
GitHub (trending+rel) ──┤ (max 50) (max 8) (Gemini Flash)
|
||||
Tavily (5 queries) ─────┘
|
||||
```
|
||||
|
||||
**Typical run:** ~100 raw articles → 50 scored → 8 enriched → 5-7 curated picks
|
||||
|
||||
---
|
||||
|
||||
## Cost Breakdown
|
||||
|
||||
| Component | Monthly Cost | Notes |
|
||||
|-----------|-------------|-------|
|
||||
| Gemini Flash API | ~$2-3/month | ~7 calls/day, ~30K tokens each |
|
||||
| Tavily API | Free | 1000 queries/month free tier covers it |
|
||||
| GitHub API | Free | Personal access token, 5000 req/h |
|
||||
| twitterapi.io | ~$10/month | Optional — bird CLI is free |
|
||||
| OpenClaw cron model | Varies | Depends on your model choice |
|
||||
| **Total** | **~$5/month** | Without twitterapi.io |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Issue | Fix |
|
||||
|-------|-----|
|
||||
| "GEMINI_API_KEY not set" | Add to LaunchAgent plist or export in shell |
|
||||
| Reddit 429 (rate limit) | Normal with 2h spacing. Reduce subreddits or increase --hours |
|
||||
| Reddit 404 on a sub | Sub may be private/quarantined. Remove from config. |
|
||||
| bird CLI not found | Install bird or remove scan_twitter_ai.sh call |
|
||||
| "No new stories found" | RSS feeds may all be read. Wait for new articles. |
|
||||
| LLM editor timeout | Increase TIMEOUT_SEC in llm_editor.py |
|
||||
| Pipeline takes too long | Increase cron timeout: `openclaw cron edit <id> --timeout 120` |
|
||||
| GitHub rate limit | Set GH_TOKEN env var for 5000 req/h (vs 60/h) |
|
||||
| Duplicate stories | Adjust --dedup-threshold in quality_score.py (default 0.80) |
|
||||
|
||||
---
|
||||
|
||||
## Learning & Feedback Loop
|
||||
|
||||
The system learns from your editorial decisions:
|
||||
|
||||
1. **During the day:** The scanner presents picks. You approve or skip them.
|
||||
2. **At night:** `update_editorial_profile.py` analyzes your patterns.
|
||||
3. **Next scan:** The LLM editor reads the updated profile and adjusts.
|
||||
|
||||
To log decisions, create `~/.openclaw/workspace/memory/editorial_decisions.md`:
|
||||
```
|
||||
[2026-03-01T10:00:00-05:00] APPROVED | Story Title Here | https://url | category
|
||||
[2026-03-01T10:00:00-05:00] SKIPPED | Another Story | https://url | category
|
||||
[2026-03-01T14:00:00-05:00] MANUAL_DRAFT | Story I Found Myself | https://url | category
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Credits
|
||||
|
||||
Built by [Jacob Ben David](https://github.com/jacob-bd) with [OpenClaw](https://github.com/openclaw/openclaw), Gemini Flash, and a collection of free/low-cost APIs.
|
||||
Inspired by the `tech-news-digest` ClawHub skill (v3.14.0 by dinstein).
|
||||
|
||||
## License
|
||||
|
||||
MIT — use it however you want. If you build something cool with it, let me know!
|
||||
@@ -0,0 +1,76 @@
|
||||
# Editorial Profile — [YOUR_CHANNEL_NAME]
|
||||
|
||||
> This profile is read by the AI editor (llm_editor.py) on every news scan.
|
||||
> It captures what you pick, what you skip, and what makes a story worth posting.
|
||||
> The "Approval History Stats" section is updated automatically by
|
||||
> update_editorial_profile.py based on your approval/rejection decisions.
|
||||
|
||||
## Identity
|
||||
- Channel: [YOUR_CHANNEL_NAME] on [Platform]
|
||||
- Editor: [Your Name]
|
||||
- Voice: [Describe your editorial voice — e.g., "Sharp, concise, no fluff. Breaking news > opinion. Facts > speculation."]
|
||||
|
||||
## What You Always Pick (high confidence)
|
||||
<!-- Stories in these categories almost always get posted. -->
|
||||
<!-- Examples — customize to YOUR interests: -->
|
||||
- Major AI company announcements (product launches, acquisitions, partnerships)
|
||||
- New model/architecture releases (especially novel approaches and benchmarks)
|
||||
- AI security incidents (hacks, prompt injection, model attacks)
|
||||
- AI + geopolitics (military applications, government regulation, trade policy)
|
||||
- Open-source model releases that challenge frontier models
|
||||
- Major funding rounds and M&A deals over $100M
|
||||
|
||||
## What You Usually Pick (medium confidence)
|
||||
<!-- Sometimes posted, depends on the angle and timing. -->
|
||||
- NVIDIA and chip industry news (when market-moving)
|
||||
- Google/Apple AI product launches
|
||||
- Creative AI tools (image/video/music generation)
|
||||
- Major partnerships between tech companies
|
||||
- Original research reports with novel data points
|
||||
|
||||
## What You Usually Skip (anti-patterns)
|
||||
<!-- The LLM editor will learn to avoid these over time. -->
|
||||
- Enterprise SaaS funding rounds under $50M
|
||||
- Generic "AI will change everything" opinion pieces
|
||||
- Routine product updates without a unique angle
|
||||
- Earnings reports (unless market-moving)
|
||||
- Crypto/blockchain/NFT/web3 crossover stories
|
||||
- Routine job market news (hiring, small layoffs)
|
||||
- Conference/event announcements
|
||||
- Podcast/interview promotions
|
||||
|
||||
## Emerging Interests (watch for these)
|
||||
<!-- Topics you're starting to pay attention to. -->
|
||||
- GitHub repos gaining rapid traction (star velocity signals)
|
||||
- Tools/frameworks before they become mainstream
|
||||
- AI regulation and policy shifts
|
||||
- [Add your emerging interests here]
|
||||
|
||||
## Source Trust Ranking
|
||||
<!-- Higher-tier sources get priority in scoring. -->
|
||||
<!-- Customize to match YOUR blogwatcher feed names. -->
|
||||
- Tier 1 (Wire): Bloomberg, Reuters, CNBC, Axios, Politico
|
||||
- Tier 2 (Tech Press): TechCrunch, The Verge, Ars Technica, Wired, The Decoder, 404 Media
|
||||
- Tier 3 (Aggregator): VentureBeat, SiliconANGLE, 9to5Google, Crunchbase News
|
||||
- Tier 4 (Community): Reddit (r/singularity, r/ClaudeAI, r/LocalLLaMA), Hacker News
|
||||
- Tier 5 (Primary): Company blogs, GitHub repos, research papers
|
||||
|
||||
## Story Selection Rules
|
||||
1. Every scan MUST produce at least 5 stories (broaden scope if needed)
|
||||
2. No exact duplicates of previously posted stories
|
||||
3. Same event from different angles = OK if the angle is genuinely new
|
||||
4. Prefer concrete news (X acquired Y, X launched Z) over speculation
|
||||
5. Prefer exclusives and scoops over recycled takes
|
||||
6. Max 2 stories from the same source per scan (diversity)
|
||||
7. Include 1 sentence summary explaining WHY this story matters
|
||||
8. If a GitHub repo is trending AND relevant, include it
|
||||
|
||||
## Approval History Stats
|
||||
<!-- This section is auto-populated by update_editorial_profile.py -->
|
||||
<!-- It analyzes your approval/rejection patterns and updates nightly -->
|
||||
- No decisions logged yet.
|
||||
- Tracking begins when you approve/skip stories in editorial_decisions.md.
|
||||
|
||||
## Scanner Blind Spots
|
||||
<!-- Auto-detected: topics you manually seek out but the scanner misses -->
|
||||
<!-- Populated by update_editorial_profile.py after enough decisions -->
|
||||
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Enrich top articles with full text content for better LLM curation.
|
||||
|
||||
Reads pipe-delimited articles, fetches full text for the top N articles
|
||||
using Cloudflare Markdown for Agents (preferred) or HTML extraction (fallback).
|
||||
|
||||
Appends full text as a pipe field: TITLE|URL|SOURCE|TIER|FULLTEXT:text
|
||||
|
||||
Usage:
|
||||
python3 enrich_top_articles.py --input articles.txt [--max 10] [--max-chars 1500]
|
||||
"""
|
||||
|
||||
import re
|
||||
import sys
|
||||
import argparse
|
||||
import ssl
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from html.parser import HTMLParser
|
||||
from urllib.request import Request, urlopen
|
||||
from urllib.error import HTTPError, URLError
|
||||
from urllib.parse import urlparse
|
||||
|
||||
_SSL_CTX = ssl.create_default_context()
|
||||
|
||||
TIMEOUT = 8
|
||||
MAX_WORKERS = 4
|
||||
USER_AGENT = "NewsScanner/1.0 (article enrichment)"
|
||||
|
||||
# Domains to skip enrichment (paywalled, JS-heavy, or not articles)
|
||||
SKIP_DOMAINS = {
|
||||
"twitter.com", "x.com",
|
||||
"reddit.com", "old.reddit.com",
|
||||
"github.com",
|
||||
"youtube.com", "youtu.be",
|
||||
"nytimes.com", "bloomberg.com", "wsj.com", "ft.com",
|
||||
"arxiv.org",
|
||||
}
|
||||
|
||||
|
||||
class TextExtractor(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._text = []
|
||||
self._skip = False
|
||||
self._skip_tags = {"script", "style", "nav", "footer", "header", "aside", "noscript"}
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
if tag in self._skip_tags:
|
||||
self._skip = True
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
if tag in self._skip_tags:
|
||||
self._skip = False
|
||||
if tag in ("p", "br", "div", "h1", "h2", "h3", "li"):
|
||||
self._text.append("\n")
|
||||
|
||||
def handle_data(self, data):
|
||||
if not self._skip:
|
||||
self._text.append(data)
|
||||
|
||||
def get_text(self):
|
||||
raw = "".join(self._text)
|
||||
raw = re.sub(r"[ \t]+", " ", raw)
|
||||
raw = re.sub(r"\n{3,}", "\n\n", raw)
|
||||
return raw.strip()
|
||||
|
||||
|
||||
def fetch_full_text(url, max_chars=1500):
|
||||
"""Fetch article full text via CF Markdown or HTML extraction."""
|
||||
domain = urlparse(url).netloc.lower().lstrip("www.")
|
||||
if domain in SKIP_DOMAINS:
|
||||
return ""
|
||||
|
||||
try:
|
||||
req = Request(url, headers={
|
||||
"Accept": "text/markdown, text/html;q=0.9",
|
||||
"User-Agent": USER_AGENT,
|
||||
})
|
||||
with urlopen(req, timeout=TIMEOUT, context=_SSL_CTX) as resp:
|
||||
content_type = resp.headers.get("Content-Type", "")
|
||||
raw = resp.read()
|
||||
|
||||
if raw[:2] == b"\x1f\x8b":
|
||||
import gzip
|
||||
raw = gzip.decompress(raw)
|
||||
|
||||
text = raw.decode("utf-8", errors="replace")
|
||||
|
||||
if "text/markdown" in content_type:
|
||||
return text[:max_chars]
|
||||
|
||||
article_match = re.search(r"<article[^>]*>(.*?)</article>", text, re.DOTALL | re.IGNORECASE)
|
||||
fragment = article_match.group(1) if article_match else text
|
||||
extractor = TextExtractor()
|
||||
try:
|
||||
extractor.feed(fragment)
|
||||
except Exception:
|
||||
return ""
|
||||
extracted = extractor.get_text()
|
||||
if len(extracted) < 80:
|
||||
return ""
|
||||
return extracted[:max_chars]
|
||||
|
||||
except (HTTPError, URLError, OSError):
|
||||
return ""
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Enrich top articles with full text")
|
||||
parser.add_argument('--input', '-i', required=True, help='Input pipe-delimited file')
|
||||
parser.add_argument('--max', type=int, default=10, help='Max articles to enrich (default: 10)')
|
||||
parser.add_argument('--max-chars', type=int, default=1500, help='Max chars per article (default: 1500)')
|
||||
args = parser.parse_args()
|
||||
|
||||
articles = []
|
||||
try:
|
||||
with open(args.input, 'r') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
articles.append(line)
|
||||
except FileNotFoundError:
|
||||
print(f"Error: file not found: {args.input}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if not articles:
|
||||
print("No articles to enrich", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
to_enrich = articles[:args.max]
|
||||
pass_through = articles[args.max:]
|
||||
|
||||
results = {}
|
||||
enriched_count = 0
|
||||
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool:
|
||||
futures = {}
|
||||
for i, line in enumerate(to_enrich):
|
||||
parts = line.split('|')
|
||||
if len(parts) >= 2:
|
||||
url = parts[1]
|
||||
futures[pool.submit(fetch_full_text, url, args.max_chars)] = i
|
||||
|
||||
for future in as_completed(futures):
|
||||
idx = futures[future]
|
||||
text = future.result()
|
||||
if text:
|
||||
results[idx] = text
|
||||
enriched_count += 1
|
||||
|
||||
for i, line in enumerate(to_enrich):
|
||||
if i in results:
|
||||
clean_text = results[i].replace('|', ' ').replace('\n', ' ').strip()
|
||||
clean_text = re.sub(r'\s+', ' ', clean_text)
|
||||
print(f"{line}|FULLTEXT:{clean_text[:args.max_chars]}")
|
||||
else:
|
||||
print(line)
|
||||
|
||||
for line in pass_through:
|
||||
print(line)
|
||||
|
||||
print(f" Done: {enriched_count}/{len(to_enrich)} articles enriched", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,251 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Fetch Reddit posts via JSON API with score filtering and noise reduction.
|
||||
|
||||
Replaces blogwatcher RSS for Reddit. Uses Reddit's public JSON API
|
||||
(no authentication required). Outputs pipe-delimited TITLE|URL|SOURCE format.
|
||||
|
||||
Usage:
|
||||
python3 fetch_reddit_news.py [--hours 24] [--min-score 20]
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import ssl
|
||||
import sys
|
||||
import time
|
||||
import argparse
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from urllib.request import Request, urlopen
|
||||
from urllib.error import HTTPError, URLError
|
||||
|
||||
_SSL_CTX = ssl.create_default_context()
|
||||
|
||||
# ── Configuration ────────────────────────────────────────────────────
|
||||
TIMEOUT = 20
|
||||
MAX_WORKERS = 3
|
||||
RETRY_COUNT = 1
|
||||
RETRY_DELAY = 3
|
||||
USER_AGENT = "NewsScanner/1.0 (bot; reddit-scanner)"
|
||||
|
||||
# Subreddit configs: dict with optional "flairs" list for flair filtering.
|
||||
# When "flairs" is set, only posts matching those flairs are included
|
||||
# (case-insensitive substring match on link_flair_text).
|
||||
SUBREDDITS = [
|
||||
# AI-focused subs — no flair filter needed (already on-topic)
|
||||
{"sub": "LocalLLaMA", "sort": "hot", "limit": 25, "min_score": 30},
|
||||
{"sub": "singularity", "sort": "hot", "limit": 25, "min_score": 50},
|
||||
{"sub": "ChatGPT", "sort": "hot", "limit": 25, "min_score": 50},
|
||||
{"sub": "Anthropic", "sort": "hot", "limit": 25, "min_score": 30},
|
||||
|
||||
# Flair-filtered subs — use flairs to cut noise, lower score threshold
|
||||
{"sub": "MachineLearning", "sort": "hot", "limit": 25, "min_score": 30,
|
||||
"flairs": ["[N]", "[R]", "[P]"]},
|
||||
{"sub": "technology", "sort": "hot", "limit": 25, "min_score": 30,
|
||||
"flairs": ["AI", "Artificial Intelligence"]},
|
||||
{"sub": "OpenAI", "sort": "hot", "limit": 25, "min_score": 30,
|
||||
"flairs": ["News"]},
|
||||
{"sub": "artificial", "sort": "hot", "limit": 25, "min_score": 20,
|
||||
"flairs": ["News"]},
|
||||
{"sub": "ClaudeAI", "sort": "hot", "limit": 25, "min_score": 20,
|
||||
"flairs": ["News"]},
|
||||
|
||||
# New subs — added via flair filtering (too noisy without)
|
||||
{"sub": "Futurology", "sort": "hot", "limit": 25, "min_score": 30,
|
||||
"flairs": ["AI", "Artificial Intelligence", "Robotics/Automation"]},
|
||||
{"sub": "ArtificialIntelligence", "sort": "hot", "limit": 25, "min_score": 20,
|
||||
"flairs": ["News"]},
|
||||
{"sub": "Bard", "sort": "hot", "limit": 25, "min_score": 20,
|
||||
"flairs": ["News"]},
|
||||
{"sub": "GeminiAI", "sort": "hot", "limit": 25, "min_score": 20,
|
||||
"flairs": ["News"]},
|
||||
]
|
||||
|
||||
# Reddit noise filter — skip questions, rants, memes
|
||||
NOISE_START = re.compile(
|
||||
r'^(Why|How|What|Can|Does|Is|Has|Are|Do|Should|Would|Could|Anyone|'
|
||||
r'Help|Rant|Vent|Am I|ELI5|CMV|PSA|Unpopular|Hot take|DAE|TIL|'
|
||||
r'Gah|Kindly explain|Seriously|From Frustration|Gemini Memory|'
|
||||
r'I just|I don.t|My experience|Thank you|Appreciation|Shoutout|'
|
||||
r'Just deleted|Thanks to everyone|I.m happy to report|'
|
||||
r'Overtaken!|F that|RIP|Goodbye)',
|
||||
re.IGNORECASE
|
||||
)
|
||||
|
||||
# AI relevance keywords (must match at least one)
|
||||
SHORT_KW = re.compile(r'\b(AI|AGI|LLM|GPU|TPU|RAG)\b', re.IGNORECASE)
|
||||
LONG_KW = re.compile(
|
||||
r'artificial intelligence|machine learning|deep learning|language model|'
|
||||
r'GPT|Claude|Gemini|ChatGPT|OpenAI|Anthropic|Google AI|DeepMind|'
|
||||
r'agentic|neural network|transformer|diffusion|generative AI|gen AI|'
|
||||
r'Llama|Mistral|Hugging Face|inference|training|fine-tuning|'
|
||||
r'open.source|NVIDIA|DeepSeek|Grok|xAI|Qwen|Codex|Copilot|'
|
||||
r'Meta AI|Cohere|Perplexity|multimodal|reasoning model|'
|
||||
r'acquisition|funding|valuation|launch|release|benchmark',
|
||||
re.IGNORECASE
|
||||
)
|
||||
|
||||
|
||||
def is_noise(title):
|
||||
"""Return True if title looks like Reddit noise (questions, rants, etc)."""
|
||||
t = title.strip()
|
||||
if NOISE_START.match(t):
|
||||
return True
|
||||
if t.endswith('?'):
|
||||
return True
|
||||
if len(t) < 20:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def is_ai_relevant(title):
|
||||
"""Return True if title contains AI-related keywords."""
|
||||
return bool(SHORT_KW.search(title) or LONG_KW.search(title))
|
||||
|
||||
|
||||
def flair_matches(post_flair, allowed_flairs):
|
||||
"""Check if a post's flair matches any in the allowed list (case-insensitive)."""
|
||||
if not post_flair:
|
||||
return False
|
||||
pf = post_flair.lower().strip()
|
||||
for af in allowed_flairs:
|
||||
if af.lower() in pf:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def fetch_subreddit(subreddit, sort, limit, min_score, cutoff, flairs=None):
|
||||
"""Fetch posts from a single subreddit. If flairs is set, only matching posts."""
|
||||
url = f"https://www.reddit.com/r/{subreddit}/{sort}.json?limit={limit}&raw_json=1"
|
||||
|
||||
for attempt in range(RETRY_COUNT + 1):
|
||||
try:
|
||||
req = Request(url, headers={
|
||||
'User-Agent': USER_AGENT,
|
||||
'Accept': 'text/html,application/json',
|
||||
})
|
||||
with urlopen(req, timeout=TIMEOUT, context=_SSL_CTX) as resp:
|
||||
data = json.loads(resp.read().decode('utf-8'))
|
||||
|
||||
posts = []
|
||||
for child in data.get('data', {}).get('children', []):
|
||||
post = child.get('data', {})
|
||||
if not post:
|
||||
continue
|
||||
|
||||
created_utc = post.get('created_utc', 0)
|
||||
post_time = datetime.fromtimestamp(created_utc, tz=timezone.utc)
|
||||
if post_time < cutoff:
|
||||
continue
|
||||
|
||||
score = post.get('score', 0)
|
||||
if score < min_score:
|
||||
continue
|
||||
|
||||
if post.get('stickied', False):
|
||||
continue
|
||||
|
||||
title = post.get('title', '').strip()
|
||||
if not title:
|
||||
continue
|
||||
|
||||
if flairs:
|
||||
post_flair = post.get('link_flair_text', '')
|
||||
if not flair_matches(post_flair, flairs):
|
||||
continue
|
||||
|
||||
if is_noise(title):
|
||||
continue
|
||||
|
||||
ai_focused = subreddit.lower() in {
|
||||
'localllama', 'machinelearning', 'chatgpt', 'openai',
|
||||
'artificial', 'anthropic', 'claudeai', 'singularity',
|
||||
'artificialintelligence', 'bard', 'geminiai',
|
||||
}
|
||||
if not ai_focused and not is_ai_relevant(title):
|
||||
continue
|
||||
|
||||
permalink = f"https://www.reddit.com{post.get('permalink', '')}"
|
||||
external_url = post.get('url', '')
|
||||
is_self = post.get('is_self', True)
|
||||
|
||||
if is_self or 'reddit.com' in external_url or 'redd.it' in external_url:
|
||||
link = permalink
|
||||
else:
|
||||
link = external_url
|
||||
|
||||
title_clean = title.replace('|', ' -')
|
||||
num_comments = post.get('num_comments', 0)
|
||||
|
||||
posts.append({
|
||||
'title': title_clean,
|
||||
'url': link,
|
||||
'source': f"r/{subreddit}",
|
||||
'score': score,
|
||||
'comments': num_comments,
|
||||
})
|
||||
|
||||
return posts
|
||||
|
||||
except HTTPError as e:
|
||||
if e.code == 429 and attempt < RETRY_COUNT:
|
||||
time.sleep(10)
|
||||
continue
|
||||
elif e.code == 403:
|
||||
print(f" Warning: r/{subreddit} is private/quarantined", file=sys.stderr)
|
||||
return []
|
||||
print(f" Warning: r/{subreddit}: HTTP {e.code}", file=sys.stderr)
|
||||
except (URLError, OSError) as e:
|
||||
print(f" Warning: r/{subreddit}: network error", file=sys.stderr)
|
||||
except Exception as e:
|
||||
print(f" Warning: r/{subreddit}: {e}", file=sys.stderr)
|
||||
|
||||
if attempt < RETRY_COUNT:
|
||||
time.sleep(RETRY_DELAY)
|
||||
|
||||
return []
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Fetch Reddit posts via JSON API")
|
||||
parser.add_argument('--hours', type=int, default=24, help='Hours lookback (default: 24)')
|
||||
parser.add_argument('--min-score', type=int, default=0, help='Override min score for all subs')
|
||||
args = parser.parse_args()
|
||||
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(hours=args.hours)
|
||||
|
||||
all_posts = []
|
||||
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool:
|
||||
futures = {}
|
||||
for cfg in SUBREDDITS:
|
||||
sub = cfg["sub"]
|
||||
sort = cfg.get("sort", "hot")
|
||||
limit = cfg.get("limit", 25)
|
||||
min_score = cfg.get("min_score", 30)
|
||||
flairs = cfg.get("flairs", None)
|
||||
effective_min = args.min_score if args.min_score > 0 else min_score
|
||||
future = pool.submit(fetch_subreddit, sub, sort, limit, effective_min, cutoff, flairs)
|
||||
futures[future] = sub
|
||||
|
||||
for future in as_completed(futures):
|
||||
posts = future.result()
|
||||
all_posts.extend(posts)
|
||||
|
||||
all_posts.sort(key=lambda x: -x['score'])
|
||||
|
||||
seen_urls = set()
|
||||
unique_posts = []
|
||||
for post in all_posts:
|
||||
if post['url'] not in seen_urls:
|
||||
seen_urls.add(post['url'])
|
||||
unique_posts.append(post)
|
||||
|
||||
for post in unique_posts:
|
||||
print(f"{post['title']}|{post['url']}|{post['source']}")
|
||||
|
||||
print(f" Done: {len(unique_posts)} posts from {len(SUBREDDITS)} subreddits", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Fetch AI tweets via twitterapi.io as a supplement to bird CLI.
|
||||
|
||||
Uses twitterapi.io's search endpoint for keyword-based Twitter searches.
|
||||
This catches breaking news that might be missed by the bird CLI's
|
||||
account-based approach.
|
||||
|
||||
Output: pipe-delimited TITLE|URL|SOURCE format.
|
||||
Sources tagged with "(tweet)" when no external article URL exists.
|
||||
|
||||
Usage:
|
||||
python3 fetch_twitter_api.py [--max-queries 3]
|
||||
|
||||
Environment:
|
||||
TWITTERAPI_IO_KEY — required
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import ssl
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
from urllib.request import Request, urlopen
|
||||
from urllib.error import HTTPError, URLError
|
||||
|
||||
_SSL_CTX = ssl.create_default_context()
|
||||
|
||||
TIMEOUT = 15
|
||||
API_BASE = "https://api.twitterapi.io/twitter"
|
||||
|
||||
# Search queries — focused on breaking AI news
|
||||
SEARCH_QUERIES = [
|
||||
'"breaking" (AI OR "artificial intelligence" OR LLM) -is:retweet lang:en',
|
||||
'(Anthropic OR Claude OR OpenAI) (announce OR launch OR release) -is:retweet lang:en',
|
||||
'(AI OR "artificial intelligence") (acquisition OR merger OR billion OR deal) -is:retweet lang:en',
|
||||
]
|
||||
|
||||
# Minimum engagement to filter noise
|
||||
MIN_LIKES = 50
|
||||
MIN_FOLLOWERS = 5000
|
||||
|
||||
|
||||
def search_twitter(query, api_key, max_results=10):
|
||||
"""Search tweets via twitterapi.io."""
|
||||
from urllib.parse import urlencode
|
||||
|
||||
params = urlencode({
|
||||
"query": query,
|
||||
"queryType": "Latest",
|
||||
})
|
||||
url = f"{API_BASE}/tweet/advanced_search?{params}"
|
||||
|
||||
req = Request(url, headers={
|
||||
"X-API-Key": api_key,
|
||||
"User-Agent": "NewsScanner/1.0",
|
||||
})
|
||||
|
||||
try:
|
||||
with urlopen(req, timeout=TIMEOUT, context=_SSL_CTX) as resp:
|
||||
data = json.loads(resp.read().decode('utf-8'))
|
||||
return data.get("tweets", [])
|
||||
except HTTPError as e:
|
||||
if e.code == 401:
|
||||
print(" Error: Invalid TWITTERAPI_IO_KEY", file=sys.stderr)
|
||||
elif e.code == 429:
|
||||
print(" Warning: twitterapi.io rate limit", file=sys.stderr)
|
||||
else:
|
||||
print(f" Warning: twitterapi.io HTTP {e.code}", file=sys.stderr)
|
||||
return []
|
||||
except Exception as e:
|
||||
print(f" Warning: twitterapi.io error: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
|
||||
def extract_url_from_tweet(tweet):
|
||||
"""
|
||||
Extract the first external URL from a tweet.
|
||||
Returns (url, is_tweet_only):
|
||||
- (external_url, False) if an article URL was found in entities
|
||||
- (tweet_url, True) if only the tweet's own URL is available
|
||||
- ("", True) if no URL could be constructed
|
||||
"""
|
||||
entities = tweet.get("entities", {})
|
||||
urls = entities.get("urls", [])
|
||||
for u in urls:
|
||||
expanded = u.get("expanded_url", u.get("url", ""))
|
||||
if expanded and "twitter.com" not in expanded and "t.co" not in expanded and "x.com" not in expanded:
|
||||
return expanded, False
|
||||
|
||||
author = tweet.get("author", {})
|
||||
screen_name = author.get("userName", "")
|
||||
tweet_id = tweet.get("id", "")
|
||||
if screen_name and tweet_id:
|
||||
return f"https://x.com/{screen_name}/status/{tweet_id}", True
|
||||
return "", True
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Fetch AI tweets via twitterapi.io")
|
||||
parser.add_argument('--max-queries', type=int, default=3,
|
||||
help='Max queries to run (default: 3)')
|
||||
args = parser.parse_args()
|
||||
|
||||
api_key = os.environ.get("TWITTERAPI_IO_KEY", "")
|
||||
if not api_key:
|
||||
print(" Warning: TWITTERAPI_IO_KEY not set, skipping", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
seen_urls = set()
|
||||
all_results = []
|
||||
|
||||
queries = SEARCH_QUERIES[:args.max_queries]
|
||||
|
||||
for query in queries:
|
||||
tweets = search_twitter(query, api_key)
|
||||
for tweet in tweets:
|
||||
likes = tweet.get("likeCount", 0)
|
||||
author = tweet.get("author", {})
|
||||
followers = author.get("followers", 0)
|
||||
|
||||
if likes < MIN_LIKES and followers < MIN_FOLLOWERS:
|
||||
continue
|
||||
|
||||
text = tweet.get("text", "").strip()
|
||||
if not text:
|
||||
continue
|
||||
|
||||
title = re.sub(r'https?://\S+', '', text).strip()
|
||||
title = title.replace('\n', ' ').replace('|', ' -')
|
||||
title = re.sub(r'\s+', ' ', title)
|
||||
|
||||
if len(title) > 200:
|
||||
title = title[:197] + "..."
|
||||
if len(title) < 15:
|
||||
continue
|
||||
|
||||
url, is_tweet_only = extract_url_from_tweet(tweet)
|
||||
if not url or url in seen_urls:
|
||||
continue
|
||||
seen_urls.add(url)
|
||||
|
||||
screen_name = author.get("userName", "unknown")
|
||||
source_tag = f"X/@{screen_name} (tweet)" if is_tweet_only else f"X/@{screen_name}"
|
||||
all_results.append(f"{title}|{url}|{source_tag}")
|
||||
|
||||
for line in all_results:
|
||||
print(line)
|
||||
|
||||
print(f" Done: {len(all_results)} tweets from {len(queries)} queries", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Fetch AI news via Tavily web search API.
|
||||
|
||||
Uses Tavily's search API (free tier: 1000 queries/month) to find
|
||||
breaking AI news that RSS feeds might miss.
|
||||
|
||||
Output: pipe-delimited TITLE|URL|SOURCE format.
|
||||
|
||||
Usage:
|
||||
python3 fetch_web_news.py [--max-queries 5] [--max-results 5]
|
||||
|
||||
Environment:
|
||||
TAVILY_API_KEY — required
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import argparse
|
||||
import ssl
|
||||
from datetime import datetime, timezone
|
||||
from urllib.request import Request, urlopen
|
||||
from urllib.error import HTTPError, URLError
|
||||
|
||||
_SSL_CTX = ssl.create_default_context()
|
||||
|
||||
TIMEOUT = 15
|
||||
TAVILY_API = "https://api.tavily.com/search"
|
||||
|
||||
# Focused search queries — customize for your editorial focus
|
||||
SEARCH_QUERIES = [
|
||||
"AI artificial intelligence breaking news today",
|
||||
"Anthropic Claude OpenAI latest announcement",
|
||||
"AI acquisition merger funding billion",
|
||||
"AI model release launch new",
|
||||
"AI regulation government policy",
|
||||
]
|
||||
|
||||
# Domains to skip (already covered by RSS feeds)
|
||||
SKIP_DOMAINS = {
|
||||
"reddit.com", "twitter.com", "x.com", "youtube.com",
|
||||
"github.com", "arxiv.org",
|
||||
}
|
||||
|
||||
|
||||
def search_tavily(query, api_key, max_results=5):
|
||||
"""Execute a Tavily search and return results."""
|
||||
payload = json.dumps({
|
||||
"query": query,
|
||||
"search_depth": "basic",
|
||||
"max_results": max_results,
|
||||
"include_answer": False,
|
||||
"include_raw_content": False,
|
||||
"days": 2,
|
||||
}).encode('utf-8')
|
||||
|
||||
req = Request(TAVILY_API, data=payload, headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"User-Agent": "NewsScanner/1.0",
|
||||
})
|
||||
|
||||
try:
|
||||
with urlopen(req, timeout=TIMEOUT, context=_SSL_CTX) as resp:
|
||||
data = json.loads(resp.read().decode('utf-8'))
|
||||
return data.get("results", [])
|
||||
except HTTPError as e:
|
||||
if e.code == 401:
|
||||
print(" Error: Invalid TAVILY_API_KEY", file=sys.stderr)
|
||||
elif e.code == 429:
|
||||
print(" Warning: Tavily rate limit reached", file=sys.stderr)
|
||||
else:
|
||||
print(f" Warning: Tavily HTTP {e.code}", file=sys.stderr)
|
||||
return []
|
||||
except Exception as e:
|
||||
print(f" Warning: Tavily error: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
|
||||
def get_domain(url):
|
||||
"""Extract domain from URL."""
|
||||
try:
|
||||
from urllib.parse import urlparse
|
||||
return urlparse(url).netloc.lower().lstrip("www.")
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Fetch AI news via Tavily search")
|
||||
parser.add_argument('--max-queries', type=int, default=3,
|
||||
help='Max search queries to run (default: 3, saves API quota)')
|
||||
parser.add_argument('--max-results', type=int, default=5,
|
||||
help='Results per query (default: 5)')
|
||||
args = parser.parse_args()
|
||||
|
||||
api_key = os.environ.get("TAVILY_API_KEY", "")
|
||||
if not api_key:
|
||||
print(" Warning: TAVILY_API_KEY not set, skipping web search", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
seen_urls = set()
|
||||
all_results = []
|
||||
|
||||
queries = SEARCH_QUERIES[:args.max_queries]
|
||||
|
||||
for query in queries:
|
||||
results = search_tavily(query, api_key, args.max_results)
|
||||
for r in results:
|
||||
url = r.get("url", "")
|
||||
title = r.get("title", "").strip()
|
||||
if not url or not title:
|
||||
continue
|
||||
|
||||
domain = get_domain(url)
|
||||
if domain in SKIP_DOMAINS:
|
||||
continue
|
||||
if url in seen_urls:
|
||||
continue
|
||||
from urllib.parse import urlparse
|
||||
path = urlparse(url).path.rstrip('/')
|
||||
if not path or path in ('/technology', '/tech', '/ai', '/tech/ai'):
|
||||
continue
|
||||
seen_urls.add(url)
|
||||
|
||||
title_clean = title.replace('|', ' -')
|
||||
source = f"Tavily/{domain}" if domain else "Tavily/Web"
|
||||
|
||||
all_results.append(f"{title_clean}|{url}|{source}")
|
||||
|
||||
for line in all_results:
|
||||
print(line)
|
||||
|
||||
print(f" Done: {len(all_results)} articles from {len(queries)} queries", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+135
@@ -0,0 +1,135 @@
|
||||
#!/bin/bash
|
||||
# AI News Filter for the News Scan Pipeline
|
||||
# Entry-based filtering with source tiers and Reddit noise filter
|
||||
# Output: TITLE|URL|SOURCE|TIER (pipe-delimited, sorted by tier)
|
||||
# Requires: blogwatcher CLI
|
||||
|
||||
python3 << 'PYEOF'
|
||||
import subprocess, sys, re
|
||||
|
||||
result = subprocess.run(
|
||||
['/usr/local/bin/blogwatcher', 'articles'],
|
||||
capture_output=True, text=True, timeout=90
|
||||
)
|
||||
raw = result.stdout
|
||||
|
||||
# ── Keywords ─────────────────────────────────────────────────────────
|
||||
# Short keywords (<=3 chars) need word boundaries to prevent substring matches
|
||||
# e.g., "AI" must not match "affairs", "explain", "maintain"
|
||||
SHORT_KEYWORDS = ['AI', 'AGI', 'LLM', 'GPU', 'TPU', 'RAG', 'API']
|
||||
|
||||
LONG_KEYWORDS = [
|
||||
'artificial intelligence', 'machine learning', 'deep learning',
|
||||
'language model', 'GPT', 'Claude', 'Gemini', 'ChatGPT',
|
||||
'OpenAI', 'Anthropic', 'Google AI', 'Microsoft AI', 'DeepMind',
|
||||
'agentic', 'agent', 'neural network', 'transformer', 'diffusion',
|
||||
'generative AI', 'gen AI', 'reasoning model', 'multimodal',
|
||||
'vision model', 'text-to-image', 'text-to-video', 'Sora', 'DALL-E',
|
||||
'Stable Diffusion', 'Midjourney', 'Llama', 'Mistral', 'Hugging Face',
|
||||
'inference', 'training', 'fine-tuning', 'embedding', 'vector',
|
||||
'context window', 'benchmark', 'open source', 'open-source',
|
||||
'robotics', 'autonomous', 'chip', 'NVIDIA',
|
||||
'acquisition', 'funding', 'Series', 'valuation',
|
||||
'launch', 'release', 'rollout', 'deploy',
|
||||
'OpenClaw', 'Qwen', 'DeepSeek', 'Grok', 'xAI',
|
||||
'Nano Banana', 'Meta AI', 'Cohere', 'Perplexity', 'Codex',
|
||||
'Copilot', 'GitHub Copilot', 'Amazon Q', 'Bedrock',
|
||||
]
|
||||
|
||||
EXCLUDE_KEYWORDS = [
|
||||
'layoffs', 'hiring', 'conference', 'event', 'podcast', 'interview',
|
||||
'salary', 'office', 'real estate', 'delivery', 'e-commerce',
|
||||
'crypto', 'blockchain', 'NFT', 'web3',
|
||||
]
|
||||
|
||||
# Build combined pattern: word-bounded short keywords + substring long keywords
|
||||
short_pat = r'\b(' + '|'.join(re.escape(k) for k in SHORT_KEYWORDS) + r')\b'
|
||||
long_pat = '|'.join(re.escape(k) for k in sorted(LONG_KEYWORDS, key=len, reverse=True))
|
||||
ai_pattern = re.compile(short_pat + '|' + long_pat, re.IGNORECASE)
|
||||
|
||||
exclude_pattern = re.compile(
|
||||
r'\b(' + '|'.join(re.escape(k) for k in EXCLUDE_KEYWORDS) + r')\b',
|
||||
re.IGNORECASE
|
||||
)
|
||||
|
||||
# ── Source tiers (names must match blogwatcher exactly) ──────────────
|
||||
# Customize: add your RSS feed names here with their trust tier
|
||||
SOURCE_TIERS = {
|
||||
# T1: Wire services + official AI lab blogs
|
||||
'Reuters Tech': 1, 'Bloomberg Tech': 1, 'Axios AI': 1, 'CNBC Tech': 1,
|
||||
'OpenAI Blog': 1,
|
||||
# T2: Tech press + priority bloggers
|
||||
'TechCrunch AI': 2, 'The Verge': 2, 'THE DECODER': 2, 'VentureBeat AI': 2,
|
||||
'Ars Technica': 2, '404 Media': 2, '9to5Google': 2, 'TestingCatalog': 2,
|
||||
'Crunchbase News': 2, 'Wired AI': 2, 'MIT Tech Review': 2, 'Google AI Blog': 2,
|
||||
'Hugging Face Blog': 2, 'Simon Willison': 2, 'Latent Space': 2,
|
||||
# T3: Aggregator / community press / analysis
|
||||
'Hacker News AI': 3, 'SiliconANGLE AI': 3, 'AI News': 3,
|
||||
'Gary Marcus': 3, 'Bens Bites': 3,
|
||||
}
|
||||
|
||||
# Reddit discussion noise (questions, complaints, memes)
|
||||
REDDIT_NOISE_START = re.compile(
|
||||
r'^(Why|How|What|Can|Does|Is|Has|Are|Do|Should|Would|Could|Anyone|'
|
||||
r'Help|Rant|Vent|Am I|ELI5|CMV|PSA|Unpopular|Hot take|DAE|TIL|'
|
||||
r'Gah|Kindly explain|Seriously|From Frustration|Gemini Memory)',
|
||||
re.IGNORECASE
|
||||
)
|
||||
|
||||
def get_tier(source, title):
|
||||
if source in SOURCE_TIERS:
|
||||
return SOURCE_TIERS[source]
|
||||
if source.startswith('r/') or 'reddit.com' in source:
|
||||
title_s = title.strip()
|
||||
if REDDIT_NOISE_START.match(title_s):
|
||||
return 99
|
||||
if title_s.endswith('?'):
|
||||
return 99
|
||||
if len(title_s) < 20:
|
||||
return 99
|
||||
return 4
|
||||
if source.startswith('http'):
|
||||
if 'bloomberg.com' in source: return 1
|
||||
if 'cnbc.com' in source: return 1
|
||||
if 'reuters.com' in source: return 1
|
||||
if 'techcrunch.com' in source: return 2
|
||||
if 'theverge.com' in source: return 2
|
||||
if 'wired.com' in source: return 2
|
||||
return 3
|
||||
return 3
|
||||
|
||||
# ── Parse blogwatcher entries ────────────────────────────────────────
|
||||
title = url = source = None
|
||||
results = []
|
||||
|
||||
for line in raw.split('\n'):
|
||||
stripped = line.strip()
|
||||
|
||||
m = re.match(r'\[(\d+)\]\s*\[new\]\s*(.*)', stripped)
|
||||
if m:
|
||||
if title and url and source is not None:
|
||||
tier = get_tier(source, title)
|
||||
if tier != 99:
|
||||
if ai_pattern.search(title) and not exclude_pattern.search(title):
|
||||
results.append((title, url, source, tier))
|
||||
title = m.group(2).strip()
|
||||
url = source = None
|
||||
continue
|
||||
|
||||
if stripped.startswith('Blog:'):
|
||||
source = stripped[5:].strip()
|
||||
elif stripped.startswith('URL:'):
|
||||
url = stripped[4:].strip()
|
||||
|
||||
if title and url and source is not None:
|
||||
tier = get_tier(source, title)
|
||||
if tier != 99:
|
||||
if ai_pattern.search(title) and not exclude_pattern.search(title):
|
||||
results.append((title, url, source, tier))
|
||||
|
||||
results.sort(key=lambda x: x[3])
|
||||
|
||||
for t, u, s, tier in results:
|
||||
t_clean = t.replace('|', ' —')
|
||||
print(f"{t_clean}|{u}|{s}|{tier}")
|
||||
PYEOF
|
||||
@@ -0,0 +1,303 @@
|
||||
#!/usr/bin/env python3
|
||||
"""GitHub Trending & Emerging Repo Scanner.
|
||||
|
||||
Scans GitHub for trending AI/ML repos using three strategies:
|
||||
1. Emerging: repos created in the last 7 days with 50+ stars
|
||||
2. Velocity: established repos (1000+ stars) gaining traction fast
|
||||
3. Releases: new releases from key AI repos (SDKs, models, tools)
|
||||
|
||||
Output format: TITLE|URL|SOURCE|TIER
|
||||
Compatible with the news scan pipeline.
|
||||
|
||||
Uses only stdlib — no pip packages. Auth via GH_TOKEN env var if available
|
||||
(5000 req/h), falls back to unauthenticated (60 req/h).
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# ── Configuration ──────────────────────────────────────────────────
|
||||
STATE_FILE = Path(os.path.expanduser(
|
||||
"~/.openclaw/workspace/memory/github_trending_state.json"
|
||||
))
|
||||
API_BASE = "https://api.github.com/search/repositories"
|
||||
HEADERS = {
|
||||
"Accept": "application/vnd.github+json",
|
||||
"User-Agent": "NewsScanner/1.0",
|
||||
}
|
||||
_gh_token = os.environ.get("GH_TOKEN", "")
|
||||
if _gh_token:
|
||||
HEADERS["Authorization"] = f"token {_gh_token}"
|
||||
REQUEST_TIMEOUT = 30
|
||||
|
||||
# Topics to scan (each generates a separate API call)
|
||||
TOPICS = ["ai", "llm", "agents", "generative-ai", "large-language-model"]
|
||||
|
||||
EMERGING_WINDOW_DAYS = 7
|
||||
EMERGING_MIN_STARS = 50
|
||||
|
||||
VELOCITY_MIN_STARS = 1000
|
||||
VELOCITY_PUSHED_DAYS = 30
|
||||
VELOCITY_GROWTH_MIN = 50
|
||||
VELOCITY_ALWAYS_IF = 10000
|
||||
|
||||
MAX_OUTPUT = 15
|
||||
TIER = 3
|
||||
|
||||
# Key AI repos to monitor for releases (owner/repo)
|
||||
# Customize: add repos relevant to your audience
|
||||
RELEASE_REPOS = [
|
||||
"openai/openai-python",
|
||||
"anthropics/anthropic-sdk-python",
|
||||
"huggingface/transformers",
|
||||
"langchain-ai/langchain",
|
||||
"ollama/ollama",
|
||||
"vllm-project/vllm",
|
||||
"run-llama/llama_index",
|
||||
"microsoft/autogen",
|
||||
"crewAIInc/crewAI",
|
||||
"BerriAI/litellm",
|
||||
"ggerganov/llama.cpp",
|
||||
"mozilla/readability",
|
||||
"deepseek-ai/DeepSeek-V3",
|
||||
"QwenLM/Qwen",
|
||||
"meta-llama/llama",
|
||||
"google/gemma.cpp",
|
||||
]
|
||||
RELEASE_WINDOW_DAYS = 3
|
||||
|
||||
|
||||
def iso_date(dt):
|
||||
return dt.strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def log(msg):
|
||||
print(msg, file=sys.stderr)
|
||||
|
||||
|
||||
_rate_limited = False
|
||||
|
||||
|
||||
def github_search(query, sort="stars", order="desc", per_page=10):
|
||||
global _rate_limited
|
||||
if _rate_limited:
|
||||
return None
|
||||
|
||||
params = urllib.parse.urlencode({
|
||||
"q": query, "sort": sort, "order": order, "per_page": per_page,
|
||||
})
|
||||
url = f"{API_BASE}?{params}"
|
||||
|
||||
req = urllib.request.Request(url, headers=HEADERS)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT) as resp:
|
||||
remaining = resp.headers.get("X-RateLimit-Remaining", "?")
|
||||
log(f" API call OK — rate limit remaining: {remaining}")
|
||||
if remaining != "?" and int(remaining) <= 1:
|
||||
log("WARNING: GitHub API rate limit nearly exhausted.")
|
||||
_rate_limited = True
|
||||
data = json.loads(resp.read().decode())
|
||||
return data.get("items", [])
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 403:
|
||||
log("WARNING: Rate limited (HTTP 403).")
|
||||
_rate_limited = True
|
||||
else:
|
||||
log(f"WARNING: GitHub API error {e.code}: {e.reason}")
|
||||
return None
|
||||
except urllib.error.URLError as e:
|
||||
log(f"WARNING: Network error: {e.reason}")
|
||||
return None
|
||||
except Exception as e:
|
||||
log(f"WARNING: Unexpected error: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def detect_language(repo):
|
||||
lang = repo.get("language")
|
||||
return lang if lang else "Mixed"
|
||||
|
||||
|
||||
def load_state():
|
||||
if STATE_FILE.exists():
|
||||
try:
|
||||
with open(STATE_FILE, "r") as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
return {"repos": {}, "last_run": None}
|
||||
|
||||
|
||||
def save_state(state):
|
||||
STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(STATE_FILE, "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
|
||||
|
||||
def scan_emerging():
|
||||
cutoff = iso_date(datetime.now(timezone.utc) - timedelta(days=EMERGING_WINDOW_DAYS))
|
||||
results = []
|
||||
for topic in TOPICS:
|
||||
if _rate_limited:
|
||||
break
|
||||
query = f"topic:{topic} created:>{cutoff} stars:>{EMERGING_MIN_STARS}"
|
||||
log(f"[Emerging] topic={topic}")
|
||||
repos = github_search(query, sort="stars", order="desc", per_page=10)
|
||||
if repos is None:
|
||||
continue
|
||||
for repo in repos:
|
||||
full_name = repo["full_name"]
|
||||
stars = repo.get("stargazers_count", 0)
|
||||
desc = (repo.get("description") or "No description").replace("|", "-")
|
||||
url = repo.get("html_url", f"https://github.com/{full_name}")
|
||||
lang = detect_language(repo)
|
||||
title = f"[GitHub EMERGING] {full_name} (+{stars} stars): {desc}"
|
||||
results.append((title, url, f"GitHub/{lang}", stars, full_name))
|
||||
time.sleep(0.5)
|
||||
return results
|
||||
|
||||
|
||||
def scan_velocity(state):
|
||||
pushed_cutoff = iso_date(datetime.now(timezone.utc) - timedelta(days=VELOCITY_PUSHED_DAYS))
|
||||
old_repos = state.get("repos", {})
|
||||
new_repos = {}
|
||||
results = []
|
||||
velocity_topics = ["ai", "llm", "large-language-model"]
|
||||
|
||||
for topic in velocity_topics:
|
||||
if _rate_limited:
|
||||
break
|
||||
query = f"topic:{topic} stars:>{VELOCITY_MIN_STARS} pushed:>{pushed_cutoff}"
|
||||
log(f"[Velocity] topic={topic}")
|
||||
repos = github_search(query, sort="stars", order="desc", per_page=10)
|
||||
if repos is None:
|
||||
continue
|
||||
for repo in repos:
|
||||
full_name = repo["full_name"]
|
||||
stars = repo.get("stargazers_count", 0)
|
||||
desc = (repo.get("description") or "No description").replace("|", "-")
|
||||
url = repo.get("html_url", f"https://github.com/{full_name}")
|
||||
lang = detect_language(repo)
|
||||
new_repos[full_name] = {"stars": stars}
|
||||
prev_stars = old_repos.get(full_name, {}).get("stars")
|
||||
growth = (stars - prev_stars) if prev_stars is not None else 0
|
||||
|
||||
if growth >= VELOCITY_GROWTH_MIN:
|
||||
title = f"[GitHub TRENDING] {full_name} (+{growth} stars): {desc}"
|
||||
results.append((title, url, f"GitHub/{lang}", growth, full_name))
|
||||
elif stars >= VELOCITY_ALWAYS_IF:
|
||||
title = f"[GitHub HOT] {full_name} ({stars:,} total stars): {desc}"
|
||||
results.append((title, url, f"GitHub/{lang}", 0, full_name))
|
||||
time.sleep(0.5)
|
||||
|
||||
merged_repos = {**old_repos}
|
||||
merged_repos.update(new_repos)
|
||||
return results, merged_repos
|
||||
|
||||
|
||||
def scan_releases():
|
||||
global _rate_limited
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=RELEASE_WINDOW_DAYS)
|
||||
results = []
|
||||
|
||||
for repo_name in RELEASE_REPOS:
|
||||
if _rate_limited:
|
||||
break
|
||||
url = f"https://api.github.com/repos/{repo_name}/releases?per_page=3"
|
||||
req = urllib.request.Request(url, headers=HEADERS)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT) as resp:
|
||||
remaining = resp.headers.get("X-RateLimit-Remaining", "?")
|
||||
if remaining != "?" and int(remaining) <= 2:
|
||||
_rate_limited = True
|
||||
break
|
||||
releases = json.loads(resp.read().decode())
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
continue
|
||||
continue
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
for release in releases:
|
||||
if release.get("draft", False) or release.get("prerelease", False):
|
||||
continue
|
||||
published = release.get("published_at", "")
|
||||
if not published:
|
||||
continue
|
||||
try:
|
||||
pub_dt = datetime.fromisoformat(published.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
continue
|
||||
if pub_dt < cutoff:
|
||||
break
|
||||
|
||||
tag = release.get("tag_name", "")
|
||||
name = release.get("name", tag)
|
||||
html_url = release.get("html_url", f"https://github.com/{repo_name}")
|
||||
body = (release.get("body") or "")[:100].replace("|", "-").replace("\n", " ")
|
||||
|
||||
title = f"[GitHub RELEASE] {repo_name} {tag}: {name}"
|
||||
if body:
|
||||
title += f" — {body}"
|
||||
results.append((title, html_url, "GitHub/Releases", 0, repo_name))
|
||||
break
|
||||
time.sleep(0.3)
|
||||
return results
|
||||
|
||||
|
||||
def main():
|
||||
log("=" * 60)
|
||||
log(f"GitHub Trending Scanner — {datetime.now(timezone.utc).isoformat()}")
|
||||
log("=" * 60)
|
||||
|
||||
state = load_state()
|
||||
seen_repos = set()
|
||||
output_lines = []
|
||||
|
||||
log("\n--- Strategy 1: Emerging repos ---")
|
||||
emerging = scan_emerging()
|
||||
emerging.sort(key=lambda x: x[3], reverse=True)
|
||||
for title, url, source, velocity, full_name in emerging:
|
||||
if full_name not in seen_repos:
|
||||
seen_repos.add(full_name)
|
||||
output_lines.append((title, url, source, TIER))
|
||||
|
||||
log("\n--- Strategy 2: Velocity ---")
|
||||
velocity_results, merged_repos = scan_velocity(state)
|
||||
velocity_results.sort(key=lambda x: x[3], reverse=True)
|
||||
for title, url, source, velocity, full_name in velocity_results:
|
||||
if full_name not in seen_repos:
|
||||
seen_repos.add(full_name)
|
||||
output_lines.append((title, url, source, TIER))
|
||||
|
||||
log("\n--- Strategy 3: Releases ---")
|
||||
releases = scan_releases()
|
||||
for title, url, source, _, full_name in releases:
|
||||
if full_name not in seen_repos:
|
||||
seen_repos.add(full_name)
|
||||
output_lines.append((title, url, source, TIER))
|
||||
|
||||
output_lines = output_lines[:MAX_OUTPUT]
|
||||
|
||||
if not output_lines:
|
||||
log("No trending repos found.")
|
||||
else:
|
||||
for title, url, source, tier in output_lines:
|
||||
print(f"{title}|{url}|{source}|{tier}")
|
||||
|
||||
state["repos"] = merged_repos
|
||||
state["last_run"] = datetime.now(timezone.utc).isoformat()
|
||||
save_state(state)
|
||||
log("Done.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import urllib.parse
|
||||
main()
|
||||
@@ -0,0 +1,437 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
llm_editor.py - AI Editor for Automated News Scanning
|
||||
======================================================
|
||||
Replaces deterministic keyword filtering with Gemini Flash AI-powered
|
||||
story selection. Reads candidate articles, an editorial profile, and
|
||||
recent post history, then calls Gemini to pick the top stories.
|
||||
|
||||
Usage:
|
||||
python3 llm_editor.py --file candidates.txt [--github github.txt]
|
||||
|
||||
Input format (pipe-delimited, one per line):
|
||||
TITLE|URL|SOURCE
|
||||
TITLE|URL|SOURCE|TIER (tier is optional, ignored by LLM)
|
||||
|
||||
Output (stdout, one JSON object per line):
|
||||
{"rank": 1, "title": "...", "url": "...", "source": "...",
|
||||
"type": "rss", "summary": "...", "category": "..."}
|
||||
|
||||
Logs picked stories to scanner_presented.md (append).
|
||||
All status/debug messages go to stderr.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# ── Paths (customize to your workspace) ──────────────────────────────
|
||||
WORKSPACE = Path(os.environ.get("OPENCLAW_WORKSPACE",
|
||||
os.path.expanduser("~/.openclaw/workspace")))
|
||||
MEMORY = WORKSPACE / "memory"
|
||||
EDITORIAL_PROFILE = MEMORY / "editorial_profile.md"
|
||||
SCANNER_PRESENTED = MEMORY / "scanner_presented.md"
|
||||
NEWS_LOG = MEMORY / "news_log.md"
|
||||
|
||||
# ── Configuration ────────────────────────────────────────────────────
|
||||
GEMINI_MODEL = "gemini-3-flash-preview"
|
||||
GEMINI_URL = (
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/"
|
||||
f"{GEMINI_MODEL}:generateContent"
|
||||
)
|
||||
TEMPERATURE = 0.3
|
||||
TIMEOUT_SEC = 120
|
||||
MAX_ARTICLES = 500
|
||||
VALID_CATEGORIES = {
|
||||
"ai_product", "m_and_a", "model_release", "security", "geopolitics",
|
||||
"github_trending", "gaming", "fintech", "hardware", "open_source", "other"
|
||||
}
|
||||
|
||||
|
||||
def log(msg):
|
||||
ts = datetime.now().strftime("%H:%M:%S")
|
||||
print(f"[llm_editor {ts}] {msg}", file=sys.stderr)
|
||||
|
||||
|
||||
def estimate_tokens(text):
|
||||
return len(text) // 4
|
||||
|
||||
|
||||
def parse_articles(filepath):
|
||||
articles = []
|
||||
try:
|
||||
with open(filepath, "r") as f:
|
||||
for line_num, line in enumerate(f, 1):
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
parts = line.split("|")
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
articles.append({
|
||||
"title": parts[0].strip(),
|
||||
"url": parts[1].strip(),
|
||||
"source": parts[2].strip(),
|
||||
})
|
||||
except FileNotFoundError:
|
||||
log(f"ERROR: File not found: {filepath}")
|
||||
sys.exit(1)
|
||||
return articles
|
||||
|
||||
|
||||
def load_file_safe(path, tail_lines=None):
|
||||
try:
|
||||
with open(path, "r") as f:
|
||||
lines = f.readlines()
|
||||
if tail_lines and len(lines) > tail_lines:
|
||||
lines = lines[-tail_lines:]
|
||||
return "".join(lines)
|
||||
except FileNotFoundError:
|
||||
return ""
|
||||
except Exception as e:
|
||||
log(f" Error reading {path}: {e}")
|
||||
return ""
|
||||
|
||||
|
||||
def filter_already_posted(articles):
|
||||
"""
|
||||
Deterministic URL pre-filter: remove candidates whose URL already
|
||||
appears in news_log.md or scanner_presented.md.
|
||||
"""
|
||||
full_log = load_file_safe(NEWS_LOG)
|
||||
if not full_log:
|
||||
return articles
|
||||
|
||||
presented_log = load_file_safe(SCANNER_PRESENTED)
|
||||
|
||||
url_pattern = re.compile(r'https?://[^\s|>\]\)"\']+')
|
||||
posted_urls = set()
|
||||
for text in [full_log, presented_log]:
|
||||
for url in url_pattern.findall(text):
|
||||
url = url.rstrip(".,;:)")
|
||||
# Skip your own channel links (customize this pattern)
|
||||
# if "t.me/yourchannel" in url:
|
||||
# continue
|
||||
posted_urls.add(url)
|
||||
|
||||
if not posted_urls:
|
||||
return articles
|
||||
|
||||
filtered = []
|
||||
removed = 0
|
||||
for a in articles:
|
||||
candidate_url = a["url"].rstrip(".,;:)")
|
||||
if candidate_url in posted_urls:
|
||||
log(f" PRE-FILTERED (already posted): {a['title'][:60]}")
|
||||
removed += 1
|
||||
else:
|
||||
filtered.append(a)
|
||||
|
||||
log(f"Pre-filtered {removed} candidates (already posted)")
|
||||
return filtered
|
||||
|
||||
|
||||
def build_prompt(articles, github_articles, editorial_profile, recent_posts, top_n):
|
||||
article_list = []
|
||||
for i, a in enumerate(articles, 1):
|
||||
article_list.append(f" {i}. [{a['source']}] {a['title']}\n URL: {a['url']}")
|
||||
articles_text = "\n".join(article_list)
|
||||
|
||||
github_text = ""
|
||||
if github_articles:
|
||||
gh_list = []
|
||||
for i, g in enumerate(github_articles, 1):
|
||||
gh_list.append(f" {i}. [{g['source']}] {g['title']}\n URL: {g['url']}")
|
||||
github_text = (
|
||||
"\n\n## GitHub Trending Repos\n"
|
||||
"These are trending GitHub repositories. Include any that are genuinely\n"
|
||||
"newsworthy for your audience.\n\n"
|
||||
+ "\n".join(gh_list)
|
||||
)
|
||||
|
||||
prompt = f"""You are the AI editor for an automated news channel. Your job is to select
|
||||
the top {top_n} stories from the candidate list below.
|
||||
|
||||
## Editorial Profile
|
||||
{editorial_profile}
|
||||
|
||||
## Recently Posted Stories (do NOT pick duplicates of these)
|
||||
{recent_posts if recent_posts else '(No recent posts available)'}
|
||||
|
||||
## Candidate Articles
|
||||
{articles_text}
|
||||
{github_text}
|
||||
|
||||
## Your Task
|
||||
Select exactly {top_n} stories from the candidates above. Rank them by
|
||||
newsworthiness for the target audience.
|
||||
|
||||
## Rules
|
||||
1. Return EXACTLY {top_n} stories — no more, no fewer.
|
||||
2. Do NOT pick stories that duplicate recently posted stories (same event).
|
||||
If a candidate covers the SAME EVENT as a recently posted story — even
|
||||
from a different source or with a different headline — do NOT pick it.
|
||||
3. Maximum 2 stories from the same source.
|
||||
4. Include a 1-sentence summary explaining WHY each story matters.
|
||||
5. Rank by newsworthiness: breaking news > major deals > product launches > analysis.
|
||||
6. Prefer concrete news (X acquired Y, X launched Z) over speculation or opinion.
|
||||
7. If a GitHub repo is trending AND relevant to the audience, include it.
|
||||
8. Assign each story a category from this list:
|
||||
ai_product, m_and_a, model_release, security, geopolitics,
|
||||
github_trending, gaming, fintech, hardware, open_source, other
|
||||
|
||||
## Required JSON Output Format
|
||||
Return a JSON array of exactly {top_n} objects, each with these fields:
|
||||
[
|
||||
{{
|
||||
"rank": 1,
|
||||
"title": "Story headline",
|
||||
"url": "https://...",
|
||||
"source": "Source name",
|
||||
"type": "rss, twitter, or github (use twitter for X/Twitter sources)",
|
||||
"summary": "One sentence why this matters.",
|
||||
"category": "category_from_list_above"
|
||||
}}
|
||||
]
|
||||
|
||||
Return ONLY the JSON array. No markdown, no commentary, no code fences."""
|
||||
return prompt
|
||||
|
||||
|
||||
def call_gemini(prompt, api_key):
|
||||
url = f"{GEMINI_URL}?key={api_key}"
|
||||
|
||||
payload = {
|
||||
"contents": [{"parts": [{"text": prompt}]}],
|
||||
"generationConfig": {
|
||||
"temperature": TEMPERATURE,
|
||||
"responseMimeType": "application/json",
|
||||
}
|
||||
}
|
||||
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
url, data=data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
|
||||
token_est = estimate_tokens(prompt)
|
||||
log(f"Sending prompt to Gemini Flash (~{token_est} tokens)")
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT_SEC) as resp:
|
||||
body = resp.read().decode("utf-8")
|
||||
result = json.loads(body)
|
||||
except urllib.error.HTTPError as e:
|
||||
error_body = e.read().decode("utf-8") if e.fp else "(no body)"
|
||||
log(f"API HTTP error {e.code}: {error_body[:500]}")
|
||||
return None
|
||||
except urllib.error.URLError as e:
|
||||
log(f"API connection error: {e.reason}")
|
||||
return None
|
||||
except Exception as e:
|
||||
log(f"API call failed: {e}")
|
||||
return None
|
||||
|
||||
try:
|
||||
text = result["candidates"][0]["content"]["parts"][0]["text"]
|
||||
except (KeyError, IndexError) as e:
|
||||
log(f"Unexpected API response structure: {e}")
|
||||
return None
|
||||
|
||||
try:
|
||||
picks = json.loads(text)
|
||||
if isinstance(picks, list):
|
||||
return picks
|
||||
if isinstance(picks, dict) and "stories" in picks:
|
||||
return picks["stories"]
|
||||
return None
|
||||
except json.JSONDecodeError:
|
||||
match = re.search(r'\[\s*\{.*?\}\s*\]', text, re.DOTALL)
|
||||
if match:
|
||||
try:
|
||||
picks = json.loads(match.group())
|
||||
if isinstance(picks, list):
|
||||
return picks
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
log(f"Could not parse LLM response. First 500 chars: {text[:500]}")
|
||||
return None
|
||||
|
||||
|
||||
def fallback_picks(articles, github_articles, top_n):
|
||||
log("FALLBACK: Using raw article order (no LLM judgment)")
|
||||
all_candidates = articles.copy()
|
||||
if github_articles:
|
||||
all_candidates.extend(github_articles)
|
||||
|
||||
picks = []
|
||||
seen_sources = {}
|
||||
for a in all_candidates:
|
||||
src = a["source"]
|
||||
if seen_sources.get(src, 0) >= 2:
|
||||
continue
|
||||
seen_sources[src] = seen_sources.get(src, 0) + 1
|
||||
if "github.com" in a["url"]:
|
||||
article_type = "github"
|
||||
elif "x.com/" in a.get("url", "") or "twitter.com/" in a.get("url", "") or "X/" in a.get("source", ""):
|
||||
article_type = "twitter"
|
||||
else:
|
||||
article_type = "rss"
|
||||
picks.append({
|
||||
"rank": len(picks) + 1,
|
||||
"title": a["title"],
|
||||
"url": a["url"],
|
||||
"source": a["source"],
|
||||
"type": article_type,
|
||||
"summary": "(Fallback: no AI summary available)",
|
||||
"category": "other",
|
||||
})
|
||||
if len(picks) >= top_n:
|
||||
break
|
||||
return picks
|
||||
|
||||
|
||||
def validate_picks(picks, top_n):
|
||||
validated = []
|
||||
for i, pick in enumerate(picks):
|
||||
if not isinstance(pick, dict):
|
||||
continue
|
||||
entry = {
|
||||
"rank": pick.get("rank", i + 1),
|
||||
"title": pick.get("title", "(no title)"),
|
||||
"url": pick.get("url", ""),
|
||||
"source": pick.get("source", "unknown"),
|
||||
"type": pick.get("type", "rss"),
|
||||
"summary": pick.get("summary", ""),
|
||||
"category": pick.get("category", "other"),
|
||||
}
|
||||
if entry["category"] not in VALID_CATEGORIES:
|
||||
entry["category"] = "other"
|
||||
if entry["type"] not in ("rss", "twitter", "github"):
|
||||
entry["type"] = "rss"
|
||||
validated.append(entry)
|
||||
|
||||
for i, v in enumerate(validated):
|
||||
v["rank"] = i + 1
|
||||
|
||||
if len(validated) != top_n:
|
||||
log(f" Warning: expected {top_n} picks, got {len(validated)}")
|
||||
return validated
|
||||
|
||||
|
||||
def log_to_scanner_presented(picks):
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
today_header = f"## {today}"
|
||||
ts = datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
|
||||
|
||||
try:
|
||||
existing = ""
|
||||
if SCANNER_PRESENTED.exists():
|
||||
existing = SCANNER_PRESENTED.read_text()
|
||||
|
||||
with open(SCANNER_PRESENTED, "a") as f:
|
||||
if today_header not in existing:
|
||||
f.write(f"\n{today_header}\n\n")
|
||||
for pick in picks:
|
||||
f.write(f"[{ts}] {pick['title']} | {pick['url']}\n")
|
||||
|
||||
log(f"Logged {len(picks)} picks to scanner_presented.md")
|
||||
except Exception as e:
|
||||
log(f"Warning: could not log to scanner_presented.md: {e}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="AI Editor — selects top stories using Gemini Flash"
|
||||
)
|
||||
parser.add_argument("--file", "-f", required=True,
|
||||
help="Path to article candidates file")
|
||||
parser.add_argument("--github", "-g",
|
||||
help="Path to GitHub trending repos file")
|
||||
parser.add_argument("--dry-run", action="store_true",
|
||||
help="Build prompt and print to stderr, but don't call API")
|
||||
args = parser.parse_args()
|
||||
|
||||
api_key = os.environ.get("GEMINI_API_KEY")
|
||||
if not api_key:
|
||||
log("ERROR: GEMINI_API_KEY environment variable not set")
|
||||
sys.exit(1)
|
||||
|
||||
top_n = int(os.environ.get("TOP_N", "7"))
|
||||
log(f"Configuration: top_n={top_n}, model={GEMINI_MODEL}")
|
||||
|
||||
log(f"Loading articles from {args.file}")
|
||||
articles = parse_articles(args.file)
|
||||
log(f" Loaded {len(articles)} candidates")
|
||||
if len(articles) > MAX_ARTICLES:
|
||||
articles = articles[:MAX_ARTICLES]
|
||||
|
||||
if not articles:
|
||||
log("ERROR: No articles found in input file")
|
||||
sys.exit(1)
|
||||
|
||||
github_articles = []
|
||||
if args.github:
|
||||
github_articles = parse_articles(args.github)
|
||||
log(f" Loaded {len(github_articles)} GitHub repos")
|
||||
|
||||
log("Running deterministic URL pre-filter")
|
||||
articles = filter_already_posted(articles)
|
||||
if github_articles:
|
||||
github_articles = filter_already_posted(github_articles)
|
||||
|
||||
total_candidates = len(articles) + len(github_articles)
|
||||
if top_n > total_candidates:
|
||||
top_n = total_candidates
|
||||
|
||||
log("Loading editorial profile")
|
||||
editorial_profile = load_file_safe(EDITORIAL_PROFILE)
|
||||
if not editorial_profile:
|
||||
editorial_profile = (
|
||||
"Select stories about AI, LLMs, tech deals, and security.\n"
|
||||
"Prefer breaking news and concrete announcements over opinion."
|
||||
)
|
||||
|
||||
log("Loading recent post history for dedup")
|
||||
recent_presented = load_file_safe(SCANNER_PRESENTED, tail_lines=60)
|
||||
recent_news_log = load_file_safe(NEWS_LOG, tail_lines=150)
|
||||
recent_posts = ""
|
||||
if recent_presented:
|
||||
recent_posts += "### scanner_presented.md (recent)\n" + recent_presented + "\n"
|
||||
if recent_news_log:
|
||||
recent_posts += "### news_log.md (recent)\n" + recent_news_log + "\n"
|
||||
|
||||
prompt = build_prompt(articles, github_articles, editorial_profile, recent_posts, top_n)
|
||||
prompt_tokens = estimate_tokens(prompt)
|
||||
log(f"Prompt built: ~{prompt_tokens} estimated tokens")
|
||||
|
||||
if args.dry_run:
|
||||
log("DRY RUN — printing prompt to stderr")
|
||||
print(prompt, file=sys.stderr)
|
||||
return
|
||||
|
||||
picks = call_gemini(prompt, api_key)
|
||||
|
||||
if picks is None:
|
||||
picks = fallback_picks(articles, github_articles, top_n)
|
||||
else:
|
||||
log(f"LLM returned {len(picks)} picks")
|
||||
picks = validate_picks(picks, top_n)
|
||||
|
||||
for pick in picks:
|
||||
print(json.dumps(pick, ensure_ascii=False))
|
||||
|
||||
log_to_scanner_presented(picks)
|
||||
log(f"Done. {len(picks)} stories selected.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+413
@@ -0,0 +1,413 @@
|
||||
#!/bin/bash
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# news_scan_deduped.sh — Automated News Scan Pipeline v2
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
#
|
||||
# Orchestrates six data sources and pipes them through quality scoring,
|
||||
# enrichment, and Gemini Flash (llm_editor.py) for AI-powered curation.
|
||||
#
|
||||
# Flow:
|
||||
# 1. RSS via blogwatcher (25 feeds)
|
||||
# 2. Reddit via JSON API (13 subreddits, score-filtered)
|
||||
# 3. Twitter via bird CLI + twitterapi.io
|
||||
# 4. GitHub trending + releases
|
||||
# 5. Tavily web search (breaking news supplement)
|
||||
# 6. All → quality_score.py → enrich_top_articles.py → llm_editor.py
|
||||
# 7. blogwatcher read-all
|
||||
#
|
||||
# Usage:
|
||||
# ./news_scan_deduped.sh # default: top 7 picks
|
||||
# ./news_scan_deduped.sh --top 5 # top 5 picks
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
|
||||
# ── Parse arguments ──────────────────────────────────────────────────
|
||||
TOP_N=7
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--top) TOP_N="$2"; shift 2 ;;
|
||||
-h|--help)
|
||||
echo "Usage: $0 [--top N]"
|
||||
echo " --top N Number of stories to curate (default: 7)"
|
||||
exit 0
|
||||
;;
|
||||
*) echo "Unknown arg: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
export TOP_N
|
||||
|
||||
# ── Temp files (cleaned up on exit) ─────────────────────────────────
|
||||
ARTICLES_FILE=$(mktemp /tmp/newscan_articles.XXXXXX)
|
||||
REDDIT_FILE=$(mktemp /tmp/newscan_reddit.XXXXXX)
|
||||
TAVILY_FILE=$(mktemp /tmp/newscan_tavily.XXXXXX)
|
||||
TWITTER_API_FILE=$(mktemp /tmp/newscan_twitterapi.XXXXXX)
|
||||
SCORED_FILE=$(mktemp /tmp/newscan_scored.XXXXXX)
|
||||
ENRICHED_FILE=$(mktemp /tmp/newscan_enriched.XXXXXX)
|
||||
PERSISTENT_CANDIDATES="$SCRIPT_DIR/../memory/last_scan_candidates.txt"
|
||||
PERSISTENT_GITHUB="$SCRIPT_DIR/../memory/last_scan_github.txt"
|
||||
GITHUB_FILE=$(mktemp /tmp/newscan_github.XXXXXX)
|
||||
TWITTER_RAW=$(mktemp /tmp/newscan_twitter.XXXXXX)
|
||||
PICKS_FILE=$(mktemp /tmp/newscan_picks.XXXXXX)
|
||||
|
||||
cleanup() {
|
||||
rm -f "$ARTICLES_FILE" "$REDDIT_FILE" "$TAVILY_FILE" "$TWITTER_API_FILE" \
|
||||
"$SCORED_FILE" "$ENRICHED_FILE" "$GITHUB_FILE" "$TWITTER_RAW" "$PICKS_FILE"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# ── Counters for stats ───────────────────────────────────────────────
|
||||
RSS_COUNT=0
|
||||
REDDIT_COUNT=0
|
||||
TWITTER_COUNT=0
|
||||
TWITTER_API_COUNT=0
|
||||
GITHUB_COUNT=0
|
||||
TAVILY_COUNT=0
|
||||
PICKS_COUNT=0
|
||||
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo " News Scanner v2 (top $TOP_N)"
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo ""
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# SOURCE 1: RSS via blogwatcher (25 feeds)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo "[1/5] Scanning RSS feeds..."
|
||||
|
||||
/usr/local/bin/timeout 90s /usr/local/bin/blogwatcher scan > /dev/null 2>&1 || echo " Warning: RSS scan timed out (continuing)"
|
||||
|
||||
python3 -c '
|
||||
import sys, subprocess, re
|
||||
|
||||
outpath = sys.argv[1]
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["/usr/local/bin/blogwatcher", "articles"],
|
||||
capture_output=True, text=True, timeout=30
|
||||
)
|
||||
raw = result.stdout
|
||||
except Exception as e:
|
||||
print(f" Warning: Could not run blogwatcher articles: {e}", file=sys.stderr)
|
||||
raw = ""
|
||||
|
||||
lines = raw.split("\n")
|
||||
articles = []
|
||||
i = 0
|
||||
|
||||
while i < len(lines):
|
||||
line = lines[i].strip()
|
||||
m = re.match(r"^\[\d+\]\s+\[new\]\s+(.+)$", line)
|
||||
if m:
|
||||
title = m.group(1).strip()
|
||||
title = title.replace("|", " -")
|
||||
source = ""
|
||||
url = ""
|
||||
for j in range(i + 1, min(i + 5, len(lines))):
|
||||
next_line = lines[j].strip()
|
||||
if next_line.startswith("Blog:"):
|
||||
source = next_line[5:].strip().replace("|", " -")
|
||||
elif next_line.startswith("URL:"):
|
||||
url = next_line[4:].strip()
|
||||
if title and url:
|
||||
articles.append(f"{title}|{url}|{source}")
|
||||
i += 1
|
||||
|
||||
with open(outpath, "w") as f:
|
||||
for a in articles:
|
||||
f.write(a + "\n")
|
||||
|
||||
print(f" Extracted {len(articles)} new RSS articles", file=sys.stderr)
|
||||
' "$ARTICLES_FILE"
|
||||
|
||||
RSS_COUNT=$(wc -l < "$ARTICLES_FILE" | tr -d ' ')
|
||||
echo " Found $RSS_COUNT articles from RSS feeds"
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# SOURCE 2: Reddit via JSON API (score-filtered)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo ""
|
||||
echo "[2/5] Scanning Reddit (JSON API)..."
|
||||
|
||||
if /usr/local/bin/timeout 60s python3 "$SCRIPT_DIR/fetch_reddit_news.py" --hours 24 > "$REDDIT_FILE" 2>/dev/null; then
|
||||
REDDIT_COUNT=$(wc -l < "$REDDIT_FILE" | tr -d ' ')
|
||||
echo " Found $REDDIT_COUNT Reddit posts (score-filtered)"
|
||||
cat "$REDDIT_FILE" >> "$ARTICLES_FILE"
|
||||
else
|
||||
echo " Warning: Reddit scan failed (continuing without)"
|
||||
REDDIT_COUNT=0
|
||||
fi
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# SOURCE 3: Twitter/X (bird CLI + twitterapi.io)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo ""
|
||||
echo "[3/5] Scanning X/Twitter..."
|
||||
|
||||
# 3a: bird CLI (primary — account-based)
|
||||
if /usr/local/bin/timeout 90s "$SCRIPT_DIR/scan_twitter_ai.sh" > "$TWITTER_RAW" 2>&1; then
|
||||
echo " bird CLI scan completed"
|
||||
else
|
||||
echo " Warning: bird CLI scan timed out or failed (continuing)"
|
||||
fi
|
||||
|
||||
if [ -s "$TWITTER_RAW" ]; then
|
||||
TWITTER_COUNT=$(python3 -c '
|
||||
import sys, re
|
||||
|
||||
twitter_file = sys.argv[1]
|
||||
articles_file = sys.argv[2]
|
||||
count = 0
|
||||
|
||||
with open(twitter_file, "r") as f:
|
||||
lines = f.readlines()
|
||||
|
||||
with open(articles_file, "a") as out:
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
if line.startswith(("===", "---", "Scanning", "Tier", "Breaking", "Product", "CEO")):
|
||||
continue
|
||||
text = line.replace("|", " -")
|
||||
urls = re.findall(r"(https?://\S+)", line)
|
||||
external_url = ""
|
||||
tweet_url = ""
|
||||
for u in urls:
|
||||
if "x.com/" in u or "twitter.com/" in u or "t.co/" in u:
|
||||
if not tweet_url:
|
||||
tweet_url = u
|
||||
else:
|
||||
if not external_url:
|
||||
external_url = u
|
||||
if external_url:
|
||||
out.write(f"{text}|{external_url}|X/Twitter\n")
|
||||
else:
|
||||
url = tweet_url
|
||||
out.write(f"{text}|{url}|X/Twitter (tweet)\n")
|
||||
count += 1
|
||||
|
||||
print(count)
|
||||
' "$TWITTER_RAW" "$ARTICLES_FILE")
|
||||
echo " bird CLI: $TWITTER_COUNT tweets"
|
||||
else
|
||||
TWITTER_COUNT=0
|
||||
fi
|
||||
|
||||
# 3b: twitterapi.io (supplement — keyword search)
|
||||
if /usr/local/bin/timeout 30s python3 "$SCRIPT_DIR/fetch_twitter_api.py" --max-queries 2 > "$TWITTER_API_FILE" 2>/dev/null; then
|
||||
TWITTER_API_COUNT=$(wc -l < "$TWITTER_API_FILE" | tr -d ' ')
|
||||
echo " twitterapi.io: $TWITTER_API_COUNT tweets"
|
||||
cat "$TWITTER_API_FILE" >> "$ARTICLES_FILE"
|
||||
else
|
||||
echo " Warning: twitterapi.io scan failed (continuing)"
|
||||
TWITTER_API_COUNT=0
|
||||
fi
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# SOURCE 4: GitHub Trending + Releases
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo ""
|
||||
echo "[4/5] Scanning GitHub trending + releases..."
|
||||
|
||||
if /usr/local/bin/timeout 45s python3 "$SCRIPT_DIR/github_trending.py" > "$GITHUB_FILE" 2>/dev/null; then
|
||||
GITHUB_COUNT=$(wc -l < "$GITHUB_FILE" | tr -d ' ')
|
||||
echo " Found $GITHUB_COUNT trending/release repos"
|
||||
else
|
||||
echo " Warning: GitHub scan timed out or failed (continuing)"
|
||||
GITHUB_COUNT=0
|
||||
fi
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# SOURCE 5: Tavily Web Search (breaking news supplement)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo ""
|
||||
echo "[5/5] Tavily web search..."
|
||||
|
||||
if /usr/local/bin/timeout 30s python3 "$SCRIPT_DIR/fetch_web_news.py" --max-queries 3 --max-results 5 > "$TAVILY_FILE" 2>/dev/null; then
|
||||
TAVILY_COUNT=$(wc -l < "$TAVILY_FILE" | tr -d ' ')
|
||||
echo " Found $TAVILY_COUNT web articles"
|
||||
cat "$TAVILY_FILE" >> "$ARTICLES_FILE"
|
||||
else
|
||||
echo " Warning: Tavily scan failed (continuing)"
|
||||
TAVILY_COUNT=0
|
||||
fi
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# QUALITY SCORING PRE-FILTER
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo ""
|
||||
TOTAL_RAW=$((RSS_COUNT + REDDIT_COUNT + TWITTER_COUNT + TWITTER_API_COUNT + TAVILY_COUNT))
|
||||
echo "Quality scoring ($TOTAL_RAW candidates)..."
|
||||
|
||||
if [ "$TOTAL_RAW" -gt 0 ]; then
|
||||
python3 "$SCRIPT_DIR/quality_score.py" --input "$ARTICLES_FILE" --max 50 > "$SCORED_FILE" 2>/dev/null
|
||||
SCORED_COUNT=$(wc -l < "$SCORED_FILE" | tr -d ' ')
|
||||
echo " Top $SCORED_COUNT articles after scoring + dedup"
|
||||
else
|
||||
cp "$ARTICLES_FILE" "$SCORED_FILE"
|
||||
SCORED_COUNT=0
|
||||
fi
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# ARTICLE ENRICHMENT (full text for top articles)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo ""
|
||||
echo "Enriching top articles with full text..."
|
||||
|
||||
if [ "$SCORED_COUNT" -gt 0 ]; then
|
||||
if /usr/local/bin/timeout 60s python3 "$SCRIPT_DIR/enrich_top_articles.py" --input "$SCORED_FILE" --max 8 --max-chars 1200 > "$ENRICHED_FILE" 2>/dev/null; then
|
||||
echo " Enrichment complete"
|
||||
else
|
||||
echo " Warning: Enrichment failed (using scored articles without full text)"
|
||||
cp "$SCORED_FILE" "$ENRICHED_FILE"
|
||||
fi
|
||||
else
|
||||
cp "$SCORED_FILE" "$ENRICHED_FILE"
|
||||
fi
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# LLM EDITORIAL FILTER (Gemini Flash via llm_editor.py)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo ""
|
||||
echo "Running LLM editorial filter (Gemini Flash)..."
|
||||
|
||||
TOTAL_CANDIDATES=$((TOTAL_RAW + GITHUB_COUNT))
|
||||
echo " Pipeline: ${TOTAL_RAW} raw -> ${SCORED_COUNT:-$TOTAL_RAW} scored -> LLM"
|
||||
|
||||
if [ "$TOTAL_CANDIDATES" -eq 0 ]; then
|
||||
echo ""
|
||||
echo "No new stories found from any source. Nothing to curate."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
LLM_CMD="python3 $SCRIPT_DIR/llm_editor.py --file $ENRICHED_FILE"
|
||||
if [ -s "$GITHUB_FILE" ]; then
|
||||
LLM_CMD="$LLM_CMD --github $GITHUB_FILE"
|
||||
fi
|
||||
|
||||
LLM_SUCCESS=true
|
||||
if eval "$LLM_CMD" > "$PICKS_FILE" 2>/tmp/llm_editor.log; then
|
||||
PICKS_COUNT=$(wc -l < "$PICKS_FILE" | tr -d ' ')
|
||||
echo " LLM selected $PICKS_COUNT stories"
|
||||
else
|
||||
echo " Warning: LLM editor failed (see /tmp/llm_editor.log)"
|
||||
LLM_SUCCESS=false
|
||||
fi
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# FORMAT & DISPLAY OUTPUT
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo ""
|
||||
cp "$ENRICHED_FILE" "$PERSISTENT_CANDIDATES" 2>/dev/null
|
||||
cp "$GITHUB_FILE" "$PERSISTENT_GITHUB" 2>/dev/null
|
||||
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo " TOP PICKS"
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo ""
|
||||
|
||||
if [ "$LLM_SUCCESS" = false ] || [ ! -s "$PICKS_FILE" ]; then
|
||||
echo "Warning: LLM curation unavailable — showing raw top articles:"
|
||||
echo ""
|
||||
head -"$TOP_N" "$ENRICHED_FILE" | while IFS='|' read -r title url source rest; do
|
||||
is_tweet=""
|
||||
if echo "$source" | grep -q "(tweet)"; then
|
||||
is_tweet="yes"
|
||||
fi
|
||||
if [ -n "$is_tweet" ]; then
|
||||
echo " [tweet] $title"
|
||||
else
|
||||
echo " * $title"
|
||||
fi
|
||||
if [ -n "$url" ]; then
|
||||
if [ -n "$is_tweet" ]; then
|
||||
echo " View tweet: $url"
|
||||
else
|
||||
echo " Link: $url"
|
||||
fi
|
||||
fi
|
||||
source_clean=$(echo "$source" | sed 's/ (tweet)//')
|
||||
if [ -n "$source_clean" ]; then
|
||||
echo " Source: $source_clean"
|
||||
fi
|
||||
echo ""
|
||||
echo "---"
|
||||
echo ""
|
||||
done
|
||||
else
|
||||
python3 -c '
|
||||
import sys, json
|
||||
|
||||
picks_file = sys.argv[1]
|
||||
|
||||
EMOJI_MAP = {
|
||||
"rss": "[article]",
|
||||
"twitter": "[tweet]",
|
||||
"github": "[github]",
|
||||
}
|
||||
|
||||
with open(picks_file, "r") as f:
|
||||
lines = f.readlines()
|
||||
|
||||
total = sum(1 for l in lines if l.strip())
|
||||
|
||||
for i, line in enumerate(lines):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
pick = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
rank = pick.get("rank", "?")
|
||||
title = pick.get("title", "(no title)")
|
||||
summary = pick.get("summary", "")
|
||||
url = pick.get("url", "")
|
||||
source = pick.get("source", "unknown")
|
||||
category = pick.get("category", "other")
|
||||
story_type = pick.get("type", "rss")
|
||||
|
||||
is_tweet = "(tweet)" in source
|
||||
|
||||
if is_tweet:
|
||||
tag = "[tweet]"
|
||||
else:
|
||||
tag = EMOJI_MAP.get(story_type, "[article]")
|
||||
|
||||
print(f"{rank}. {tag} {title}")
|
||||
if summary:
|
||||
print(f" Why: {summary}")
|
||||
if url:
|
||||
if is_tweet:
|
||||
print(f" View tweet: {url}")
|
||||
else:
|
||||
print(f" Link: {url}")
|
||||
source_display = source.replace(" (tweet)", "")
|
||||
print(f" Source: {source_display} [{category}]")
|
||||
print()
|
||||
if i < len(lines) - 1:
|
||||
print("---")
|
||||
print()
|
||||
' "$PICKS_FILE"
|
||||
fi
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# CLEANUP: Mark articles as read in blogwatcher
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo "Marking RSS articles as read..."
|
||||
echo "y" | /usr/local/bin/blogwatcher read-all > /dev/null 2>&1 || echo " Warning: Could not mark articles as read"
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# STATS
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo "Sources: $RSS_COUNT RSS + $REDDIT_COUNT Reddit + $((TWITTER_COUNT + TWITTER_API_COUNT)) Twitter + $GITHUB_COUNT GitHub + $TAVILY_COUNT Tavily"
|
||||
echo "Pipeline: $TOTAL_CANDIDATES raw -> ${SCORED_COUNT:-N/A} scored -> $PICKS_COUNT curated picks"
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
@@ -0,0 +1,173 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quality scoring pre-filter for the news scan pipeline.
|
||||
|
||||
Reads pipe-delimited articles (TITLE|URL|SOURCE or TITLE|URL|SOURCE|TIER),
|
||||
scores them based on source tier, title quality, freshness signals, and
|
||||
deduplicates by title similarity.
|
||||
|
||||
Outputs the top N articles in the same pipe-delimited format, sorted by score.
|
||||
|
||||
Usage:
|
||||
python3 quality_score.py --input articles.txt [--max 50] [--dedup-threshold 0.80]
|
||||
"""
|
||||
|
||||
import sys
|
||||
import re
|
||||
import argparse
|
||||
from difflib import SequenceMatcher
|
||||
|
||||
# ── Source priority scoring ──────────────────────────────────────────
|
||||
# Higher = better. Customize to match your blogwatcher feed names.
|
||||
PRIORITY_SOURCES = {
|
||||
# T1: Wire services + official AI lab blogs (+5 bonus)
|
||||
'Reuters Tech': 5, 'Bloomberg Tech': 5, 'Axios AI': 5, 'CNBC Tech': 5,
|
||||
'OpenAI Blog': 5,
|
||||
# T2: Tech press + priority bloggers (+3 bonus)
|
||||
'TechCrunch AI': 3, 'The Verge': 3, 'THE DECODER': 3, 'VentureBeat AI': 3,
|
||||
'Ars Technica': 3, '404 Media': 3, 'Wired AI': 3, 'MIT Tech Review': 3,
|
||||
'Google AI Blog': 3, 'Hugging Face Blog': 3, 'Simon Willison': 3,
|
||||
'Latent Space': 3, 'Crunchbase News': 3,
|
||||
# T3: Aggregators (+1 bonus)
|
||||
'Hacker News AI': 1, 'SiliconANGLE AI': 1, 'AI News': 1,
|
||||
'Gary Marcus': 1, 'Bens Bites': 1,
|
||||
# X/Twitter (+2 — original source, not aggregated)
|
||||
'X/Twitter': 2,
|
||||
}
|
||||
|
||||
# High-value keywords that boost score
|
||||
HIGH_VALUE_KEYWORDS = re.compile(
|
||||
r'\b(acqui|merger|billion|partnership|launch|release|'
|
||||
r'announce|breakthrough|regulation|ban|security|vulnerability|'
|
||||
r'open.source|Pentagon|military|government|antitrust)\b',
|
||||
re.IGNORECASE
|
||||
)
|
||||
|
||||
# Signal words for breaking/exclusive news
|
||||
BREAKING_KEYWORDS = re.compile(
|
||||
r'\b(breaking|exclusive|just in|confirmed|leaked|first look|'
|
||||
r'officially|unveil|reveal)\b',
|
||||
re.IGNORECASE
|
||||
)
|
||||
|
||||
|
||||
def title_similarity(t1, t2):
|
||||
"""Fast title similarity using SequenceMatcher."""
|
||||
return SequenceMatcher(None, t1.lower(), t2.lower()).ratio()
|
||||
|
||||
|
||||
def compute_score(title, source, tier_str):
|
||||
"""Compute a quality score for an article."""
|
||||
score = 0
|
||||
|
||||
score += PRIORITY_SOURCES.get(source, 0)
|
||||
|
||||
if source.startswith('r/'):
|
||||
score += 1
|
||||
|
||||
if source.startswith('GitHub'):
|
||||
score += 2
|
||||
|
||||
try:
|
||||
tier = int(tier_str) if tier_str else 3
|
||||
except ValueError:
|
||||
tier = 3
|
||||
if tier == 1:
|
||||
score += 4
|
||||
elif tier == 2:
|
||||
score += 2
|
||||
elif tier == 3:
|
||||
score += 1
|
||||
|
||||
hv_matches = HIGH_VALUE_KEYWORDS.findall(title)
|
||||
score += min(len(hv_matches) * 2, 6)
|
||||
|
||||
if BREAKING_KEYWORDS.search(title):
|
||||
score += 3
|
||||
|
||||
title_len = len(title)
|
||||
if title_len < 30:
|
||||
score -= 1
|
||||
elif 50 <= title_len <= 150:
|
||||
score += 1
|
||||
|
||||
return score
|
||||
|
||||
|
||||
def deduplicate(articles, threshold=0.80):
|
||||
"""Remove near-duplicate articles by title similarity. Keep highest-scored."""
|
||||
unique = []
|
||||
for article in articles:
|
||||
is_dup = False
|
||||
for existing in unique:
|
||||
sim = title_similarity(article['title'], existing['title'])
|
||||
if sim >= threshold:
|
||||
is_dup = True
|
||||
if article['score'] > existing['score']:
|
||||
unique.remove(existing)
|
||||
unique.append(article)
|
||||
break
|
||||
if not is_dup:
|
||||
unique.append(article)
|
||||
return unique
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Quality scoring pre-filter")
|
||||
parser.add_argument('--input', '-i', required=True, help='Input pipe-delimited file')
|
||||
parser.add_argument('--max', type=int, default=50, help='Max articles to output (default: 50)')
|
||||
parser.add_argument('--dedup-threshold', type=float, default=0.80,
|
||||
help='Title similarity threshold for dedup (default: 0.80)')
|
||||
args = parser.parse_args()
|
||||
|
||||
articles = []
|
||||
try:
|
||||
with open(args.input, 'r') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
parts = line.split('|')
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
title = parts[0]
|
||||
url = parts[1]
|
||||
source = parts[2]
|
||||
tier = parts[3] if len(parts) > 3 else ''
|
||||
|
||||
score = compute_score(title, source, tier)
|
||||
articles.append({
|
||||
'title': title,
|
||||
'url': url,
|
||||
'source': source,
|
||||
'tier': tier,
|
||||
'score': score,
|
||||
'line': line,
|
||||
})
|
||||
except FileNotFoundError:
|
||||
print(f"Error: file not found: {args.input}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if not articles:
|
||||
print("No articles to score", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
articles.sort(key=lambda x: -x['score'])
|
||||
unique = deduplicate(articles, args.dedup_threshold)
|
||||
unique.sort(key=lambda x: -x['score'])
|
||||
output = unique[:args.max]
|
||||
|
||||
for article in output:
|
||||
if article['tier']:
|
||||
print(f"{article['title']}|{article['url']}|{article['source']}|{article['tier']}")
|
||||
else:
|
||||
print(f"{article['title']}|{article['url']}|{article['source']}")
|
||||
|
||||
total = len(articles)
|
||||
deduped = total - len(unique)
|
||||
final = len(output)
|
||||
print(f" Done: {total} in -> {deduped} dupes removed -> {final} out", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+88
@@ -0,0 +1,88 @@
|
||||
#!/bin/bash
|
||||
# Twitter/X AI News Scanner
|
||||
# Scans official accounts, reporters/leakers, and trending AI topics
|
||||
# Requires: bird CLI (https://github.com/nicholasgasior/bird)
|
||||
|
||||
set -e
|
||||
|
||||
BIRD="/usr/local/bin/bird"
|
||||
|
||||
# Auth check: bird reads AUTH_TOKEN and CT0 from env vars automatically.
|
||||
# If not in env (e.g., SSH session), fall back to Chrome cookies.
|
||||
if [ -z "$AUTH_TOKEN" ] || [ -z "$CT0" ]; then
|
||||
if [ -d "$HOME/Library/Application Support/Google/Chrome" ]; then
|
||||
BIRD_EXTRA="--cookie-source chrome"
|
||||
else
|
||||
echo "Warning: No X auth available (no AUTH_TOKEN/CT0 env vars, no Chrome cookies)"
|
||||
echo "Twitter scan skipped."
|
||||
exit 0
|
||||
fi
|
||||
else
|
||||
BIRD_EXTRA=""
|
||||
fi
|
||||
|
||||
echo "Scanning X/Twitter for AI news..."
|
||||
|
||||
# Tier 1: Official AI company accounts (announcements)
|
||||
OFFICIAL_ACCOUNTS=(
|
||||
"OpenAI"
|
||||
"AnthropicAI"
|
||||
"GoogleAI"
|
||||
"Google"
|
||||
"HuggingFace"
|
||||
"MetaAI"
|
||||
"MistralAI"
|
||||
"DeepMind"
|
||||
"xAI"
|
||||
"NVIDIAAIDev"
|
||||
"Apple"
|
||||
"MicrosoftAI"
|
||||
)
|
||||
|
||||
# Tier 2: Reporters, leakers, and fast-signal accounts (break news first)
|
||||
# Customize: add reporters who cover your beat
|
||||
REPORTER_ACCOUNTS=(
|
||||
"btibor91"
|
||||
"testingcatalog"
|
||||
"kylewiggers"
|
||||
"dseetharaman"
|
||||
"rachelmetz"
|
||||
"CadeMetz"
|
||||
"inafried"
|
||||
"_philschmid"
|
||||
"rohanpaul_ai"
|
||||
)
|
||||
|
||||
# Tier 3: CEO/thought leader accounts (context, not breaking)
|
||||
CEO_ACCOUNTS=(
|
||||
"sama"
|
||||
"darioamodei"
|
||||
"ylecun"
|
||||
"karpathy"
|
||||
"elonmusk"
|
||||
)
|
||||
|
||||
echo "Scanning official accounts..."
|
||||
for acct in "${OFFICIAL_ACCOUNTS[@]}"; do
|
||||
timeout 8s $BIRD $BIRD_EXTRA search "from:$acct" -n 3 --plain 2>/dev/null | head -20 || true
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Scanning reporters & leakers..."
|
||||
for acct in "${REPORTER_ACCOUNTS[@]}"; do
|
||||
timeout 8s $BIRD $BIRD_EXTRA search "from:$acct" -n 3 --plain 2>/dev/null | head -20 || true
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Breaking AI news search..."
|
||||
timeout 10s $BIRD $BIRD_EXTRA search just launched OR now available OR rolling out OR just released AI model -filter:replies -filter:retweets -n 8 --plain 2>/dev/null | head -40 || true
|
||||
|
||||
echo ""
|
||||
echo "Product launches & announcements..."
|
||||
timeout 10s $BIRD $BIRD_EXTRA search introducing OR announcing AI OR LLM OR model -filter:replies -filter:retweets -n 8 --plain 2>/dev/null | head -40 || true
|
||||
|
||||
echo ""
|
||||
echo "CEO signals (context only)..."
|
||||
for acct in "${CEO_ACCOUNTS[@]}"; do
|
||||
timeout 8s $BIRD $BIRD_EXTRA search "from:$acct" -n 2 --plain 2>/dev/null | head -15 || true
|
||||
done
|
||||
@@ -0,0 +1,148 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Editorial Profile Updater
|
||||
Analyzes approval/rejection patterns and updates the editorial profile.
|
||||
|
||||
Usage: python3 update_editorial_profile.py [--dry-run]
|
||||
|
||||
Reads: memory/editorial_decisions.md
|
||||
Updates: memory/editorial_profile.md (Approval History Stats section)
|
||||
|
||||
Set OPENCLAW_WORKSPACE env var or defaults to ~/.openclaw/workspace
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
WORKSPACE = Path(os.environ.get("OPENCLAW_WORKSPACE",
|
||||
os.path.expanduser("~/.openclaw/workspace")))
|
||||
DECISIONS_PATH = WORKSPACE / "memory" / "editorial_decisions.md"
|
||||
PROFILE_PATH = WORKSPACE / "memory" / "editorial_profile.md"
|
||||
|
||||
|
||||
def parse_decisions():
|
||||
if not DECISIONS_PATH.exists():
|
||||
return []
|
||||
decisions = []
|
||||
with open(DECISIONS_PATH) as f:
|
||||
for line in f:
|
||||
m = re.match(
|
||||
r"\[(.*?)\]\s*(APPROVED|SKIPPED|MANUAL_DRAFT)\s*\|\s*(.*?)\s*\|\s*(.*?)\s*\|\s*(.*)",
|
||||
line.strip()
|
||||
)
|
||||
if m:
|
||||
decisions.append({
|
||||
"timestamp": m.group(1),
|
||||
"action": m.group(2),
|
||||
"title": m.group(3),
|
||||
"url": m.group(4),
|
||||
"category": m.group(5).strip(),
|
||||
})
|
||||
return decisions
|
||||
|
||||
|
||||
def analyze_patterns(decisions):
|
||||
if not decisions:
|
||||
return "- No decisions logged yet.\n- Tracking begins when you approve/skip stories.\n"
|
||||
|
||||
category_stats = defaultdict(lambda: {"approved": 0, "skipped": 0, "manual_draft": 0})
|
||||
total_approved = 0
|
||||
total_skipped = 0
|
||||
total_manual = 0
|
||||
|
||||
for d in decisions:
|
||||
cat = d["category"]
|
||||
action = d["action"].lower()
|
||||
category_stats[cat][action] += 1
|
||||
if action == "approved":
|
||||
total_approved += 1
|
||||
elif action == "manual_draft":
|
||||
total_manual += 1
|
||||
else:
|
||||
total_skipped += 1
|
||||
|
||||
total = total_approved + total_skipped + total_manual
|
||||
approval_rate = (total_approved / max(total_approved + total_skipped, 1) * 100)
|
||||
|
||||
report = f"- Total decisions: {total}\n"
|
||||
report += f"- Scanner approved: {total_approved} ({approval_rate:.0f}% of scanner stories)\n"
|
||||
report += f"- Scanner skipped: {total_skipped}\n"
|
||||
report += f"- Manual drafts: {total_manual} (stories you sought out yourself)\n"
|
||||
report += "- Last updated: " + datetime.now().strftime("%Y-%m-%d") + "\n\n"
|
||||
report += "Category breakdown:\n"
|
||||
for cat, stats in sorted(category_stats.items(),
|
||||
key=lambda x: -(x[1]["approved"] + x[1]["manual_draft"])):
|
||||
a = stats["approved"]
|
||||
s = stats["skipped"]
|
||||
m = stats["manual_draft"]
|
||||
scanner_total = a + s
|
||||
rate = (a / scanner_total * 100) if scanner_total > 0 else 0
|
||||
parts = []
|
||||
if a:
|
||||
parts.append(f"{a} approved")
|
||||
if s:
|
||||
parts.append(f"{s} skipped")
|
||||
if m:
|
||||
parts.append(f"{m} manual")
|
||||
if scanner_total > 0:
|
||||
report += f" - {cat}: {', '.join(parts)} ({rate:.0f}% scanner approval)\n"
|
||||
else:
|
||||
report += f" - {cat}: {', '.join(parts)} (manual only)\n"
|
||||
|
||||
# Blind spot analysis
|
||||
blind_spots = []
|
||||
for cat, stats in category_stats.items():
|
||||
m = stats["manual_draft"]
|
||||
a = stats["approved"]
|
||||
if m > 0 and m > a:
|
||||
blind_spots.append((cat, m, a))
|
||||
|
||||
if blind_spots:
|
||||
report += "\n## Scanner Blind Spots\n"
|
||||
report += "Topics you manually seek out but the scanner rarely catches:\n"
|
||||
for cat, manual_count, scanner_count in sorted(blind_spots, key=lambda x: -x[1]):
|
||||
report += f" - **{cat}**: {manual_count} manual draft(s) vs {scanner_count} scanner catch(es). "
|
||||
if scanner_count == 0:
|
||||
report += "Scanner never found this topic.\n"
|
||||
else:
|
||||
report += "Consider adding more RSS feeds or keywords.\n"
|
||||
|
||||
return report
|
||||
|
||||
|
||||
def update_profile(analysis, dry_run=False):
|
||||
profile = PROFILE_PATH.read_text()
|
||||
marker = "## Approval History Stats"
|
||||
blind_marker = "## Scanner Blind Spots"
|
||||
|
||||
if blind_marker in profile:
|
||||
profile = profile[:profile.index(blind_marker)]
|
||||
|
||||
if marker in profile:
|
||||
before = profile[:profile.index(marker)]
|
||||
new_section = f"{marker}\n{analysis}"
|
||||
updated = before + new_section
|
||||
else:
|
||||
updated = profile + f"\n{marker}\n{analysis}"
|
||||
|
||||
if dry_run:
|
||||
print("DRY RUN -- would update profile with:")
|
||||
print(new_section if marker in profile else f"\n{marker}\n{analysis}")
|
||||
else:
|
||||
PROFILE_PATH.write_text(updated)
|
||||
print(f"Updated {PROFILE_PATH}")
|
||||
|
||||
|
||||
def main():
|
||||
dry_run = "--dry-run" in sys.argv
|
||||
decisions = parse_decisions()
|
||||
analysis = analyze_patterns(decisions)
|
||||
print(analysis)
|
||||
update_profile(analysis, dry_run)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user