admin/OpenClaw-DeepReeder
1
0
Fork 0
mirror of https://github.com/astonysh/OpenClaw-DeepReeder.git synced 2026-08-14 00:57:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
17 Commits 1 Branch 0 Tags
main
Commit Graph
17 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Tony Li d7ed514cc5 docs: update cn README with FIRECRAWL_API_KEY config 2026-02-22 15:47:52 +01:00
Tony Li d89896cbe6 docs: update translated READMEs with FIRECRAWL_API_KEY config 2026-02-22 15:47:03 +01:00
Tony Li 0c91f0068e feat(parser): add Firecrawl fallback for paywalled and bot-protected content 
Integrates the Firecrawl API into GenericParser. If native HTTP requests fail
(e.g., 403 Forbidden) or returning nothing due to paywalls, we automatically
fallback to Firecrawl to extract Markdown content using pseudosearch engine features.

Requires FIRECRAWL_API_KEY environment variable to activate.
 2026-02-22 15:41:34 +01:00
Tony Li b700e86ee4 fix(security): resolve security issues reported by bot 
- Restrict loose URL trigger in manifest.json to only respond to explicit read commands
- Add security warning and wrap external content in <external_content> tags in storage.py to prevent prompt injections
- Add requirements.txt for local deployment
- Support environment variables for Twitter/Nitter custom API instances
- Fix pyproject.toml legacy build backend
 2026-02-21 20:53:28 +01:00
Tony Li 3ac62458b6 Merge pull request #4 from BlueBirdBack/feat/twitter-profile-url-support 
feat: support X profile URLs in DeepReader
 2026-02-21 16:52:06 +01:00
Bruce Bell (B3) 7e3549806e feat: support X profile URLs in twitter parser 2026-02-21 16:45:49 +01:00
Tony Li 4b9b3a1d63 Merge pull request #3 from BlueBirdBack/reliability-config-and-twitter-guardrails 
feat: implement env config + add Nitter fetch guardrails
 2026-02-21 16:07:39 +01:00
Bruce Bell (B3) 23fa006c8f feat: implement env config and harden nitter fallback fetch 2026-02-21 16:06:15 +01:00
Tony Li f230208777 Merge pull request #2 from BlueBirdBack/security-hardening-ssrf-yaml-limits 
security: harden URL fetches + YAML frontmatter handling
 2026-02-21 16:03:34 +01:00
Bruce Bell (B3) bd73a0f9b1 security: harden URL fetches and frontmatter serialization 2026-02-21 16:00:53 +01:00
Tony Li 6b835b1d8b feat: integrate notebooklm-py for Audio Overview and Notebook artifact generation 2026-02-20 14:46:57 +01:00
Tony Li a20843abc5 🎯 Reposition as default OpenClaw web gateway for ClawHub SEO 
- Rewrite README.md with 'Use When' pattern for ClawHub discovery
- Add SKILL.md for ClawHub detail page display
- Update manifest.json with SEO-optimized description and expanded tags
- Update pyproject.toml with new keywords and ClawHub URL
- Sync all translations (zh, es, fr, ko, ja, ar) with new positioning
- Add 'Why DeepReader?' comparison table
- Add ClawHub install command: npx clawhub@latest install deepreader
 2026-02-18 07:30:12 +01:00
Tony Li b8c452baaa 🟠 Add Reddit parser + update all READMEs 
Reddit Parser:
- Uses Reddit's native .json URL suffix (zero dependencies)
- Extracts post title, body, author, score, flair
- Top comments sorted by score (up to 15)
- Nested reply threads (up to 3 levels deep)
- Media detection (images, galleries, Reddit video)
- Proper error handling (429 rate limit, 404, 403)

Core:
- Added is_reddit_url() to utils.py
- Registered RedditParser in router.py
- Added reddit_reading capability to manifest.json

README:
- Updated all 7 language versions with Reddit section
  (EN, 中文, Español, 한국어, 日本語, العربية, Français)
 2026-02-16 14:22:31 +01:00
Tony Li a45f09dfe3 🐦 Integrate FxTwitter API + multilingual README 
Twitter/X Parser:
- Primary: FxTwitter API (zero-dep, structured JSON)
  - Regular tweets with engagement stats
  - X Articles (long-form content) with full text
  - Quoted tweets with nested content
  - Media extraction (images, videos, GIFs)
- Fallback: Nitter instances (reply thread extraction)
- Credits: inspired by x-tweet-fetcher by ythx-101

README:
- Updated with FxTwitter integration details
- Added credits section
- Added multilingual versions:
  - 🇨🇳 中文 (README_zh.md)
  - 🇪🇸 Español (README_es.md)
  - 🇰🇷 한국어 (README_ko.md)
  - 🇯🇵 日本語 (README_ja.md)
  - 🇸🇦 العربية (README_ar.md)
  - 🇫🇷 Français (README_fr.md)
 2026-02-16 14:09:53 +01:00
Tony Li 21de5c01f7 Merge remote: use local versions of README and LICENSE 2026-02-16 13:59:19 +01:00
Tony Li d59ae38920 🦞 Initial release: OpenClaw DeepReeder v1.0.0 
Autonomous web content ingestion engine for AI agents.

Features:
- Generic article/blog parser (Trafilatura + BeautifulSoup fallback)
- Twitter/X parser (via Nitter instances)
- YouTube transcript parser
- Clean Markdown output with YAML frontmatter
- Automatic URL detection and routing
- Structured memory storage
 2026-02-16 13:58:54 +01:00
Tony Li 15f832da00 Initial commit 2026-02-16 13:54:51 +01:00