mirror of
https://github.com/astonysh/OpenClaw-DeepReeder.git
synced 2026-08-14 00:57:54 +00:00
Tony Li
d59ae38920
Autonomous web content ingestion engine for AI agents. Features: - Generic article/blog parser (Trafilatura + BeautifulSoup fallback) - Twitter/X parser (via Nitter instances) - YouTube transcript parser - Clean Markdown output with YAML frontmatter - Automatic URL detection and routing - Structured memory storage
25 lines
479 B
Plaintext
25 lines
479 B
Plaintext
# DeepReader Skill - Dependencies
|
|
# Web scraping & content extraction
|
|
trafilatura>=1.12.0
|
|
requests>=2.31.0
|
|
lxml>=5.1.0
|
|
lxml-html-clean>=0.4.0
|
|
|
|
# YouTube transcript extraction
|
|
youtube-transcript-api>=0.6.2
|
|
|
|
# Data validation & settings
|
|
pydantic>=2.5.0
|
|
pydantic-settings>=2.1.0
|
|
|
|
# URL parsing & utilities
|
|
tldextract>=5.1.0
|
|
|
|
# HTML parsing (fallback)
|
|
beautifulsoup4>=4.12.0
|
|
|
|
# Unique ID generation (stdlib, listed for clarity)
|
|
# uuid - built-in
|
|
# re - built-in
|
|
# hashlib - built-in
|