mirror of
https://github.com/astonysh/OpenClaw-DeepReeder.git
synced 2026-08-14 00:57:54 +00:00
🎯 Reposition as default OpenClaw web gateway for ClawHub SEO
- Rewrite README.md with 'Use When' pattern for ClawHub discovery - Add SKILL.md for ClawHub detail page display - Update manifest.json with SEO-optimized description and expanded tags - Update pyproject.toml with new keywords and ClawHub URL - Sync all translations (zh, es, fr, ko, ja, ar) with new positioning - Add 'Why DeepReader?' comparison table - Add ClawHub install command: npx clawhub@latest install deepreader
This commit is contained in:
@@ -1,23 +1,54 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
# 🦞 OpenClaw DeepReader
|
||||
|
||||
> **Autonomous web content ingestion engine for AI agents.**
|
||||
> **The default web content gateway for OpenClaw agents.** Read X (Twitter), Reddit, YouTube, and any webpage — zero config, zero API keys.
|
||||
|
||||
DeepReeder intercepts URLs from user messages, scrapes content intelligently using specialized parsers, formats it into clean Markdown with YAML frontmatter, and saves it to the agent's long-term memory.
|
||||
DeepReader is the built-in content reader for the [OpenClaw](https://github.com/anthropics/openclaw) agent framework. Paste any URL into a conversation, and DeepReader automatically fetches, parses, and saves high-quality Markdown to your agent's long-term memory. Built for social media and the modern web.
|
||||
|
||||
🌍 **Translations**: [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ Features
|
||||
## ⚡ Install
|
||||
|
||||
| Parser | Sources | Method |
|
||||
|--------|---------|--------|
|
||||
| 🌐 **Generic** | Blogs, articles, docs | [Trafilatura](https://trafilatura.readthedocs.io/) with BeautifulSoup fallback |
|
||||
| 🐦 **Twitter / X** | Tweets, threads, X Articles | **FxTwitter API** (primary) + Nitter (fallback) |
|
||||
| 🟠 **Reddit** | Posts + comment threads | **Reddit .json API** (zero-config) |
|
||||
| 🎬 **YouTube** | Video transcripts | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
```bash
|
||||
npx clawhub@latest install deepreader
|
||||
```
|
||||
|
||||
### 🐦 Twitter / X — Deep Integration
|
||||
Or install manually:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Use When
|
||||
|
||||
- You need to **read a tweet, thread, or X article** and add it to OpenClaw's memory
|
||||
- You need to **ingest a Reddit post** with top comments and discussion context
|
||||
- You want to **save a YouTube transcript** for later reference or analysis
|
||||
- You want to **clip any blog, article, or documentation page** into clean Markdown
|
||||
- Your agent needs a **default web reader** that just works — no API keys, no setup
|
||||
|
||||
---
|
||||
|
||||
## ✨ Supported Sources
|
||||
|
||||
| Parser | Sources | Method | API Key? |
|
||||
|--------|---------|--------|----------|
|
||||
| 🐦 **Twitter / X** | Tweets, threads, X Articles | [FxTwitter API](https://github.com/FxEmbed/FxEmbed) + Nitter fallback | ❌ None |
|
||||
| 🟠 **Reddit** | Posts + comment threads | Reddit `.json` API | ❌ None |
|
||||
| 🎬 **YouTube** | Video transcripts | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) | ❌ None |
|
||||
| 🌐 **Any URL** | Blogs, articles, docs | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup | ❌ None |
|
||||
|
||||
**Zero API keys. Zero login. Zero rate limits. Just paste and read.**
|
||||
|
||||
---
|
||||
|
||||
## 🐦 Twitter / X — Deep Integration
|
||||
|
||||
Powered by [FxTwitter](https://github.com/FxEmbed/FxEmbed) API with Nitter fallback. Inspired by [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
|
||||
|
||||
@@ -31,7 +62,7 @@ Powered by [FxTwitter](https://github.com/FxEmbed/FxEmbed) API with Nitter fallb
|
||||
| Reply threads | ✅ Via Nitter fallback (first 5) |
|
||||
| Engagement stats | ✅ ❤️ likes, 🔁 RTs, 👁️ views, 🔖 bookmarks |
|
||||
|
||||
### 🟠 Reddit — Native JSON Integration
|
||||
## 🟠 Reddit — Native JSON Integration
|
||||
|
||||
Uses Reddit's built-in `.json` URL suffix — **no API keys, no OAuth, no registration**.
|
||||
|
||||
@@ -45,9 +76,38 @@ Uses Reddit's built-in `.json` URL suffix — **no API keys, no OAuth, no regist
|
||||
| Post stats | ✅ ⬆️ score, 💬 comment count, upvote ratio |
|
||||
| Flair tags | ✅ Included |
|
||||
|
||||
**No API keys. No login. No rate limits.**
|
||||
---
|
||||
|
||||
### Output Format
|
||||
## 🚀 Quick Start
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# Read a tweet → saves to agent memory
|
||||
result = run("Check out this tweet: https://x.com/elonmusk/status/123456")
|
||||
|
||||
# Read a Reddit discussion → captures post + top comments
|
||||
result = run("Great thread: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
|
||||
# Read a YouTube video → saves full transcript
|
||||
result = run("Watch this: https://youtube.com/watch?v=dQw4w9WgXcQ")
|
||||
|
||||
# Read any article → extracts clean content
|
||||
result = run("Interesting read: https://example.com/blog/ai-agents-2026")
|
||||
|
||||
# Batch process multiple URLs at once
|
||||
result = run("""
|
||||
Here are some links to read:
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://example.com/article
|
||||
""")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📄 Output Format
|
||||
|
||||
Every piece of content is saved as a `.md` file with structured YAML frontmatter:
|
||||
|
||||
@@ -80,53 +140,6 @@ Post body goes here...
|
||||
|
||||
---
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# Create a virtual environment
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# Process a single URL
|
||||
result = run("Check out this article: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# Process a tweet (uses FxTwitter API automatically)
|
||||
result = run("Interesting thread: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Process a Reddit post (uses .json API automatically)
|
||||
result = run("Great discussion: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# Process multiple URLs at once
|
||||
result = run("""
|
||||
Here are some links:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
@@ -146,7 +159,7 @@ deepreader_skill/
|
||||
└── youtube.py # YouTube transcript parser
|
||||
```
|
||||
|
||||
### Parser Selection Strategy
|
||||
### Router Strategy
|
||||
|
||||
```
|
||||
URL detected → is Twitter/X? → FxTwitter API → Nitter fallback
|
||||
@@ -159,7 +172,7 @@ URL detected → is Twitter/X? → FxTwitter API → Nitter fallback
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
DeepReeder uses sensible defaults out of the box. Configuration can be customized via environment variables:
|
||||
DeepReader uses sensible defaults out of the box. Configuration can be customized via environment variables:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
@@ -168,6 +181,20 @@ DeepReeder uses sensible defaults out of the box. Configuration can be customize
|
||||
|
||||
---
|
||||
|
||||
## 💡 Why DeepReader?
|
||||
|
||||
| Feature | DeepReader | Manual scraping | Browser tools |
|
||||
|---------|-----------|----------------|---------------|
|
||||
| **Trigger** | Automatic on URL | Manual code | Manual action |
|
||||
| **Twitter/X** | ✅ Full support | ❌ Blocked | ⚠️ Partial |
|
||||
| **Reddit threads** | ✅ + comments | ⚠️ Complex | ⚠️ Slow |
|
||||
| **YouTube transcripts** | ✅ Built-in | ❌ Separate tool | ❌ Not available |
|
||||
| **API keys needed** | ❌ None | ✅ Often | ✅ Sometimes |
|
||||
| **Output format** | Clean Markdown | Raw HTML | Screenshots |
|
||||
| **Memory integration** | ✅ Auto-save | ❌ Manual | ❌ Manual |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Credits
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — Public API for fetching Twitter/X content
|
||||
|
||||
+37
-57
@@ -1,49 +1,33 @@
|
||||
<div dir="rtl">
|
||||
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
# 🦞 OpenClaw DeepReader
|
||||
|
||||
> **محرك استيعاب محتوى الويب الذاتي لوكلاء الذكاء الاصطناعي.**
|
||||
> **بوابة محتوى الويب الافتراضية لوكلاء OpenClaw.** يقرأ X (Twitter)، Reddit، YouTube وأي صفحة ويب — بدون إعداد، بدون مفاتيح API.
|
||||
|
||||
DeepReader هو قارئ المحتوى المدمج في إطار عمل [OpenClaw](https://github.com/anthropics/openclaw). الصق أي رابط في المحادثة، وسيقوم DeepReader تلقائياً بجلب وتحليل وحفظ Markdown عالي الجودة في ذاكرة الوكيل طويلة المدى.
|
||||
|
||||
🌍 **الترجمات**: [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ الميزات
|
||||
## ⚡ التثبيت
|
||||
|
||||
| المحلل | المصادر | الطريقة |
|
||||
|--------|---------|---------|
|
||||
| 🌐 **عام** | مدونات، مقالات، وثائق | [Trafilatura](https://trafilatura.readthedocs.io/) مع BeautifulSoup احتياطي |
|
||||
| 🐦 **Twitter / X** | تغريدات، سلاسل، مقالات X | **FxTwitter API** (رئيسي) + Nitter (احتياطي) |
|
||||
| 🟠 **Reddit** | منشورات + سلاسل تعليقات | **Reddit .json API** (بدون إعداد) |
|
||||
| 🎬 **YouTube** | نصوص الفيديو | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
<div dir="ltr">
|
||||
|
||||
### 🟠 Reddit — تكامل JSON الأصلي
|
||||
```bash
|
||||
npx clawhub@latest install deepreader
|
||||
```
|
||||
|
||||
يستخدم لاحقة URL `.json` المدمجة في Reddit — **بدون مفاتيح API، بدون OAuth، بدون تسجيل**.
|
||||
</div>
|
||||
|
||||
| نوع المحتوى | الدعم |
|
||||
|-------------|-------|
|
||||
| منشورات نصية | ✅ نص Markdown كامل |
|
||||
| منشورات الروابط | ✅ URL + بيانات وصفية |
|
||||
| أفضل التعليقات (مرتبة حسب النقاط) | ✅ حتى 15 تعليقاً |
|
||||
| سلاسل الردود المتداخلة | ✅ حتى 3 مستويات عمق |
|
||||
| الوسائط (صور، معارض، فيديو) | ✅ استخراج الروابط |
|
||||
| إحصائيات المنشور | ✅ ⬆️ النقاط، 💬 عدد التعليقات |
|
||||
| علامات Flair | ✅ مضمّنة |
|
||||
|
||||
**بدون مفاتيح API. بدون تسجيل دخول. بدون حدود للسرعة.**
|
||||
|
||||
---
|
||||
|
||||
## 📦 التثبيت
|
||||
أو التثبيت يدوياً:
|
||||
|
||||
<div dir="ltr">
|
||||
|
||||
```bash
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
@@ -51,6 +35,29 @@ pip install -e .
|
||||
|
||||
---
|
||||
|
||||
## 🎯 استخدمه عندما
|
||||
|
||||
- تحتاج لـ **قراءة تغريدة أو سلسلة أو مقال X** وإضافتها لذاكرة OpenClaw
|
||||
- تحتاج لـ **استيعاب منشور Reddit** مع أفضل التعليقات
|
||||
- تريد **حفظ نص فيديو YouTube** للمراجعة لاحقاً
|
||||
- تريد **قص أي مدونة أو مقال** إلى Markdown نظيف
|
||||
- وكيلك يحتاج **قارئ ويب افتراضي** يعمل فوراً — بدون مفاتيح API
|
||||
|
||||
---
|
||||
|
||||
## ✨ المصادر المدعومة
|
||||
|
||||
| المحلل | المصادر | الطريقة | مفتاح API؟ |
|
||||
|--------|---------|---------|-----------|
|
||||
| 🐦 **Twitter / X** | تغريدات، سلاسل | FxTwitter API + Nitter | ❌ لا |
|
||||
| 🟠 **Reddit** | منشورات + تعليقات | Reddit `.json` API | ❌ لا |
|
||||
| 🎬 **YouTube** | نصوص الفيديو | youtube-transcript-api | ❌ لا |
|
||||
| 🌐 **أي رابط** | مدونات، مقالات | Trafilatura + BeautifulSoup | ❌ لا |
|
||||
|
||||
**بدون مفاتيح API. بدون تسجيل دخول. بدون حدود. الصق واقرأ.**
|
||||
|
||||
---
|
||||
|
||||
## 🚀 البداية السريعة
|
||||
|
||||
<div dir="ltr">
|
||||
@@ -58,36 +65,9 @@ pip install -e .
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
result = run("اطلع على هذا: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
result = run("اطلع على هذا: https://x.com/user/status/123456")
|
||||
result = run("نقاش رائع: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ الهيكل
|
||||
|
||||
<div dir="ltr">
|
||||
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # نقطة الدخول — دالة run()
|
||||
├── manifest.json # بيانات المهارة الوصفية
|
||||
├── requirements.txt # قائمة التبعيات
|
||||
├── core/
|
||||
│ ├── router.py # منطق توجيه URL → المحلل
|
||||
│ ├── storage.py # إنشاء وحفظ ملفات Markdown
|
||||
│ └── utils.py # استخراج URL ودوال مساعدة
|
||||
└── parsers/
|
||||
├── base.py # المحلل الأساسي
|
||||
├── generic.py # محلل المقالات العام
|
||||
├── twitter.py # محلل Twitter/X
|
||||
├── reddit.py # محلل Reddit (.json API)
|
||||
└── youtube.py # محلل نصوص YouTube
|
||||
result = run("شاهد هذا: https://youtube.com/watch?v=dQw4w9WgXcQ")
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
+72
-55
@@ -1,28 +1,59 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
# 🦞 OpenClaw DeepReader
|
||||
|
||||
> **Motor autónomo de ingestión de contenido web para agentes de IA.**
|
||||
> **La pasarela de contenido web predeterminada para agentes OpenClaw.** Lee X (Twitter), Reddit, YouTube y cualquier página web — sin configuración, sin claves API.
|
||||
|
||||
DeepReeder intercepta URLs de los mensajes de usuario, extrae contenido de forma inteligente usando parsers especializados, lo formatea en Markdown limpio con metadatos YAML frontmatter, y lo guarda en la memoria a largo plazo del agente.
|
||||
DeepReader es el lector de contenido integrado para el framework de agentes [OpenClaw](https://github.com/anthropics/openclaw). Pega cualquier URL en una conversación, y DeepReader automáticamente obtiene, analiza y guarda Markdown de alta calidad en la memoria a largo plazo del agente. Diseñado para redes sociales y la web moderna.
|
||||
|
||||
🌍 **Traducciones**: [English](README.md) · [中文](README_zh.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ Características
|
||||
## ⚡ Instalación
|
||||
|
||||
| Parser | Fuentes | Método |
|
||||
|--------|---------|--------|
|
||||
| 🌐 **Genérico** | Blogs, artículos, documentación | [Trafilatura](https://trafilatura.readthedocs.io/) con fallback BeautifulSoup |
|
||||
| 🐦 **Twitter / X** | Tweets, hilos, X Articles | **FxTwitter API** (principal) + Nitter (fallback) |
|
||||
| 🟠 **Reddit** | Posts + hilos de comentarios | **Reddit .json API** (sin configuración) |
|
||||
| 🎬 **YouTube** | Transcripciones de vídeo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
```bash
|
||||
npx clawhub@latest install deepreader
|
||||
```
|
||||
|
||||
### 🐦 Twitter / X — Integración Profunda
|
||||
O instala manualmente:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Úsalo Cuando
|
||||
|
||||
- Necesites **leer un tweet, hilo o artículo de X** y añadirlo a la memoria de OpenClaw
|
||||
- Necesites **ingerir un post de Reddit** con los comentarios principales y contexto de discusión
|
||||
- Quieras **guardar una transcripción de YouTube** para referencia o análisis posterior
|
||||
- Quieras **recortar cualquier blog, artículo o documentación** en Markdown limpio
|
||||
- Tu agente necesite un **lector web predeterminado** que simplemente funcione — sin claves API, sin configuración
|
||||
|
||||
---
|
||||
|
||||
## ✨ Fuentes Soportadas
|
||||
|
||||
| Parser | Fuentes | Método | ¿Clave API? |
|
||||
|--------|---------|--------|-------------|
|
||||
| 🐦 **Twitter / X** | Tweets, hilos, X Articles | [FxTwitter API](https://github.com/FxEmbed/FxEmbed) + Nitter fallback | ❌ Ninguna |
|
||||
| 🟠 **Reddit** | Posts + hilos de comentarios | Reddit `.json` API | ❌ Ninguna |
|
||||
| 🎬 **YouTube** | Transcripciones de vídeo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) | ❌ Ninguna |
|
||||
| 🌐 **Cualquier URL** | Blogs, artículos, docs | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup | ❌ Ninguna |
|
||||
|
||||
**Sin claves API. Sin inicio de sesión. Sin límites. Pega y lee.**
|
||||
|
||||
---
|
||||
|
||||
## 🐦 Twitter / X — Integración Profunda
|
||||
|
||||
Impulsado por la API de [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspirado en [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
|
||||
|
||||
| Tipo de Contenido | Soporte |
|
||||
|-------------------|---------|
|
||||
|-------------------|---------|
|
||||
| Tweets regulares | ✅ Texto completo + estadísticas |
|
||||
| Tweets largos (Twitter Blue) | ✅ Texto completo |
|
||||
| X Articles (contenido largo) | ✅ Artículo completo + recuento de palabras |
|
||||
@@ -31,12 +62,12 @@ Impulsado por la API de [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspira
|
||||
| Hilos de respuestas | ✅ Vía Nitter fallback (primeras 5) |
|
||||
| Estadísticas de interacción | ✅ ❤️ likes, 🔁 RTs, 👁️ vistas, 🔖 marcadores |
|
||||
|
||||
### 🟠 Reddit — Integración JSON Nativa
|
||||
## 🟠 Reddit — Integración JSON Nativa
|
||||
|
||||
Usa el sufijo `.json` nativo de Reddit — **sin claves API, sin OAuth, sin registro**.
|
||||
|
||||
| Tipo de Contenido | Soporte |
|
||||
|-------------------|---------|
|
||||
|-------------------|---------|
|
||||
| Self posts (texto) | ✅ Cuerpo completo en Markdown |
|
||||
| Link posts | ✅ URL + metadatos |
|
||||
| Comentarios principales (por puntuación) | ✅ Hasta 15 comentarios |
|
||||
@@ -45,25 +76,6 @@ Usa el sufijo `.json` nativo de Reddit — **sin claves API, sin OAuth, sin regi
|
||||
| Estadísticas del post | ✅ ⬆️ puntuación, 💬 comentarios, ratio de votos |
|
||||
| Etiquetas Flair | ✅ Incluidas |
|
||||
|
||||
**Sin claves API. Sin inicio de sesión. Sin límites de velocidad.**
|
||||
|
||||
---
|
||||
|
||||
## 📦 Instalación
|
||||
|
||||
```bash
|
||||
# Clonar el repositorio
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# Crear entorno virtual
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Instalar dependencias
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Inicio Rápido
|
||||
@@ -71,27 +83,26 @@ pip install -e .
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# Procesar una sola URL
|
||||
result = run("Mira este artículo: https://example.com/blog/post")
|
||||
print(result)
|
||||
# Leer un tweet → guarda en la memoria del agente
|
||||
result = run("Mira este tweet: https://x.com/elonmusk/status/123456")
|
||||
|
||||
# Procesar un tweet
|
||||
result = run("Hilo interesante: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Procesar un post de Reddit
|
||||
# Leer una discusión de Reddit → captura post + comentarios
|
||||
result = run("Gran discusión: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# Procesar múltiples URLs
|
||||
# Leer un vídeo de YouTube → guarda transcripción completa
|
||||
result = run("Mira esto: https://youtube.com/watch?v=dQw4w9WgXcQ")
|
||||
|
||||
# Leer cualquier artículo → extrae contenido limpio
|
||||
result = run("Lectura interesante: https://example.com/blog/ai-agents-2026")
|
||||
|
||||
# Procesamiento por lotes de múltiples URLs
|
||||
result = run("""
|
||||
Aquí hay algunos enlaces:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://example.com/article
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -102,6 +113,7 @@ print(result)
|
||||
deepreader_skill/
|
||||
├── __init__.py # Punto de entrada — función run()
|
||||
├── manifest.json # Metadatos del skill y configuración de triggers
|
||||
├── SKILL.md # Descripción para ClawHub
|
||||
├── requirements.txt # Dependencias
|
||||
├── core/
|
||||
│ ├── router.py # Lógica de enrutamiento URL → Parser
|
||||
@@ -115,15 +127,6 @@ deepreader_skill/
|
||||
└── youtube.py # Parser de transcripciones de YouTube
|
||||
```
|
||||
|
||||
### Estrategia de Selección de Parser
|
||||
|
||||
```
|
||||
URL detectada → ¿Twitter/X? → FxTwitter API → Nitter fallback
|
||||
→ ¿Reddit? → .json suffix API
|
||||
→ ¿YouTube? → youtube-transcript-api
|
||||
→ ¿otro? → Trafilatura (genérico)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuración
|
||||
@@ -135,6 +138,20 @@ URL detectada → ¿Twitter/X? → FxTwitter API → Nitter fallback
|
||||
|
||||
---
|
||||
|
||||
## 💡 ¿Por Qué DeepReader?
|
||||
|
||||
| Característica | DeepReader | Scraping manual | Herramientas de navegador |
|
||||
|---------------|-----------|----------------|--------------------------|
|
||||
| **Activación** | Automática por URL | Código manual | Acción manual |
|
||||
| **Twitter/X** | ✅ Soporte completo | ❌ Bloqueado | ⚠️ Parcial |
|
||||
| **Hilos Reddit** | ✅ + comentarios | ⚠️ Complejo | ⚠️ Lento |
|
||||
| **Transcripciones YouTube** | ✅ Integrado | ❌ Herramienta separada | ❌ No disponible |
|
||||
| **Claves API** | ❌ Ninguna | ✅ Frecuentemente | ✅ A veces |
|
||||
| **Formato de salida** | Markdown limpio | HTML crudo | Capturas de pantalla |
|
||||
| **Integración de memoria** | ✅ Auto-guardado | ❌ Manual | ❌ Manual |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Créditos
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — API pública para obtener contenido de Twitter/X
|
||||
|
||||
+70
-44
@@ -1,23 +1,54 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
# 🦞 OpenClaw DeepReader
|
||||
|
||||
> **Moteur autonome d'ingestion de contenu web pour agents IA.**
|
||||
> **La passerelle de contenu web par défaut pour les agents OpenClaw.** Lit X (Twitter), Reddit, YouTube et toute page web — zéro configuration, zéro clé API.
|
||||
|
||||
DeepReeder intercepte les URLs des messages utilisateur, extrait le contenu intelligemment à l'aide de parsers spécialisés, le formate en Markdown propre avec des métadonnées YAML frontmatter, et le sauvegarde dans la mémoire à long terme de l'agent.
|
||||
DeepReader est le lecteur de contenu intégré pour le framework d'agents [OpenClaw](https://github.com/anthropics/openclaw). Collez n'importe quelle URL dans une conversation, et DeepReader récupère, analyse et sauvegarde automatiquement du Markdown de haute qualité dans la mémoire à long terme de l'agent. Conçu pour les réseaux sociaux et le web moderne.
|
||||
|
||||
🌍 **Traductions** : [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [العربية](README_ar.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ Fonctionnalités
|
||||
## ⚡ Installation
|
||||
|
||||
| Parser | Sources | Méthode |
|
||||
|--------|---------|---------|
|
||||
| 🌐 **Générique** | Blogs, articles, documentation | [Trafilatura](https://trafilatura.readthedocs.io/) avec fallback BeautifulSoup |
|
||||
| 🐦 **Twitter / X** | Tweets, fils, X Articles | **FxTwitter API** (principal) + Nitter (fallback) |
|
||||
| 🟠 **Reddit** | Posts + fils de commentaires | **Reddit .json API** (sans configuration) |
|
||||
| 🎬 **YouTube** | Transcriptions vidéo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
```bash
|
||||
npx clawhub@latest install deepreader
|
||||
```
|
||||
|
||||
### 🐦 Twitter / X — Intégration Approfondie
|
||||
Ou installation manuelle :
|
||||
|
||||
```bash
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Utilisez Quand
|
||||
|
||||
- Vous avez besoin de **lire un tweet, un fil ou un article X** et de l'ajouter à la mémoire d'OpenClaw
|
||||
- Vous avez besoin d'**ingérer un post Reddit** avec les meilleurs commentaires et le contexte de discussion
|
||||
- Vous voulez **sauvegarder une transcription YouTube** pour référence ou analyse ultérieure
|
||||
- Vous voulez **clipper n'importe quel blog, article ou documentation** en Markdown propre
|
||||
- Votre agent a besoin d'un **lecteur web par défaut** qui fonctionne tout simplement — sans clé API, sans configuration
|
||||
|
||||
---
|
||||
|
||||
## ✨ Sources Supportées
|
||||
|
||||
| Parser | Sources | Méthode | Clé API ? |
|
||||
|--------|---------|---------|-----------|
|
||||
| 🐦 **Twitter / X** | Tweets, fils, X Articles | [FxTwitter API](https://github.com/FxEmbed/FxEmbed) + Nitter fallback | ❌ Aucune |
|
||||
| 🟠 **Reddit** | Posts + fils de commentaires | Reddit `.json` API | ❌ Aucune |
|
||||
| 🎬 **YouTube** | Transcriptions vidéo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) | ❌ Aucune |
|
||||
| 🌐 **Toute URL** | Blogs, articles, docs | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup | ❌ Aucune |
|
||||
|
||||
**Zéro clé API. Zéro connexion. Zéro limite. Collez et lisez.**
|
||||
|
||||
---
|
||||
|
||||
## 🐦 Twitter / X — Intégration Approfondie
|
||||
|
||||
Propulsé par l'API [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspiré par [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
|
||||
|
||||
@@ -31,7 +62,7 @@ Propulsé par l'API [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspiré pa
|
||||
| Fils de réponses | ✅ Via Nitter fallback (5 premières) |
|
||||
| Statistiques d'engagement | ✅ ❤️ likes, 🔁 RTs, 👁️ vues, 🔖 signets |
|
||||
|
||||
### 🟠 Reddit — Intégration JSON Native
|
||||
## 🟠 Reddit — Intégration JSON Native
|
||||
|
||||
Utilise le suffixe URL `.json` intégré de Reddit — **sans clé API, sans OAuth, sans inscription**.
|
||||
|
||||
@@ -45,20 +76,6 @@ Utilise le suffixe URL `.json` intégré de Reddit — **sans clé API, sans OAu
|
||||
| Statistiques du post | ✅ ⬆️ score, 💬 commentaires, ratio de votes |
|
||||
| Tags Flair | ✅ Inclus |
|
||||
|
||||
**Sans clé API. Sans connexion. Sans limite de débit.**
|
||||
|
||||
---
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
```bash
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Démarrage Rapide
|
||||
@@ -66,23 +83,26 @@ pip install -e .
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# Traiter une URL
|
||||
result = run("Regarde cet article : https://example.com/blog/post")
|
||||
print(result)
|
||||
# Lire un tweet → sauvegarde dans la mémoire de l'agent
|
||||
result = run("Regarde ce tweet : https://x.com/elonmusk/status/123456")
|
||||
|
||||
# Traiter un post Reddit
|
||||
# Lire une discussion Reddit → capture post + commentaires
|
||||
result = run("Super discussion : https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# Traiter plusieurs URLs
|
||||
# Lire une vidéo YouTube → sauvegarde la transcription complète
|
||||
result = run("Regarde ça : https://youtube.com/watch?v=dQw4w9WgXcQ")
|
||||
|
||||
# Lire n'importe quel article → extrait le contenu propre
|
||||
result = run("Lecture intéressante : https://example.com/blog/ai-agents-2026")
|
||||
|
||||
# Traitement par lots de plusieurs URLs
|
||||
result = run("""
|
||||
Voici quelques liens :
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://example.com/article
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -93,6 +113,7 @@ print(result)
|
||||
deepreader_skill/
|
||||
├── __init__.py # Point d'entrée — fonction run()
|
||||
├── manifest.json # Métadonnées du skill
|
||||
├── SKILL.md # Description pour ClawHub
|
||||
├── requirements.txt # Dépendances
|
||||
├── core/
|
||||
│ ├── router.py # Routage URL → Parser
|
||||
@@ -106,15 +127,6 @@ deepreader_skill/
|
||||
└── youtube.py # Parser YouTube
|
||||
```
|
||||
|
||||
### Stratégie de Sélection
|
||||
|
||||
```
|
||||
URL détectée → Twitter/X? → FxTwitter API → Nitter fallback
|
||||
→ Reddit? → .json suffix API
|
||||
→ YouTube? → youtube-transcript-api
|
||||
→ autre? → Trafilatura (générique)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuration
|
||||
@@ -126,6 +138,20 @@ URL détectée → Twitter/X? → FxTwitter API → Nitter fallback
|
||||
|
||||
---
|
||||
|
||||
## 💡 Pourquoi DeepReader ?
|
||||
|
||||
| Fonctionnalité | DeepReader | Scraping manuel | Outils navigateur |
|
||||
|----------------|-----------|----------------|-------------------|
|
||||
| **Déclenchement** | Automatique par URL | Code manuel | Action manuelle |
|
||||
| **Twitter/X** | ✅ Support complet | ❌ Bloqué | ⚠️ Partiel |
|
||||
| **Fils Reddit** | ✅ + commentaires | ⚠️ Complexe | ⚠️ Lent |
|
||||
| **Transcriptions YouTube** | ✅ Intégré | ❌ Outil séparé | ❌ Non disponible |
|
||||
| **Clés API** | ❌ Aucune | ✅ Souvent | ✅ Parfois |
|
||||
| **Format sortie** | Markdown propre | HTML brut | Captures d'écran |
|
||||
| **Intégration mémoire** | ✅ Auto-sauvegarde | ❌ Manuel | ❌ Manuel |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Remerciements
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — API publique pour Twitter/X
|
||||
|
||||
+46
-123
@@ -1,68 +1,50 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
# 🦞 OpenClaw DeepReader
|
||||
|
||||
> **AIエージェント向け自律型Webコンテンツ取り込みエンジン。**
|
||||
> **OpenClawエージェントのデフォルトWebコンテンツゲートウェイ。** X(Twitter)、Reddit、YouTube、あらゆるWebページを読み取り — ゼロ設定、ゼロAPIキー。
|
||||
|
||||
DeepReederはユーザーメッセージからURLを自動検出し、専用パーサーを使ってコンテンツをインテリジェントにスクレイピングし、YAMLフロントマター付きのクリーンなMarkdownに変換して、エージェントの長期メモリに保存します。
|
||||
DeepReaderは[OpenClaw](https://github.com/anthropics/openclaw)エージェントフレームワークの組み込みコンテンツリーダーです。会話にURLを貼り付けるだけで、DeepReaderが自動的にフェッチ、パース、高品質なMarkdownをエージェントの長期メモリに保存します。
|
||||
|
||||
🌍 **翻訳**: [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ 機能
|
||||
## ⚡ インストール
|
||||
|
||||
| パーサー | ソース | 方法 |
|
||||
|---------|--------|------|
|
||||
| 🌐 **汎用** | ブログ、記事、ドキュメント | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup フォールバック |
|
||||
| 🐦 **Twitter / X** | ツイート、スレッド、Xアーティクル | **FxTwitter API**(メイン)+ Nitter(フォールバック) |
|
||||
| 🟠 **Reddit** | 投稿 + コメントスレッド | **Reddit .json API**(設定不要) |
|
||||
| 🎬 **YouTube** | 動画字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
```bash
|
||||
npx clawhub@latest install deepreader
|
||||
```
|
||||
|
||||
### 🐦 Twitter / X — ディープインテグレーション
|
||||
または手動:
|
||||
|
||||
[FxTwitter](https://github.com/FxEmbed/FxEmbed) APIベース。[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)にインスパイアされました。
|
||||
|
||||
| コンテンツタイプ | サポート |
|
||||
|----------------|---------|
|
||||
| 通常のツイート | ✅ 全文 + エンゲージメント統計 |
|
||||
| 長文ツイート(Twitter Blue) | ✅ 全文 |
|
||||
| Xアーティクル(長文コンテンツ) | ✅ 完全な記事 + 単語数 |
|
||||
| 引用ツイート | ✅ ネストされたコンテンツ含む |
|
||||
| メディア(画像、動画、GIF) | ✅ URL抽出 |
|
||||
| リプライスレッド | ✅ Nitterフォールバック経由(最初の5件) |
|
||||
| エンゲージメント統計 | ✅ ❤️ いいね、🔁 RT、👁️ 閲覧、🔖 ブックマーク |
|
||||
|
||||
### 🟠 Reddit — ネイティブJSON統合
|
||||
|
||||
RedditのビルトインURL `.json` サフィックスを使用 — **APIキー不要、OAuth不要、登録不要**。
|
||||
|
||||
| コンテンツタイプ | サポート |
|
||||
|----------------|---------|
|
||||
| セルフ投稿(テキスト) | ✅ 完全なMarkdown本文 |
|
||||
| リンク投稿 | ✅ URL + メタデータ |
|
||||
| 人気コメント(スコア順) | ✅ 最大15件 |
|
||||
| ネストされた返信スレッド | ✅ 最大3階層 |
|
||||
| メディア(画像、ギャラリー、動画) | ✅ URL抽出 |
|
||||
| 投稿統計 | ✅ ⬆️ スコア、💬 コメント数、投票率 |
|
||||
| Flairタグ | ✅ 含む |
|
||||
|
||||
**APIキー不要。ログイン不要。レート制限なし。**
|
||||
```bash
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📦 インストール
|
||||
## 🎯 使用シナリオ
|
||||
|
||||
```bash
|
||||
# リポジトリをクローン
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
- **ツイート、スレッド、Xアーティクル**を読んでOpenClawメモリに追加
|
||||
- **Reddit投稿**をコメントと共にインジェスト
|
||||
- **YouTube字幕**を保存して後で参照
|
||||
- **ブログ、記事、ドキュメント**をMarkdownにクリップ
|
||||
- **デフォルトWebリーダー**として — APIキー不要
|
||||
|
||||
# 仮想環境を作成
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
---
|
||||
|
||||
# 依存関係をインストール
|
||||
pip install -e .
|
||||
```
|
||||
## ✨ 対応ソース
|
||||
|
||||
| パーサー | ソース | 方法 | APIキー? |
|
||||
|---------|--------|------|----------|
|
||||
| 🐦 **Twitter / X** | ツイート、スレッド | [FxTwitter API](https://github.com/FxEmbed/FxEmbed) + Nitter | ❌ なし |
|
||||
| 🟠 **Reddit** | 投稿 + コメント | Reddit `.json` API | ❌ なし |
|
||||
| 🎬 **YouTube** | 動画字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) | ❌ なし |
|
||||
| 🌐 **すべてのURL** | ブログ、記事 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup | ❌ なし |
|
||||
|
||||
**APIキーゼロ。ログインゼロ。レート制限ゼロ。貼り付けて読むだけ。**
|
||||
|
||||
---
|
||||
|
||||
@@ -71,94 +53,35 @@ pip install -e .
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# 単一URLを処理
|
||||
result = run("この記事をチェック: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# ツイートを処理
|
||||
result = run("興味深いスレッド: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Redditの投稿を処理
|
||||
result = run("このツイートをチェック: https://x.com/elonmusk/status/123456")
|
||||
result = run("素晴らしい議論: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# 複数のURLを一括処理
|
||||
result = run("""
|
||||
いくつかのリンクがあります:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
""")
|
||||
print(result)
|
||||
result = run("これを見て: https://youtube.com/watch?v=dQw4w9WgXcQ")
|
||||
result = run("興味深い記事: https://example.com/blog/ai-agents-2026")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ アーキテクチャ
|
||||
## 💡 なぜDeepReader?
|
||||
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # エントリポイント — run() 関数
|
||||
├── manifest.json # スキルメタデータとトリガー設定
|
||||
├── requirements.txt # 依存関係リスト
|
||||
├── core/
|
||||
│ ├── router.py # URL → パーサールーティングロジック
|
||||
│ ├── storage.py # Markdownファイル生成・保存
|
||||
│ └── utils.py # URL抽出とユーティリティ関数
|
||||
└── parsers/
|
||||
├── base.py # 抽象基底パーサーとParseResultモデル
|
||||
├── generic.py # 汎用記事/ブログパーサー
|
||||
├── twitter.py # Twitter/Xパーサー(FxTwitter + Nitter)
|
||||
├── reddit.py # Redditパーサー(.json API)
|
||||
└── youtube.py # YouTube字幕パーサー
|
||||
```
|
||||
|
||||
### パーサー選択戦略
|
||||
|
||||
```
|
||||
URL検出 → Twitter/X? → FxTwitter API → Nitterフォールバック
|
||||
→ Reddit? → .jsonサフィックスAPI
|
||||
→ YouTube? → youtube-transcript-api
|
||||
→ その他 → Trafilatura(汎用)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 設定
|
||||
|
||||
| 変数 | デフォルト | 説明 |
|
||||
|------|-----------|------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | コンテンツの保存先 |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | ログの詳細レベル |
|
||||
| 機能 | DeepReader | 手動スクレイピング | ブラウザツール |
|
||||
|------|-----------|-----------------|--------------|
|
||||
| **トリガー** | URL自動 | コード必要 | 手動 |
|
||||
| **Twitter/X** | ✅ 完全 | ❌ ブロック | ⚠️ 部分 |
|
||||
| **Reddit** | ✅ + コメント | ⚠️ 複雑 | ⚠️ 遅い |
|
||||
| **YouTube字幕** | ✅ 内蔵 | ❌ 別ツール | ❌ 不可 |
|
||||
| **APIキー** | ❌ 不要 | ✅ 必要 | ✅ 時々 |
|
||||
| **出力** | Markdown | HTML | スクリーンショット |
|
||||
| **メモリ統合** | ✅ 自動 | ❌ 手動 | ❌ 手動 |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 クレジット
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — Twitter/Xコンテンツ取得用パブリックAPI
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — FxTwitter統合アプローチのインスピレーション
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — 高性能Webコンテンツ抽出
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — YouTube字幕取得
|
||||
|
||||
---
|
||||
|
||||
## 🤝 コントリビューション
|
||||
|
||||
コントリビューション歓迎です!
|
||||
|
||||
1. リポジトリをForkします
|
||||
2. フィーチャーブランチを作成 (`git checkout -b feature/amazing-parser`)
|
||||
3. 変更をコミット (`git commit -m '素晴らしいパーサーを追加'`)
|
||||
4. ブランチをプッシュ (`git push origin feature/amazing-parser`)
|
||||
5. Pull Requestを開きます
|
||||
|
||||
---
|
||||
- **[FxTwitter](https://github.com/FxEmbed/FxEmbed)** · **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** · **[Trafilatura](https://trafilatura.readthedocs.io/)** · **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)**
|
||||
|
||||
## 📄 ライセンス
|
||||
|
||||
このプロジェクトは**MITライセンス**の下でライセンスされています — 詳細は[LICENSE](LICENSE)ファイルをご覧ください。
|
||||
**MITライセンス** — [LICENSE](LICENSE)参照。
|
||||
|
||||
---
|
||||
|
||||
|
||||
+70
-53
@@ -1,23 +1,54 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
# 🦞 OpenClaw DeepReader
|
||||
|
||||
> **AI 에이전트를 위한 자율 웹 콘텐츠 수집 엔진.**
|
||||
> **OpenClaw 에이전트의 기본 웹 콘텐츠 게이트웨이.** X (Twitter), Reddit, YouTube 및 모든 웹페이지 읽기 — 제로 설정, 제로 API 키.
|
||||
|
||||
DeepReeder는 사용자 메시지에서 URL을 자동으로 감지하고, 전문 파서를 사용하여 콘텐츠를 지능적으로 스크래핑하며, YAML 프론트매터가 포함된 깔끔한 Markdown으로 변환하여 에이전트의 장기 메모리에 저장합니다.
|
||||
DeepReader는 [OpenClaw](https://github.com/anthropics/openclaw) 에이전트 프레임워크의 내장 콘텐츠 리더입니다. 대화에 URL을 붙여넣으면 DeepReader가 자동으로 가져오고, 파싱하고, 고품질 Markdown을 에이전트의 장기 메모리에 저장합니다. 소셜 미디어와 현대 웹을 위해 설계되었습니다.
|
||||
|
||||
🌍 **번역**: [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [日本語](README_ja.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ 기능
|
||||
## ⚡ 설치
|
||||
|
||||
| 파서 | 소스 | 방법 |
|
||||
|------|------|------|
|
||||
| 🌐 **범용** | 블로그, 기사, 문서 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup 대체 |
|
||||
| 🐦 **Twitter / X** | 트윗, 스레드, X 아티클 | **FxTwitter API** (주력) + Nitter (대체) |
|
||||
| 🟠 **Reddit** | 게시물 + 댓글 스레드 | **Reddit .json API** (제로 설정) |
|
||||
| 🎬 **YouTube** | 동영상 자막 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
```bash
|
||||
npx clawhub@latest install deepreader
|
||||
```
|
||||
|
||||
### 🐦 Twitter / X — 심층 통합
|
||||
또는 수동 설치:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 사용 시나리오
|
||||
|
||||
- **트윗, 스레드 또는 X 아티클**을 읽어서 OpenClaw 메모리에 추가해야 할 때
|
||||
- **Reddit 게시물**을 인기 댓글과 토론 컨텍스트와 함께 수집해야 할 때
|
||||
- **YouTube 자막**을 나중에 참조하거나 분석하기 위해 저장하고 싶을 때
|
||||
- **블로그, 기사 또는 문서 페이지**를 깔끔한 Markdown으로 클리핑하고 싶을 때
|
||||
- 에이전트에 **기본 웹 리더**가 필요할 때 — API 키 없이, 설정 없이 바로 작동
|
||||
|
||||
---
|
||||
|
||||
## ✨ 지원 소스
|
||||
|
||||
| 파서 | 소스 | 방법 | API 키? |
|
||||
|------|------|------|---------|
|
||||
| 🐦 **Twitter / X** | 트윗, 스레드, X 아티클 | [FxTwitter API](https://github.com/FxEmbed/FxEmbed) + Nitter 대체 | ❌ 없음 |
|
||||
| 🟠 **Reddit** | 게시물 + 댓글 스레드 | Reddit `.json` API | ❌ 없음 |
|
||||
| 🎬 **YouTube** | 동영상 자막 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) | ❌ 없음 |
|
||||
| 🌐 **모든 URL** | 블로그, 기사, 문서 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup | ❌ 없음 |
|
||||
|
||||
**API 키 제로. 로그인 제로. 속도 제한 제로. 붙여넣기만 하면 읽기.**
|
||||
|
||||
---
|
||||
|
||||
## 🐦 Twitter / X — 심층 통합
|
||||
|
||||
[FxTwitter](https://github.com/FxEmbed/FxEmbed) API 기반. [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)에서 영감을 받았습니다.
|
||||
|
||||
@@ -31,7 +62,7 @@ DeepReeder는 사용자 메시지에서 URL을 자동으로 감지하고, 전문
|
||||
| 답글 스레드 | ✅ Nitter 대체를 통해 (처음 5개) |
|
||||
| 참여 통계 | ✅ ❤️ 좋아요, 🔁 리트윗, 👁️ 조회, 🔖 북마크 |
|
||||
|
||||
### 🟠 Reddit — 네이티브 JSON 통합
|
||||
## 🟠 Reddit — 네이티브 JSON 통합
|
||||
|
||||
Reddit의 내장 `.json` URL 접미사 사용 — **API 키 불필요, OAuth 불필요, 등록 불필요**.
|
||||
|
||||
@@ -45,25 +76,6 @@ Reddit의 내장 `.json` URL 접미사 사용 — **API 키 불필요, OAuth 불
|
||||
| 게시물 통계 | ✅ ⬆️ 점수, 💬 댓글 수, 추천 비율 |
|
||||
| Flair 태그 | ✅ 포함 |
|
||||
|
||||
**API 키 불필요. 로그인 불필요. 속도 제한 없음.**
|
||||
|
||||
---
|
||||
|
||||
## 📦 설치
|
||||
|
||||
```bash
|
||||
# 저장소 클론
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# 가상 환경 생성
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# 의존성 설치
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 빠른 시작
|
||||
@@ -71,27 +83,26 @@ pip install -e .
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# 단일 URL 처리
|
||||
result = run("이 기사를 확인하세요: https://example.com/blog/post")
|
||||
print(result)
|
||||
# 트윗 읽기 → 에이전트 메모리에 저장
|
||||
result = run("이 트윗을 확인하세요: https://x.com/elonmusk/status/123456")
|
||||
|
||||
# 트윗 처리
|
||||
result = run("흥미로운 스레드: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Reddit 게시물 처리
|
||||
# Reddit 토론 읽기 → 게시물 + 인기 댓글 캡처
|
||||
result = run("좋은 토론: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# 여러 URL 한번에 처리
|
||||
# YouTube 비디오 읽기 → 전체 자막 저장
|
||||
result = run("이것을 보세요: https://youtube.com/watch?v=dQw4w9WgXcQ")
|
||||
|
||||
# 모든 기사 읽기 → 깨끗한 콘텐츠 추출
|
||||
result = run("흥미로운 글: https://example.com/blog/ai-agents-2026")
|
||||
|
||||
# 여러 URL 일괄 처리
|
||||
result = run("""
|
||||
여기 몇 가지 링크가 있습니다:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://example.com/article
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -102,6 +113,7 @@ print(result)
|
||||
deepreader_skill/
|
||||
├── __init__.py # 진입점 — run() 함수
|
||||
├── manifest.json # 스킬 메타데이터 및 트리거 설정
|
||||
├── SKILL.md # ClawHub 스킬 설명
|
||||
├── requirements.txt # 의존성 목록
|
||||
├── core/
|
||||
│ ├── router.py # URL → 파서 라우팅 로직
|
||||
@@ -115,15 +127,6 @@ deepreader_skill/
|
||||
└── youtube.py # YouTube 자막 파서
|
||||
```
|
||||
|
||||
### 파서 선택 전략
|
||||
|
||||
```
|
||||
URL 감지 → Twitter/X? → FxTwitter API → Nitter 대체
|
||||
→ Reddit? → .json 접미사 API
|
||||
→ YouTube? → youtube-transcript-api
|
||||
→ 기타 → Trafilatura (범용)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 설정
|
||||
@@ -135,6 +138,20 @@ URL 감지 → Twitter/X? → FxTwitter API → Nitter 대체
|
||||
|
||||
---
|
||||
|
||||
## 💡 왜 DeepReader인가?
|
||||
|
||||
| 기능 | DeepReader | 수동 스크래핑 | 브라우저 도구 |
|
||||
|------|-----------|-------------|-------------|
|
||||
| **트리거** | URL 자동 트리거 | 코드 작성 필요 | 수동 작업 |
|
||||
| **Twitter/X** | ✅ 완전 지원 | ❌ 차단됨 | ⚠️ 부분적 |
|
||||
| **Reddit 스레드** | ✅ + 댓글 | ⚠️ 복잡 | ⚠️ 느림 |
|
||||
| **YouTube 자막** | ✅ 내장 | ❌ 별도 도구 | ❌ 불가 |
|
||||
| **API 키** | ❌ 불필요 | ✅ 자주 필요 | ✅ 때때로 |
|
||||
| **출력 형식** | 깔끔한 Markdown | 원시 HTML | 스크린샷 |
|
||||
| **메모리 통합** | ✅ 자동 저장 | ❌ 수동 | ❌ 수동 |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 크레딧
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — Twitter/X 콘텐츠 가져오기용 공개 API
|
||||
|
||||
+70
-55
@@ -1,23 +1,54 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
# 🦞 OpenClaw DeepReader
|
||||
|
||||
> **面向 AI 智能体的自主网页内容摄取引擎。**
|
||||
> **OpenClaw 的默认 Web 内容读取入口。** 读取 X(Twitter)、Reddit、YouTube 和任意网页 — 零配置,零 API 密钥。
|
||||
|
||||
DeepReeder 自动拦截用户消息中的 URL,使用专用解析器智能抓取内容,将其格式化为带有 YAML 前置信息的干净 Markdown,并保存到智能体的长期记忆中。
|
||||
DeepReader 是 [OpenClaw](https://github.com/anthropics/openclaw) 智能体框架的内置内容读取器。在对话中粘贴任意 URL,DeepReader 会自动抓取、解析并将高质量 Markdown 保存到智能体的长期记忆中。专为社交媒体和现代网页设计。
|
||||
|
||||
🌍 **其他语言**: [English](README.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ 功能特性
|
||||
## ⚡ 安装
|
||||
|
||||
| 解析器 | 来源 | 方法 |
|
||||
|--------|------|------|
|
||||
| 🌐 **通用** | 博客、文章、文档 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup 备用方案 |
|
||||
| 🐦 **Twitter / X** | 推文、线程、X 文章 | **FxTwitter API**(主力) + Nitter(备用) |
|
||||
| 🟠 **Reddit** | 帖子 + 评论线程 | **Reddit .json API**(零配置) |
|
||||
| 🎬 **YouTube** | 视频字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
```bash
|
||||
npx clawhub@latest install deepreader
|
||||
```
|
||||
|
||||
### 🐦 Twitter / X — 深度整合
|
||||
或手动安装:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 使用场景
|
||||
|
||||
- 需要**读取推文、线程或 X 文章**并添加到 OpenClaw 的记忆中
|
||||
- 需要**摄入 Reddit 帖子**,包括热门评论和讨论上下文
|
||||
- 想要**保存 YouTube 字幕**,以便后续参考或分析
|
||||
- 想要**提取任意博客、文章或文档页面**为干净的 Markdown
|
||||
- 你的智能体需要一个**默认的 Web 读取器** — 开箱即用,无需 API 密钥
|
||||
|
||||
---
|
||||
|
||||
## ✨ 支持的内容源
|
||||
|
||||
| 解析器 | 来源 | 方法 | API 密钥? |
|
||||
|--------|------|------|-----------|
|
||||
| 🐦 **Twitter / X** | 推文、线程、X 文章 | [FxTwitter API](https://github.com/FxEmbed/FxEmbed) + Nitter 备用 | ❌ 无需 |
|
||||
| 🟠 **Reddit** | 帖子 + 评论线程 | Reddit `.json` API | ❌ 无需 |
|
||||
| 🎬 **YouTube** | 视频字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) | ❌ 无需 |
|
||||
| 🌐 **任意 URL** | 博客、文章、文档 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup | ❌ 无需 |
|
||||
|
||||
**零 API 密钥。零登录。零速率限制。粘贴即读。**
|
||||
|
||||
---
|
||||
|
||||
## 🐦 Twitter / X — 深度整合
|
||||
|
||||
基于 [FxTwitter](https://github.com/FxEmbed/FxEmbed) API,灵感来自 [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)。
|
||||
|
||||
@@ -31,7 +62,7 @@ DeepReeder 自动拦截用户消息中的 URL,使用专用解析器智能抓
|
||||
| 回复线程 | ✅ 通过 Nitter 备用方案(前5条) |
|
||||
| 互动数据 | ✅ ❤️ 喜欢、🔁 转发、👁️ 浏览、🔖 书签 |
|
||||
|
||||
### 🟠 Reddit — 原生 JSON 整合
|
||||
## 🟠 Reddit — 原生 JSON 整合
|
||||
|
||||
使用 Reddit 内置的 `.json` URL 后缀 — **无需 API 密钥、无需 OAuth、无需注册**。
|
||||
|
||||
@@ -45,25 +76,6 @@ DeepReeder 自动拦截用户消息中的 URL,使用专用解析器智能抓
|
||||
| 帖子统计 | ✅ ⬆️ 评分、💬 评论数、点赞比例 |
|
||||
| Flair 标签 | ✅ 包含 |
|
||||
|
||||
**无需 API 密钥。无需登录。无速率限制。**
|
||||
|
||||
---
|
||||
|
||||
## 📦 安装
|
||||
|
||||
```bash
|
||||
# 克隆仓库
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# 创建虚拟环境
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# 安装依赖
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 快速开始
|
||||
@@ -71,27 +83,26 @@ pip install -e .
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# 处理单个 URL
|
||||
result = run("看看这篇文章: https://example.com/blog/post")
|
||||
print(result)
|
||||
# 读取推文 → 保存到智能体记忆
|
||||
result = run("看看这条推文: https://x.com/elonmusk/status/123456")
|
||||
|
||||
# 处理推文(自动使用 FxTwitter API)
|
||||
result = run("有趣的推文: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# 处理 Reddit 帖子(自动使用 .json API)
|
||||
# 读取 Reddit 讨论 → 捕获帖子 + 热门评论
|
||||
result = run("精彩讨论: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# 读取 YouTube 视频 → 保存完整字幕
|
||||
result = run("看这个: https://youtube.com/watch?v=dQw4w9WgXcQ")
|
||||
|
||||
# 读取任意文章 → 提取干净内容
|
||||
result = run("有趣的文章: https://example.com/blog/ai-agents-2026")
|
||||
|
||||
# 批量处理多个 URL
|
||||
result = run("""
|
||||
这里有一些链接:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://example.com/article
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -102,6 +113,7 @@ print(result)
|
||||
deepreader_skill/
|
||||
├── __init__.py # 入口 — run() 函数
|
||||
├── manifest.json # 技能元数据与触发配置
|
||||
├── SKILL.md # ClawHub 技能说明
|
||||
├── requirements.txt # 依赖列表
|
||||
├── core/
|
||||
│ ├── router.py # URL → 解析器路由逻辑
|
||||
@@ -115,21 +127,10 @@ deepreader_skill/
|
||||
└── youtube.py # YouTube 字幕解析器
|
||||
```
|
||||
|
||||
### 解析器选择策略
|
||||
|
||||
```
|
||||
检测到 URL → Twitter/X? → FxTwitter API → Nitter 备用
|
||||
→ Reddit? → .json 后缀 API
|
||||
→ YouTube? → youtube-transcript-api
|
||||
→ 其他 → Trafilatura(通用)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 配置
|
||||
|
||||
DeepReeder 开箱即用,使用合理的默认值。可通过环境变量自定义配置:
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | 保存内容的路径 |
|
||||
@@ -137,6 +138,20 @@ DeepReeder 开箱即用,使用合理的默认值。可通过环境变量自定
|
||||
|
||||
---
|
||||
|
||||
## 💡 为什么选择 DeepReader?
|
||||
|
||||
| 特性 | DeepReader | 手动抓取 | 浏览器工具 |
|
||||
|------|-----------|---------|-----------|
|
||||
| **触发方式** | URL 自动触发 | 需写代码 | 手动操作 |
|
||||
| **Twitter/X** | ✅ 完整支持 | ❌ 被封锁 | ⚠️ 部分支持 |
|
||||
| **Reddit 线程** | ✅ + 评论 | ⚠️ 复杂 | ⚠️ 慢 |
|
||||
| **YouTube 字幕** | ✅ 内置 | ❌ 需额外工具 | ❌ 不可用 |
|
||||
| **API 密钥** | ❌ 无需 | ✅ 通常需要 | ✅ 有时需要 |
|
||||
| **输出格式** | 干净 Markdown | 原始 HTML | 截图 |
|
||||
| **记忆整合** | ✅ 自动保存 | ❌ 手动 | ❌ 手动 |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 致谢
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — 获取 Twitter/X 内容的公共 API
|
||||
@@ -148,7 +163,7 @@ DeepReeder 开箱即用,使用合理的默认值。可通过环境变量自定
|
||||
|
||||
## 🤝 贡献
|
||||
|
||||
欢迎贡献!您可以:
|
||||
欢迎贡献!
|
||||
|
||||
1. Fork 仓库
|
||||
2. 创建功能分支 (`git checkout -b feature/amazing-parser`)
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
name: DeepReader
|
||||
description: The default web content reader for OpenClaw. Reads X (Twitter), Reddit, YouTube, and any webpage into clean Markdown — zero API keys required. Use when you need to ingest social media posts, articles, or video transcripts into agent memory.
|
||||
---
|
||||
|
||||
# DeepReader
|
||||
|
||||
The default web content reader for OpenClaw agents. Automatically detects URLs in messages, fetches content using specialized parsers, and saves clean Markdown with YAML frontmatter to agent memory.
|
||||
|
||||
## Use when
|
||||
|
||||
1. A user shares a **tweet, thread, or X article** and you need to read its content
|
||||
2. A user shares a **Reddit post** and you need the discussion + top comments
|
||||
3. A user shares a **YouTube video** and you need the transcript
|
||||
4. A user shares **any blog, article, or documentation URL** and you need the text
|
||||
5. You need to **batch-read multiple URLs** from a single message
|
||||
|
||||
## Supported sources
|
||||
|
||||
| Source | Method | API Key? |
|
||||
|--------|--------|----------|
|
||||
| Twitter / X | FxTwitter API + Nitter fallback | None |
|
||||
| Reddit | .json suffix API | None |
|
||||
| YouTube | youtube-transcript-api | None |
|
||||
| Any URL | Trafilatura + BeautifulSoup | None |
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# Automatic — triggered when message contains URLs
|
||||
result = run("Check this out: https://x.com/user/status/123456")
|
||||
|
||||
# Reddit post with comments
|
||||
result = run("https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
|
||||
# YouTube transcript
|
||||
result = run("https://youtube.com/watch?v=dQw4w9WgXcQ")
|
||||
|
||||
# Any webpage
|
||||
result = run("https://example.com/blog/interesting-article")
|
||||
|
||||
# Multiple URLs at once
|
||||
result = run("""
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/
|
||||
https://example.com/article
|
||||
""")
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
Content is saved as `.md` files with structured YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: "Tweet by @user"
|
||||
source_url: "https://x.com/user/status/123456"
|
||||
domain: "x.com"
|
||||
parser: "twitter"
|
||||
ingested_at: "2026-02-16T12:00:00Z"
|
||||
content_hash: "sha256:..."
|
||||
word_count: 350
|
||||
---
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Where to save ingested content |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | Logging verbosity |
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
URL detected → is Twitter/X? → FxTwitter API → Nitter fallback
|
||||
→ is Reddit? → .json suffix API
|
||||
→ is YouTube? → youtube-transcript-api
|
||||
→ otherwise → Trafilatura (generic)
|
||||
```
|
||||
|
||||
Triggers automatically when any message contains `https://` or `http://`.
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "DeepReader",
|
||||
"description": "Autonomous web content ingestion engine. Intercepts URLs from user messages, scrapes content intelligently using specialized parsers (generic articles, Twitter/X, YouTube), formats it into clean Markdown with YAML frontmatter metadata, and saves it to the agent's long-term memory directory.",
|
||||
"description": "The default web content reader for OpenClaw. Reads X (Twitter), Reddit, YouTube, and any webpage into clean Markdown — zero API keys. Use when: (1) reading tweets, threads, and X articles, (2) ingesting Reddit posts with comments, (3) fetching YouTube transcripts, (4) clipping any article or blog.",
|
||||
"version": "1.0.0",
|
||||
"author": "OpenClaw",
|
||||
"entry_point": "deepreader_skill",
|
||||
@@ -21,16 +21,28 @@
|
||||
"capabilities": [
|
||||
"web_scraping",
|
||||
"content_extraction",
|
||||
"youtube_transcription",
|
||||
"twitter_reading",
|
||||
"x_reading",
|
||||
"reddit_reading",
|
||||
"markdown_generation"
|
||||
"youtube_transcription",
|
||||
"markdown_generation",
|
||||
"url_reading",
|
||||
"social_media_ingestion",
|
||||
"web_reader"
|
||||
],
|
||||
"memory_path": "../../memory/inbox/",
|
||||
"tags": [
|
||||
"reader",
|
||||
"web-reader",
|
||||
"twitter",
|
||||
"x",
|
||||
"reddit",
|
||||
"youtube",
|
||||
"scraper",
|
||||
"ingestion",
|
||||
"memory"
|
||||
"memory",
|
||||
"default",
|
||||
"content-extraction",
|
||||
"social-media"
|
||||
]
|
||||
}
|
||||
+17
-3
@@ -3,16 +3,29 @@ requires = ["setuptools>=68.0", "wheel"]
|
||||
build-backend = "setuptools.backends._legacy:_Backend"
|
||||
|
||||
[project]
|
||||
name = "openclaw-deepreeder"
|
||||
name = "openclaw-deepreader"
|
||||
version = "1.0.0"
|
||||
description = "Autonomous web content ingestion engine for OpenClaw agents. Scrapes articles, Twitter/X posts, and YouTube transcripts into clean Markdown."
|
||||
description = "The default web content reader for OpenClaw agents. Reads X (Twitter), Reddit, YouTube, and any webpage into clean Markdown with zero API keys."
|
||||
readme = "README.md"
|
||||
license = {text = "MIT"}
|
||||
requires-python = ">=3.10"
|
||||
authors = [
|
||||
{name = "OpenClaw", email = "astonysh@users.noreply.github.com"},
|
||||
]
|
||||
keywords = ["web-scraping", "content-extraction", "markdown", "ai-agent", "openclaw"]
|
||||
keywords = [
|
||||
"openclaw",
|
||||
"web-reader",
|
||||
"content-extraction",
|
||||
"twitter-reader",
|
||||
"x-reader",
|
||||
"reddit-reader",
|
||||
"youtube-transcript",
|
||||
"ai-agent",
|
||||
"web-scraping",
|
||||
"markdown",
|
||||
"social-media",
|
||||
"deepreader",
|
||||
]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Intended Audience :: Developers",
|
||||
@@ -41,6 +54,7 @@ dependencies = [
|
||||
Homepage = "https://github.com/astonysh/OpenClaw-DeepReeder"
|
||||
Repository = "https://github.com/astonysh/OpenClaw-DeepReeder"
|
||||
Issues = "https://github.com/astonysh/OpenClaw-DeepReeder/issues"
|
||||
ClawHub = "https://clawhub.ai"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
include = ["deepreader_skill*"]
|
||||
|
||||
Reference in New Issue
Block a user