🟠 Add Reddit parser + update all READMEs

Reddit Parser:
- Uses Reddit's native .json URL suffix (zero dependencies)
- Extracts post title, body, author, score, flair
- Top comments sorted by score (up to 15)
- Nested reply threads (up to 3 levels deep)
- Media detection (images, galleries, Reddit video)
- Proper error handling (429 rate limit, 404, 403)

Core:
- Added is_reddit_url() to utils.py
- Registered RedditParser in router.py
- Added reddit_reading capability to manifest.json

README:
- Updated all 7 language versions with Reddit section
  (EN, 中文, Español, 한국어, 日本語, العربية, Français)
This commit is contained in:
Tony Li
2026-02-16 14:22:31 +01:00
parent a45f09dfe3
commit b8c452baaa
11 changed files with 605 additions and 195 deletions
+44 -35
View File
@@ -14,6 +14,7 @@ DeepReeder intercepts URLs from user messages, scrapes content intelligently usi
|--------|---------|--------|
| 🌐 **Generic** | Blogs, articles, docs | [Trafilatura](https://trafilatura.readthedocs.io/) with BeautifulSoup fallback |
| 🐦 **Twitter / X** | Tweets, threads, X Articles | **FxTwitter API** (primary) + Nitter (fallback) |
| 🟠 **Reddit** | Posts + comment threads | **Reddit .json API** (zero-config) |
| 🎬 **YouTube** | Video transcripts | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
### 🐦 Twitter / X — Deep Integration
@@ -30,6 +31,20 @@ Powered by [FxTwitter](https://github.com/FxEmbed/FxEmbed) API with Nitter fallb
| Reply threads | ✅ Via Nitter fallback (first 5) |
| Engagement stats | ✅ ❤️ likes, 🔁 RTs, 👁️ views, 🔖 bookmarks |
### 🟠 Reddit — Native JSON Integration
Uses Reddit's built-in `.json` URL suffix — **no API keys, no OAuth, no registration**.
| Content Type | Support |
|-------------|---------|
| Self posts (text) | ✅ Full markdown body |
| Link posts | ✅ URL + metadata |
| Top comments (sorted by score) | ✅ Up to 15 comments |
| Nested reply threads | ✅ Up to 3 levels deep |
| Media (images, galleries, video) | ✅ URLs extracted |
| Post stats | ✅ ⬆️ score, 💬 comment count, upvote ratio |
| Flair tags | ✅ Included |
**No API keys. No login. No rate limits.**
### Output Format
@@ -38,18 +53,29 @@ Every piece of content is saved as a `.md` file with structured YAML frontmatter
```yaml
---
title: "Article Title"
source_url: "https://example.com/article"
domain: "example.com"
parser: "generic"
title: "[r/python] How I built an AI agent framework"
source_url: "https://www.reddit.com/r/python/comments/abc123/..."
domain: "reddit.com"
parser: "reddit"
ingested_at: "2026-02-16T12:00:00Z"
content_hash: "sha256:abc123..."
word_count: 1500
word_count: 2500
---
# Article Title
# How I built an AI agent framework
The clean, extracted content goes here...
**r/python** · u/developer123 · 2026-02-16 12:00 UTC
📊 ⬆️ 847 (96% upvoted) · 💬 234 comments · 🏷️ Discussion
---
Post body goes here...
---
### 💬 Top Comments
**u/expert_dev** (⬆️ 342):
> This is a really well-structured approach...
```
---
@@ -84,37 +110,21 @@ print(result)
result = run("Interesting thread: https://x.com/elonmusk/status/123456")
print(result)
# Process a Reddit post (uses .json API automatically)
result = run("Great discussion: https://www.reddit.com/r/python/comments/abc123/my_post/")
print(result)
# Process multiple URLs at once
result = run("""
Here are some links:
https://example.com/article
https://youtube.com/watch?v=dQw4w9WgXcQ
https://x.com/user/status/123456
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
""")
print(result)
```
### Example Output
```
📚 DeepReader — Processed 3 URL(s):
✅ How to Build AI Agents
Source: https://example.com/article
Saved to: memory/inbox/2026-02-16_how-to-build-ai-agents.md
Content: 3,200 characters
✅ Tweet by @elonmusk (Mon Feb 16 12:00:00 +0000 2026)
Source: https://x.com/elonmusk/status/123456
Saved to: memory/inbox/2026-02-16_tweet-by-elonmusk.md
Content: 480 characters
✅ Rick Astley - Never Gonna Give You Up
Source: https://youtube.com/watch?v=dQw4w9WgXcQ
Saved to: memory/inbox/2026-02-16_rick-astley-never-gonna.md
Content: 15,000 characters
```
---
## 🏗️ Architecture
@@ -132,18 +142,17 @@ deepreader_skill/
├── base.py # Abstract base parser & ParseResult model
├── generic.py # Generic article/blog parser (Trafilatura)
├── twitter.py # Twitter/X parser (FxTwitter + Nitter)
├── reddit.py # Reddit parser (.json API)
└── youtube.py # YouTube transcript parser
```
### Twitter Parser Strategy
### Parser Selection Strategy
```
URL detected → FxTwitter API (primary)
↓ success? → ✅ Rich result (stats, media, articles)
↓ failure?
Nitter instances (fallback)
↓ success? → ✅ Basic result + reply threads
↓ failure? → ❌ Graceful error with diagnostics
URL detected → is Twitter/X? → FxTwitter API → Nitter fallback
→ is Reddit? → .json suffix API
→ is YouTube? → youtube-transcript-api
→ otherwise → Trafilatura (generic)
```
---
+20 -83
View File
@@ -4,8 +4,6 @@
> **محرك استيعاب محتوى الويب الذاتي لوكلاء الذكاء الاصطناعي.**
يعترض DeepReeder عناوين URL من رسائل المستخدم، ويستخرج المحتوى بذكاء باستخدام محللات متخصصة، ويحوله إلى Markdown نظيف مع بيانات YAML الوصفية، ويحفظه في الذاكرة طويلة المدى للوكيل.
🌍 **الترجمات**: [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [Français](README_fr.md)
---
@@ -16,21 +14,22 @@
|--------|---------|---------|
| 🌐 **عام** | مدونات، مقالات، وثائق | [Trafilatura](https://trafilatura.readthedocs.io/) مع BeautifulSoup احتياطي |
| 🐦 **Twitter / X** | تغريدات، سلاسل، مقالات X | **FxTwitter API** (رئيسي) + Nitter (احتياطي) |
| 🟠 **Reddit** | منشورات + سلاسل تعليقات | **Reddit .json API** (بدون إعداد) |
| 🎬 **YouTube** | نصوص الفيديو | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
### 🐦 Twitter / X — تكامل عميق
### 🟠 Reddit — تكامل JSON الأصلي
مدعوم بواجهة برمجة تطبيقات [FxTwitter](https://github.com/FxEmbed/FxEmbed). مستلهم من [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
يستخدم لاحقة URL `.json` المدمجة في Reddit — **بدون مفاتيح API، بدون OAuth، بدون تسجيل**.
| نوع المحتوى | الدعم |
|-------------|-------|
| التغريدات العادية | ✅ نص كامل + إحصائيات التفاعل |
| التغريدات الطويلة (Twitter Blue) | ✅ نص كامل |
| مقالات X (محتوى طويل) | ✅ مقال كامل + عدد الكلمات |
| التغريدات المقتبسة | ✅ محتوى متداخل مضمّن |
| الوسائط (صور، فيديو، GIF) | ✅ استخراج الروابط |
| سلاسل الردود | ✅ عبر Nitter الاحتياطي (أول 5) |
| إحصائيات التفاعل | ✅ ❤️ إعجابات، 🔁 إعادة تغريد، 👁️ مشاهدات، 🔖 إشارات مرجعية |
| منشورات نصية | ✅ نص Markdown كامل |
| منشورات الروابط | ✅ URL + بيانات وصفية |
| أفضل التعليقات (مرتبة حسب النقاط) | ✅ حتى 15 تعليقاً |
| سلاسل الردود المتداخلة | ✅ حتى 3 مستويات عمق |
| الوسائط (صور، معارض، فيديو) | ✅ استخراج الروابط |
| إحصائيات المنشور | ✅ ⬆️ النقاط، 💬 عدد التعليقات |
| علامات Flair | ✅ مضمّنة |
**بدون مفاتيح API. بدون تسجيل دخول. بدون حدود للسرعة.**
@@ -41,15 +40,10 @@
<div dir="ltr">
```bash
# استنساخ المستودع
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
cd OpenClaw-DeepReeder
# إنشاء بيئة افتراضية
python3 -m venv .venv
source .venv/bin/activate
# تثبيت التبعيات
pip install -e .
```
@@ -64,21 +58,10 @@ pip install -e .
```python
from deepreader_skill import run
# معالجة عنوان URL واحد
result = run("اطلع على هذا المقال: https://example.com/blog/post")
result = run("اطلع على هذا: https://example.com/blog/post")
print(result)
# معالجة تغريدة (يستخدم FxTwitter API تلقائياً)
result = run("سلسلة مثيرة: https://x.com/elonmusk/status/123456")
print(result)
# معالجة عناوين URL متعددة دفعة واحدة
result = run("""
إليك بعض الروابط:
https://example.com/article
https://youtube.com/watch?v=dQw4w9WgXcQ
https://x.com/user/status/123456
""")
result = run("نقاش رائع: https://www.reddit.com/r/python/comments/abc123/my_post/")
print(result)
```
@@ -93,78 +76,32 @@ print(result)
```
deepreader_skill/
├── __init__.py # نقطة الدخول — دالة run()
├── manifest.json # بيانات المهارة الوصفية وإعدادات المشغلات
├── manifest.json # بيانات المهارة الوصفية
├── requirements.txt # قائمة التبعيات
├── core/
│ ├── router.py # منطق توجيه URL → المحلل
│ ├── storage.py # إنشاء وحفظ ملفات Markdown
│ └── utils.py # استخراج URL ودوال مساعدة
└── parsers/
├── base.py # المحلل الأساسي المجرد ونموذج ParseResult
├── generic.py # محلل المقالات/المدونات العام
├── twitter.py # محلل Twitter/X (FxTwitter + Nitter)
├── base.py # المحلل الأساسي
├── generic.py # محلل المقالات العام
├── twitter.py # محلل Twitter/X
├── reddit.py # محلل Reddit (.json API)
└── youtube.py # محلل نصوص YouTube
```
</div>
### استراتيجية محلل Twitter
<div dir="ltr">
```
اكتشاف URL → FxTwitter API (رئيسي)
↓ نجاح؟ → ✅ نتيجة غنية (إحصائيات، وسائط، مقالات)
↓ فشل؟
مثيلات Nitter (احتياطي)
↓ نجاح؟ → ✅ نتيجة أساسية + سلاسل الردود
↓ فشل؟ → ❌ رسالة خطأ ودية مع التشخيص
```
</div>
---
## 🔧 الإعدادات
يعمل DeepReeder مباشرة مع إعدادات افتراضية معقولة. يمكن تخصيص الإعدادات عبر متغيرات البيئة:
| المتغير | الافتراضي | الوصف |
|---------|-----------|-------|
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | مسار حفظ المحتوى |
| `DEEPREEDER_LOG_LEVEL` | `INFO` | مستوى تفصيل السجلات |
---
## 🙏 شكر وتقدير
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — واجهة برمجة تطبيقات عامة لجلب محتوى Twitter/X
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — مصدر إلهام لنهج تكامل FxTwitter
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — استخراج محتوى الويب القوي
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — جلب نصوص YouTube
---
## 🤝 المساهمة
المساهمات مرحب بها! يمكنك:
1. عمل Fork للمستودع
2. إنشاء فرع ميزة (`git checkout -b feature/amazing-parser`)
3. التزام بتغييراتك (`git commit -m 'إضافة محلل رائع'`)
4. دفع الفرع (`git push origin feature/amazing-parser`)
5. فتح Pull Request
---
- **[FxTwitter](https://github.com/FxEmbed/FxEmbed)** · **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** · **[Trafilatura](https://trafilatura.readthedocs.io/)** · **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)**
## 📄 الترخيص
هذا المشروع مرخص بموجب **رخصة MIT** — راجع ملف [LICENSE](LICENSE) للتفاصيل.
**رخصة MIT** — راجع ملف [LICENSE](LICENSE) للتفاصيل.
---
<p align="center">
صنع بـ 🦞 بواسطة <a href="https://github.com/astonysh">OpenClaw</a>
</p>
<p align="center">صنع بـ 🦞 بواسطة <a href="https://github.com/astonysh">OpenClaw</a></p>
</div>
+28 -11
View File
@@ -14,6 +14,7 @@ DeepReeder intercepta URLs de los mensajes de usuario, extrae contenido de forma
|--------|---------|--------|
| 🌐 **Genérico** | Blogs, artículos, documentación | [Trafilatura](https://trafilatura.readthedocs.io/) con fallback BeautifulSoup |
| 🐦 **Twitter / X** | Tweets, hilos, X Articles | **FxTwitter API** (principal) + Nitter (fallback) |
| 🟠 **Reddit** | Posts + hilos de comentarios | **Reddit .json API** (sin configuración) |
| 🎬 **YouTube** | Transcripciones de vídeo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
### 🐦 Twitter / X — Integración Profunda
@@ -30,6 +31,20 @@ Impulsado por la API de [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspira
| Hilos de respuestas | ✅ Vía Nitter fallback (primeras 5) |
| Estadísticas de interacción | ✅ ❤️ likes, 🔁 RTs, 👁️ vistas, 🔖 marcadores |
### 🟠 Reddit — Integración JSON Nativa
Usa el sufijo `.json` nativo de Reddit — **sin claves API, sin OAuth, sin registro**.
| Tipo de Contenido | Soporte |
|-------------------|---------|
| Self posts (texto) | ✅ Cuerpo completo en Markdown |
| Link posts | ✅ URL + metadatos |
| Comentarios principales (por puntuación) | ✅ Hasta 15 comentarios |
| Hilos de respuestas anidados | ✅ Hasta 3 niveles |
| Medios (imágenes, galerías, vídeo) | ✅ URLs extraídas |
| Estadísticas del post | ✅ ⬆️ puntuación, 💬 comentarios, ratio de votos |
| Etiquetas Flair | ✅ Incluidas |
**Sin claves API. Sin inicio de sesión. Sin límites de velocidad.**
---
@@ -60,16 +75,21 @@ from deepreader_skill import run
result = run("Mira este artículo: https://example.com/blog/post")
print(result)
# Procesar un tweet (usa FxTwitter API automáticamente)
# Procesar un tweet
result = run("Hilo interesante: https://x.com/elonmusk/status/123456")
print(result)
# Procesar múltiples URLs a la vez
# Procesar un post de Reddit
result = run("Gran discusión: https://www.reddit.com/r/python/comments/abc123/my_post/")
print(result)
# Procesar múltiples URLs
result = run("""
Aquí hay algunos enlaces:
https://example.com/article
https://youtube.com/watch?v=dQw4w9WgXcQ
https://x.com/user/status/123456
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
""")
print(result)
```
@@ -91,26 +111,23 @@ deepreader_skill/
├── base.py # Parser base abstracto y modelo ParseResult
├── generic.py # Parser genérico de artículos/blogs
├── twitter.py # Parser Twitter/X (FxTwitter + Nitter)
├── reddit.py # Parser Reddit (.json API)
└── youtube.py # Parser de transcripciones de YouTube
```
### Estrategia del Parser de Twitter
### Estrategia de Selección de Parser
```
URL detectada → FxTwitter API (principal)
↓ ¿éxito? → ✅ Resultado enriquecido (stats, media, artículos)
↓ ¿fallo?
Instancias Nitter (fallback)
↓ ¿éxito? → ✅ Resultado básico + hilos de respuestas
↓ ¿fallo? → ❌ Error descriptivo con diagnóstico
URL detectada → ¿Twitter/X? → FxTwitter API → Nitter fallback
→ ¿Reddit? → .json suffix API
→ ¿YouTube? → youtube-transcript-api
→ ¿otro? → Trafilatura (genérico)
```
---
## 🔧 Configuración
DeepReeder funciona listo para usar con valores predeterminados sensatos. Se puede personalizar mediante variables de entorno:
| Variable | Predeterminado | Descripción |
|----------|---------------|-------------|
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Ruta para guardar contenido |
+44 -36
View File
@@ -14,6 +14,7 @@ DeepReeder intercepte les URLs des messages utilisateur, extrait le contenu inte
|--------|---------|---------|
| 🌐 **Générique** | Blogs, articles, documentation | [Trafilatura](https://trafilatura.readthedocs.io/) avec fallback BeautifulSoup |
| 🐦 **Twitter / X** | Tweets, fils, X Articles | **FxTwitter API** (principal) + Nitter (fallback) |
| 🟠 **Reddit** | Posts + fils de commentaires | **Reddit .json API** (sans configuration) |
| 🎬 **YouTube** | Transcriptions vidéo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
### 🐦 Twitter / X — Intégration Approfondie
@@ -30,6 +31,20 @@ Propulsé par l'API [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspiré pa
| Fils de réponses | ✅ Via Nitter fallback (5 premières) |
| Statistiques d'engagement | ✅ ❤️ likes, 🔁 RTs, 👁️ vues, 🔖 signets |
### 🟠 Reddit — Intégration JSON Native
Utilise le suffixe URL `.json` intégré de Reddit — **sans clé API, sans OAuth, sans inscription**.
| Type de Contenu | Support |
|----------------|---------|
| Self posts (texte) | ✅ Corps Markdown complet |
| Link posts | ✅ URL + métadonnées |
| Meilleurs commentaires (par score) | ✅ Jusqu'à 15 commentaires |
| Fils de réponses imbriqués | ✅ Jusqu'à 3 niveaux |
| Médias (images, galeries, vidéo) | ✅ URLs extraites |
| Statistiques du post | ✅ ⬆️ score, 💬 commentaires, ratio de votes |
| Tags Flair | ✅ Inclus |
**Sans clé API. Sans connexion. Sans limite de débit.**
---
@@ -37,15 +52,10 @@ Propulsé par l'API [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspiré pa
## 📦 Installation
```bash
# Cloner le dépôt
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
cd OpenClaw-DeepReeder
# Créer un environnement virtuel
python3 -m venv .venv
source .venv/bin/activate
# Installer les dépendances
pip install -e .
```
@@ -56,20 +66,21 @@ pip install -e .
```python
from deepreader_skill import run
# Traiter une seule URL
# Traiter une URL
result = run("Regarde cet article : https://example.com/blog/post")
print(result)
# Traiter un tweet (utilise automatiquement l'API FxTwitter)
result = run("Fil intéressant : https://x.com/elonmusk/status/123456")
# Traiter un post Reddit
result = run("Super discussion : https://www.reddit.com/r/python/comments/abc123/my_post/")
print(result)
# Traiter plusieurs URLs en une fois
# Traiter plusieurs URLs
result = run("""
Voici quelques liens :
https://example.com/article
https://youtube.com/watch?v=dQw4w9WgXcQ
https://x.com/user/status/123456
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
""")
print(result)
```
@@ -81,49 +92,46 @@ print(result)
```
deepreader_skill/
├── __init__.py # Point d'entrée — fonction run()
├── manifest.json # Métadonnées du skill et configuration des triggers
├── requirements.txt # Liste des dépendances
├── manifest.json # Métadonnées du skill
├── requirements.txt # Dépendances
├── core/
│ ├── router.py # Logique de routage URL → Parser
│ ├── storage.py # Génération et sauvegarde des fichiers Markdown
│ └── utils.py # Extraction d'URLs et fonctions utilitaires
│ ├── router.py # Routage URL → Parser
│ ├── storage.py # Génération et sauvegarde Markdown
│ └── utils.py # Extraction d'URLs et utilitaires
└── parsers/
├── base.py # Parser de base abstrait et modèle ParseResult
├── generic.py # Parser générique d'articles/blogs
├── base.py # Parser de base abstrait
├── generic.py # Parser générique (Trafilatura)
├── twitter.py # Parser Twitter/X (FxTwitter + Nitter)
── youtube.py # Parser de transcriptions YouTube
── reddit.py # Parser Reddit (.json API)
└── youtube.py # Parser YouTube
```
### Stratégie du Parser Twitter
### Stratégie de Sélection
```
URL détectée → FxTwitter API (principal)
↓ succès ? → ✅ Résultat enrichi (stats, médias, articles)
↓ échec ?
Instances Nitter (fallback)
↓ succès ? → ✅ Résultat basique + fils de réponses
↓ échec ? → ❌ Message d'erreur explicatif avec diagnostic
URL détectée → Twitter/X? → FxTwitter API → Nitter fallback
→ Reddit? → .json suffix API
→ YouTube? → youtube-transcript-api
→ autre? → Trafilatura (générique)
```
---
## 🔧 Configuration
DeepReeder fonctionne immédiatement avec des valeurs par défaut raisonnables. La configuration peut être personnalisée via des variables d'environnement :
| Variable | Par défaut | Description |
|----------|-----------|-------------|
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Chemin de sauvegarde du contenu |
| `DEEPREEDER_LOG_LEVEL` | `INFO` | Niveau de verbosité des journaux |
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Chemin de sauvegarde |
| `DEEPREEDER_LOG_LEVEL` | `INFO` | Niveau de verbosité |
---
## 🙏 Remerciements
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — API publique pour récupérer le contenu Twitter/X
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — Inspiration pour l'approche d'intégration FxTwitter
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — Extraction robuste de contenu web
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — Récupération de transcriptions YouTube
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — API publique pour Twitter/X
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — Inspiration pour l'intégration FxTwitter
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — Extraction de contenu web
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — Transcriptions YouTube
---
@@ -132,16 +140,16 @@ DeepReeder fonctionne immédiatement avec des valeurs par défaut raisonnables.
Les contributions sont les bienvenues !
1. Forkez le dépôt
2. Créez une branche de fonctionnalité (`git checkout -b feature/parser-genial`)
3. Commitez vos changements (`git commit -m 'Ajouter un parser génial'`)
4. Poussez la branche (`git push origin feature/parser-genial`)
2. Créez une branche (`git checkout -b feature/parser-genial`)
3. Commitez (`git commit -m 'Ajouter un parser génial'`)
4. Poussez (`git push origin feature/parser-genial`)
5. Ouvrez une Pull Request
---
## 📄 Licence
Ce projet est sous licence **MIT** — consultez le fichier [LICENSE](LICENSE) pour plus de détails.
**Licence MIT** — consultez [LICENSE](LICENSE) pour plus de détails.
---
+27 -10
View File
@@ -14,6 +14,7 @@ DeepReederはユーザーメッセージからURLを自動検出し、専用パ
|---------|--------|------|
| 🌐 **汎用** | ブログ、記事、ドキュメント | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup フォールバック |
| 🐦 **Twitter / X** | ツイート、スレッド、Xアーティクル | **FxTwitter API**(メイン)+ Nitter(フォールバック) |
| 🟠 **Reddit** | 投稿 + コメントスレッド | **Reddit .json API**(設定不要) |
| 🎬 **YouTube** | 動画字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
### 🐦 Twitter / X — ディープインテグレーション
@@ -30,6 +31,20 @@ DeepReederはユーザーメッセージからURLを自動検出し、専用パ
| リプライスレッド | ✅ Nitterフォールバック経由(最初の5件) |
| エンゲージメント統計 | ✅ ❤️ いいね、🔁 RT、👁️ 閲覧、🔖 ブックマーク |
### 🟠 Reddit — ネイティブJSON統合
RedditのビルトインURL `.json` サフィックスを使用 — **APIキー不要、OAuth不要、登録不要**
| コンテンツタイプ | サポート |
|----------------|---------|
| セルフ投稿(テキスト) | ✅ 完全なMarkdown本文 |
| リンク投稿 | ✅ URL + メタデータ |
| 人気コメント(スコア順) | ✅ 最大15件 |
| ネストされた返信スレッド | ✅ 最大3階層 |
| メディア(画像、ギャラリー、動画) | ✅ URL抽出 |
| 投稿統計 | ✅ ⬆️ スコア、💬 コメント数、投票率 |
| Flairタグ | ✅ 含む |
**APIキー不要。ログイン不要。レート制限なし。**
---
@@ -60,16 +75,21 @@ from deepreader_skill import run
result = run("この記事をチェック: https://example.com/blog/post")
print(result)
# ツイートを処理(自動的にFxTwitter APIを使用)
# ツイートを処理
result = run("興味深いスレッド: https://x.com/elonmusk/status/123456")
print(result)
# Redditの投稿を処理
result = run("素晴らしい議論: https://www.reddit.com/r/python/comments/abc123/my_post/")
print(result)
# 複数のURLを一括処理
result = run("""
いくつかのリンクがあります:
https://example.com/article
https://youtube.com/watch?v=dQw4w9WgXcQ
https://x.com/user/status/123456
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
""")
print(result)
```
@@ -91,26 +111,23 @@ deepreader_skill/
├── base.py # 抽象基底パーサーとParseResultモデル
├── generic.py # 汎用記事/ブログパーサー
├── twitter.py # Twitter/Xパーサー(FxTwitter + Nitter
├── reddit.py # Redditパーサー(.json API
└── youtube.py # YouTube字幕パーサー
```
### Twitterパーサー戦略
### パーサー選択戦略
```
URL検出 → FxTwitter API(メイン)
↓ 成功? → ✅ リッチな結果(統計、メディア、記事)
↓ 失敗?
Nitterインスタンス(フォールバック
↓ 成功? → ✅ 基本結果 + リプライスレッド
↓ 失敗? → ❌ 診断付きエラーメッセージ
URL検出 → Twitter/X → FxTwitter API → Nitterフォールバック
→ Reddit → .jsonサフィックスAPI
→ YouTube → youtube-transcript-api
→ その他 → Trafilatura(汎用
```
---
## 🔧 設定
DeepReederはデフォルト設定ですぐに使えます。環境変数でカスタマイズ可能です:
| 変数 | デフォルト | 説明 |
|------|-----------|------|
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | コンテンツの保存先 |
+27 -10
View File
@@ -14,6 +14,7 @@ DeepReeder는 사용자 메시지에서 URL을 자동으로 감지하고, 전문
|------|------|------|
| 🌐 **범용** | 블로그, 기사, 문서 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup 대체 |
| 🐦 **Twitter / X** | 트윗, 스레드, X 아티클 | **FxTwitter API** (주력) + Nitter (대체) |
| 🟠 **Reddit** | 게시물 + 댓글 스레드 | **Reddit .json API** (제로 설정) |
| 🎬 **YouTube** | 동영상 자막 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
### 🐦 Twitter / X — 심층 통합
@@ -30,6 +31,20 @@ DeepReeder는 사용자 메시지에서 URL을 자동으로 감지하고, 전문
| 답글 스레드 | ✅ Nitter 대체를 통해 (처음 5개) |
| 참여 통계 | ✅ ❤️ 좋아요, 🔁 리트윗, 👁️ 조회, 🔖 북마크 |
### 🟠 Reddit — 네이티브 JSON 통합
Reddit의 내장 `.json` URL 접미사 사용 — **API 키 불필요, OAuth 불필요, 등록 불필요**.
| 콘텐츠 유형 | 지원 |
|------------|------|
| 셀프 포스트 (텍스트) | ✅ 전체 Markdown 본문 |
| 링크 포스트 | ✅ URL + 메타데이터 |
| 인기 댓글 (점수순 정렬) | ✅ 최대 15개 댓글 |
| 중첩 답글 스레드 | ✅ 최대 3단계 깊이 |
| 미디어 (이미지, 갤러리, 동영상) | ✅ URL 추출 |
| 게시물 통계 | ✅ ⬆️ 점수, 💬 댓글 수, 추천 비율 |
| Flair 태그 | ✅ 포함 |
**API 키 불필요. 로그인 불필요. 속도 제한 없음.**
---
@@ -60,16 +75,21 @@ from deepreader_skill import run
result = run("이 기사를 확인하세요: https://example.com/blog/post")
print(result)
# 트윗 처리 (자동으로 FxTwitter API 사용)
# 트윗 처리
result = run("흥미로운 스레드: https://x.com/elonmusk/status/123456")
print(result)
# Reddit 게시물 처리
result = run("좋은 토론: https://www.reddit.com/r/python/comments/abc123/my_post/")
print(result)
# 여러 URL 한번에 처리
result = run("""
여기 몇 가지 링크가 있습니다:
https://example.com/article
https://youtube.com/watch?v=dQw4w9WgXcQ
https://x.com/user/status/123456
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
""")
print(result)
```
@@ -91,26 +111,23 @@ deepreader_skill/
├── base.py # 추상 기본 파서 및 ParseResult 모델
├── generic.py # 범용 기사/블로그 파서
├── twitter.py # Twitter/X 파서 (FxTwitter + Nitter)
├── reddit.py # Reddit 파서 (.json API)
└── youtube.py # YouTube 자막 파서
```
### Twitter 파서 전략
### 파서 선택 전략
```
URL 감지 → FxTwitter API (주력)
↓ 성공? → ✅ 풍부한 결과 (통계, 미디어, 기사)
↓ 실패?
Nitter 인스턴스 (대체)
↓ 성공? → ✅ 기본 결과 + 답글 스레드
↓ 실패? → ❌ 친절한 오류 메시지 및 진단
URL 감지 → Twitter/X? → FxTwitter API → Nitter 대체
→ Reddit? → .json 접미사 API
→ YouTube? → youtube-transcript-api
→ 기타 → Trafilatura (범용)
```
---
## 🔧 설정
DeepReeder는 합리적인 기본값으로 바로 사용할 수 있습니다. 환경 변수로 설정을 변경할 수 있습니다:
| 변수 | 기본값 | 설명 |
|------|--------|------|
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | 콘텐츠 저장 경로 |
+26 -7
View File
@@ -14,6 +14,7 @@ DeepReeder 自动拦截用户消息中的 URL,使用专用解析器智能抓
|--------|------|------|
| 🌐 **通用** | 博客、文章、文档 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup 备用方案 |
| 🐦 **Twitter / X** | 推文、线程、X 文章 | **FxTwitter API**(主力) + Nitter(备用) |
| 🟠 **Reddit** | 帖子 + 评论线程 | **Reddit .json API**(零配置) |
| 🎬 **YouTube** | 视频字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
### 🐦 Twitter / X — 深度整合
@@ -30,6 +31,20 @@ DeepReeder 自动拦截用户消息中的 URL,使用专用解析器智能抓
| 回复线程 | ✅ 通过 Nitter 备用方案(前5条) |
| 互动数据 | ✅ ❤️ 喜欢、🔁 转发、👁️ 浏览、🔖 书签 |
### 🟠 Reddit — 原生 JSON 整合
使用 Reddit 内置的 `.json` URL 后缀 — **无需 API 密钥、无需 OAuth、无需注册**
| 内容类型 | 支持 |
|---------|------|
| 自发帖(文本) | ✅ 完整 Markdown 正文 |
| 链接帖 | ✅ URL + 元数据 |
| 热门评论(按评分排序) | ✅ 最多15条评论 |
| 嵌套回复线程 | ✅ 最多3层深度 |
| 媒体(图片、图集、视频) | ✅ URL 提取 |
| 帖子统计 | ✅ ⬆️ 评分、💬 评论数、点赞比例 |
| Flair 标签 | ✅ 包含 |
**无需 API 密钥。无需登录。无速率限制。**
---
@@ -64,12 +79,17 @@ print(result)
result = run("有趣的推文: https://x.com/elonmusk/status/123456")
print(result)
# 处理 Reddit 帖子(自动使用 .json API
result = run("精彩讨论: https://www.reddit.com/r/python/comments/abc123/my_post/")
print(result)
# 批量处理多个 URL
result = run("""
这里有一些链接:
https://example.com/article
https://youtube.com/watch?v=dQw4w9WgXcQ
https://x.com/user/status/123456
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
""")
print(result)
```
@@ -91,18 +111,17 @@ deepreader_skill/
├── base.py # 抽象基类与 ParseResult 模型
├── generic.py # 通用文章/博客解析器
├── twitter.py # Twitter/X 解析器(FxTwitter + Nitter
├── reddit.py # Reddit 解析器(.json API
└── youtube.py # YouTube 字幕解析器
```
### Twitter 解析器策略
### 解析器选择策略
```
检测到 URL → FxTwitter API(主力)
↓ 成功? → ✅ 丰富结果(数据、媒体、文章)
↓ 失败?
Nitter 实例(备用)
↓ 成功? → ✅ 基础结果 + 回复线程
↓ 失败? → ❌ 友好的错误信息与诊断
检测到 URL → Twitter/X FxTwitter API → Nitter 备用
→ Reddit → .json 后缀 API
→ YouTube → youtube-transcript-api
→ 其他 → Trafilatura(通用)
```
---
+3 -1
View File
@@ -11,9 +11,10 @@ from __future__ import annotations
import logging
from typing import Sequence
from ..core.utils import is_twitter_url, is_youtube_url
from ..core.utils import is_reddit_url, is_twitter_url, is_youtube_url
from ..parsers.base import BaseParser, ParseResult
from ..parsers.generic import GenericParser
from ..parsers.reddit import RedditParser
from ..parsers.twitter import TwitterParser
from ..parsers.youtube import YouTubeParser
@@ -60,6 +61,7 @@ class ParserRouter:
self._parsers.extend([
TwitterParser(),
YouTubeParser(),
RedditParser(),
])
# Fallback parser (always matches)
+11
View File
@@ -132,6 +132,10 @@ def content_hash(text: str) -> str:
_TWITTER_DOMAINS = {"twitter.com", "x.com", "mobile.twitter.com", "mobile.x.com"}
_YOUTUBE_DOMAINS = {"youtube.com", "youtu.be", "www.youtube.com", "m.youtube.com"}
_REDDIT_DOMAINS = {
"reddit.com", "www.reddit.com", "old.reddit.com",
"new.reddit.com", "np.reddit.com", "m.reddit.com",
}
def is_twitter_url(url: str) -> bool:
@@ -147,6 +151,13 @@ def is_youtube_url(url: str) -> bool:
return hostname in _YOUTUBE_DOMAINS
def is_reddit_url(url: str) -> bool:
"""Return ``True`` if *url* points to Reddit."""
parsed = urlparse(url)
hostname = parsed.hostname or ""
return hostname in _REDDIT_DOMAINS and "/comments/" in parsed.path
def extract_youtube_video_id(url: str) -> str | None:
"""Extract the video ID from a YouTube URL.
+8 -2
View File
@@ -23,8 +23,14 @@
"content_extraction",
"youtube_transcription",
"twitter_reading",
"reddit_reading",
"markdown_generation"
],
"memory_path": "../../memory/inbox/",
"tags": ["reader", "scraper", "ingestion", "memory"]
}
"tags": [
"reader",
"scraper",
"ingestion",
"memory"
]
}
+367
View File
@@ -0,0 +1,367 @@
"""
DeepReader Skill - Reddit Parser
==================================
Fetches Reddit posts and comments using Reddit's native ``.json``
URL suffix — **zero dependencies, zero API keys, zero configuration**.
How it works
------------
Reddit natively supports appending ``.json`` to any post URL, returning
the full post data + comment tree as structured JSON. For example::
https://www.reddit.com/r/python/comments/abc123/my_post/
→ https://www.reddit.com/r/python/comments/abc123/my_post/.json
This approach requires:
- No API keys or OAuth tokens
- No external libraries (uses only ``urllib`` from stdlib)
- No registration or app creation
The only requirement is a descriptive ``User-Agent`` header to avoid
Reddit's generic rate-limiting (HTTP 429).
Content extracted
-----------------
- Post title, body (selftext), author, score, flair
- Subreddit info
- Top-level comments sorted by score (configurable limit)
- Nested reply threads (configurable depth)
- Media URLs (images, videos, galleries)
- Crosspost/link detection
"""
from __future__ import annotations
import json
import logging
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from urllib.parse import urlparse
from .base import BaseParser, ParseResult
logger = logging.getLogger("deepreader.parsers.reddit")
class RedditParser(BaseParser):
"""Parse Reddit posts and comments via the native .json API."""
name = "reddit"
timeout = 30
# Maximum number of top-level comments to include.
max_comments: int = 15
# Maximum reply nesting depth to include.
max_reply_depth: int = 3
# User-Agent for Reddit requests (required to avoid 429).
reddit_user_agent: str = "DeepReeder/1.0 (OpenClaw Skill; +https://github.com/astonysh/OpenClaw-DeepReeder)"
def can_handle(self, url: str) -> bool:
"""Return ``True`` for reddit.com URLs that look like post links."""
from ..core.utils import is_reddit_url
return is_reddit_url(url)
def parse(self, url: str) -> ParseResult:
"""Fetch a Reddit post + comments via the .json suffix."""
json_url = self._build_json_url(url)
if not json_url:
return ParseResult.failure(
url,
"Could not build a valid Reddit JSON URL. "
"Expected format: https://www.reddit.com/r/subreddit/comments/id/title/",
)
max_attempts = 2
last_error = ""
for attempt in range(max_attempts):
try:
req = urllib.request.Request(
json_url,
headers={
"User-Agent": self.reddit_user_agent,
"Accept": "application/json",
},
)
with urllib.request.urlopen(req, timeout=self.timeout) as resp:
data = json.loads(resp.read().decode())
return self._build_result(url, data)
except urllib.error.HTTPError as exc:
if exc.code == 429:
last_error = "Reddit rate limit hit (HTTP 429). Try again later."
if attempt < max_attempts - 1:
time.sleep(2)
continue
elif exc.code == 404:
last_error = "Post not found (HTTP 404). It may be deleted or private."
elif exc.code == 403:
last_error = "Access denied (HTTP 403). The subreddit may be private."
else:
last_error = f"HTTP {exc.code}: {exc.reason}"
return ParseResult.failure(url, last_error)
except urllib.error.URLError:
last_error = "Network error: failed to reach Reddit"
if attempt < max_attempts - 1:
time.sleep(1)
continue
except Exception as exc: # noqa: BLE001
last_error = f"Unexpected error: {exc}"
logger.warning("Reddit parser unexpected error: %s", exc)
break
return ParseResult.failure(url, last_error)
# ==================================================================
# Result Builder
# ==================================================================
def _build_result(self, original_url: str, data: list) -> ParseResult:
"""Transform Reddit JSON response into a ParseResult."""
from ..core.utils import clean_text, generate_excerpt
if not isinstance(data, list) or len(data) < 1:
return ParseResult.failure(
original_url, "Unexpected Reddit JSON structure."
)
# --- Post data (first element) ---
try:
post = data[0]["data"]["children"][0]["data"]
except (KeyError, IndexError, TypeError):
return ParseResult.failure(
original_url, "Could not extract post data from Reddit JSON."
)
title = post.get("title", "Untitled")
author = post.get("author", "[deleted]")
subreddit = post.get("subreddit_name_prefixed", "r/unknown")
score = post.get("score", 0)
upvote_ratio = post.get("upvote_ratio", 0)
num_comments = post.get("num_comments", 0)
created_utc = post.get("created_utc", 0)
selftext = post.get("selftext", "")
flair = post.get("link_flair_text", "")
is_self = post.get("is_self", True)
post_url = post.get("url", "")
permalink = post.get("permalink", "")
# Format timestamp
timestamp = ""
if created_utc:
dt = datetime.fromtimestamp(created_utc, tz=timezone.utc)
timestamp = dt.strftime("%Y-%m-%d %H:%M UTC")
# --- Build content ---
content_parts: list[str] = []
# Header
content_parts.append(f"# {title}\n")
content_parts.append(
f"**{subreddit}** · u/{author} · {timestamp}\n"
)
# Stats line
stats_line = (
f"⬆️ {score:,} ({upvote_ratio:.0%} upvoted) · "
f"💬 {num_comments:,} comments"
)
if flair:
stats_line += f" · 🏷️ {flair}"
content_parts.append(f"📊 {stats_line}\n")
content_parts.append("---\n")
# Post body
if selftext:
content_parts.append(selftext)
elif not is_self and post_url:
content_parts.append(f"🔗 **Link post**: [{post_url}]({post_url})")
# --- Media detection ---
media_parts = self._extract_media(post)
if media_parts:
content_parts.append("\n\n---\n### 🖼️ Media\n")
content_parts.extend(media_parts)
# --- Comments (second element) ---
if len(data) >= 2:
comments = self._extract_comments(data[1])
if comments:
content_parts.append("\n\n---\n### 💬 Top Comments\n")
content_parts.extend(comments)
full_content = clean_text("\n".join(content_parts))
# --- Tags ---
tags = ["reddit", subreddit.lower()]
if flair:
tags.append(flair.lower().replace(" ", "-"))
if not is_self:
tags.append("link-post")
return ParseResult(
url=original_url,
title=f"[{subreddit}] {title}",
content=full_content,
author=f"u/{author}",
excerpt=generate_excerpt(full_content),
tags=tags,
)
# ==================================================================
# Comment Extraction
# ==================================================================
def _extract_comments(self, comment_listing: dict) -> list[str]:
"""Extract top-level comments sorted by score."""
try:
children = comment_listing["data"]["children"]
except (KeyError, TypeError):
return []
# Filter out "more" comment stubs and deleted comments
comments = []
for child in children:
if child.get("kind") != "t1":
continue
cdata = child.get("data", {})
body = cdata.get("body", "")
if not body or body == "[deleted]" or body == "[removed]":
continue
comments.append(cdata)
# Sort by score (highest first)
comments.sort(key=lambda c: c.get("score", 0), reverse=True)
# Build formatted comment list
result: list[str] = []
for cdata in comments[: self.max_comments]:
author = cdata.get("author", "[deleted]")
score = cdata.get("score", 0)
body = cdata.get("body", "").strip()
# Format the comment
result.append(f"**u/{author}** (⬆️ {score:,}):\n")
# Indent comment body as blockquote
quoted_body = "\n".join(f"> {line}" for line in body.split("\n"))
result.append(f"{quoted_body}\n")
# Extract nested replies (limited depth)
replies = cdata.get("replies")
if replies and isinstance(replies, dict):
nested = self._extract_nested_replies(replies, depth=1)
if nested:
result.extend(nested)
result.append("") # blank line between comments
return result
def _extract_nested_replies(
self, reply_listing: dict, depth: int,
) -> list[str]:
"""Recursively extract nested replies up to max_reply_depth."""
if depth >= self.max_reply_depth:
return []
try:
children = reply_listing["data"]["children"]
except (KeyError, TypeError):
return []
result: list[str] = []
indent = " " * depth
for child in children[:5]: # limit nested replies
if child.get("kind") != "t1":
continue
cdata = child.get("data", {})
body = cdata.get("body", "")
if not body or body in ("[deleted]", "[removed]"):
continue
author = cdata.get("author", "[deleted]")
score = cdata.get("score", 0)
result.append(f"{indent}↳ **u/{author}** (⬆️ {score:,}):\n")
quoted = "\n".join(
f"{indent}> {line}" for line in body.strip().split("\n")
)
result.append(f"{quoted}\n")
# Recurse deeper
sub_replies = cdata.get("replies")
if sub_replies and isinstance(sub_replies, dict):
nested = self._extract_nested_replies(sub_replies, depth + 1)
if nested:
result.extend(nested)
return result
# ==================================================================
# Media Extraction
# ==================================================================
def _extract_media(self, post: dict) -> list[str]:
"""Extract media URLs from a Reddit post."""
parts: list[str] = []
# Direct image URL
url = post.get("url", "")
if any(url.endswith(ext) for ext in (".jpg", ".jpeg", ".png", ".gif", ".webp")):
parts.append(f"![Image]({url})\n")
# Reddit gallery
if post.get("is_gallery"):
media_metadata = post.get("media_metadata", {})
for i, (_, media) in enumerate(media_metadata.items(), 1):
if media.get("status") == "valid" and media.get("s", {}).get("u"):
img_url = media["s"]["u"].replace("&amp;", "&")
parts.append(f"![Gallery image {i}]({img_url})\n")
# Reddit video
reddit_video = (post.get("media") or {}).get("reddit_video", {})
if reddit_video.get("fallback_url"):
parts.append(f"🎬 Video: [{reddit_video['fallback_url']}]({reddit_video['fallback_url']})\n")
# External video (e.g. YouTube embed)
if post.get("is_video") and not reddit_video:
parts.append(f"🎬 External video: [{url}]({url})\n")
return parts
# ==================================================================
# URL Utilities
# ==================================================================
@staticmethod
def _build_json_url(url: str) -> str | None:
"""Convert a Reddit post URL to its .json API endpoint.
Examples::
https://www.reddit.com/r/python/comments/abc123/my_post/
→ https://www.reddit.com/r/python/comments/abc123/my_post/.json
https://old.reddit.com/r/python/comments/abc123/
→ https://www.reddit.com/r/python/comments/abc123/.json
"""
parsed = urlparse(url)
# Validate it looks like a Reddit post URL
if not parsed.path or "/comments/" not in parsed.path:
return None
# Normalize to www.reddit.com
path = parsed.path.rstrip("/")
# Append .json
json_url = f"https://www.reddit.com{path}/.json"
return json_url