mirror of
https://github.com/astonysh/OpenClaw-DeepReeder.git
synced 2026-08-14 09:02:11 +00:00
🟠 Add Reddit parser + update all READMEs
Reddit Parser: - Uses Reddit's native .json URL suffix (zero dependencies) - Extracts post title, body, author, score, flair - Top comments sorted by score (up to 15) - Nested reply threads (up to 3 levels deep) - Media detection (images, galleries, Reddit video) - Proper error handling (429 rate limit, 404, 403) Core: - Added is_reddit_url() to utils.py - Registered RedditParser in router.py - Added reddit_reading capability to manifest.json README: - Updated all 7 language versions with Reddit section (EN, 中文, Español, 한국어, 日本語, العربية, Français)
This commit is contained in:
@@ -14,6 +14,7 @@ DeepReeder intercepts URLs from user messages, scrapes content intelligently usi
|
||||
|--------|---------|--------|
|
||||
| 🌐 **Generic** | Blogs, articles, docs | [Trafilatura](https://trafilatura.readthedocs.io/) with BeautifulSoup fallback |
|
||||
| 🐦 **Twitter / X** | Tweets, threads, X Articles | **FxTwitter API** (primary) + Nitter (fallback) |
|
||||
| 🟠 **Reddit** | Posts + comment threads | **Reddit .json API** (zero-config) |
|
||||
| 🎬 **YouTube** | Video transcripts | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — Deep Integration
|
||||
@@ -30,6 +31,20 @@ Powered by [FxTwitter](https://github.com/FxEmbed/FxEmbed) API with Nitter fallb
|
||||
| Reply threads | ✅ Via Nitter fallback (first 5) |
|
||||
| Engagement stats | ✅ ❤️ likes, 🔁 RTs, 👁️ views, 🔖 bookmarks |
|
||||
|
||||
### 🟠 Reddit — Native JSON Integration
|
||||
|
||||
Uses Reddit's built-in `.json` URL suffix — **no API keys, no OAuth, no registration**.
|
||||
|
||||
| Content Type | Support |
|
||||
|-------------|---------|
|
||||
| Self posts (text) | ✅ Full markdown body |
|
||||
| Link posts | ✅ URL + metadata |
|
||||
| Top comments (sorted by score) | ✅ Up to 15 comments |
|
||||
| Nested reply threads | ✅ Up to 3 levels deep |
|
||||
| Media (images, galleries, video) | ✅ URLs extracted |
|
||||
| Post stats | ✅ ⬆️ score, 💬 comment count, upvote ratio |
|
||||
| Flair tags | ✅ Included |
|
||||
|
||||
**No API keys. No login. No rate limits.**
|
||||
|
||||
### Output Format
|
||||
@@ -38,18 +53,29 @@ Every piece of content is saved as a `.md` file with structured YAML frontmatter
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: "Article Title"
|
||||
source_url: "https://example.com/article"
|
||||
domain: "example.com"
|
||||
parser: "generic"
|
||||
title: "[r/python] How I built an AI agent framework"
|
||||
source_url: "https://www.reddit.com/r/python/comments/abc123/..."
|
||||
domain: "reddit.com"
|
||||
parser: "reddit"
|
||||
ingested_at: "2026-02-16T12:00:00Z"
|
||||
content_hash: "sha256:abc123..."
|
||||
word_count: 1500
|
||||
word_count: 2500
|
||||
---
|
||||
|
||||
# Article Title
|
||||
# How I built an AI agent framework
|
||||
|
||||
The clean, extracted content goes here...
|
||||
**r/python** · u/developer123 · 2026-02-16 12:00 UTC
|
||||
📊 ⬆️ 847 (96% upvoted) · 💬 234 comments · 🏷️ Discussion
|
||||
|
||||
---
|
||||
|
||||
Post body goes here...
|
||||
|
||||
---
|
||||
### 💬 Top Comments
|
||||
|
||||
**u/expert_dev** (⬆️ 342):
|
||||
> This is a really well-structured approach...
|
||||
```
|
||||
|
||||
---
|
||||
@@ -84,37 +110,21 @@ print(result)
|
||||
result = run("Interesting thread: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Process a Reddit post (uses .json API automatically)
|
||||
result = run("Great discussion: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# Process multiple URLs at once
|
||||
result = run("""
|
||||
Here are some links:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
### Example Output
|
||||
|
||||
```
|
||||
📚 DeepReader — Processed 3 URL(s):
|
||||
|
||||
✅ How to Build AI Agents
|
||||
Source: https://example.com/article
|
||||
Saved to: memory/inbox/2026-02-16_how-to-build-ai-agents.md
|
||||
Content: 3,200 characters
|
||||
|
||||
✅ Tweet by @elonmusk (Mon Feb 16 12:00:00 +0000 2026)
|
||||
Source: https://x.com/elonmusk/status/123456
|
||||
Saved to: memory/inbox/2026-02-16_tweet-by-elonmusk.md
|
||||
Content: 480 characters
|
||||
|
||||
✅ Rick Astley - Never Gonna Give You Up
|
||||
Source: https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
Saved to: memory/inbox/2026-02-16_rick-astley-never-gonna.md
|
||||
Content: 15,000 characters
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
@@ -132,18 +142,17 @@ deepreader_skill/
|
||||
├── base.py # Abstract base parser & ParseResult model
|
||||
├── generic.py # Generic article/blog parser (Trafilatura)
|
||||
├── twitter.py # Twitter/X parser (FxTwitter + Nitter)
|
||||
├── reddit.py # Reddit parser (.json API)
|
||||
└── youtube.py # YouTube transcript parser
|
||||
```
|
||||
|
||||
### Twitter Parser Strategy
|
||||
### Parser Selection Strategy
|
||||
|
||||
```
|
||||
URL detected → FxTwitter API (primary)
|
||||
↓ success? → ✅ Rich result (stats, media, articles)
|
||||
↓ failure?
|
||||
Nitter instances (fallback)
|
||||
↓ success? → ✅ Basic result + reply threads
|
||||
↓ failure? → ❌ Graceful error with diagnostics
|
||||
URL detected → is Twitter/X? → FxTwitter API → Nitter fallback
|
||||
→ is Reddit? → .json suffix API
|
||||
→ is YouTube? → youtube-transcript-api
|
||||
→ otherwise → Trafilatura (generic)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
+20
-83
@@ -4,8 +4,6 @@
|
||||
|
||||
> **محرك استيعاب محتوى الويب الذاتي لوكلاء الذكاء الاصطناعي.**
|
||||
|
||||
يعترض DeepReeder عناوين URL من رسائل المستخدم، ويستخرج المحتوى بذكاء باستخدام محللات متخصصة، ويحوله إلى Markdown نظيف مع بيانات YAML الوصفية، ويحفظه في الذاكرة طويلة المدى للوكيل.
|
||||
|
||||
🌍 **الترجمات**: [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
@@ -16,21 +14,22 @@
|
||||
|--------|---------|---------|
|
||||
| 🌐 **عام** | مدونات، مقالات، وثائق | [Trafilatura](https://trafilatura.readthedocs.io/) مع BeautifulSoup احتياطي |
|
||||
| 🐦 **Twitter / X** | تغريدات، سلاسل، مقالات X | **FxTwitter API** (رئيسي) + Nitter (احتياطي) |
|
||||
| 🟠 **Reddit** | منشورات + سلاسل تعليقات | **Reddit .json API** (بدون إعداد) |
|
||||
| 🎬 **YouTube** | نصوص الفيديو | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — تكامل عميق
|
||||
### 🟠 Reddit — تكامل JSON الأصلي
|
||||
|
||||
مدعوم بواجهة برمجة تطبيقات [FxTwitter](https://github.com/FxEmbed/FxEmbed). مستلهم من [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
|
||||
يستخدم لاحقة URL `.json` المدمجة في Reddit — **بدون مفاتيح API، بدون OAuth، بدون تسجيل**.
|
||||
|
||||
| نوع المحتوى | الدعم |
|
||||
|-------------|-------|
|
||||
| التغريدات العادية | ✅ نص كامل + إحصائيات التفاعل |
|
||||
| التغريدات الطويلة (Twitter Blue) | ✅ نص كامل |
|
||||
| مقالات X (محتوى طويل) | ✅ مقال كامل + عدد الكلمات |
|
||||
| التغريدات المقتبسة | ✅ محتوى متداخل مضمّن |
|
||||
| الوسائط (صور، فيديو، GIF) | ✅ استخراج الروابط |
|
||||
| سلاسل الردود | ✅ عبر Nitter الاحتياطي (أول 5) |
|
||||
| إحصائيات التفاعل | ✅ ❤️ إعجابات، 🔁 إعادة تغريد، 👁️ مشاهدات، 🔖 إشارات مرجعية |
|
||||
| منشورات نصية | ✅ نص Markdown كامل |
|
||||
| منشورات الروابط | ✅ URL + بيانات وصفية |
|
||||
| أفضل التعليقات (مرتبة حسب النقاط) | ✅ حتى 15 تعليقاً |
|
||||
| سلاسل الردود المتداخلة | ✅ حتى 3 مستويات عمق |
|
||||
| الوسائط (صور، معارض، فيديو) | ✅ استخراج الروابط |
|
||||
| إحصائيات المنشور | ✅ ⬆️ النقاط، 💬 عدد التعليقات |
|
||||
| علامات Flair | ✅ مضمّنة |
|
||||
|
||||
**بدون مفاتيح API. بدون تسجيل دخول. بدون حدود للسرعة.**
|
||||
|
||||
@@ -41,15 +40,10 @@
|
||||
<div dir="ltr">
|
||||
|
||||
```bash
|
||||
# استنساخ المستودع
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# إنشاء بيئة افتراضية
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# تثبيت التبعيات
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
@@ -64,21 +58,10 @@ pip install -e .
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# معالجة عنوان URL واحد
|
||||
result = run("اطلع على هذا المقال: https://example.com/blog/post")
|
||||
result = run("اطلع على هذا: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# معالجة تغريدة (يستخدم FxTwitter API تلقائياً)
|
||||
result = run("سلسلة مثيرة: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# معالجة عناوين URL متعددة دفعة واحدة
|
||||
result = run("""
|
||||
إليك بعض الروابط:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
""")
|
||||
result = run("نقاش رائع: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
```
|
||||
|
||||
@@ -93,78 +76,32 @@ print(result)
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # نقطة الدخول — دالة run()
|
||||
├── manifest.json # بيانات المهارة الوصفية وإعدادات المشغلات
|
||||
├── manifest.json # بيانات المهارة الوصفية
|
||||
├── requirements.txt # قائمة التبعيات
|
||||
├── core/
|
||||
│ ├── router.py # منطق توجيه URL → المحلل
|
||||
│ ├── storage.py # إنشاء وحفظ ملفات Markdown
|
||||
│ └── utils.py # استخراج URL ودوال مساعدة
|
||||
└── parsers/
|
||||
├── base.py # المحلل الأساسي المجرد ونموذج ParseResult
|
||||
├── generic.py # محلل المقالات/المدونات العام
|
||||
├── twitter.py # محلل Twitter/X (FxTwitter + Nitter)
|
||||
├── base.py # المحلل الأساسي
|
||||
├── generic.py # محلل المقالات العام
|
||||
├── twitter.py # محلل Twitter/X
|
||||
├── reddit.py # محلل Reddit (.json API)
|
||||
└── youtube.py # محلل نصوص YouTube
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
### استراتيجية محلل Twitter
|
||||
|
||||
<div dir="ltr">
|
||||
|
||||
```
|
||||
اكتشاف URL → FxTwitter API (رئيسي)
|
||||
↓ نجاح؟ → ✅ نتيجة غنية (إحصائيات، وسائط، مقالات)
|
||||
↓ فشل؟
|
||||
مثيلات Nitter (احتياطي)
|
||||
↓ نجاح؟ → ✅ نتيجة أساسية + سلاسل الردود
|
||||
↓ فشل؟ → ❌ رسالة خطأ ودية مع التشخيص
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 🔧 الإعدادات
|
||||
|
||||
يعمل DeepReeder مباشرة مع إعدادات افتراضية معقولة. يمكن تخصيص الإعدادات عبر متغيرات البيئة:
|
||||
|
||||
| المتغير | الافتراضي | الوصف |
|
||||
|---------|-----------|-------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | مسار حفظ المحتوى |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | مستوى تفصيل السجلات |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 شكر وتقدير
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — واجهة برمجة تطبيقات عامة لجلب محتوى Twitter/X
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — مصدر إلهام لنهج تكامل FxTwitter
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — استخراج محتوى الويب القوي
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — جلب نصوص YouTube
|
||||
|
||||
---
|
||||
|
||||
## 🤝 المساهمة
|
||||
|
||||
المساهمات مرحب بها! يمكنك:
|
||||
|
||||
1. عمل Fork للمستودع
|
||||
2. إنشاء فرع ميزة (`git checkout -b feature/amazing-parser`)
|
||||
3. التزام بتغييراتك (`git commit -m 'إضافة محلل رائع'`)
|
||||
4. دفع الفرع (`git push origin feature/amazing-parser`)
|
||||
5. فتح Pull Request
|
||||
|
||||
---
|
||||
- **[FxTwitter](https://github.com/FxEmbed/FxEmbed)** · **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** · **[Trafilatura](https://trafilatura.readthedocs.io/)** · **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)**
|
||||
|
||||
## 📄 الترخيص
|
||||
|
||||
هذا المشروع مرخص بموجب **رخصة MIT** — راجع ملف [LICENSE](LICENSE) للتفاصيل.
|
||||
**رخصة MIT** — راجع ملف [LICENSE](LICENSE) للتفاصيل.
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
صنع بـ 🦞 بواسطة <a href="https://github.com/astonysh">OpenClaw</a>
|
||||
</p>
|
||||
<p align="center">صنع بـ 🦞 بواسطة <a href="https://github.com/astonysh">OpenClaw</a></p>
|
||||
|
||||
</div>
|
||||
|
||||
+28
-11
@@ -14,6 +14,7 @@ DeepReeder intercepta URLs de los mensajes de usuario, extrae contenido de forma
|
||||
|--------|---------|--------|
|
||||
| 🌐 **Genérico** | Blogs, artículos, documentación | [Trafilatura](https://trafilatura.readthedocs.io/) con fallback BeautifulSoup |
|
||||
| 🐦 **Twitter / X** | Tweets, hilos, X Articles | **FxTwitter API** (principal) + Nitter (fallback) |
|
||||
| 🟠 **Reddit** | Posts + hilos de comentarios | **Reddit .json API** (sin configuración) |
|
||||
| 🎬 **YouTube** | Transcripciones de vídeo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — Integración Profunda
|
||||
@@ -30,6 +31,20 @@ Impulsado por la API de [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspira
|
||||
| Hilos de respuestas | ✅ Vía Nitter fallback (primeras 5) |
|
||||
| Estadísticas de interacción | ✅ ❤️ likes, 🔁 RTs, 👁️ vistas, 🔖 marcadores |
|
||||
|
||||
### 🟠 Reddit — Integración JSON Nativa
|
||||
|
||||
Usa el sufijo `.json` nativo de Reddit — **sin claves API, sin OAuth, sin registro**.
|
||||
|
||||
| Tipo de Contenido | Soporte |
|
||||
|-------------------|---------|
|
||||
| Self posts (texto) | ✅ Cuerpo completo en Markdown |
|
||||
| Link posts | ✅ URL + metadatos |
|
||||
| Comentarios principales (por puntuación) | ✅ Hasta 15 comentarios |
|
||||
| Hilos de respuestas anidados | ✅ Hasta 3 niveles |
|
||||
| Medios (imágenes, galerías, vídeo) | ✅ URLs extraídas |
|
||||
| Estadísticas del post | ✅ ⬆️ puntuación, 💬 comentarios, ratio de votos |
|
||||
| Etiquetas Flair | ✅ Incluidas |
|
||||
|
||||
**Sin claves API. Sin inicio de sesión. Sin límites de velocidad.**
|
||||
|
||||
---
|
||||
@@ -60,16 +75,21 @@ from deepreader_skill import run
|
||||
result = run("Mira este artículo: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# Procesar un tweet (usa FxTwitter API automáticamente)
|
||||
# Procesar un tweet
|
||||
result = run("Hilo interesante: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Procesar múltiples URLs a la vez
|
||||
# Procesar un post de Reddit
|
||||
result = run("Gran discusión: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# Procesar múltiples URLs
|
||||
result = run("""
|
||||
Aquí hay algunos enlaces:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
@@ -91,26 +111,23 @@ deepreader_skill/
|
||||
├── base.py # Parser base abstracto y modelo ParseResult
|
||||
├── generic.py # Parser genérico de artículos/blogs
|
||||
├── twitter.py # Parser Twitter/X (FxTwitter + Nitter)
|
||||
├── reddit.py # Parser Reddit (.json API)
|
||||
└── youtube.py # Parser de transcripciones de YouTube
|
||||
```
|
||||
|
||||
### Estrategia del Parser de Twitter
|
||||
### Estrategia de Selección de Parser
|
||||
|
||||
```
|
||||
URL detectada → FxTwitter API (principal)
|
||||
↓ ¿éxito? → ✅ Resultado enriquecido (stats, media, artículos)
|
||||
↓ ¿fallo?
|
||||
Instancias Nitter (fallback)
|
||||
↓ ¿éxito? → ✅ Resultado básico + hilos de respuestas
|
||||
↓ ¿fallo? → ❌ Error descriptivo con diagnóstico
|
||||
URL detectada → ¿Twitter/X? → FxTwitter API → Nitter fallback
|
||||
→ ¿Reddit? → .json suffix API
|
||||
→ ¿YouTube? → youtube-transcript-api
|
||||
→ ¿otro? → Trafilatura (genérico)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuración
|
||||
|
||||
DeepReeder funciona listo para usar con valores predeterminados sensatos. Se puede personalizar mediante variables de entorno:
|
||||
|
||||
| Variable | Predeterminado | Descripción |
|
||||
|----------|---------------|-------------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Ruta para guardar contenido |
|
||||
|
||||
+44
-36
@@ -14,6 +14,7 @@ DeepReeder intercepte les URLs des messages utilisateur, extrait le contenu inte
|
||||
|--------|---------|---------|
|
||||
| 🌐 **Générique** | Blogs, articles, documentation | [Trafilatura](https://trafilatura.readthedocs.io/) avec fallback BeautifulSoup |
|
||||
| 🐦 **Twitter / X** | Tweets, fils, X Articles | **FxTwitter API** (principal) + Nitter (fallback) |
|
||||
| 🟠 **Reddit** | Posts + fils de commentaires | **Reddit .json API** (sans configuration) |
|
||||
| 🎬 **YouTube** | Transcriptions vidéo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — Intégration Approfondie
|
||||
@@ -30,6 +31,20 @@ Propulsé par l'API [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspiré pa
|
||||
| Fils de réponses | ✅ Via Nitter fallback (5 premières) |
|
||||
| Statistiques d'engagement | ✅ ❤️ likes, 🔁 RTs, 👁️ vues, 🔖 signets |
|
||||
|
||||
### 🟠 Reddit — Intégration JSON Native
|
||||
|
||||
Utilise le suffixe URL `.json` intégré de Reddit — **sans clé API, sans OAuth, sans inscription**.
|
||||
|
||||
| Type de Contenu | Support |
|
||||
|----------------|---------|
|
||||
| Self posts (texte) | ✅ Corps Markdown complet |
|
||||
| Link posts | ✅ URL + métadonnées |
|
||||
| Meilleurs commentaires (par score) | ✅ Jusqu'à 15 commentaires |
|
||||
| Fils de réponses imbriqués | ✅ Jusqu'à 3 niveaux |
|
||||
| Médias (images, galeries, vidéo) | ✅ URLs extraites |
|
||||
| Statistiques du post | ✅ ⬆️ score, 💬 commentaires, ratio de votes |
|
||||
| Tags Flair | ✅ Inclus |
|
||||
|
||||
**Sans clé API. Sans connexion. Sans limite de débit.**
|
||||
|
||||
---
|
||||
@@ -37,15 +52,10 @@ Propulsé par l'API [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspiré pa
|
||||
## 📦 Installation
|
||||
|
||||
```bash
|
||||
# Cloner le dépôt
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# Créer un environnement virtuel
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Installer les dépendances
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
@@ -56,20 +66,21 @@ pip install -e .
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# Traiter une seule URL
|
||||
# Traiter une URL
|
||||
result = run("Regarde cet article : https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# Traiter un tweet (utilise automatiquement l'API FxTwitter)
|
||||
result = run("Fil intéressant : https://x.com/elonmusk/status/123456")
|
||||
# Traiter un post Reddit
|
||||
result = run("Super discussion : https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# Traiter plusieurs URLs en une fois
|
||||
# Traiter plusieurs URLs
|
||||
result = run("""
|
||||
Voici quelques liens :
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
@@ -81,49 +92,46 @@ print(result)
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # Point d'entrée — fonction run()
|
||||
├── manifest.json # Métadonnées du skill et configuration des triggers
|
||||
├── requirements.txt # Liste des dépendances
|
||||
├── manifest.json # Métadonnées du skill
|
||||
├── requirements.txt # Dépendances
|
||||
├── core/
|
||||
│ ├── router.py # Logique de routage URL → Parser
|
||||
│ ├── storage.py # Génération et sauvegarde des fichiers Markdown
|
||||
│ └── utils.py # Extraction d'URLs et fonctions utilitaires
|
||||
│ ├── router.py # Routage URL → Parser
|
||||
│ ├── storage.py # Génération et sauvegarde Markdown
|
||||
│ └── utils.py # Extraction d'URLs et utilitaires
|
||||
└── parsers/
|
||||
├── base.py # Parser de base abstrait et modèle ParseResult
|
||||
├── generic.py # Parser générique d'articles/blogs
|
||||
├── base.py # Parser de base abstrait
|
||||
├── generic.py # Parser générique (Trafilatura)
|
||||
├── twitter.py # Parser Twitter/X (FxTwitter + Nitter)
|
||||
└── youtube.py # Parser de transcriptions YouTube
|
||||
├── reddit.py # Parser Reddit (.json API)
|
||||
└── youtube.py # Parser YouTube
|
||||
```
|
||||
|
||||
### Stratégie du Parser Twitter
|
||||
### Stratégie de Sélection
|
||||
|
||||
```
|
||||
URL détectée → FxTwitter API (principal)
|
||||
↓ succès ? → ✅ Résultat enrichi (stats, médias, articles)
|
||||
↓ échec ?
|
||||
Instances Nitter (fallback)
|
||||
↓ succès ? → ✅ Résultat basique + fils de réponses
|
||||
↓ échec ? → ❌ Message d'erreur explicatif avec diagnostic
|
||||
URL détectée → Twitter/X? → FxTwitter API → Nitter fallback
|
||||
→ Reddit? → .json suffix API
|
||||
→ YouTube? → youtube-transcript-api
|
||||
→ autre? → Trafilatura (générique)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
DeepReeder fonctionne immédiatement avec des valeurs par défaut raisonnables. La configuration peut être personnalisée via des variables d'environnement :
|
||||
|
||||
| Variable | Par défaut | Description |
|
||||
|----------|-----------|-------------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Chemin de sauvegarde du contenu |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | Niveau de verbosité des journaux |
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Chemin de sauvegarde |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | Niveau de verbosité |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Remerciements
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — API publique pour récupérer le contenu Twitter/X
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — Inspiration pour l'approche d'intégration FxTwitter
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — Extraction robuste de contenu web
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — Récupération de transcriptions YouTube
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — API publique pour Twitter/X
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — Inspiration pour l'intégration FxTwitter
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — Extraction de contenu web
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — Transcriptions YouTube
|
||||
|
||||
---
|
||||
|
||||
@@ -132,16 +140,16 @@ DeepReeder fonctionne immédiatement avec des valeurs par défaut raisonnables.
|
||||
Les contributions sont les bienvenues !
|
||||
|
||||
1. Forkez le dépôt
|
||||
2. Créez une branche de fonctionnalité (`git checkout -b feature/parser-genial`)
|
||||
3. Commitez vos changements (`git commit -m 'Ajouter un parser génial'`)
|
||||
4. Poussez la branche (`git push origin feature/parser-genial`)
|
||||
2. Créez une branche (`git checkout -b feature/parser-genial`)
|
||||
3. Commitez (`git commit -m 'Ajouter un parser génial'`)
|
||||
4. Poussez (`git push origin feature/parser-genial`)
|
||||
5. Ouvrez une Pull Request
|
||||
|
||||
---
|
||||
|
||||
## 📄 Licence
|
||||
|
||||
Ce projet est sous licence **MIT** — consultez le fichier [LICENSE](LICENSE) pour plus de détails.
|
||||
**Licence MIT** — consultez [LICENSE](LICENSE) pour plus de détails.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+27
-10
@@ -14,6 +14,7 @@ DeepReederはユーザーメッセージからURLを自動検出し、専用パ
|
||||
|---------|--------|------|
|
||||
| 🌐 **汎用** | ブログ、記事、ドキュメント | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup フォールバック |
|
||||
| 🐦 **Twitter / X** | ツイート、スレッド、Xアーティクル | **FxTwitter API**(メイン)+ Nitter(フォールバック) |
|
||||
| 🟠 **Reddit** | 投稿 + コメントスレッド | **Reddit .json API**(設定不要) |
|
||||
| 🎬 **YouTube** | 動画字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — ディープインテグレーション
|
||||
@@ -30,6 +31,20 @@ DeepReederはユーザーメッセージからURLを自動検出し、専用パ
|
||||
| リプライスレッド | ✅ Nitterフォールバック経由(最初の5件) |
|
||||
| エンゲージメント統計 | ✅ ❤️ いいね、🔁 RT、👁️ 閲覧、🔖 ブックマーク |
|
||||
|
||||
### 🟠 Reddit — ネイティブJSON統合
|
||||
|
||||
RedditのビルトインURL `.json` サフィックスを使用 — **APIキー不要、OAuth不要、登録不要**。
|
||||
|
||||
| コンテンツタイプ | サポート |
|
||||
|----------------|---------|
|
||||
| セルフ投稿(テキスト) | ✅ 完全なMarkdown本文 |
|
||||
| リンク投稿 | ✅ URL + メタデータ |
|
||||
| 人気コメント(スコア順) | ✅ 最大15件 |
|
||||
| ネストされた返信スレッド | ✅ 最大3階層 |
|
||||
| メディア(画像、ギャラリー、動画) | ✅ URL抽出 |
|
||||
| 投稿統計 | ✅ ⬆️ スコア、💬 コメント数、投票率 |
|
||||
| Flairタグ | ✅ 含む |
|
||||
|
||||
**APIキー不要。ログイン不要。レート制限なし。**
|
||||
|
||||
---
|
||||
@@ -60,16 +75,21 @@ from deepreader_skill import run
|
||||
result = run("この記事をチェック: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# ツイートを処理(自動的にFxTwitter APIを使用)
|
||||
# ツイートを処理
|
||||
result = run("興味深いスレッド: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Redditの投稿を処理
|
||||
result = run("素晴らしい議論: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# 複数のURLを一括処理
|
||||
result = run("""
|
||||
いくつかのリンクがあります:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
@@ -91,26 +111,23 @@ deepreader_skill/
|
||||
├── base.py # 抽象基底パーサーとParseResultモデル
|
||||
├── generic.py # 汎用記事/ブログパーサー
|
||||
├── twitter.py # Twitter/Xパーサー(FxTwitter + Nitter)
|
||||
├── reddit.py # Redditパーサー(.json API)
|
||||
└── youtube.py # YouTube字幕パーサー
|
||||
```
|
||||
|
||||
### Twitterパーサー戦略
|
||||
### パーサー選択戦略
|
||||
|
||||
```
|
||||
URL検出 → FxTwitter API(メイン)
|
||||
↓ 成功? → ✅ リッチな結果(統計、メディア、記事)
|
||||
↓ 失敗?
|
||||
Nitterインスタンス(フォールバック)
|
||||
↓ 成功? → ✅ 基本結果 + リプライスレッド
|
||||
↓ 失敗? → ❌ 診断付きエラーメッセージ
|
||||
URL検出 → Twitter/X? → FxTwitter API → Nitterフォールバック
|
||||
→ Reddit? → .jsonサフィックスAPI
|
||||
→ YouTube? → youtube-transcript-api
|
||||
→ その他 → Trafilatura(汎用)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 設定
|
||||
|
||||
DeepReederはデフォルト設定ですぐに使えます。環境変数でカスタマイズ可能です:
|
||||
|
||||
| 変数 | デフォルト | 説明 |
|
||||
|------|-----------|------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | コンテンツの保存先 |
|
||||
|
||||
+27
-10
@@ -14,6 +14,7 @@ DeepReeder는 사용자 메시지에서 URL을 자동으로 감지하고, 전문
|
||||
|------|------|------|
|
||||
| 🌐 **범용** | 블로그, 기사, 문서 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup 대체 |
|
||||
| 🐦 **Twitter / X** | 트윗, 스레드, X 아티클 | **FxTwitter API** (주력) + Nitter (대체) |
|
||||
| 🟠 **Reddit** | 게시물 + 댓글 스레드 | **Reddit .json API** (제로 설정) |
|
||||
| 🎬 **YouTube** | 동영상 자막 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — 심층 통합
|
||||
@@ -30,6 +31,20 @@ DeepReeder는 사용자 메시지에서 URL을 자동으로 감지하고, 전문
|
||||
| 답글 스레드 | ✅ Nitter 대체를 통해 (처음 5개) |
|
||||
| 참여 통계 | ✅ ❤️ 좋아요, 🔁 리트윗, 👁️ 조회, 🔖 북마크 |
|
||||
|
||||
### 🟠 Reddit — 네이티브 JSON 통합
|
||||
|
||||
Reddit의 내장 `.json` URL 접미사 사용 — **API 키 불필요, OAuth 불필요, 등록 불필요**.
|
||||
|
||||
| 콘텐츠 유형 | 지원 |
|
||||
|------------|------|
|
||||
| 셀프 포스트 (텍스트) | ✅ 전체 Markdown 본문 |
|
||||
| 링크 포스트 | ✅ URL + 메타데이터 |
|
||||
| 인기 댓글 (점수순 정렬) | ✅ 최대 15개 댓글 |
|
||||
| 중첩 답글 스레드 | ✅ 최대 3단계 깊이 |
|
||||
| 미디어 (이미지, 갤러리, 동영상) | ✅ URL 추출 |
|
||||
| 게시물 통계 | ✅ ⬆️ 점수, 💬 댓글 수, 추천 비율 |
|
||||
| Flair 태그 | ✅ 포함 |
|
||||
|
||||
**API 키 불필요. 로그인 불필요. 속도 제한 없음.**
|
||||
|
||||
---
|
||||
@@ -60,16 +75,21 @@ from deepreader_skill import run
|
||||
result = run("이 기사를 확인하세요: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# 트윗 처리 (자동으로 FxTwitter API 사용)
|
||||
# 트윗 처리
|
||||
result = run("흥미로운 스레드: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Reddit 게시물 처리
|
||||
result = run("좋은 토론: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# 여러 URL 한번에 처리
|
||||
result = run("""
|
||||
여기 몇 가지 링크가 있습니다:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
@@ -91,26 +111,23 @@ deepreader_skill/
|
||||
├── base.py # 추상 기본 파서 및 ParseResult 모델
|
||||
├── generic.py # 범용 기사/블로그 파서
|
||||
├── twitter.py # Twitter/X 파서 (FxTwitter + Nitter)
|
||||
├── reddit.py # Reddit 파서 (.json API)
|
||||
└── youtube.py # YouTube 자막 파서
|
||||
```
|
||||
|
||||
### Twitter 파서 전략
|
||||
### 파서 선택 전략
|
||||
|
||||
```
|
||||
URL 감지 → FxTwitter API (주력)
|
||||
↓ 성공? → ✅ 풍부한 결과 (통계, 미디어, 기사)
|
||||
↓ 실패?
|
||||
Nitter 인스턴스 (대체)
|
||||
↓ 성공? → ✅ 기본 결과 + 답글 스레드
|
||||
↓ 실패? → ❌ 친절한 오류 메시지 및 진단
|
||||
URL 감지 → Twitter/X? → FxTwitter API → Nitter 대체
|
||||
→ Reddit? → .json 접미사 API
|
||||
→ YouTube? → youtube-transcript-api
|
||||
→ 기타 → Trafilatura (범용)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 설정
|
||||
|
||||
DeepReeder는 합리적인 기본값으로 바로 사용할 수 있습니다. 환경 변수로 설정을 변경할 수 있습니다:
|
||||
|
||||
| 변수 | 기본값 | 설명 |
|
||||
|------|--------|------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | 콘텐츠 저장 경로 |
|
||||
|
||||
+26
-7
@@ -14,6 +14,7 @@ DeepReeder 自动拦截用户消息中的 URL,使用专用解析器智能抓
|
||||
|--------|------|------|
|
||||
| 🌐 **通用** | 博客、文章、文档 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup 备用方案 |
|
||||
| 🐦 **Twitter / X** | 推文、线程、X 文章 | **FxTwitter API**(主力) + Nitter(备用) |
|
||||
| 🟠 **Reddit** | 帖子 + 评论线程 | **Reddit .json API**(零配置) |
|
||||
| 🎬 **YouTube** | 视频字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — 深度整合
|
||||
@@ -30,6 +31,20 @@ DeepReeder 自动拦截用户消息中的 URL,使用专用解析器智能抓
|
||||
| 回复线程 | ✅ 通过 Nitter 备用方案(前5条) |
|
||||
| 互动数据 | ✅ ❤️ 喜欢、🔁 转发、👁️ 浏览、🔖 书签 |
|
||||
|
||||
### 🟠 Reddit — 原生 JSON 整合
|
||||
|
||||
使用 Reddit 内置的 `.json` URL 后缀 — **无需 API 密钥、无需 OAuth、无需注册**。
|
||||
|
||||
| 内容类型 | 支持 |
|
||||
|---------|------|
|
||||
| 自发帖(文本) | ✅ 完整 Markdown 正文 |
|
||||
| 链接帖 | ✅ URL + 元数据 |
|
||||
| 热门评论(按评分排序) | ✅ 最多15条评论 |
|
||||
| 嵌套回复线程 | ✅ 最多3层深度 |
|
||||
| 媒体(图片、图集、视频) | ✅ URL 提取 |
|
||||
| 帖子统计 | ✅ ⬆️ 评分、💬 评论数、点赞比例 |
|
||||
| Flair 标签 | ✅ 包含 |
|
||||
|
||||
**无需 API 密钥。无需登录。无速率限制。**
|
||||
|
||||
---
|
||||
@@ -64,12 +79,17 @@ print(result)
|
||||
result = run("有趣的推文: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# 处理 Reddit 帖子(自动使用 .json API)
|
||||
result = run("精彩讨论: https://www.reddit.com/r/python/comments/abc123/my_post/")
|
||||
print(result)
|
||||
|
||||
# 批量处理多个 URL
|
||||
result = run("""
|
||||
这里有一些链接:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
https://www.reddit.com/r/MachineLearning/comments/xyz789/new_paper/
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
@@ -91,18 +111,17 @@ deepreader_skill/
|
||||
├── base.py # 抽象基类与 ParseResult 模型
|
||||
├── generic.py # 通用文章/博客解析器
|
||||
├── twitter.py # Twitter/X 解析器(FxTwitter + Nitter)
|
||||
├── reddit.py # Reddit 解析器(.json API)
|
||||
└── youtube.py # YouTube 字幕解析器
|
||||
```
|
||||
|
||||
### Twitter 解析器策略
|
||||
### 解析器选择策略
|
||||
|
||||
```
|
||||
检测到 URL → FxTwitter API(主力)
|
||||
↓ 成功? → ✅ 丰富结果(数据、媒体、文章)
|
||||
↓ 失败?
|
||||
Nitter 实例(备用)
|
||||
↓ 成功? → ✅ 基础结果 + 回复线程
|
||||
↓ 失败? → ❌ 友好的错误信息与诊断
|
||||
检测到 URL → Twitter/X? → FxTwitter API → Nitter 备用
|
||||
→ Reddit? → .json 后缀 API
|
||||
→ YouTube? → youtube-transcript-api
|
||||
→ 其他 → Trafilatura(通用)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -11,9 +11,10 @@ from __future__ import annotations
|
||||
import logging
|
||||
from typing import Sequence
|
||||
|
||||
from ..core.utils import is_twitter_url, is_youtube_url
|
||||
from ..core.utils import is_reddit_url, is_twitter_url, is_youtube_url
|
||||
from ..parsers.base import BaseParser, ParseResult
|
||||
from ..parsers.generic import GenericParser
|
||||
from ..parsers.reddit import RedditParser
|
||||
from ..parsers.twitter import TwitterParser
|
||||
from ..parsers.youtube import YouTubeParser
|
||||
|
||||
@@ -60,6 +61,7 @@ class ParserRouter:
|
||||
self._parsers.extend([
|
||||
TwitterParser(),
|
||||
YouTubeParser(),
|
||||
RedditParser(),
|
||||
])
|
||||
|
||||
# Fallback parser (always matches)
|
||||
|
||||
@@ -132,6 +132,10 @@ def content_hash(text: str) -> str:
|
||||
|
||||
_TWITTER_DOMAINS = {"twitter.com", "x.com", "mobile.twitter.com", "mobile.x.com"}
|
||||
_YOUTUBE_DOMAINS = {"youtube.com", "youtu.be", "www.youtube.com", "m.youtube.com"}
|
||||
_REDDIT_DOMAINS = {
|
||||
"reddit.com", "www.reddit.com", "old.reddit.com",
|
||||
"new.reddit.com", "np.reddit.com", "m.reddit.com",
|
||||
}
|
||||
|
||||
|
||||
def is_twitter_url(url: str) -> bool:
|
||||
@@ -147,6 +151,13 @@ def is_youtube_url(url: str) -> bool:
|
||||
return hostname in _YOUTUBE_DOMAINS
|
||||
|
||||
|
||||
def is_reddit_url(url: str) -> bool:
|
||||
"""Return ``True`` if *url* points to Reddit."""
|
||||
parsed = urlparse(url)
|
||||
hostname = parsed.hostname or ""
|
||||
return hostname in _REDDIT_DOMAINS and "/comments/" in parsed.path
|
||||
|
||||
|
||||
def extract_youtube_video_id(url: str) -> str | None:
|
||||
"""Extract the video ID from a YouTube URL.
|
||||
|
||||
|
||||
@@ -23,8 +23,14 @@
|
||||
"content_extraction",
|
||||
"youtube_transcription",
|
||||
"twitter_reading",
|
||||
"reddit_reading",
|
||||
"markdown_generation"
|
||||
],
|
||||
"memory_path": "../../memory/inbox/",
|
||||
"tags": ["reader", "scraper", "ingestion", "memory"]
|
||||
}
|
||||
"tags": [
|
||||
"reader",
|
||||
"scraper",
|
||||
"ingestion",
|
||||
"memory"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,367 @@
|
||||
"""
|
||||
DeepReader Skill - Reddit Parser
|
||||
==================================
|
||||
Fetches Reddit posts and comments using Reddit's native ``.json``
|
||||
URL suffix — **zero dependencies, zero API keys, zero configuration**.
|
||||
|
||||
How it works
|
||||
------------
|
||||
Reddit natively supports appending ``.json`` to any post URL, returning
|
||||
the full post data + comment tree as structured JSON. For example::
|
||||
|
||||
https://www.reddit.com/r/python/comments/abc123/my_post/
|
||||
→ https://www.reddit.com/r/python/comments/abc123/my_post/.json
|
||||
|
||||
This approach requires:
|
||||
- No API keys or OAuth tokens
|
||||
- No external libraries (uses only ``urllib`` from stdlib)
|
||||
- No registration or app creation
|
||||
|
||||
The only requirement is a descriptive ``User-Agent`` header to avoid
|
||||
Reddit's generic rate-limiting (HTTP 429).
|
||||
|
||||
Content extracted
|
||||
-----------------
|
||||
- Post title, body (selftext), author, score, flair
|
||||
- Subreddit info
|
||||
- Top-level comments sorted by score (configurable limit)
|
||||
- Nested reply threads (configurable depth)
|
||||
- Media URLs (images, videos, galleries)
|
||||
- Crosspost/link detection
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from .base import BaseParser, ParseResult
|
||||
|
||||
logger = logging.getLogger("deepreader.parsers.reddit")
|
||||
|
||||
|
||||
class RedditParser(BaseParser):
|
||||
"""Parse Reddit posts and comments via the native .json API."""
|
||||
|
||||
name = "reddit"
|
||||
timeout = 30
|
||||
|
||||
# Maximum number of top-level comments to include.
|
||||
max_comments: int = 15
|
||||
|
||||
# Maximum reply nesting depth to include.
|
||||
max_reply_depth: int = 3
|
||||
|
||||
# User-Agent for Reddit requests (required to avoid 429).
|
||||
reddit_user_agent: str = "DeepReeder/1.0 (OpenClaw Skill; +https://github.com/astonysh/OpenClaw-DeepReeder)"
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
"""Return ``True`` for reddit.com URLs that look like post links."""
|
||||
from ..core.utils import is_reddit_url
|
||||
return is_reddit_url(url)
|
||||
|
||||
def parse(self, url: str) -> ParseResult:
|
||||
"""Fetch a Reddit post + comments via the .json suffix."""
|
||||
json_url = self._build_json_url(url)
|
||||
if not json_url:
|
||||
return ParseResult.failure(
|
||||
url,
|
||||
"Could not build a valid Reddit JSON URL. "
|
||||
"Expected format: https://www.reddit.com/r/subreddit/comments/id/title/",
|
||||
)
|
||||
|
||||
max_attempts = 2
|
||||
last_error = ""
|
||||
|
||||
for attempt in range(max_attempts):
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
json_url,
|
||||
headers={
|
||||
"User-Agent": self.reddit_user_agent,
|
||||
"Accept": "application/json",
|
||||
},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=self.timeout) as resp:
|
||||
data = json.loads(resp.read().decode())
|
||||
|
||||
return self._build_result(url, data)
|
||||
|
||||
except urllib.error.HTTPError as exc:
|
||||
if exc.code == 429:
|
||||
last_error = "Reddit rate limit hit (HTTP 429). Try again later."
|
||||
if attempt < max_attempts - 1:
|
||||
time.sleep(2)
|
||||
continue
|
||||
elif exc.code == 404:
|
||||
last_error = "Post not found (HTTP 404). It may be deleted or private."
|
||||
elif exc.code == 403:
|
||||
last_error = "Access denied (HTTP 403). The subreddit may be private."
|
||||
else:
|
||||
last_error = f"HTTP {exc.code}: {exc.reason}"
|
||||
return ParseResult.failure(url, last_error)
|
||||
|
||||
except urllib.error.URLError:
|
||||
last_error = "Network error: failed to reach Reddit"
|
||||
if attempt < max_attempts - 1:
|
||||
time.sleep(1)
|
||||
continue
|
||||
|
||||
except Exception as exc: # noqa: BLE001
|
||||
last_error = f"Unexpected error: {exc}"
|
||||
logger.warning("Reddit parser unexpected error: %s", exc)
|
||||
break
|
||||
|
||||
return ParseResult.failure(url, last_error)
|
||||
|
||||
# ==================================================================
|
||||
# Result Builder
|
||||
# ==================================================================
|
||||
|
||||
def _build_result(self, original_url: str, data: list) -> ParseResult:
|
||||
"""Transform Reddit JSON response into a ParseResult."""
|
||||
from ..core.utils import clean_text, generate_excerpt
|
||||
|
||||
if not isinstance(data, list) or len(data) < 1:
|
||||
return ParseResult.failure(
|
||||
original_url, "Unexpected Reddit JSON structure."
|
||||
)
|
||||
|
||||
# --- Post data (first element) ---
|
||||
try:
|
||||
post = data[0]["data"]["children"][0]["data"]
|
||||
except (KeyError, IndexError, TypeError):
|
||||
return ParseResult.failure(
|
||||
original_url, "Could not extract post data from Reddit JSON."
|
||||
)
|
||||
|
||||
title = post.get("title", "Untitled")
|
||||
author = post.get("author", "[deleted]")
|
||||
subreddit = post.get("subreddit_name_prefixed", "r/unknown")
|
||||
score = post.get("score", 0)
|
||||
upvote_ratio = post.get("upvote_ratio", 0)
|
||||
num_comments = post.get("num_comments", 0)
|
||||
created_utc = post.get("created_utc", 0)
|
||||
selftext = post.get("selftext", "")
|
||||
flair = post.get("link_flair_text", "")
|
||||
is_self = post.get("is_self", True)
|
||||
post_url = post.get("url", "")
|
||||
permalink = post.get("permalink", "")
|
||||
|
||||
# Format timestamp
|
||||
timestamp = ""
|
||||
if created_utc:
|
||||
dt = datetime.fromtimestamp(created_utc, tz=timezone.utc)
|
||||
timestamp = dt.strftime("%Y-%m-%d %H:%M UTC")
|
||||
|
||||
# --- Build content ---
|
||||
content_parts: list[str] = []
|
||||
|
||||
# Header
|
||||
content_parts.append(f"# {title}\n")
|
||||
content_parts.append(
|
||||
f"**{subreddit}** · u/{author} · {timestamp}\n"
|
||||
)
|
||||
|
||||
# Stats line
|
||||
stats_line = (
|
||||
f"⬆️ {score:,} ({upvote_ratio:.0%} upvoted) · "
|
||||
f"💬 {num_comments:,} comments"
|
||||
)
|
||||
if flair:
|
||||
stats_line += f" · 🏷️ {flair}"
|
||||
content_parts.append(f"📊 {stats_line}\n")
|
||||
|
||||
content_parts.append("---\n")
|
||||
|
||||
# Post body
|
||||
if selftext:
|
||||
content_parts.append(selftext)
|
||||
elif not is_self and post_url:
|
||||
content_parts.append(f"🔗 **Link post**: [{post_url}]({post_url})")
|
||||
|
||||
# --- Media detection ---
|
||||
media_parts = self._extract_media(post)
|
||||
if media_parts:
|
||||
content_parts.append("\n\n---\n### 🖼️ Media\n")
|
||||
content_parts.extend(media_parts)
|
||||
|
||||
# --- Comments (second element) ---
|
||||
if len(data) >= 2:
|
||||
comments = self._extract_comments(data[1])
|
||||
if comments:
|
||||
content_parts.append("\n\n---\n### 💬 Top Comments\n")
|
||||
content_parts.extend(comments)
|
||||
|
||||
full_content = clean_text("\n".join(content_parts))
|
||||
|
||||
# --- Tags ---
|
||||
tags = ["reddit", subreddit.lower()]
|
||||
if flair:
|
||||
tags.append(flair.lower().replace(" ", "-"))
|
||||
if not is_self:
|
||||
tags.append("link-post")
|
||||
|
||||
return ParseResult(
|
||||
url=original_url,
|
||||
title=f"[{subreddit}] {title}",
|
||||
content=full_content,
|
||||
author=f"u/{author}",
|
||||
excerpt=generate_excerpt(full_content),
|
||||
tags=tags,
|
||||
)
|
||||
|
||||
# ==================================================================
|
||||
# Comment Extraction
|
||||
# ==================================================================
|
||||
|
||||
def _extract_comments(self, comment_listing: dict) -> list[str]:
|
||||
"""Extract top-level comments sorted by score."""
|
||||
try:
|
||||
children = comment_listing["data"]["children"]
|
||||
except (KeyError, TypeError):
|
||||
return []
|
||||
|
||||
# Filter out "more" comment stubs and deleted comments
|
||||
comments = []
|
||||
for child in children:
|
||||
if child.get("kind") != "t1":
|
||||
continue
|
||||
cdata = child.get("data", {})
|
||||
body = cdata.get("body", "")
|
||||
if not body or body == "[deleted]" or body == "[removed]":
|
||||
continue
|
||||
comments.append(cdata)
|
||||
|
||||
# Sort by score (highest first)
|
||||
comments.sort(key=lambda c: c.get("score", 0), reverse=True)
|
||||
|
||||
# Build formatted comment list
|
||||
result: list[str] = []
|
||||
for cdata in comments[: self.max_comments]:
|
||||
author = cdata.get("author", "[deleted]")
|
||||
score = cdata.get("score", 0)
|
||||
body = cdata.get("body", "").strip()
|
||||
|
||||
# Format the comment
|
||||
result.append(f"**u/{author}** (⬆️ {score:,}):\n")
|
||||
# Indent comment body as blockquote
|
||||
quoted_body = "\n".join(f"> {line}" for line in body.split("\n"))
|
||||
result.append(f"{quoted_body}\n")
|
||||
|
||||
# Extract nested replies (limited depth)
|
||||
replies = cdata.get("replies")
|
||||
if replies and isinstance(replies, dict):
|
||||
nested = self._extract_nested_replies(replies, depth=1)
|
||||
if nested:
|
||||
result.extend(nested)
|
||||
|
||||
result.append("") # blank line between comments
|
||||
|
||||
return result
|
||||
|
||||
def _extract_nested_replies(
|
||||
self, reply_listing: dict, depth: int,
|
||||
) -> list[str]:
|
||||
"""Recursively extract nested replies up to max_reply_depth."""
|
||||
if depth >= self.max_reply_depth:
|
||||
return []
|
||||
|
||||
try:
|
||||
children = reply_listing["data"]["children"]
|
||||
except (KeyError, TypeError):
|
||||
return []
|
||||
|
||||
result: list[str] = []
|
||||
indent = " " * depth
|
||||
|
||||
for child in children[:5]: # limit nested replies
|
||||
if child.get("kind") != "t1":
|
||||
continue
|
||||
cdata = child.get("data", {})
|
||||
body = cdata.get("body", "")
|
||||
if not body or body in ("[deleted]", "[removed]"):
|
||||
continue
|
||||
|
||||
author = cdata.get("author", "[deleted]")
|
||||
score = cdata.get("score", 0)
|
||||
|
||||
result.append(f"{indent}↳ **u/{author}** (⬆️ {score:,}):\n")
|
||||
quoted = "\n".join(
|
||||
f"{indent}> {line}" for line in body.strip().split("\n")
|
||||
)
|
||||
result.append(f"{quoted}\n")
|
||||
|
||||
# Recurse deeper
|
||||
sub_replies = cdata.get("replies")
|
||||
if sub_replies and isinstance(sub_replies, dict):
|
||||
nested = self._extract_nested_replies(sub_replies, depth + 1)
|
||||
if nested:
|
||||
result.extend(nested)
|
||||
|
||||
return result
|
||||
|
||||
# ==================================================================
|
||||
# Media Extraction
|
||||
# ==================================================================
|
||||
|
||||
def _extract_media(self, post: dict) -> list[str]:
|
||||
"""Extract media URLs from a Reddit post."""
|
||||
parts: list[str] = []
|
||||
|
||||
# Direct image URL
|
||||
url = post.get("url", "")
|
||||
if any(url.endswith(ext) for ext in (".jpg", ".jpeg", ".png", ".gif", ".webp")):
|
||||
parts.append(f"\n")
|
||||
|
||||
# Reddit gallery
|
||||
if post.get("is_gallery"):
|
||||
media_metadata = post.get("media_metadata", {})
|
||||
for i, (_, media) in enumerate(media_metadata.items(), 1):
|
||||
if media.get("status") == "valid" and media.get("s", {}).get("u"):
|
||||
img_url = media["s"]["u"].replace("&", "&")
|
||||
parts.append(f"\n")
|
||||
|
||||
# Reddit video
|
||||
reddit_video = (post.get("media") or {}).get("reddit_video", {})
|
||||
if reddit_video.get("fallback_url"):
|
||||
parts.append(f"🎬 Video: [{reddit_video['fallback_url']}]({reddit_video['fallback_url']})\n")
|
||||
|
||||
# External video (e.g. YouTube embed)
|
||||
if post.get("is_video") and not reddit_video:
|
||||
parts.append(f"🎬 External video: [{url}]({url})\n")
|
||||
|
||||
return parts
|
||||
|
||||
# ==================================================================
|
||||
# URL Utilities
|
||||
# ==================================================================
|
||||
|
||||
@staticmethod
|
||||
def _build_json_url(url: str) -> str | None:
|
||||
"""Convert a Reddit post URL to its .json API endpoint.
|
||||
|
||||
Examples::
|
||||
https://www.reddit.com/r/python/comments/abc123/my_post/
|
||||
→ https://www.reddit.com/r/python/comments/abc123/my_post/.json
|
||||
|
||||
https://old.reddit.com/r/python/comments/abc123/
|
||||
→ https://www.reddit.com/r/python/comments/abc123/.json
|
||||
"""
|
||||
parsed = urlparse(url)
|
||||
|
||||
# Validate it looks like a Reddit post URL
|
||||
if not parsed.path or "/comments/" not in parsed.path:
|
||||
return None
|
||||
|
||||
# Normalize to www.reddit.com
|
||||
path = parsed.path.rstrip("/")
|
||||
|
||||
# Append .json
|
||||
json_url = f"https://www.reddit.com{path}/.json"
|
||||
return json_url
|
||||
Reference in New Issue
Block a user