mirror of
https://github.com/astonysh/OpenClaw-DeepReeder.git
synced 2026-08-14 00:57:54 +00:00
🐦 Integrate FxTwitter API + multilingual README
Twitter/X Parser: - Primary: FxTwitter API (zero-dep, structured JSON) - Regular tweets with engagement stats - X Articles (long-form content) with full text - Quoted tweets with nested content - Media extraction (images, videos, GIFs) - Fallback: Nitter instances (reply thread extraction) - Credits: inspired by x-tweet-fetcher by ythx-101 README: - Updated with FxTwitter integration details - Added credits section - Added multilingual versions: - 🇨🇳 中文 (README_zh.md) - 🇪🇸 Español (README_es.md) - 🇰🇷 한국어 (README_ko.md) - 🇯🇵 日本語 (README_ja.md) - 🇸🇦 العربية (README_ar.md) - 🇫🇷 Français (README_fr.md)
This commit is contained in:
@@ -4,6 +4,8 @@
|
||||
|
||||
DeepReeder intercepts URLs from user messages, scrapes content intelligently using specialized parsers, formats it into clean Markdown with YAML frontmatter, and saves it to the agent's long-term memory.
|
||||
|
||||
🌍 **Translations**: [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ Features
|
||||
@@ -11,9 +13,25 @@ DeepReeder intercepts URLs from user messages, scrapes content intelligently usi
|
||||
| Parser | Sources | Method |
|
||||
|--------|---------|--------|
|
||||
| 🌐 **Generic** | Blogs, articles, docs | [Trafilatura](https://trafilatura.readthedocs.io/) with BeautifulSoup fallback |
|
||||
| 🐦 **Twitter / X** | Tweets & threads | Nitter instance proxying |
|
||||
| 🐦 **Twitter / X** | Tweets, threads, X Articles | **FxTwitter API** (primary) + Nitter (fallback) |
|
||||
| 🎬 **YouTube** | Video transcripts | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — Deep Integration
|
||||
|
||||
Powered by [FxTwitter](https://github.com/FxEmbed/FxEmbed) API with Nitter fallback. Inspired by [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
|
||||
|
||||
| Content Type | Support |
|
||||
|-------------|---------|
|
||||
| Regular tweets | ✅ Full text + engagement stats |
|
||||
| Long tweets (Twitter Blue) | ✅ Full text |
|
||||
| X Articles (long-form) | ✅ Complete article text + word count |
|
||||
| Quoted tweets | ✅ Nested content included |
|
||||
| Media (images, video, GIF) | ✅ URLs extracted |
|
||||
| Reply threads | ✅ Via Nitter fallback (first 5) |
|
||||
| Engagement stats | ✅ ❤️ likes, 🔁 RTs, 👁️ views, 🔖 bookmarks |
|
||||
|
||||
**No API keys. No login. No rate limits.**
|
||||
|
||||
### Output Format
|
||||
|
||||
Every piece of content is saved as a `.md` file with structured YAML frontmatter:
|
||||
@@ -62,6 +80,10 @@ from deepreader_skill import run
|
||||
result = run("Check out this article: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# Process a tweet (uses FxTwitter API automatically)
|
||||
result = run("Interesting thread: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Process multiple URLs at once
|
||||
result = run("""
|
||||
Here are some links:
|
||||
@@ -79,18 +101,18 @@ print(result)
|
||||
|
||||
✅ How to Build AI Agents
|
||||
Source: https://example.com/article
|
||||
Saved to: memory/inbox/20260216_120000_how-to-build-ai-agents.md
|
||||
Content: 3200 characters
|
||||
Saved to: memory/inbox/2026-02-16_how-to-build-ai-agents.md
|
||||
Content: 3,200 characters
|
||||
|
||||
✅ Tweet by @elonmusk (Mon Feb 16 12:00:00 +0000 2026)
|
||||
Source: https://x.com/elonmusk/status/123456
|
||||
Saved to: memory/inbox/2026-02-16_tweet-by-elonmusk.md
|
||||
Content: 480 characters
|
||||
|
||||
✅ Rick Astley - Never Gonna Give You Up
|
||||
Source: https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
Saved to: memory/inbox/20260216_120001_rick-astley-never-gonna.md
|
||||
Content: 15000 characters
|
||||
|
||||
✅ @user's tweet
|
||||
Source: https://x.com/user/status/123456
|
||||
Saved to: memory/inbox/20260216_120002_user-tweet.md
|
||||
Content: 280 characters
|
||||
Saved to: memory/inbox/2026-02-16_rick-astley-never-gonna.md
|
||||
Content: 15,000 characters
|
||||
```
|
||||
|
||||
---
|
||||
@@ -107,10 +129,21 @@ deepreader_skill/
|
||||
│ ├── storage.py # Markdown file generation & saving
|
||||
│ └── utils.py # URL extraction & helper utilities
|
||||
└── parsers/
|
||||
├── base.py # Abstract base parser & ParseResult model
|
||||
├── generic.py # Generic article/blog parser
|
||||
├── twitter.py # Twitter/X parser (via Nitter)
|
||||
└── youtube.py # YouTube transcript parser
|
||||
├── base.py # Abstract base parser & ParseResult model
|
||||
├── generic.py # Generic article/blog parser (Trafilatura)
|
||||
├── twitter.py # Twitter/X parser (FxTwitter + Nitter)
|
||||
└── youtube.py # YouTube transcript parser
|
||||
```
|
||||
|
||||
### Twitter Parser Strategy
|
||||
|
||||
```
|
||||
URL detected → FxTwitter API (primary)
|
||||
↓ success? → ✅ Rich result (stats, media, articles)
|
||||
↓ failure?
|
||||
Nitter instances (fallback)
|
||||
↓ success? → ✅ Basic result + reply threads
|
||||
↓ failure? → ❌ Graceful error with diagnostics
|
||||
```
|
||||
|
||||
---
|
||||
@@ -126,6 +159,15 @@ DeepReeder uses sensible defaults out of the box. Configuration can be customize
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Credits
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — Public API for fetching Twitter/X content
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — Inspiration for the FxTwitter integration approach
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — Robust web content extraction
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — YouTube transcript fetching
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Feel free to:
|
||||
@@ -144,13 +186,6 @@ This project is licensed under the **MIT License** — see the [LICENSE](LICENSE
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Links
|
||||
|
||||
- **Repository**: [github.com/astonysh/OpenClaw-DeepReeder](https://github.com/astonysh/OpenClaw-DeepReeder)
|
||||
- **Issues**: [github.com/astonysh/OpenClaw-DeepReeder/issues](https://github.com/astonysh/OpenClaw-DeepReeder/issues)
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
Built with 🦞 by <a href="https://github.com/astonysh">OpenClaw</a>
|
||||
</p>
|
||||
|
||||
+170
@@ -0,0 +1,170 @@
|
||||
<div dir="rtl">
|
||||
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
|
||||
> **محرك استيعاب محتوى الويب الذاتي لوكلاء الذكاء الاصطناعي.**
|
||||
|
||||
يعترض DeepReeder عناوين URL من رسائل المستخدم، ويستخرج المحتوى بذكاء باستخدام محللات متخصصة، ويحوله إلى Markdown نظيف مع بيانات YAML الوصفية، ويحفظه في الذاكرة طويلة المدى للوكيل.
|
||||
|
||||
🌍 **الترجمات**: [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ الميزات
|
||||
|
||||
| المحلل | المصادر | الطريقة |
|
||||
|--------|---------|---------|
|
||||
| 🌐 **عام** | مدونات، مقالات، وثائق | [Trafilatura](https://trafilatura.readthedocs.io/) مع BeautifulSoup احتياطي |
|
||||
| 🐦 **Twitter / X** | تغريدات، سلاسل، مقالات X | **FxTwitter API** (رئيسي) + Nitter (احتياطي) |
|
||||
| 🎬 **YouTube** | نصوص الفيديو | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — تكامل عميق
|
||||
|
||||
مدعوم بواجهة برمجة تطبيقات [FxTwitter](https://github.com/FxEmbed/FxEmbed). مستلهم من [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
|
||||
|
||||
| نوع المحتوى | الدعم |
|
||||
|-------------|-------|
|
||||
| التغريدات العادية | ✅ نص كامل + إحصائيات التفاعل |
|
||||
| التغريدات الطويلة (Twitter Blue) | ✅ نص كامل |
|
||||
| مقالات X (محتوى طويل) | ✅ مقال كامل + عدد الكلمات |
|
||||
| التغريدات المقتبسة | ✅ محتوى متداخل مضمّن |
|
||||
| الوسائط (صور، فيديو، GIF) | ✅ استخراج الروابط |
|
||||
| سلاسل الردود | ✅ عبر Nitter الاحتياطي (أول 5) |
|
||||
| إحصائيات التفاعل | ✅ ❤️ إعجابات، 🔁 إعادة تغريد، 👁️ مشاهدات، 🔖 إشارات مرجعية |
|
||||
|
||||
**بدون مفاتيح API. بدون تسجيل دخول. بدون حدود للسرعة.**
|
||||
|
||||
---
|
||||
|
||||
## 📦 التثبيت
|
||||
|
||||
<div dir="ltr">
|
||||
|
||||
```bash
|
||||
# استنساخ المستودع
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# إنشاء بيئة افتراضية
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# تثبيت التبعيات
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 🚀 البداية السريعة
|
||||
|
||||
<div dir="ltr">
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# معالجة عنوان URL واحد
|
||||
result = run("اطلع على هذا المقال: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# معالجة تغريدة (يستخدم FxTwitter API تلقائياً)
|
||||
result = run("سلسلة مثيرة: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# معالجة عناوين URL متعددة دفعة واحدة
|
||||
result = run("""
|
||||
إليك بعض الروابط:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ الهيكل
|
||||
|
||||
<div dir="ltr">
|
||||
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # نقطة الدخول — دالة run()
|
||||
├── manifest.json # بيانات المهارة الوصفية وإعدادات المشغلات
|
||||
├── requirements.txt # قائمة التبعيات
|
||||
├── core/
|
||||
│ ├── router.py # منطق توجيه URL → المحلل
|
||||
│ ├── storage.py # إنشاء وحفظ ملفات Markdown
|
||||
│ └── utils.py # استخراج URL ودوال مساعدة
|
||||
└── parsers/
|
||||
├── base.py # المحلل الأساسي المجرد ونموذج ParseResult
|
||||
├── generic.py # محلل المقالات/المدونات العام
|
||||
├── twitter.py # محلل Twitter/X (FxTwitter + Nitter)
|
||||
└── youtube.py # محلل نصوص YouTube
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
### استراتيجية محلل Twitter
|
||||
|
||||
<div dir="ltr">
|
||||
|
||||
```
|
||||
اكتشاف URL → FxTwitter API (رئيسي)
|
||||
↓ نجاح؟ → ✅ نتيجة غنية (إحصائيات، وسائط، مقالات)
|
||||
↓ فشل؟
|
||||
مثيلات Nitter (احتياطي)
|
||||
↓ نجاح؟ → ✅ نتيجة أساسية + سلاسل الردود
|
||||
↓ فشل؟ → ❌ رسالة خطأ ودية مع التشخيص
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 🔧 الإعدادات
|
||||
|
||||
يعمل DeepReeder مباشرة مع إعدادات افتراضية معقولة. يمكن تخصيص الإعدادات عبر متغيرات البيئة:
|
||||
|
||||
| المتغير | الافتراضي | الوصف |
|
||||
|---------|-----------|-------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | مسار حفظ المحتوى |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | مستوى تفصيل السجلات |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 شكر وتقدير
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — واجهة برمجة تطبيقات عامة لجلب محتوى Twitter/X
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — مصدر إلهام لنهج تكامل FxTwitter
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — استخراج محتوى الويب القوي
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — جلب نصوص YouTube
|
||||
|
||||
---
|
||||
|
||||
## 🤝 المساهمة
|
||||
|
||||
المساهمات مرحب بها! يمكنك:
|
||||
|
||||
1. عمل Fork للمستودع
|
||||
2. إنشاء فرع ميزة (`git checkout -b feature/amazing-parser`)
|
||||
3. التزام بتغييراتك (`git commit -m 'إضافة محلل رائع'`)
|
||||
4. دفع الفرع (`git push origin feature/amazing-parser`)
|
||||
5. فتح Pull Request
|
||||
|
||||
---
|
||||
|
||||
## 📄 الترخيص
|
||||
|
||||
هذا المشروع مرخص بموجب **رخصة MIT** — راجع ملف [LICENSE](LICENSE) للتفاصيل.
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
صنع بـ 🦞 بواسطة <a href="https://github.com/astonysh">OpenClaw</a>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
|
||||
> **Motor autónomo de ingestión de contenido web para agentes de IA.**
|
||||
|
||||
DeepReeder intercepta URLs de los mensajes de usuario, extrae contenido de forma inteligente usando parsers especializados, lo formatea en Markdown limpio con metadatos YAML frontmatter, y lo guarda en la memoria a largo plazo del agente.
|
||||
|
||||
🌍 **Traducciones**: [English](README.md) · [中文](README_zh.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ Características
|
||||
|
||||
| Parser | Fuentes | Método |
|
||||
|--------|---------|--------|
|
||||
| 🌐 **Genérico** | Blogs, artículos, documentación | [Trafilatura](https://trafilatura.readthedocs.io/) con fallback BeautifulSoup |
|
||||
| 🐦 **Twitter / X** | Tweets, hilos, X Articles | **FxTwitter API** (principal) + Nitter (fallback) |
|
||||
| 🎬 **YouTube** | Transcripciones de vídeo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — Integración Profunda
|
||||
|
||||
Impulsado por la API de [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspirado en [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
|
||||
|
||||
| Tipo de Contenido | Soporte |
|
||||
|-------------------|---------|
|
||||
| Tweets regulares | ✅ Texto completo + estadísticas |
|
||||
| Tweets largos (Twitter Blue) | ✅ Texto completo |
|
||||
| X Articles (contenido largo) | ✅ Artículo completo + recuento de palabras |
|
||||
| Tweets citados | ✅ Contenido anidado incluido |
|
||||
| Medios (imágenes, vídeo, GIF) | ✅ URLs extraídas |
|
||||
| Hilos de respuestas | ✅ Vía Nitter fallback (primeras 5) |
|
||||
| Estadísticas de interacción | ✅ ❤️ likes, 🔁 RTs, 👁️ vistas, 🔖 marcadores |
|
||||
|
||||
**Sin claves API. Sin inicio de sesión. Sin límites de velocidad.**
|
||||
|
||||
---
|
||||
|
||||
## 📦 Instalación
|
||||
|
||||
```bash
|
||||
# Clonar el repositorio
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# Crear entorno virtual
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Instalar dependencias
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Inicio Rápido
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# Procesar una sola URL
|
||||
result = run("Mira este artículo: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# Procesar un tweet (usa FxTwitter API automáticamente)
|
||||
result = run("Hilo interesante: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Procesar múltiples URLs a la vez
|
||||
result = run("""
|
||||
Aquí hay algunos enlaces:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Arquitectura
|
||||
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # Punto de entrada — función run()
|
||||
├── manifest.json # Metadatos del skill y configuración de triggers
|
||||
├── requirements.txt # Dependencias
|
||||
├── core/
|
||||
│ ├── router.py # Lógica de enrutamiento URL → Parser
|
||||
│ ├── storage.py # Generación y guardado de archivos Markdown
|
||||
│ └── utils.py # Extracción de URLs y utilidades
|
||||
└── parsers/
|
||||
├── base.py # Parser base abstracto y modelo ParseResult
|
||||
├── generic.py # Parser genérico de artículos/blogs
|
||||
├── twitter.py # Parser Twitter/X (FxTwitter + Nitter)
|
||||
└── youtube.py # Parser de transcripciones de YouTube
|
||||
```
|
||||
|
||||
### Estrategia del Parser de Twitter
|
||||
|
||||
```
|
||||
URL detectada → FxTwitter API (principal)
|
||||
↓ ¿éxito? → ✅ Resultado enriquecido (stats, media, artículos)
|
||||
↓ ¿fallo?
|
||||
Instancias Nitter (fallback)
|
||||
↓ ¿éxito? → ✅ Resultado básico + hilos de respuestas
|
||||
↓ ¿fallo? → ❌ Error descriptivo con diagnóstico
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuración
|
||||
|
||||
DeepReeder funciona listo para usar con valores predeterminados sensatos. Se puede personalizar mediante variables de entorno:
|
||||
|
||||
| Variable | Predeterminado | Descripción |
|
||||
|----------|---------------|-------------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Ruta para guardar contenido |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | Nivel de detalle del registro |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Créditos
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — API pública para obtener contenido de Twitter/X
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — Inspiración para la integración de FxTwitter
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — Extracción robusta de contenido web
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — Obtención de transcripciones de YouTube
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contribuir
|
||||
|
||||
¡Las contribuciones son bienvenidas!
|
||||
|
||||
1. Haz fork del repositorio
|
||||
2. Crea una rama de funcionalidad (`git checkout -b feature/parser-increible`)
|
||||
3. Haz commit de tus cambios (`git commit -m 'Agregar parser increíble'`)
|
||||
4. Haz push a la rama (`git push origin feature/parser-increible`)
|
||||
5. Abre un Pull Request
|
||||
|
||||
---
|
||||
|
||||
## 📄 Licencia
|
||||
|
||||
Este proyecto está licenciado bajo la **Licencia MIT** — consulta el archivo [LICENSE](LICENSE) para más detalles.
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
Construido con 🦞 por <a href="https://github.com/astonysh">OpenClaw</a>
|
||||
</p>
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
|
||||
> **Moteur autonome d'ingestion de contenu web pour agents IA.**
|
||||
|
||||
DeepReeder intercepte les URLs des messages utilisateur, extrait le contenu intelligemment à l'aide de parsers spécialisés, le formate en Markdown propre avec des métadonnées YAML frontmatter, et le sauvegarde dans la mémoire à long terme de l'agent.
|
||||
|
||||
🌍 **Traductions** : [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [العربية](README_ar.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ Fonctionnalités
|
||||
|
||||
| Parser | Sources | Méthode |
|
||||
|--------|---------|---------|
|
||||
| 🌐 **Générique** | Blogs, articles, documentation | [Trafilatura](https://trafilatura.readthedocs.io/) avec fallback BeautifulSoup |
|
||||
| 🐦 **Twitter / X** | Tweets, fils, X Articles | **FxTwitter API** (principal) + Nitter (fallback) |
|
||||
| 🎬 **YouTube** | Transcriptions vidéo | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — Intégration Approfondie
|
||||
|
||||
Propulsé par l'API [FxTwitter](https://github.com/FxEmbed/FxEmbed). Inspiré par [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher).
|
||||
|
||||
| Type de Contenu | Support |
|
||||
|----------------|---------|
|
||||
| Tweets classiques | ✅ Texte complet + statistiques d'engagement |
|
||||
| Tweets longs (Twitter Blue) | ✅ Texte complet |
|
||||
| X Articles (contenu long) | ✅ Article complet + nombre de mots |
|
||||
| Tweets cités | ✅ Contenu imbriqué inclus |
|
||||
| Médias (images, vidéo, GIF) | ✅ URLs extraites |
|
||||
| Fils de réponses | ✅ Via Nitter fallback (5 premières) |
|
||||
| Statistiques d'engagement | ✅ ❤️ likes, 🔁 RTs, 👁️ vues, 🔖 signets |
|
||||
|
||||
**Sans clé API. Sans connexion. Sans limite de débit.**
|
||||
|
||||
---
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
```bash
|
||||
# Cloner le dépôt
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# Créer un environnement virtuel
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Installer les dépendances
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Démarrage Rapide
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# Traiter une seule URL
|
||||
result = run("Regarde cet article : https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# Traiter un tweet (utilise automatiquement l'API FxTwitter)
|
||||
result = run("Fil intéressant : https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# Traiter plusieurs URLs en une fois
|
||||
result = run("""
|
||||
Voici quelques liens :
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # Point d'entrée — fonction run()
|
||||
├── manifest.json # Métadonnées du skill et configuration des triggers
|
||||
├── requirements.txt # Liste des dépendances
|
||||
├── core/
|
||||
│ ├── router.py # Logique de routage URL → Parser
|
||||
│ ├── storage.py # Génération et sauvegarde des fichiers Markdown
|
||||
│ └── utils.py # Extraction d'URLs et fonctions utilitaires
|
||||
└── parsers/
|
||||
├── base.py # Parser de base abstrait et modèle ParseResult
|
||||
├── generic.py # Parser générique d'articles/blogs
|
||||
├── twitter.py # Parser Twitter/X (FxTwitter + Nitter)
|
||||
└── youtube.py # Parser de transcriptions YouTube
|
||||
```
|
||||
|
||||
### Stratégie du Parser Twitter
|
||||
|
||||
```
|
||||
URL détectée → FxTwitter API (principal)
|
||||
↓ succès ? → ✅ Résultat enrichi (stats, médias, articles)
|
||||
↓ échec ?
|
||||
Instances Nitter (fallback)
|
||||
↓ succès ? → ✅ Résultat basique + fils de réponses
|
||||
↓ échec ? → ❌ Message d'erreur explicatif avec diagnostic
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
DeepReeder fonctionne immédiatement avec des valeurs par défaut raisonnables. La configuration peut être personnalisée via des variables d'environnement :
|
||||
|
||||
| Variable | Par défaut | Description |
|
||||
|----------|-----------|-------------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | Chemin de sauvegarde du contenu |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | Niveau de verbosité des journaux |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Remerciements
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — API publique pour récupérer le contenu Twitter/X
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — Inspiration pour l'approche d'intégration FxTwitter
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — Extraction robuste de contenu web
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — Récupération de transcriptions YouTube
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contribuer
|
||||
|
||||
Les contributions sont les bienvenues !
|
||||
|
||||
1. Forkez le dépôt
|
||||
2. Créez une branche de fonctionnalité (`git checkout -b feature/parser-genial`)
|
||||
3. Commitez vos changements (`git commit -m 'Ajouter un parser génial'`)
|
||||
4. Poussez la branche (`git push origin feature/parser-genial`)
|
||||
5. Ouvrez une Pull Request
|
||||
|
||||
---
|
||||
|
||||
## 📄 Licence
|
||||
|
||||
Ce projet est sous licence **MIT** — consultez le fichier [LICENSE](LICENSE) pour plus de détails.
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
Construit avec 🦞 par <a href="https://github.com/astonysh">OpenClaw</a>
|
||||
</p>
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
|
||||
> **AIエージェント向け自律型Webコンテンツ取り込みエンジン。**
|
||||
|
||||
DeepReederはユーザーメッセージからURLを自動検出し、専用パーサーを使ってコンテンツをインテリジェントにスクレイピングし、YAMLフロントマター付きのクリーンなMarkdownに変換して、エージェントの長期メモリに保存します。
|
||||
|
||||
🌍 **翻訳**: [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [한국어](README_ko.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ 機能
|
||||
|
||||
| パーサー | ソース | 方法 |
|
||||
|---------|--------|------|
|
||||
| 🌐 **汎用** | ブログ、記事、ドキュメント | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup フォールバック |
|
||||
| 🐦 **Twitter / X** | ツイート、スレッド、Xアーティクル | **FxTwitter API**(メイン)+ Nitter(フォールバック) |
|
||||
| 🎬 **YouTube** | 動画字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — ディープインテグレーション
|
||||
|
||||
[FxTwitter](https://github.com/FxEmbed/FxEmbed) APIベース。[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)にインスパイアされました。
|
||||
|
||||
| コンテンツタイプ | サポート |
|
||||
|----------------|---------|
|
||||
| 通常のツイート | ✅ 全文 + エンゲージメント統計 |
|
||||
| 長文ツイート(Twitter Blue) | ✅ 全文 |
|
||||
| Xアーティクル(長文コンテンツ) | ✅ 完全な記事 + 単語数 |
|
||||
| 引用ツイート | ✅ ネストされたコンテンツ含む |
|
||||
| メディア(画像、動画、GIF) | ✅ URL抽出 |
|
||||
| リプライスレッド | ✅ Nitterフォールバック経由(最初の5件) |
|
||||
| エンゲージメント統計 | ✅ ❤️ いいね、🔁 RT、👁️ 閲覧、🔖 ブックマーク |
|
||||
|
||||
**APIキー不要。ログイン不要。レート制限なし。**
|
||||
|
||||
---
|
||||
|
||||
## 📦 インストール
|
||||
|
||||
```bash
|
||||
# リポジトリをクローン
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# 仮想環境を作成
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# 依存関係をインストール
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 クイックスタート
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# 単一URLを処理
|
||||
result = run("この記事をチェック: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# ツイートを処理(自動的にFxTwitter APIを使用)
|
||||
result = run("興味深いスレッド: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# 複数のURLを一括処理
|
||||
result = run("""
|
||||
いくつかのリンクがあります:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ アーキテクチャ
|
||||
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # エントリポイント — run() 関数
|
||||
├── manifest.json # スキルメタデータとトリガー設定
|
||||
├── requirements.txt # 依存関係リスト
|
||||
├── core/
|
||||
│ ├── router.py # URL → パーサールーティングロジック
|
||||
│ ├── storage.py # Markdownファイル生成・保存
|
||||
│ └── utils.py # URL抽出とユーティリティ関数
|
||||
└── parsers/
|
||||
├── base.py # 抽象基底パーサーとParseResultモデル
|
||||
├── generic.py # 汎用記事/ブログパーサー
|
||||
├── twitter.py # Twitter/Xパーサー(FxTwitter + Nitter)
|
||||
└── youtube.py # YouTube字幕パーサー
|
||||
```
|
||||
|
||||
### Twitterパーサー戦略
|
||||
|
||||
```
|
||||
URL検出 → FxTwitter API(メイン)
|
||||
↓ 成功? → ✅ リッチな結果(統計、メディア、記事)
|
||||
↓ 失敗?
|
||||
Nitterインスタンス(フォールバック)
|
||||
↓ 成功? → ✅ 基本結果 + リプライスレッド
|
||||
↓ 失敗? → ❌ 診断付きエラーメッセージ
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 設定
|
||||
|
||||
DeepReederはデフォルト設定ですぐに使えます。環境変数でカスタマイズ可能です:
|
||||
|
||||
| 変数 | デフォルト | 説明 |
|
||||
|------|-----------|------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | コンテンツの保存先 |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | ログの詳細レベル |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 クレジット
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — Twitter/Xコンテンツ取得用パブリックAPI
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — FxTwitter統合アプローチのインスピレーション
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — 高性能Webコンテンツ抽出
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — YouTube字幕取得
|
||||
|
||||
---
|
||||
|
||||
## 🤝 コントリビューション
|
||||
|
||||
コントリビューション歓迎です!
|
||||
|
||||
1. リポジトリをForkします
|
||||
2. フィーチャーブランチを作成 (`git checkout -b feature/amazing-parser`)
|
||||
3. 変更をコミット (`git commit -m '素晴らしいパーサーを追加'`)
|
||||
4. ブランチをプッシュ (`git push origin feature/amazing-parser`)
|
||||
5. Pull Requestを開きます
|
||||
|
||||
---
|
||||
|
||||
## 📄 ライセンス
|
||||
|
||||
このプロジェクトは**MITライセンス**の下でライセンスされています — 詳細は[LICENSE](LICENSE)ファイルをご覧ください。
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/astonysh">OpenClaw</a>が🦞で構築
|
||||
</p>
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
|
||||
> **AI 에이전트를 위한 자율 웹 콘텐츠 수집 엔진.**
|
||||
|
||||
DeepReeder는 사용자 메시지에서 URL을 자동으로 감지하고, 전문 파서를 사용하여 콘텐츠를 지능적으로 스크래핑하며, YAML 프론트매터가 포함된 깔끔한 Markdown으로 변환하여 에이전트의 장기 메모리에 저장합니다.
|
||||
|
||||
🌍 **번역**: [English](README.md) · [中文](README_zh.md) · [Español](README_es.md) · [日本語](README_ja.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ 기능
|
||||
|
||||
| 파서 | 소스 | 방법 |
|
||||
|------|------|------|
|
||||
| 🌐 **범용** | 블로그, 기사, 문서 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup 대체 |
|
||||
| 🐦 **Twitter / X** | 트윗, 스레드, X 아티클 | **FxTwitter API** (주력) + Nitter (대체) |
|
||||
| 🎬 **YouTube** | 동영상 자막 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — 심층 통합
|
||||
|
||||
[FxTwitter](https://github.com/FxEmbed/FxEmbed) API 기반. [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)에서 영감을 받았습니다.
|
||||
|
||||
| 콘텐츠 유형 | 지원 |
|
||||
|------------|------|
|
||||
| 일반 트윗 | ✅ 전체 텍스트 + 참여 통계 |
|
||||
| 긴 트윗 (Twitter Blue) | ✅ 전체 텍스트 |
|
||||
| X 아티클 (장문) | ✅ 전체 기사 + 단어 수 |
|
||||
| 인용 트윗 | ✅ 중첩 콘텐츠 포함 |
|
||||
| 미디어 (이미지, 동영상, GIF) | ✅ URL 추출 |
|
||||
| 답글 스레드 | ✅ Nitter 대체를 통해 (처음 5개) |
|
||||
| 참여 통계 | ✅ ❤️ 좋아요, 🔁 리트윗, 👁️ 조회, 🔖 북마크 |
|
||||
|
||||
**API 키 불필요. 로그인 불필요. 속도 제한 없음.**
|
||||
|
||||
---
|
||||
|
||||
## 📦 설치
|
||||
|
||||
```bash
|
||||
# 저장소 클론
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# 가상 환경 생성
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# 의존성 설치
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 빠른 시작
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# 단일 URL 처리
|
||||
result = run("이 기사를 확인하세요: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# 트윗 처리 (자동으로 FxTwitter API 사용)
|
||||
result = run("흥미로운 스레드: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# 여러 URL 한번에 처리
|
||||
result = run("""
|
||||
여기 몇 가지 링크가 있습니다:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 아키텍처
|
||||
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # 진입점 — run() 함수
|
||||
├── manifest.json # 스킬 메타데이터 및 트리거 설정
|
||||
├── requirements.txt # 의존성 목록
|
||||
├── core/
|
||||
│ ├── router.py # URL → 파서 라우팅 로직
|
||||
│ ├── storage.py # Markdown 파일 생성 및 저장
|
||||
│ └── utils.py # URL 추출 및 유틸리티 함수
|
||||
└── parsers/
|
||||
├── base.py # 추상 기본 파서 및 ParseResult 모델
|
||||
├── generic.py # 범용 기사/블로그 파서
|
||||
├── twitter.py # Twitter/X 파서 (FxTwitter + Nitter)
|
||||
└── youtube.py # YouTube 자막 파서
|
||||
```
|
||||
|
||||
### Twitter 파서 전략
|
||||
|
||||
```
|
||||
URL 감지 → FxTwitter API (주력)
|
||||
↓ 성공? → ✅ 풍부한 결과 (통계, 미디어, 기사)
|
||||
↓ 실패?
|
||||
Nitter 인스턴스 (대체)
|
||||
↓ 성공? → ✅ 기본 결과 + 답글 스레드
|
||||
↓ 실패? → ❌ 친절한 오류 메시지 및 진단
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 설정
|
||||
|
||||
DeepReeder는 합리적인 기본값으로 바로 사용할 수 있습니다. 환경 변수로 설정을 변경할 수 있습니다:
|
||||
|
||||
| 변수 | 기본값 | 설명 |
|
||||
|------|--------|------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | 콘텐츠 저장 경로 |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | 로깅 상세 수준 |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 크레딧
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — Twitter/X 콘텐츠 가져오기용 공개 API
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — FxTwitter 통합 접근 방식에 영감
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — 강력한 웹 콘텐츠 추출
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — YouTube 자막 가져오기
|
||||
|
||||
---
|
||||
|
||||
## 🤝 기여
|
||||
|
||||
기여를 환영합니다!
|
||||
|
||||
1. 저장소를 Fork 합니다
|
||||
2. 기능 브랜치를 생성합니다 (`git checkout -b feature/amazing-parser`)
|
||||
3. 변경 사항을 커밋합니다 (`git commit -m '놀라운 파서 추가'`)
|
||||
4. 브랜치를 푸시합니다 (`git push origin feature/amazing-parser`)
|
||||
5. Pull Request를 엽니다
|
||||
|
||||
---
|
||||
|
||||
## 📄 라이선스
|
||||
|
||||
이 프로젝트는 **MIT 라이선스**에 따라 라이선스가 부여됩니다 — 자세한 내용은 [LICENSE](LICENSE) 파일을 참조하세요.
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/astonysh">OpenClaw</a>에서 🦞 로 만들었습니다
|
||||
</p>
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
# 🦞 OpenClaw DeepReeder
|
||||
|
||||
> **面向 AI 智能体的自主网页内容摄取引擎。**
|
||||
|
||||
DeepReeder 自动拦截用户消息中的 URL,使用专用解析器智能抓取内容,将其格式化为带有 YAML 前置信息的干净 Markdown,并保存到智能体的长期记忆中。
|
||||
|
||||
🌍 **其他语言**: [English](README.md) · [Español](README_es.md) · [한국어](README_ko.md) · [日本語](README_ja.md) · [العربية](README_ar.md) · [Français](README_fr.md)
|
||||
|
||||
---
|
||||
|
||||
## ✨ 功能特性
|
||||
|
||||
| 解析器 | 来源 | 方法 |
|
||||
|--------|------|------|
|
||||
| 🌐 **通用** | 博客、文章、文档 | [Trafilatura](https://trafilatura.readthedocs.io/) + BeautifulSoup 备用方案 |
|
||||
| 🐦 **Twitter / X** | 推文、线程、X 文章 | **FxTwitter API**(主力) + Nitter(备用) |
|
||||
| 🎬 **YouTube** | 视频字幕 | [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api) |
|
||||
|
||||
### 🐦 Twitter / X — 深度整合
|
||||
|
||||
基于 [FxTwitter](https://github.com/FxEmbed/FxEmbed) API,灵感来自 [x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)。
|
||||
|
||||
| 内容类型 | 支持 |
|
||||
|---------|------|
|
||||
| 普通推文 | ✅ 全文 + 互动数据 |
|
||||
| 长推文(Twitter Blue) | ✅ 完整文本 |
|
||||
| X 文章(长文) | ✅ 完整文章 + 字数统计 |
|
||||
| 引用推文 | ✅ 嵌套内容 |
|
||||
| 媒体(图片、视频、GIF) | ✅ URL 提取 |
|
||||
| 回复线程 | ✅ 通过 Nitter 备用方案(前5条) |
|
||||
| 互动数据 | ✅ ❤️ 喜欢、🔁 转发、👁️ 浏览、🔖 书签 |
|
||||
|
||||
**无需 API 密钥。无需登录。无速率限制。**
|
||||
|
||||
---
|
||||
|
||||
## 📦 安装
|
||||
|
||||
```bash
|
||||
# 克隆仓库
|
||||
git clone https://github.com/astonysh/OpenClaw-DeepReeder.git
|
||||
cd OpenClaw-DeepReeder
|
||||
|
||||
# 创建虚拟环境
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# 安装依赖
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
```python
|
||||
from deepreader_skill import run
|
||||
|
||||
# 处理单个 URL
|
||||
result = run("看看这篇文章: https://example.com/blog/post")
|
||||
print(result)
|
||||
|
||||
# 处理推文(自动使用 FxTwitter API)
|
||||
result = run("有趣的推文: https://x.com/elonmusk/status/123456")
|
||||
print(result)
|
||||
|
||||
# 批量处理多个 URL
|
||||
result = run("""
|
||||
这里有一些链接:
|
||||
https://example.com/article
|
||||
https://youtube.com/watch?v=dQw4w9WgXcQ
|
||||
https://x.com/user/status/123456
|
||||
""")
|
||||
print(result)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 架构
|
||||
|
||||
```
|
||||
deepreader_skill/
|
||||
├── __init__.py # 入口 — run() 函数
|
||||
├── manifest.json # 技能元数据与触发配置
|
||||
├── requirements.txt # 依赖列表
|
||||
├── core/
|
||||
│ ├── router.py # URL → 解析器路由逻辑
|
||||
│ ├── storage.py # Markdown 文件生成与保存
|
||||
│ └── utils.py # URL 提取和工具函数
|
||||
└── parsers/
|
||||
├── base.py # 抽象基类与 ParseResult 模型
|
||||
├── generic.py # 通用文章/博客解析器
|
||||
├── twitter.py # Twitter/X 解析器(FxTwitter + Nitter)
|
||||
└── youtube.py # YouTube 字幕解析器
|
||||
```
|
||||
|
||||
### Twitter 解析器策略
|
||||
|
||||
```
|
||||
检测到 URL → FxTwitter API(主力)
|
||||
↓ 成功? → ✅ 丰富结果(数据、媒体、文章)
|
||||
↓ 失败?
|
||||
Nitter 实例(备用)
|
||||
↓ 成功? → ✅ 基础结果 + 回复线程
|
||||
↓ 失败? → ❌ 友好的错误信息与诊断
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 配置
|
||||
|
||||
DeepReeder 开箱即用,使用合理的默认值。可通过环境变量自定义配置:
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `DEEPREEDER_MEMORY_PATH` | `../../memory/inbox/` | 保存内容的路径 |
|
||||
| `DEEPREEDER_LOG_LEVEL` | `INFO` | 日志级别 |
|
||||
|
||||
---
|
||||
|
||||
## 🙏 致谢
|
||||
|
||||
- **[FxTwitter / FixTweet](https://github.com/FxEmbed/FxEmbed)** — 获取 Twitter/X 内容的公共 API
|
||||
- **[x-tweet-fetcher](https://github.com/ythx-101/x-tweet-fetcher)** — FxTwitter 整合方案的灵感来源
|
||||
- **[Trafilatura](https://trafilatura.readthedocs.io/)** — 强大的网页内容提取工具
|
||||
- **[youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api)** — YouTube 字幕获取
|
||||
|
||||
---
|
||||
|
||||
## 🤝 贡献
|
||||
|
||||
欢迎贡献!您可以:
|
||||
|
||||
1. Fork 仓库
|
||||
2. 创建功能分支 (`git checkout -b feature/amazing-parser`)
|
||||
3. 提交更改 (`git commit -m '添加新解析器'`)
|
||||
4. 推送分支 (`git push origin feature/amazing-parser`)
|
||||
5. 提交 Pull Request
|
||||
|
||||
---
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
本项目基于 **MIT 许可证** — 查看 [LICENSE](LICENSE) 文件获取详情。
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
由 <a href="https://github.com/astonysh">OpenClaw</a> 用 🦞 构建
|
||||
</p>
|
||||
+258
-126
@@ -1,32 +1,42 @@
|
||||
"""
|
||||
DeepReader Skill - Twitter / X Parser
|
||||
=======================================
|
||||
Strategy-pattern implementation for reading tweets:
|
||||
Hybrid strategy for reading tweets:
|
||||
|
||||
1. **Primary**: Rotate through public Nitter instances to fetch tweet
|
||||
content without any API keys.
|
||||
2. **Fallback**: Gracefully degrade with informative guidance on how to
|
||||
plug in a scraping service (ZenRows, ScrapingBee) or browser cookies.
|
||||
1. **Primary**: FxTwitter public API — zero dependencies, reliable,
|
||||
supports regular tweets, long tweets, X Articles, quoted tweets,
|
||||
media, and engagement stats.
|
||||
2. **Fallback**: Rotate through public Nitter instances to fetch tweet
|
||||
content (especially useful for fetching reply threads).
|
||||
|
||||
Why Nitter?
|
||||
-----------
|
||||
Twitter's official API is paywalled and rate-limited. Nitter is an
|
||||
open-source alternative frontend that serves tweets as plain HTML,
|
||||
making extraction trivial. However, public instances may go down,
|
||||
so we rotate through several and retry.
|
||||
Why FxTwitter first?
|
||||
---------------------
|
||||
FxTwitter (api.fxtwitter.com) is a public, maintenance-free API that
|
||||
returns structured JSON. It is far more reliable than Nitter instances
|
||||
which frequently go offline. It also returns rich metadata (stats,
|
||||
media, articles) that Nitter does not expose.
|
||||
|
||||
Extending with a paid scraping service
|
||||
---------------------------------------
|
||||
If all Nitter instances fail, you can integrate a proxy/rendering
|
||||
service. See the ``_fallback_scrape`` method for detailed guidance
|
||||
on where to plug in ZenRows or browser cookies.
|
||||
Nitter as fallback
|
||||
-------------------
|
||||
Nitter HTML scraping is retained as a secondary strategy, primarily
|
||||
because it can extract reply threads (first N replies), which FxTwitter
|
||||
does not provide.
|
||||
|
||||
Credits
|
||||
--------
|
||||
FxTwitter integration inspired by `x-tweet-fetcher` by ythx-101:
|
||||
https://github.com/ythx-101/x-tweet-fetcher
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import random
|
||||
import re
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import requests
|
||||
@@ -39,7 +49,7 @@ logger = logging.getLogger("deepreader.parsers.twitter")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Known public Nitter instances (community-maintained)
|
||||
# Update this list periodically – instances come and go.
|
||||
# Used only as fallback for reply-thread extraction.
|
||||
# ---------------------------------------------------------------------------
|
||||
NITTER_INSTANCES: list[str] = [
|
||||
"https://nitter.privacydev.net",
|
||||
@@ -54,13 +64,17 @@ NITTER_INSTANCES: list[str] = [
|
||||
|
||||
|
||||
class TwitterParser(BaseParser):
|
||||
"""Parse tweets from Twitter / X via Nitter relay instances."""
|
||||
"""Parse tweets from Twitter / X.
|
||||
|
||||
Primary: FxTwitter API (structured JSON, zero deps, rich metadata).
|
||||
Fallback: Nitter HTML scraping (reply threads).
|
||||
"""
|
||||
|
||||
name = "twitter"
|
||||
timeout = 20
|
||||
timeout = 30
|
||||
|
||||
# Maximum number of Nitter instances to try before giving up.
|
||||
max_retries: int = 4
|
||||
# Maximum Nitter instances to try for reply extraction.
|
||||
max_nitter_retries: int = 3
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
"""Return ``True`` for twitter.com / x.com URLs."""
|
||||
@@ -68,49 +82,233 @@ class TwitterParser(BaseParser):
|
||||
return is_twitter_url(url)
|
||||
|
||||
def parse(self, url: str) -> ParseResult:
|
||||
"""Attempt to read a tweet via Nitter, with graceful fallbacks."""
|
||||
tweet_path = self._extract_tweet_path(url)
|
||||
if not tweet_path:
|
||||
"""Attempt to read a tweet — FxTwitter first, Nitter fallback."""
|
||||
tweet_info = self._extract_tweet_info(url)
|
||||
if not tweet_info:
|
||||
return ParseResult.failure(
|
||||
url,
|
||||
"Could not extract a valid tweet path from this URL. "
|
||||
"Expected format: https://twitter.com/user/status/123456",
|
||||
)
|
||||
|
||||
# Shuffle instances to spread load and improve resilience.
|
||||
username, tweet_id = tweet_info
|
||||
|
||||
# ----- Strategy 1: FxTwitter API (primary) -----
|
||||
result = self._parse_fxtwitter(url, username, tweet_id)
|
||||
if result.success:
|
||||
return result
|
||||
|
||||
logger.warning(
|
||||
"FxTwitter failed for %s, trying Nitter fallback: %s",
|
||||
url, result.error,
|
||||
)
|
||||
|
||||
# ----- Strategy 2: Nitter HTML scraping (fallback) -----
|
||||
tweet_path = f"{username}/status/{tweet_id}"
|
||||
nitter_result = self._parse_nitter_fallback(url, tweet_path)
|
||||
if nitter_result.success:
|
||||
return nitter_result
|
||||
|
||||
# ----- Both strategies failed -----
|
||||
return ParseResult.failure(
|
||||
url,
|
||||
f"⚠️ All strategies failed for this tweet.\n"
|
||||
f"FxTwitter error: {result.error}\n"
|
||||
f"Nitter error: {nitter_result.error}\n\n"
|
||||
f"The tweet URL was: {url}\n"
|
||||
f"This may be a deleted/private tweet, or a temporary service outage.",
|
||||
)
|
||||
|
||||
# ==================================================================
|
||||
# FxTwitter API — Primary Strategy
|
||||
# ==================================================================
|
||||
|
||||
def _parse_fxtwitter(
|
||||
self, original_url: str, username: str, tweet_id: str,
|
||||
) -> ParseResult:
|
||||
"""Fetch tweet via the FxTwitter public JSON API."""
|
||||
api_url = f"https://api.fxtwitter.com/{username}/status/{tweet_id}"
|
||||
|
||||
max_attempts = 2
|
||||
last_error = ""
|
||||
|
||||
for attempt in range(max_attempts):
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
api_url,
|
||||
headers={"User-Agent": "Mozilla/5.0"},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=self.timeout) as resp:
|
||||
data = json.loads(resp.read().decode())
|
||||
|
||||
if data.get("code") != 200:
|
||||
last_error = (
|
||||
f"FxTwitter returned code {data.get('code')}: "
|
||||
f"{data.get('message', 'Unknown')}"
|
||||
)
|
||||
return ParseResult.failure(original_url, last_error)
|
||||
|
||||
return self._build_result_from_fxtwitter(original_url, data)
|
||||
|
||||
except urllib.error.HTTPError as exc:
|
||||
last_error = f"HTTP {exc.code}: {exc.reason}"
|
||||
return ParseResult.failure(original_url, last_error)
|
||||
|
||||
except urllib.error.URLError:
|
||||
last_error = "Network error: failed to reach FxTwitter API"
|
||||
if attempt < max_attempts - 1:
|
||||
time.sleep(1)
|
||||
continue
|
||||
|
||||
except Exception as exc: # noqa: BLE001
|
||||
last_error = f"Unexpected error: {exc}"
|
||||
logger.warning("FxTwitter unexpected error: %s", exc)
|
||||
break
|
||||
|
||||
return ParseResult.failure(original_url, last_error)
|
||||
|
||||
def _build_result_from_fxtwitter(
|
||||
self, original_url: str, data: dict,
|
||||
) -> ParseResult:
|
||||
"""Transform FxTwitter JSON response into a ParseResult."""
|
||||
from ..core.utils import clean_text, generate_excerpt
|
||||
|
||||
tweet = data["tweet"]
|
||||
author = tweet.get("author", {}).get("name", "")
|
||||
screen_name = tweet.get("author", {}).get("screen_name", "")
|
||||
created_at = tweet.get("created_at", "")
|
||||
tweet_text = tweet.get("text", "")
|
||||
is_article = bool(tweet.get("article"))
|
||||
is_note = tweet.get("is_note_tweet", False)
|
||||
|
||||
# --- Engagement stats ---
|
||||
stats_line = (
|
||||
f"❤️ {tweet.get('likes', 0):,} "
|
||||
f"🔁 {tweet.get('retweets', 0):,} "
|
||||
f"🔖 {tweet.get('bookmarks', 0):,} "
|
||||
f"👁️ {tweet.get('views', 0):,} "
|
||||
f"💬 {tweet.get('replies', 0):,}"
|
||||
)
|
||||
|
||||
content_parts: list[str] = []
|
||||
|
||||
# --- X Article (long-form content) ---
|
||||
if is_article:
|
||||
article = tweet["article"]
|
||||
article_title = article.get("title", "")
|
||||
article_blocks = article.get("content", {}).get("blocks", [])
|
||||
article_text = "\n\n".join(
|
||||
b.get("text", "") for b in article_blocks if b.get("text", "")
|
||||
)
|
||||
word_count = len(article_text.split())
|
||||
|
||||
title = f"📝 {article_title}" if article_title else f"X Article by @{screen_name}"
|
||||
|
||||
content_parts.append(f"# {article_title}\n")
|
||||
content_parts.append(f"**By @{screen_name}** ({author}) · {created_at}\n")
|
||||
content_parts.append(f"📊 {stats_line}\n")
|
||||
content_parts.append(f"📐 {word_count:,} words\n")
|
||||
content_parts.append("---\n")
|
||||
content_parts.append(article_text)
|
||||
|
||||
else:
|
||||
# --- Regular tweet / note tweet ---
|
||||
title = f"Tweet by @{screen_name}"
|
||||
if created_at:
|
||||
title += f" ({created_at})"
|
||||
|
||||
content_parts.append(f"**@{screen_name}** ({author})\n")
|
||||
content_parts.append(f"🕐 {created_at}\n")
|
||||
content_parts.append(f"📊 {stats_line}\n")
|
||||
content_parts.append("---\n")
|
||||
content_parts.append(tweet_text)
|
||||
|
||||
if is_note:
|
||||
content_parts.append("\n\n> 📝 *This is a Note Tweet (long-form)*")
|
||||
|
||||
# --- Quoted tweet ---
|
||||
quote = tweet.get("quote")
|
||||
if quote:
|
||||
qt_author = quote.get("author", {}).get("screen_name", "unknown")
|
||||
qt_text = quote.get("text", "")
|
||||
content_parts.append("\n\n---\n### 🔁 Quoted Tweet\n")
|
||||
content_parts.append(f"> **@{qt_author}**: {qt_text}\n")
|
||||
qt_stats = (
|
||||
f"> ❤️ {quote.get('likes', 0):,} "
|
||||
f"🔁 {quote.get('retweets', 0):,} "
|
||||
f"👁️ {quote.get('views', 0):,}"
|
||||
)
|
||||
content_parts.append(qt_stats)
|
||||
|
||||
# --- Media ---
|
||||
media = tweet.get("media", {})
|
||||
all_media = media.get("all", [])
|
||||
if all_media:
|
||||
content_parts.append("\n\n---\n### 🖼️ Media\n")
|
||||
for i, item in enumerate(all_media, 1):
|
||||
media_type = item.get("type", "unknown")
|
||||
media_url = item.get("url", "")
|
||||
if media_type == "photo":
|
||||
content_parts.append(f"\n")
|
||||
elif media_type == "video":
|
||||
content_parts.append(f"🎬 Video: [{media_url}]({media_url})\n")
|
||||
elif media_type == "gif":
|
||||
content_parts.append(f"🎞️ GIF: [{media_url}]({media_url})\n")
|
||||
|
||||
full_content = clean_text("\n".join(content_parts))
|
||||
|
||||
tags = ["twitter"]
|
||||
if is_article:
|
||||
tags.append("x-article")
|
||||
if is_note:
|
||||
tags.append("note-tweet")
|
||||
if quote:
|
||||
tags.append("quote-tweet")
|
||||
if all_media:
|
||||
tags.append("has-media")
|
||||
|
||||
return ParseResult(
|
||||
url=original_url,
|
||||
title=title,
|
||||
content=full_content,
|
||||
author=f"@{screen_name}" if screen_name else author,
|
||||
excerpt=generate_excerpt(full_content),
|
||||
tags=tags,
|
||||
)
|
||||
|
||||
# ==================================================================
|
||||
# Nitter HTML Scraping — Fallback Strategy
|
||||
# ==================================================================
|
||||
|
||||
def _parse_nitter_fallback(self, original_url: str, tweet_path: str) -> ParseResult:
|
||||
"""Try Nitter instances as a fallback (useful for reply threads)."""
|
||||
instances = random.sample(
|
||||
NITTER_INSTANCES,
|
||||
min(self.max_retries, len(NITTER_INSTANCES)),
|
||||
min(self.max_nitter_retries, len(NITTER_INSTANCES)),
|
||||
)
|
||||
|
||||
last_error = ""
|
||||
for instance in instances:
|
||||
nitter_url = f"{instance}/{tweet_path}"
|
||||
logger.info("Trying Nitter instance: %s", nitter_url)
|
||||
logger.info("Trying Nitter fallback: %s", nitter_url)
|
||||
try:
|
||||
result = self._parse_nitter(url, nitter_url)
|
||||
result = self._parse_nitter_page(original_url, nitter_url)
|
||||
if result.success:
|
||||
return result
|
||||
last_error = result.error
|
||||
except requests.RequestException as exc:
|
||||
last_error = str(exc)
|
||||
logger.warning("Nitter instance %s failed: %s", instance, exc)
|
||||
continue
|
||||
logger.warning("Nitter %s failed: %s", instance, exc)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
last_error = str(exc)
|
||||
logger.warning("Unexpected error with %s: %s", instance, exc)
|
||||
continue
|
||||
logger.warning("Nitter unexpected error: %s", exc)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# All Nitter instances failed → fallback
|
||||
# ------------------------------------------------------------------
|
||||
return self._fallback_scrape(url, last_error)
|
||||
return ParseResult.failure(
|
||||
original_url,
|
||||
f"All Nitter instances failed. Last error: {last_error}",
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Nitter HTML Parsing
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _parse_nitter(self, original_url: str, nitter_url: str) -> ParseResult:
|
||||
def _parse_nitter_page(self, original_url: str, nitter_url: str) -> ParseResult:
|
||||
"""Fetch and parse a single Nitter page."""
|
||||
resp = requests.get(
|
||||
nitter_url,
|
||||
@@ -120,40 +318,36 @@ class TwitterParser(BaseParser):
|
||||
resp.raise_for_status()
|
||||
soup = BeautifulSoup(resp.text, "lxml")
|
||||
|
||||
# --- Tweet body ---
|
||||
tweet_div = soup.find("div", class_="tweet-content") or soup.find(
|
||||
"div", class_="main-tweet"
|
||||
)
|
||||
if not tweet_div:
|
||||
return ParseResult.failure(
|
||||
original_url,
|
||||
f"Nitter page loaded but no tweet content found at {nitter_url}",
|
||||
f"Nitter page loaded but no content found at {nitter_url}",
|
||||
)
|
||||
|
||||
tweet_text = tweet_div.get_text(separator="\n", strip=True)
|
||||
|
||||
# --- Author ---
|
||||
author_tag = soup.find("a", class_="fullname") or soup.find(
|
||||
"span", class_="username"
|
||||
)
|
||||
author = author_tag.get_text(strip=True) if author_tag else ""
|
||||
|
||||
# --- Timestamp ---
|
||||
date_tag = soup.find("span", class_="tweet-date")
|
||||
timestamp = ""
|
||||
if date_tag:
|
||||
a_tag = date_tag.find("a")
|
||||
timestamp = a_tag.get("title", "") if a_tag else date_tag.get_text(strip=True)
|
||||
|
||||
# Build a nice title
|
||||
title = f"Tweet by {author}" if author else "Tweet"
|
||||
if timestamp:
|
||||
title += f" ({timestamp})"
|
||||
|
||||
# Collect reply context if present
|
||||
# Collect reply context
|
||||
replies: list[str] = []
|
||||
reply_divs = soup.find_all("div", class_="reply")
|
||||
for rd in reply_divs[:5]: # limit to first 5 replies
|
||||
for rd in reply_divs[:5]:
|
||||
reply_content = rd.find("div", class_="tweet-content")
|
||||
if reply_content:
|
||||
replies.append(reply_content.get_text(separator=" ", strip=True))
|
||||
@@ -174,92 +368,30 @@ class TwitterParser(BaseParser):
|
||||
content=full_content,
|
||||
author=author,
|
||||
excerpt=generate_excerpt(full_content),
|
||||
tags=["twitter"],
|
||||
tags=["twitter", "nitter-fallback"],
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Fallback Strategy
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _fallback_scrape(self, url: str, last_error: str) -> ParseResult:
|
||||
"""Produce a graceful degradation result with integration guidance.
|
||||
|
||||
.. rubric:: How to extend with a paid scraping service
|
||||
|
||||
**Option A – ZenRows / ScrapingBee:**
|
||||
|
||||
1. Sign up at https://www.zenrows.com/ or https://www.scrapingbee.com/
|
||||
2. Obtain your API key.
|
||||
3. Replace the body of this method with::
|
||||
|
||||
import requests
|
||||
api_key = "YOUR_ZENROWS_API_KEY"
|
||||
params = {
|
||||
"url": url,
|
||||
"apikey": api_key,
|
||||
"js_render": "true",
|
||||
"premium_proxy": "true",
|
||||
}
|
||||
resp = requests.get("https://api.zenrows.com/v1/", params=params)
|
||||
html = resp.text
|
||||
# Then parse 'html' with BeautifulSoup to extract tweet text.
|
||||
|
||||
**Option B – Browser Cookies:**
|
||||
|
||||
1. Export your Twitter session cookies (e.g. with the *EditThisCookie*
|
||||
browser extension) as a Netscape-format ``cookies.txt`` file.
|
||||
2. Place the file at ``deepreader_skill/twitter_cookies.txt``.
|
||||
3. Modify ``_fetch_with_cookies()`` below to load and send them::
|
||||
|
||||
import http.cookiejar
|
||||
jar = http.cookiejar.MozillaCookieJar("twitter_cookies.txt")
|
||||
jar.load()
|
||||
session = requests.Session()
|
||||
session.cookies = jar
|
||||
resp = session.get(url, headers=self._get_headers())
|
||||
|
||||
**Option C – Playwright / Selenium headless browser:**
|
||||
|
||||
For the most reliable extraction, you can use a headless browser.
|
||||
This is heavier but handles JavaScript-rendered content::
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
page = browser.new_page()
|
||||
page.goto(url, wait_until="networkidle")
|
||||
html = page.content()
|
||||
browser.close()
|
||||
# Then parse 'html' as above.
|
||||
|
||||
Returns a :class:`ParseResult` with ``success=False`` and the
|
||||
guidance embedded in the error message.
|
||||
"""
|
||||
error_msg = (
|
||||
f"⚠️ All Nitter instances failed for this tweet.\n"
|
||||
f"Last error: {last_error}\n\n"
|
||||
f"The tweet URL was: {url}\n\n"
|
||||
f"💡 To improve Twitter support, consider:\n"
|
||||
f" 1. Updating the NITTER_INSTANCES list in twitter.py\n"
|
||||
f" 2. Integrating a paid scraping service (ZenRows/ScrapingBee)\n"
|
||||
f" 3. Using browser cookies for authenticated access\n"
|
||||
f" See the _fallback_scrape() docstring for detailed instructions."
|
||||
)
|
||||
logger.warning("Twitter fallback triggered for %s", url)
|
||||
return ParseResult.failure(url, error_msg)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# URL Utilities
|
||||
# ------------------------------------------------------------------
|
||||
# ==================================================================
|
||||
# URL Utilities
|
||||
# ==================================================================
|
||||
|
||||
@staticmethod
|
||||
def _extract_tweet_path(url: str) -> str | None:
|
||||
"""Extract the tweet path (``user/status/id``) from a Twitter URL.
|
||||
def _extract_tweet_info(url: str) -> tuple[str, str] | None:
|
||||
"""Extract (username, tweet_id) from a Twitter URL.
|
||||
|
||||
Returns ``None`` if the URL doesn't match the expected pattern.
|
||||
"""
|
||||
match = re.search(
|
||||
r"(?:x\.com|twitter\.com)/([a-zA-Z0-9_]{1,15})/status/(\d+)", url,
|
||||
)
|
||||
if match:
|
||||
return match.group(1), match.group(2)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _extract_tweet_path(url: str) -> str | None:
|
||||
"""Legacy: extract ``user/status/id`` path from a Twitter URL."""
|
||||
parsed = urlparse(url)
|
||||
# Match patterns like /username/status/1234567890
|
||||
match = re.match(r"^/([^/]+)/status/(\d+)", parsed.path)
|
||||
if match:
|
||||
return f"{match.group(1)}/status/{match.group(2)}"
|
||||
|
||||
Reference in New Issue
Block a user