mirror of
https://github.com/Gen-Verse/OpenClaw-RL.git
synced 2026-08-14 00:47:53 +00:00
Update README.md
This commit is contained in:
Xuyang Chen
+44
-11
@@ -2,19 +2,51 @@
|
||||
|
||||
RL training for terminal agents. The agent interacts with Docker-hosted environments and is trained with GRPO (optional PRM).
|
||||
|
||||
This workflow has two independent components:
|
||||
## Architecture & requirements
|
||||
|
||||
- **Training machine** runs task router + Ray + training scripts, and connects to workers via `WORKER_URLS`
|
||||
- **Remote workers:** run the pool server and execute tasks (Docker required): [remote/README.md](remote/README.md)
|
||||
This setup has two independent components:
|
||||
|
||||
- **Training machine**: runs task router + Ray + training scripts. Requires a GPU environment and the training dependencies. Connects to workers via `WORKER_URLS`.
|
||||
- **Remote workers:**: run the pool server and execute tasks inside Docker. Requires Docker and network reachability from the training machine (default port **18081**).
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
## Directory structure
|
||||
|
||||
- **Training machine:** GPU node/cluster with the required training dependencies.
|
||||
- **Remote workers:** Docker-capable hosts reachable from the training machine (default pool server port **18081**). Setup: [remote/README.md](remote/README.md).
|
||||
|
||||
---
|
||||
```
|
||||
terminal-rl/
|
||||
├── README.md
|
||||
├── router_server.py # Task router service
|
||||
├── generate.py # Generation entry
|
||||
├── rollout_agent.py # Agent abstraction
|
||||
├── agent_runner.py # Orchestrate agent execution
|
||||
├── env_client.py # Environment client
|
||||
├── inference_client.py # Inference client
|
||||
├── sample_builders.py # Build training samples from rollout outcomes
|
||||
├── request_utils.py
|
||||
├── custom_types.py
|
||||
├── rollout_log.py
|
||||
├── terminal-rl_qwen3-8b.sh # Qwen3-8B training script
|
||||
├── terminal-rl_qwen3-8b_prm_2nodes.sh # Qwen3-8B PRM 2-node training script
|
||||
├── agent/
|
||||
│ ├── camel_agent.py # Rollout agent based on camel-ai ChatAgent
|
||||
│ └── prm_agent.py # PRM scoring agent
|
||||
├── configs/
|
||||
│ └── rollout_qwen3.yaml # Rollout config for Qwen3 models
|
||||
├── data_utils/
|
||||
│ ├── download.py # Dataset download
|
||||
│ ├── convert_task_to_dataset.py # Convert tasks to training JSONL
|
||||
│ └── load_tasks.py
|
||||
├── remote/ # Remote workers and Docker environment
|
||||
│ ├── README.md # Worker deployment guide
|
||||
│ ├── pool_server.py # Task pool server (port 18081)
|
||||
│ ├── terminal_env.py # Terminal environment wrapper
|
||||
│ ├── run_pool_server.sh
|
||||
│ ├── setup.sh
|
||||
│ ├── compose_override.yaml
|
||||
│ └── docker_compose_utils.py
|
||||
└── dataset/ # Dataset directory
|
||||
```
|
||||
|
||||
## Instructions
|
||||
|
||||
@@ -71,7 +103,7 @@ export WANDB_KEY="your-wandb-key"
|
||||
Then run (from repo root):
|
||||
|
||||
```bash
|
||||
bash terminal-rl/terminal_qwen3_8b_rl.sh
|
||||
bash terminal-rl/terminal-rl_qwen3-8b.sh
|
||||
```
|
||||
|
||||
---
|
||||
@@ -94,7 +126,7 @@ export PRM_SGLANG_URL="http://<prm-router-ip>:<prm-router-port>"
|
||||
Then run:
|
||||
|
||||
```bash
|
||||
bash terminal-rl/terminal_qwen3_8b_prm_rl_2nodes.sh
|
||||
bash terminal-rl/terminal-rl_qwen3-8b_prm_2nodes.sh
|
||||
```
|
||||
|
||||
---
|
||||
@@ -102,4 +134,5 @@ bash terminal-rl/terminal_qwen3_8b_prm_rl_2nodes.sh
|
||||
## Notes
|
||||
|
||||
- `WORKER_URLS` must point to already-running pool servers.
|
||||
- As an example, one rollout agent implementation in this repo is based on **CAMEL** (see `terminal-rl/agent/camel_agent.py` and [CAMEL](https://github.com/camel-ai/camel)).
|
||||
- The rollout agent implementation in `terminal-rl/agent/camel_agent.py` is based on [camel-ai/camel](https://github.com/camel-ai/camel)'s `ChatAgent`.
|
||||
|
||||
|
||||
@@ -66,7 +66,7 @@ This script:
|
||||
- Sets `DATASET_DIR` and `TBENCH_OUTPUT_ROOT` under `terminal-rl/` by default.
|
||||
- Starts the pool server with `python -m terminal-rl.remote.pool_server` on `0.0.0.0:18081` (overridable via `ENV_SERVER_PORT`, `WORKER_MAX_TASKS`, `WORKER_MAX_RUNS_PER_TASK`).
|
||||
|
||||
Run in background / under a process manager as needed. Example (nohup`):
|
||||
Run in background / under a process manager as needed. Example (`nohup`):
|
||||
|
||||
```bash
|
||||
nohup bash terminal-rl/remote/run_pool_server.sh > pool_server.log 2>&1 &
|
||||
@@ -107,7 +107,7 @@ When running the pool server (via `run_pool_server.sh` or `python -m terminal-rl
|
||||
| `COMPOSE_OVERRIDE_PATH` | — | Optional Docker Compose override file. |
|
||||
|
||||
|
||||
Example with custom port and limits:
|
||||
Example with custom port and task limits:
|
||||
|
||||
```bash
|
||||
export ENV_SERVER_PORT=18082
|
||||
@@ -119,6 +119,12 @@ bash terminal-rl/remote/run_pool_server.sh
|
||||
Example using pre-built images from a registry (pull mode). Set the image source and prefix; you can build and push your own:
|
||||
|
||||
```bash
|
||||
export http_proxy="http://<proxy-host>:<port>"
|
||||
export https_proxy="http://<proxy-host>:<port>"
|
||||
export no_proxy="localhost,127.0.0.1"
|
||||
export HTTP_PROXY="$http_proxy"
|
||||
export HTTPS_PROXY="$https_proxy"
|
||||
export NO_PROXY="$no_proxy"
|
||||
export TBENCH_DOCKER_IMAGE_SOURCE=pull
|
||||
export TBENCH_DOCKER_PULL_PREFIX="ghcr.io/<your-org>/<your-image>:task-"
|
||||
export COMPOSE_OVERRIDE_PATH="terminal-rl/remote/compose_override.yaml"
|
||||
|
||||
Reference in New Issue
Block a user