From ccb79a959c148ce4f5919c530d241128042b4104 Mon Sep 17 00:00:00 2001 From: Xuyang Chen Date: Wed, 15 Apr 2026 19:24:06 -0400 Subject: [PATCH] Update README.md --- terminal-rl/README.md | 55 ++++++++++++++++++++++++++++-------- terminal-rl/remote/README.md | 10 +++++-- 2 files changed, 52 insertions(+), 13 deletions(-) diff --git a/terminal-rl/README.md b/terminal-rl/README.md index 15d01bf..e5379a1 100644 --- a/terminal-rl/README.md +++ b/terminal-rl/README.md @@ -2,19 +2,51 @@ RL training for terminal agents. The agent interacts with Docker-hosted environments and is trained with GRPO (optional PRM). -This workflow has two independent components: +## Architecture & requirements -- **Training machine** runs task router + Ray + training scripts, and connects to workers via `WORKER_URLS` -- **Remote workers:** run the pool server and execute tasks (Docker required): [remote/README.md](remote/README.md) +This setup has two independent components: + +- **Training machine**: runs task router + Ray + training scripts. Requires a GPU environment and the training dependencies. Connects to workers via `WORKER_URLS`. +- **Remote workers:**: run the pool server and execute tasks inside Docker. Requires Docker and network reachability from the training machine (default port **18081**). --- -## Prerequisites +## Directory structure -- **Training machine:** GPU node/cluster with the required training dependencies. -- **Remote workers:** Docker-capable hosts reachable from the training machine (default pool server port **18081**). Setup: [remote/README.md](remote/README.md). - ---- +``` +terminal-rl/ +├── README.md +├── router_server.py # Task router service +├── generate.py # Generation entry +├── rollout_agent.py # Agent abstraction +├── agent_runner.py # Orchestrate agent execution +├── env_client.py # Environment client +├── inference_client.py # Inference client +├── sample_builders.py # Build training samples from rollout outcomes +├── request_utils.py +├── custom_types.py +├── rollout_log.py +├── terminal-rl_qwen3-8b.sh # Qwen3-8B training script +├── terminal-rl_qwen3-8b_prm_2nodes.sh # Qwen3-8B PRM 2-node training script +├── agent/ +│ ├── camel_agent.py # Rollout agent based on camel-ai ChatAgent +│ └── prm_agent.py # PRM scoring agent +├── configs/ +│ └── rollout_qwen3.yaml # Rollout config for Qwen3 models +├── data_utils/ +│ ├── download.py # Dataset download +│ ├── convert_task_to_dataset.py # Convert tasks to training JSONL +│ └── load_tasks.py +├── remote/ # Remote workers and Docker environment +│ ├── README.md # Worker deployment guide +│ ├── pool_server.py # Task pool server (port 18081) +│ ├── terminal_env.py # Terminal environment wrapper +│ ├── run_pool_server.sh +│ ├── setup.sh +│ ├── compose_override.yaml +│ └── docker_compose_utils.py +└── dataset/ # Dataset directory +``` ## Instructions @@ -71,7 +103,7 @@ export WANDB_KEY="your-wandb-key" Then run (from repo root): ```bash -bash terminal-rl/terminal_qwen3_8b_rl.sh +bash terminal-rl/terminal-rl_qwen3-8b.sh ``` --- @@ -94,7 +126,7 @@ export PRM_SGLANG_URL="http://:" Then run: ```bash -bash terminal-rl/terminal_qwen3_8b_prm_rl_2nodes.sh +bash terminal-rl/terminal-rl_qwen3-8b_prm_2nodes.sh ``` --- @@ -102,4 +134,5 @@ bash terminal-rl/terminal_qwen3_8b_prm_rl_2nodes.sh ## Notes - `WORKER_URLS` must point to already-running pool servers. -- As an example, one rollout agent implementation in this repo is based on **CAMEL** (see `terminal-rl/agent/camel_agent.py` and [CAMEL](https://github.com/camel-ai/camel)). +- The rollout agent implementation in `terminal-rl/agent/camel_agent.py` is based on [camel-ai/camel](https://github.com/camel-ai/camel)'s `ChatAgent`. + diff --git a/terminal-rl/remote/README.md b/terminal-rl/remote/README.md index 9d14446..8f79ec1 100644 --- a/terminal-rl/remote/README.md +++ b/terminal-rl/remote/README.md @@ -66,7 +66,7 @@ This script: - Sets `DATASET_DIR` and `TBENCH_OUTPUT_ROOT` under `terminal-rl/` by default. - Starts the pool server with `python -m terminal-rl.remote.pool_server` on `0.0.0.0:18081` (overridable via `ENV_SERVER_PORT`, `WORKER_MAX_TASKS`, `WORKER_MAX_RUNS_PER_TASK`). -Run in background / under a process manager as needed. Example (nohup`): +Run in background / under a process manager as needed. Example (`nohup`): ```bash nohup bash terminal-rl/remote/run_pool_server.sh > pool_server.log 2>&1 & @@ -107,7 +107,7 @@ When running the pool server (via `run_pool_server.sh` or `python -m terminal-rl | `COMPOSE_OVERRIDE_PATH` | — | Optional Docker Compose override file. | -Example with custom port and limits: +Example with custom port and task limits: ```bash export ENV_SERVER_PORT=18082 @@ -119,6 +119,12 @@ bash terminal-rl/remote/run_pool_server.sh Example using pre-built images from a registry (pull mode). Set the image source and prefix; you can build and push your own: ```bash +export http_proxy="http://:" +export https_proxy="http://:" +export no_proxy="localhost,127.0.0.1" +export HTTP_PROXY="$http_proxy" +export HTTPS_PROXY="$https_proxy" +export NO_PROXY="$no_proxy" export TBENCH_DOCKER_IMAGE_SOURCE=pull export TBENCH_DOCKER_PULL_PREFIX="ghcr.io//:task-" export COMPOSE_OVERRIDE_PATH="terminal-rl/remote/compose_override.yaml"