Add experimental Harbor integration for GRPO environment training

[adithya-s-k]

Closes #6017

Adds an experimental integration for training on Harbor agentic task suites with GRPOTrainer via environment_factory. It lives at trl.experimental.harbor, is gated behind a new trl[harbor] extra, and is lazy-imported so non-users pay nothing.

It mirrors the structure of the OpenReward integration (#5752, #5729, #5696): a single HarborSpec maps one task suite to the three trainer slots —

from trl import GRPOConfig, GRPOTrainer
from trl.experimental.harbor import HarborSpec

spec = HarborSpec("AdithyaSK/data_agent_rl_environment_train", agent="bash", num_tasks=64)
trainer = GRPOTrainer(
    model="Qwen/Qwen3-4B",
    args=GRPOConfig(num_generations=8, max_steps=50, max_tool_calling_iterations=25),
    train_dataset=spec.train_dataset,
    environment_factory=spec.environment_factory,
    reward_funcs=spec.reward_funcs,
)
trainer.train()

• HarborEnv wraps a Harbor sandbox + verifier. TRL drives the rollout loop: it calls the env's tool methods during generation and reads env.reward after. The base agent (harness) is pluggable via agent= — a built-in name, a HarborEnv subclass, or an import/file path.
• Built-in bash harness, plus jupyter and terminal_notes example harnesses under examples/scripts/harbor/harnesses/ (folder-per-harness, each with a README listing its tools).

External agents only (for now)

Harbor supports external agents (run outside the sandbox, drive the loop via exec) and installed agents (installed into the image, run headless inside the container, trajectory parsed after). Only the external pattern is supported, because RL needs the trainer to drive generation turn-by-turn and capture the policy's tokens/log-probs + env mask — which an opaque in-container agent can't expose. A HarborEnv is therefore an external agent: tool methods exec into the sandbox, but the loop and the model under training stay in TRL. (Documented in docs/source/harbor.md.)

Notable changes

• pyproject.toml: new harbor extra (harbor>=0.13.0; python_version >= '3.12'); relaxes the vLLM cap vllm>=0.12.0,<=0.19.0 → vllm>=0.22.0. The old cap pins transformers<5, which breaks environment_factory (needs transformers>=5.2). This change is separable from the rest of the PR if you'd prefer it split out.
• E2B COPY workaround: E2B's from_dockerfile build honors RUN but silently drops files COPY'd from the build context, breaking task healthchecks that run those files (e.g. a data-pull hook). HarborEnv replicates the Dockerfile's COPY directives at runtime (upload as the sandbox user, mv into place as root).

Testing

• tests/experimental/test_harbor.py — 12 tests (agent resolution, dataset/metadata columns, reward func), green; gated by require_harbor.
• Verified end-to-end: gpt-4.1 over the bash harness loads task data and lands reward=1.0 on a data-agent task.

---

Note

Medium Risk
New experimental sandbox/RL path and a broader vLLM dependency range affect install compatibility; core training paths are unchanged unless Harbor or the new vLLM floor is used.

Overview
Adds experimental Harbor integration so GRPOTrainer can train on Harbor agentic task suites via environment_factory, following the same HarborSpec → train_dataset / environment_factory / reward_funcs pattern as OpenReward.

Core: trl.experimental.harbor introduces HarborSpec, HarborEnv (sandbox lifecycle, lazy verifier reward, dedicated async loop thread), and built-in HarborBashEnv. Harnesses are pluggable via agent= (name, import path, or HarborEnv subclass). Includes an E2B workaround that re-applies Dockerfile COPY files at runtime.

Packaging & deps: New optional extra trl[harbor] (harbor>=0.13.0, Python 3.12+). vLLM constraint relaxed from <=0.19.0 to >=0.22.0 so environment_factory can use transformers>=5.2.

Examples & docs: examples/scripts/harbor/data_agent.py GRPO script; example jupyter and terminal_notes harnesses; new docs/source/harbor.md and toctree/example table entries.

Tests: tests/experimental/test_harbor.py for agent resolution, dataset building, and outcome rewards; is_harbor_available / require_harbor helpers.

^{Reviewed by Cursor Bugbot for commit 1f6f01c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5c076888f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Use the task's verifier config

For Harbor tasks that define a [verifier] block, this constructs a fresh default VerifierConfig() instead of using self._task.config.verifier. That silently drops task-provided verifier settings such as env credentials/model parameters, a custom import_path, user, or separate verifier environment, so those valid Harbor tasks will either run the wrong verifier or fail during reward computation.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Include the cloud backend extras used by the example

The new example and docs recommend running with --env e2b, but installing trl[vllm,harbor] only pulls Harbor's base package; Harbor keeps providers such as e2b, daytona, and runloop behind optional extras. A user following the script metadata or quick-start will therefore hit an import error as soon as environment_type="e2b" is selected, even with E2B_API_KEY set.

Useful? React with 👍 / 👎.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1f6f01c. Configure here.}

[cursor]

Jupyter harness wrong async loop

High Severity

JupyterEnv._run_cell drives sandbox exec via self._loop.run_until_complete, but HarborEnv keeps that loop running forever on a background thread and routes all Harbor I/O through _run / run_coroutine_threadsafe. The first add_and_execute_code_cell call typically raises a running-loop error, so the documented jupyter harness cannot run cells during GRPO rollouts.

 

^{Reviewed by Cursor Bugbot for commit 1f6f01c. Configure here.}

[cursor]

Trial temp directories leak

Medium Severity

Each HarborEnv.reset assigns a new TrialPaths directory from tempfile.mkdtemp without deleting the prior trial folder. GRPO reuses one env instance per rollout slot across many steps, so host temp directories accumulate for the whole run and can exhaust disk space during long Harbor training.

 

^{Reviewed by Cursor Bugbot for commit 1f6f01c. Configure here.}