SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks

Research Overview

If you have trained a multi-turn agent, you know this frustration: the agent has a productive 9-turn conversation, then says something wrong in turn 10, and the whole trajectory gets marked as failure. All 9 good turns? Penalized. The training signal is wasted.

This is the credit assignment problem for multi-turn agents. Current methods like DPO compare entire trajectories: this 10-turn conversation succeeded, that one failed. But they cannot pinpoint which specific turn made the difference. It is like grading a student's exam by looking only at the final answer, not the work.

SWEET-RL (Step-WisE Evaluation from Training-time information) solves this with an asymmetric actor-critic design. The key insight: during training, you often have access to the "right answer" (reference code, target output, solution). Why not let the critic see it?

The critic sees both the conversation history AND the solution. This lets it judge whether each turn moves toward or away from the goal. The actor (the agent you deploy) never sees the solution, keeping everything realistic.

The result? Llama-3.1-8B trained with SWEET-RL matches GPT-4o on collaborative coding tasks (40.4% vs 40.4%) and beats O1-Mini (30.3%). A 6% absolute improvement over Multi-Turn DPO, the previous best method.

The Credit Assignment Problem

Consider this 5-turn coding collaboration where the agent makes good progress but introduces a bug in turn 4:

Before vs After: How Training Signal Changes

Why step-level feedback matters:

• Faster learning: Every turn provides useful gradient, not just the final outcome
• Better generalization: Good patterns are reinforced even in failed trajectories
• Clearer debugging: You can trace exactly which steps the model struggles with

SWEET-RL Architecture

SWEET-RL operates in two stages:

Stage 1: Train the Critic

The critic learns to assign advantages (how good is this action?) to each turn. It trains on preference pairs from offline trajectories:

1. Take two trajectories for the same task
2. Label the one with higher cumulative reward as "chosen"
3. Train the critic to prefer chosen trajectories using Bradley-Terry objective

The key: the critic sees training-time information (the reference solution) that the actor never sees.

Architecture Diagram Explanation: The chart above shows the asymmetric information flow. During training, conversation turns flow to both the Critic (which also receives the ground truth solution) and the Actor (which only sees history). The Critic produces step-level advantage scores that guide the Actor's learning. At deployment, only the Actor runs—the Critic is discarded.

Stage 2: Train the Actor

With the trained critic, optimize the actor policy:

1. At each turn, sample 16 candidate responses
2. Rank them by critic advantage scores
3. Top 50% become "chosen", bottom 50% become "rejected"
4. Apply DPO loss on these turn-level preferences

This converts the trajectory-level signal into step-level signal, enabling much more efficient learning.

Critic Parameterization

The advantage function uses a clever parameterization:

A(observation, action, context) =
  (1/L) * sum(log π_trained / log π_reference)

Where L is the number of tokens (for normalization). This is the mean log probability ratio between the trained and reference models.

The Asymmetric Trick

The core innovation is asymmetric information between actor and critic:

Why this works:

1. The critic can make informed judgments because it knows where the conversation should go
2. The actor learns from these judgments without ever seeing the solution
3. At deployment, the actor works without any privileged information

Training-Time Information by Industry

The "privileged information" the critic sees varies by domain. Here is what it looks like across industries:

The key requirement: you must have verifiable ground truth during training. At deployment, the agent never sees this information.

ColBench Benchmark

The paper introduces ColBench, a benchmark for multi-turn collaborative agents:

Tasks

Scale

• Training: 10K+ backend tasks, 10K frontend tasks
• Testing: 1K backend, 500 frontend (manually inspected)
• Offline trajectories: 15K backend, 6K frontend

Why ColBench?

1. Sufficient complexity: Tasks require multi-step reasoning, not single-turn solutions
2. Minimal engineering overhead: LLM simulators with ground-truth artifacts, no complex environments
3. Procedural generation: Prevents overfitting to specific examples

The benchmark is open-source and included in the official repository.

Benchmark Results

Main Results (Llama-3.1-8B-Instruct)

SWEET-RL achieves 6% absolute improvement over Multi-Turn DPO.

Comparison to Larger Models

An 8B model matches GPT-4o and beats O1-Mini. The 8B model with SWEET-RL also matches the 70B model with zero-shot prompting.

Off-Policy Learning (70B Model)

SWEET-RL works even when training a larger model on data from a smaller model:

Rejection Fine-Tuning actually hurts performance because it forces the 70B model to imitate suboptimal 8B trajectories word-by-word. SWEET-RL avoids this by using step-level rewards instead of direct imitation.

Ablation Studies

What Makes SWEET-RL Work?

Three critical components:

1. Training-time information: Without it, the critic cannot judge step quality. 9 percentage points lost.
2. Log probability parameterization: Using a regression head instead loses 4 points. LLMs generalize better with their native output format.
3. Length normalization: Without it, the model degenerates completely.

Credit Assignment Comparison

The paper compares different ways to assign step-level rewards:

LLM-as-Judge (using another LLM to rate responses) gets distracted by length and format rather than actual quality. Value function heads (standard in RL) generalize poorly to new tasks.

Data Scaling

SWEET-RL requires more initial data to train a reliable critic:

• At 3K samples: Multi-Turn DPO outperforms SWEET-RL
• At 6K+ samples: SWEET-RL catches up and surpasses
• At 15K samples: SWEET-RL significantly ahead

If you have limited data, Multi-Turn DPO may be a better choice. With sufficient data (6K+), SWEET-RL is clearly superior.

Implementation Blueprint

When to Use SWEET-RL

Use SWEET-RL when you have:

1. Multi-turn tasks (3+ turns of interaction)
2. Ground-truth artifacts for training (reference code, target outputs, solution paths)
3. Sufficient data (6K+ trajectory pairs, more is better)
4. Clear success criteria (unit tests, similarity scores, verifiable outcomes)

Do not use SWEET-RL for:

• Single-turn tasks (use regular DPO)
• Tasks without ground truth (use RLHF or LLM-as-Judge)
• Limited data scenarios (< 3K examples, use Multi-Turn DPO)

Data Qualification Checklist

Before starting SWEET-RL implementation, assess your readiness:

Score interpretation: If you cannot check all boxes, consider Multi-Turn DPO (needs only success labels) or standard fine-tuning. Missing the 6K+ volume threshold is the most common blocker.

Tech Stack

Step-by-Step Workflow

Phase 1: Data Collection

1. Define your task with clear success criteria
2. Collect 10K+ task instances with ground truth
3. Generate trajectories using your base model
4. Filter for both successful and failed attempts
5. Target: 15K+ trajectories (mix of outcomes)

Phase 2: Critic Training

1. Construct preference pairs:
   - Same task, different trajectories
   - Higher cumulative reward = chosen
2. Train critic with Bradley-Terry objective
3. Include training-time info (ground truth) as input
4. Validate on held-out preference pairs

Phase 3: Actor Training

1. For each turn in each trajectory:
   - Sample 16 candidate responses
   - Score with trained critic
   - Top 8 = chosen, bottom 8 = rejected
2. Apply DPO loss on turn-level preferences
3. Repeat for multiple epochs

Key Parameters

Adaptation to Your Domain

Customer Support Bot:

• Ground truth: Expert response transcripts
• Success metric: Resolution rate + customer satisfaction
• Turns: Typically 5-15

Code Assistant:

• Ground truth: Correct code implementation
• Success metric: Unit test pass rate
• Turns: Typically 3-10

Tutoring System:

• Ground truth: Ideal explanation path
• Success metric: Student quiz performance
• Turns: Typically 5-20

Limitations

1. Data requirements: SWEET-RL needs more data than Multi-Turn DPO to train an effective critic. At 3K samples, it underperforms.

2. Ground truth dependency: You need training-time information (reference solutions). For tasks without clear ground truth, this is not applicable.

3. Offline only: The paper tests offline RL (learning from collected data). Online learning with human feedback remains expensive.

4. Domain tested: Only validated on artifact creation (code, webpages). Generalization to other domains (dialogue, planning) is untested.

5. Compute cost: Training both critic and actor requires roughly 2x the compute of standard fine-tuning.

The Bottom Line

For ML engineers building multi-turn agents:

SWEET-RL is the current best method if you have:

• 6K+ training trajectories
• Ground truth artifacts for training
• Multi-turn tasks (3+ turns)

The 6% improvement over Multi-Turn DPO is significant, and matching GPT-4o with an 8B model means major cost savings at deployment.

For engineering managers:

Budget 2-3 weeks for integration. The official Facebook Research code provides a solid starting point. ROI calculation: if your 8B model can match GPT-4o quality, you save ~10x on inference costs at scale.

For researchers:

The asymmetric actor-critic design (critic sees privileged information, actor does not) is a generalizable pattern. The key insight is that training-time information should inform credit assignment, not just final outcomes.

Official resources:

• Paper: arxiv.org/abs/2503.15478
• Code: github.com/facebookresearch/sweet_rl
• Benchmark: ColBench (included in repo)