Youtu-Agent: Scaling Agent Productivity with Automated Generation and Hybrid Policy Optimization

Building upon the structured configuration system, Youtu-Agent provides automated mechanisms for generating complete agent configurations from high-level task descriptions. We introduce two generation paradigms tailored to different complexity levels and use cases, as illustrated in Figure 1.

How can we adapt an agent to specific tasks without expensive parameter fine-tuning? Traditional approaches face significant limitations: manual prompt engineering is labor-intensive with no guarantee of improvement, while SFT or RL requires substantial data and computational resources. To address this, the Agent Practice module integrates Training-free Group Relative Policy Optimization (Training-free GRPO) [1], enabling low-cost agent improvement through in-context experience accumulation without any gradient updates.

The core mechanism operates as follows: on a small training dataset (with or without ground truth), the agent performs multiple rollouts per task, generating diverse solution trajectories. Inspired by GRPO’s group relative optimization, an evaluator assesses the relative quality of trajectories for the same task. Rather than calculating a numerical advantage in vanilla GRPO, the LLM introspects on these diverse outputs and distill a semantic group advantage, i.e., a textual learning direction derived from contrasting successful and failed trials. Such advantage optimizes the existing experiential knowledge, serving as a refined policy for subsequent epochs. After completing practice, during online testing, these experiences are injected into the agent’s context, acting as a form of “textual LoRA” that guides reasoning without modifying model weights.

Compared with traditional learning approaches, the agent practice module requires no gradient computation, works with API-based models where fine-tuning is inaccessible, leverages small training sets effectively, and allows experiences to be accumulated continuously.

For applications requiring significant and lasting performance improvement, the Agent RL module provides a complete pipeline for end-to-end, production-grade reinforcement learning (see Figure 3). We implement custom agent logic within our framework for inference, and integrate with Agent-Lightning [3] as the connector between Youtu-Agent and modern RL framework such as VeRL [2] for distributed training. Such integration addresses two critical challenges that have historically limited agent RL: scalability to large distributed systems and stability over long-horizon tasks.

Scalability Solutions. Scaling agent RL presents unique challenges due to the complex, stateful nature of agent-environment interactions. We address these through: (1) RESTful API wrapping that encapsulates agent execution environments as standardized services, enabling seamless distribution and load balancing; (2) Ray-based concurrency for highly parallel rollout collection across distributed workers; and (3) layered timeout logic with hierarchical controls at tool, step, and episode levels to handle various potential timeout issues without fatal failures. Such infrastructure-level optimization enables stable scaling to 128 GPUs.

Stability Solutions. Long-horizon agent tasks exacerbate the “entropy explosion” problem where policies degenerate into repetitive or nonsensical actions. We mitigate this through: (1) filtering invalid and anomalous tool calls from training data to prevent learning degenerate patterns; (2) removing batch shuffling and reducing off-policy update iterations to prevent the policy from overfitting the outdated experience; and (3) correcting bias of advantage estimation in turn-level GRPO training. These techniques ensure stable training dynamics with consistent performance improvements across hundreds of training steps.

To validate the effectiveness of our framework design and tool implementations, we evaluate Youtu-Agent on established agent benchmarks. Prioritizing accessibility and reproducibility, we build our agents entirely on open-source models—specifically the DeepSeek-V3 series—without relying on proprietary APIs such as Claude or GPT. We adopt a Plan-and-Execute paradigm, where a planner agent orchestrates task decomposition and scheduling while specialized executor agents handle subtasks.

WebWalkerQA. This benchmark (680 questions) evaluates LLMs’ ability to perform multi-step, deep web navigation and question answering over real websites. Equipped with web search and crawling tools, Youtu-Agent achieves 71.47% pass@1 accuracy, as shown in Figure 4. This result demonstrates robust environment interaction and multi-step exploration capabilities.

GAIA (Text-only Subset). GAIA [4] (466 questions) tests real-world question answering requiring reasoning, multimodal understanding, web browsing, and tool-use proficiency. Since we use text-only models, we evaluate on the text-only subset. Beyond web tools, we provide document parsing and code execution capabilities to handle file attachments. Youtu-Agent achieves 72.8% pass@1 accuracy, demonstrating effective tool selection and deep reasoning abilities that validate our framework’s design for general-purpose agent applications.

These results establish Youtu-Agent as a competitive open-source baseline without requiring proprietary APIs or closed-source systems.

We evaluate the practical utility of Youtu-Agent’s automated generation mechanisms through quantitative metrics and a qualitative case study.

As introduced in Section 2.3, the Agent Practice module integrates Training-free GRPO to enable low-cost agent improvement through experience accumulation without parameter updates. We validate its effectiveness on challenging mathematical reasoning benchmarks AIME 2024 and AIME 2025 [5], using Mean@32 for robust measurement.

Experimental Setup. Our evaluation is based on ReAct [6] with a code interpreter tool. During the learning stage, we randomly sample 100 problems from the DAPO-Math-17K [7] dataset and run 3 epochs with group size $5$ and temperature $0.7$. For online testing, the temperature is set to $0.3$.

As shown in Table 2, when ground truths are available for the training samples, Training-free GRPO achieves +2.7% on AIME 2024 and +5.4% on AIME 2025 compared to ReAct, demonstrating the effectiveness of learned experiences. Notably, the variant without ground truths still achieves strong results, validating applicability to domains where labeled data is scarce. Compared to RL methods requiring 10,000+ training samples and approximately $10,000 for fine-tuning a 32B model, Training-free GRPO achieves better performance with only 100 samples and approximately $18 learning costs, offering a highly effective and accessible pathway for real-world applications. Furthermore, as shown in Figure 5, both learning and validation performance improve steadily across epochs, and the average number of tool calls is decreasing, which indicates that the agent learns more efficient problem-solving strategies.

As introduced in Section 2.4, our Agent RL module proposes modifications to both infrastructure and algorithm implementation of Agent-Lightning [3] to enable scalable and stable end-to-end RL training. The effectiveness of our optimization is confirmed through both training efficiency and stability on multiple benchmarks.

Efficiency. Compared with the Agent-Lightning (v0.2.2 official version), our infrastructure optimizations such as concurrency control, RESTful API wrapping, and hierarchical timeout logic reduce iteration time by approximately 40%, enabling efficient scaling to 128 GPUs without timeout issues (see Figure 6).

Effectiveness. To ensure fair comparison, we perform RL training of Qwen2.5-7B-Instruct on two typical tasks: 1) math/code for reasoning capabilities, and 2) search for information-seeking capabilities.

For the math/code task, we follow the experimental settings of ReTool [10] where a code interpreter (sandbox) is prepared as an external tool for assistance of solving mathematic problems. As shown in Figure 3, evaluation results confirm that the accuracy gains of 35% and 22% are respectively achieved on AIME24 and AIME25.

For the search task, we follow the experimental settings of SearchR1 [12] where a local wikipedia-2018 retrieval service (E5-HNSW64 embeddings) is prepared as an external tool for solving general QA and multi-hop QA problems. As shown in Table 4, consistent improvements are observed across all benchmarks with accuracy gains of 17%, 19%, 21%, 8%, 17%, 13%, and 10% respectively on TriviaQA, PopQA, NatureQuestions, MusiQue, HotpotQA, Bamboogle, and 2WikiMultiHop.

Training Dynamics. Figure 7 illustrates the training dynamics for comparison with the official Agent-Lightning release. After our improvements addressing “entropy explosion”, the actor’s KL divergence and gradient norm remain stable. It can be observed that critic score increases steadily, which is in line with the validation accuracy.

These experiments demonstrate that Youtu-Agent not only provides a robust execution framework but also supports effective model training for building continuously improving agents.

Recent years have seen a proliferation of agent frameworks. MetaGPT [13], AutoGen [14], and ChatDev [15] have popularized multi-agent collaboration, assigning static roles to LLMs to solve complex tasks. While effective, these frameworks often rely on predefined roles and heavy prompt tuning. Youtu-Agent advances this paradigm by automating the generation of agent configurations themselves, including executable tool code and optimized prompts, rather than just orchestrating static roles.

Research into automated agent design, such as ADAS [16] and AutoAgents [17], has explored generating agents from high-level descriptions. Youtu-Agent distinguishes itself by not only generating the agent’s design but also synthesizing the necessary tool code (Python) and providing a unified pipeline that extends from automated generation to continuous optimization.

Optimizing agents typically involves either prompt optimization or parameter updates. Reflexion [18] and ReAct [6] use verbal reinforcement and reasoning traces to improve performance without training. On the training side, Toolformer [19] and Gorilla [20] focus on fine-tuning for tool use. Youtu-Agent integrates both approaches: the Practice module enables low-cost, inference-time improvement through experience accumulation, and the Agent RL module provides a full-scale RL pipeline for end-to-end parameter optimization, addressing the stability issues often encountered in PPO/RLHF for agents.