The Problem with Scalar Rewards in Agentic Training

Standard policy-gradient methods, such as Group Relative Policy Optimization (GRPO), typically collapse multi-turn agent rollouts into a single scalar reward. This approach is inherently lossy when tasks require a sequence of distinct skills. Because the optimizer receives only a final score, it struggles to infer which specific actions or sub-competencies contributed to success, especially in environments with sparse or delayed feedback.

Subtask-Decomposed Advantage Estimation (SDAE)

Reinforcement Learning with Decomposed Subtasks (RLDS) addresses this by decomposing the reward before the policy update. The core mechanism, Subtask-Decomposed Advantage Estimation (SDAE), operates as follows:

• Taxonomy Mapping: Trajectory rewards are split into shares based on a predefined taxonomy of subtasks.
• Per-Subtask Advantage: Instead of a single global advantage, the system computes a group-relative advantage for each subtask.
• Token-Level Credit Assignment: The model distributes credit by weighting each subtask's advantage by its importance. This credit is concentrated around specific steps where a reflection indicates a subtask was executed.

Performance and Efficiency Gains

RLDS shows the most significant improvements in tasks characterized by high subtask heterogeneity. In benchmarks like ScienceWorld and FrozenLake, the method yielded gains of +11.5 and +9.8 points respectively.

Beyond accuracy, RLDS improves compute efficiency in long-horizon tasks. In ScienceWorld, the method achieved a 10.9% reduction in wall-clock time per step. This occurs because the fixed overhead of the "reflect-and-grade" process is effectively amortized over longer rollouts, making the decomposition process more economical than standard scalar-based training for complex, multi-step agentic workflows.