Shopping categories share search, evidence, option, and purchase procedures. The evolving unit is a chunk inside the shared bank.
AGENTIC RL · COMPLETE PROJECT REPORT
IntraSkill
Skill–model co-evolution
for shopping agents
A four-round applied study of one idea: let an agent policy learn while its external procedural skill is measured, revised, and retained where it still helps.
TEST SUCCESS · WITHOUT SKILLS
51.75%↗84.25%
Qwen3.5-4B · BaseQwen3.5-4B · Iter80
+32.50 percentage points
400 shared tasks · one training run
4BPolicy parameters
80Rollout steps
4Evolution rounds
10 → 8Initial → final chunks
READING GUIDE
A project overview with the details intact.
Start with the method, or enter through a specific question. Every chapter is available in English and Chinese; the reference library preserves the original engineering documents.
01 / BACKGROUND & METHODWhy should a model and its skill co-evolve?Motivation, related work, scene-level chunks, repeating train–gate intervals, paired contribution estimates.
02 / TASK, ENVIRONMENT & REWARDWhat does the agent see, and how is a purchase scored?Task anatomy, personal profile, simulator, tools, reward design, and one complete shopping trace.
03 / PROTOCOLWhat exactly was run?Splits, runtime, decoding, training denominators, validation, and provenance.
04 / RESULTSWhat improved, and what remains open?Four test conditions, paired intervals, gates, skill evolution, missing controls.
05 / BEHAVIORHow does the agent's behavior change?Generation length, success/failure steps, search, product visits, repairs, cases.
06 / OPTIMIZATIONWhat do the trainer logs reveal?160 updates: entropy, PPO diagnostics, gradient norms, throughput and truncation.
07 / REPRODUCEHow can the evidence be checked?Offline commands, numerical downloads, source hashes, project and deployment guides.
01 / BACKGROUND & METHOD
Keep mutable procedures explicit.
Train reusable behavior.
A skill is useful precisely because some SOPs should remain readable and replaceable. Yet stale guidance can restrict exploration. IntraSkill measures the residual value of minimal chunks while the policy learns under randomized omission.
Eight GRPO siblings share one chunk mask. Scaffold exposure never becomes a hidden difference inside the relative-advantage baseline.
A frozen model, paired bare/masked tasks, and chunk-level regression decide what is rewritten, retained, or retired.
Sample a shared context
20% skill-free groups; otherwise sample chunks by contribution. Eight sibling trajectories share one mask.
Update the policy
GRPO compares trajectories under the same context. Only generated tokens carry the training loss.
Investigate failures
Retry all-wrong groups with the full skill. Trace2Skill investigates remaining failures and proposes additions or rewrites.
Validate and revise
Every 20 steps, pair bare and random-mask validation at the same checkpoint. Retain positive-contribution winners.
Rᵢ − Bᵢ = Cᵢᵀβ + εᵢ
Same task, same checkpoint. No-intercept main-effect OLS; coefficients guide selection and are not independent causal guarantees.
Read the motivation, literature map, chunk definition, and full algorithm →
02 / EVALUATION
Stronger performance,
with the guidance removed.
All four conditions use the same 400 test tasks, with one completed rollout per task. Strict reward is the training objective; success requires full credit.
| Model | Guidance | Strict reward | Success |
|---|---|---|---|
| Base | None | 0.5422 | 51.75% |
| Base | Initial skill S₀ | 0.5663 | 54.50% |
| Iter80 | None | 0.8501 | 84.25% |
| Iter80 | Final skill Sₜ | 0.8671 | 86.25% |
139 tasks newly succeed
Comparing base and Iter80 without skills: 139 failures become successes, 9 successes regress, and 252 tasks keep the same outcome.
The final skill adds 2 points
Its success gain has a 95% interval of −0.50 to +4.75 points. A plain-GRPO control and repeated training seeds are needed to isolate the benefit of co-evolution.
03 / TRAINING DYNAMICS
Follow the learning process.
80 rollout steps, four 20-step rounds. 20,480 generated episodes; 10,240 admitted to the trainer after filtering.
Light: raw · Solid: 5-step mean
Reward covers all completed, scored generation, including filtered groups. Zero-variance drop rate measures discarded groups, not the prevalence of equal-reward groups. Public data cover four completed rounds, steps 1–80.
BEHAVIOR EXPLORER
Shorter generation.
A more complex interaction story.
20,480 training episodes and 3,600 evaluations. On fixed validation tasks, tokens per turn fall 43.5%; the paired change in model steps remains uncertain. Explore the outcome and weighting behind each mean.
Inspect numerical values
Training uses re-encoded saved output; evaluation uses API completion tokens. Each episode contributes one per-turn average. Outcome groups change with training. Product option refreshes do not count as new page visits.
W&B · OPTIMIZATION
Inside 160 optimizer updates.
Four finished runs, complete history, and 3,118 numerical matches with local logs. Entropy declines across rounds; PPO KL, clipping, gradient norms, and throughput add context to reward and behavior.
04 / SKILL EVOLUTION
Inspect the guidance
at each checkpoint.
Chunks can be retained, rewritten, added, or retired. Their contribution is specific to a checkpoint and mask distribution. The skill bank need not shrink monotonically.
The original Chinese skill text is shown verbatim in both languages. Scores are selection coefficients, not confidence-adjusted estimates.
Explore the failure funnel and contribution history
05 / REPRODUCIBILITY
From evidence to artifacts.
The public bundle includes step metrics, group summaries, paired task outcomes, contribution estimates, and skill texts. Rebuild the analysis without a GPU or model API.
Source repository ↗Runtime · GRPO · Trace2SkillExperiment report ↗Protocol, results, limitationsAnalysis dataset ↓JSON · 80 steps · 4 roundsFigures & provenance ↗SVG · PNG · CSV · SHA-256
python -m pip install -e ".[analysis,site]"
python scripts/analyze_training.py
python scripts/analyze_behavior.py
python scripts/analyze_wandb.py
python scripts/plot_training.py
python scripts/plot_behavior.py
python scripts/plot_wandb.py
python scripts/build_site.py
python scripts/check_publication.pyInterpretation & unfinished comparisons
- One training run. No plain-GRPO, fixed-skill, no-mask, or repeated-seed control is included in this release.
- Initial S₀ was recovered from frozen recorded contexts; its recovery did not use test outcomes and is not a newly completed calibration run.
- In the base-model ranking appendix, top4 success is 51.75%, bottom4 55.00%, and one random4 subset 54.25%. This does not establish ranking validity; four planned random subsets were not run.
- Reported scores use the local ShopSimulator protocol and frozen persona split; they are not a direct benchmark comparison against the upstream paper.