IntraSkill / ShopSimRL · Qwen3.5-4B · Iter80 · 4 rounds

Background and method: skill-model co-evolution

The project in one paragraph

IntraSkill proposes a training system in which an agent policy and an external procedural skill bank evolve together. We implemented the system for a personalized shopping agent in ShopSimulator and ran four complete training rounds with Qwen3.5-4B. The policy learns from GRPO rollouts, while a slower loop diagnoses unresolved failures, proposes reusable instructions, estimates each instruction's checkpoint-specific contribution, and updates the bank. The experiment produced a substantial improvement over the initial model under this system; it is an applied project study, not a complete causal demonstration that co-evolution outperforms every simpler RL alternative.

Why keep both a model and an external skill?

Agent skills are useful because many real tasks contain procedures that resemble operating manuals rather than permanent facts. A shopping agent should remember how to construct a query, verify evidence, select a variant, and check price constraints. The exact policy, interface, or business rule may change. Such guidance should remain readable, inspectable, and replaceable at inference time instead of being irreversibly buried in model parameters.

Pure external augmentation is also insufficient. Long or stale instructions consume context, can anchor exploration to yesterday's best procedure, and make the policy dependent on retrieval. Conversely, forcing every useful procedure into parameters discards one of the main advantages of skills: operators can audit and update them without retraining the model.

IntraSkill therefore treats skill content as a checkpoint-relative scaffold:

  • Keep external guidance that still helps. Mutable SOPs and details the current policy does not reliably execute remain in the bank.
  • Withdraw guidance that no longer contributes. Once a chunk becomes redundant, stale, or constraining, masking it creates room for unaided exploration and the gate may retire it.
  • Use good guidance to make sparse reward learnable. A hard task that is always failed gives a GRPO group no useful relative signal. Skill-assisted exploration can make some trajectories succeed, after which GRPO can reinforce the successful behavior.
  • Turn new failure modes into new guidance. Tasks that remain unsolved even with the full current skill reveal the most useful frontier for targeted analysis.

This lifecycle does not assume that every retiring chunk was literally stored in model parameters. A low estimated contribution can also reflect noise, redundancy, interactions, or insufficient power. We call the useful guidance remaining at a checkpoint the residual competence frontier.

Recent systems differ mainly in what they evolve, the unit they manipulate, and whether external guidance remains at deployment.

Direction Representative method Main mechanism Difference from IntraSkill
Trajectory-to-skill distillation Trace2Skill Holds the skill-using model fixed, proposes patches from success and failure traces in parallel, then consolidates them many-to-one into a portable skill. We use this style of evidence-grounded diagnosis for cold start and the slow loop, then place every deployable candidate behind a frozen-checkpoint intervention gate.
Skill-bank evolution during RL SkillRL Distills successful demonstrations and failure lessons into general and task-specific skills; after validation epochs, failure analysis adds or refines bank entries while the policy trains with retrieved guidance. Its principal unit is a retrieved skill entry. We use one scene-level shopping skill whose internal minimal chunks are masked, estimated, rewritten, retained, or retired.
Complete skill internalization SKILL0 Organizes skill files by task and skill category, associates each file with a dedicated validation sub-task, scores helpfulness as performance with the active skill minus performance without skills, and follows a decaying budget until no skill remains. Our target is not a zero-skill endpoint. We retire only chunks without positive residual contribution and preserve mutable SOPs that remain useful at inference.
Scaffold-to-parameter co-evolution EDGE Splits each GRPO group into experience-conditioned teacher rollouts and experience-free student rollouts. Their reward gap gates a pooled GRPO update and reverse-KL distillation; an experience bank expands and prunes online. The contexts inside an EDGE group intentionally differ. This gives a cheap gain estimate, but its pooled group baseline carries both policy sampling variation and scaffold exposure. IntraSkill keeps every sibling's prompt treatment identical and estimates chunk effects separately on a frozen checkpoint.
Unified skill lifecycle policy Skill1 Trains one policy to generate retrieval queries, re-rank skills, act with the selected skill, and distill new skills; task outcome supplies credit to all three stages. We do not train a skill writer or retriever. Our focus is a scene-level procedural bank, group-consistent policy learning, and auditable chunk-level interventions.

The table is a positioning map, not a benchmark comparison. The papers use different environments, models, rewards, and evaluation protocols. IntraSkill combines three ideas that are usually separated: skill-guided exploration, selective internalization through randomized omission, and evidence-based bank revision.

Why scene-level rather than product-category skills?

Personalized shopping contains many product categories, but the agent repeatedly performs the same underlying procedure: translate requirements and persona evidence into a query, inspect candidates, verify constraints, choose an option, and purchase. Category-specific skill files can therefore duplicate large parts of one workflow. IntraSkill assigns one skill bank to the entire shopping scene. Category facts may appear in a chunk's applicability condition, but category routing does not define the skill boundary.

The unit that evolves is the chunk inside that scene-level skill. This makes the intervention match the actual question: which reusable procedure still helps the current policy?

Framework: training intervals separated by skill gates

IntraSkill training timeline: individual GRPO rollout steps lead to a validation gate and a new skill bank, followed by the next block of rollout steps.

Cold start produces the initial bank (S_0). Training then advances in 20-step intervals. Each block in the figure is one GRPO rollout step: the group-consistent treatment supports a model update, while all-wrong groups accumulate as evidence for possible skill changes. The SkillBank remains frozen throughout the interval.

At the interval boundary, the checkpoint freezes. Full-skill retries, failure analysis, and compilation produce candidates; paired randomized validation then selects (S_{t+1}). Only after the gate does the new bank enter prompts for the next sequence of GRPO rollout steps. Model parameters can change at every step, whereas skill text changes only at these gates. Diagnostic rollouts never enter the training advantage.

The skill representation

A chunk is a minimal independent intervention

A chunk is not merely a short paragraph. It is the smallest unit that is meaningful to include or withhold in an experiment:

  1. Semantically complete: it expresses one reusable decision procedure, rather than half of a rule.
  2. Independently executable: a reader can act on it without another chunk supplying a missing step.
  3. Observable: its procedure and verification condition refer to information available in the normal agent observation.
  4. Conditioned: it states when it applies, so a local lesson does not become a universal shopping rule.
  5. Non-leaking: it excludes product IDs, private target facts, gold answers, and task-specific shortcuts.
  6. Weakly coupled: if two short instructions must always be executed together to have meaning, they are compiled into one chunk rather than entered as collinear regression factors.

Each chunk uses four fields: Applies when, Procedure, Verify, and Avoid. Here is a verbatim frozen cold-start example:

搜索查询构建 - 显式约束整合

  • Applies when: 任务包含明确属性约束（品牌、颜色、尺寸、价格、功能等）时
  • Procedure:
  • 将所有核心属性约束整合为单一搜索查询词
  • 将价格约束、产品形态、核心成分等关键要素直接融入搜索词
  • 避免使用泛化搜索词或分步搜索
  • Verify: 验证搜索结果首屏是否出现匹配所有核心约束的商品
  • Avoid: 搜索词过于宽泛或遗漏关键约束导致结果不相关

This chunk is independently maskable because it contains its own trigger, action sequence, check, and failure warning. The report quotes frozen text rather than rewriting the released artifact. Other chunks cover persona evidence, candidate comparison, option selection, and price verification.

Cold start: from trajectories to (S_0)

Cold start turns broad base-model experience into a small set of testable interventions:

  1. Run the base agent on training tasks without skills and retain labeled success and failure trajectories.
  2. Convert successes into reusable behavior cards. For failures, use a gold-aware audit to replay the trace, find the earliest causal divergence, and test a minimal repair in a live environment.
  3. Keep evidence cards separate from deployable instructions. A card is one trajectory-local hypothesis, not a chunk candidate.
  4. Consolidate many cards by mechanism. Repeated, independently supported lessons become a small set of canonical chunks; idiosyncratic tips and duplicates are removed.
  5. Run Gate A on a fixed validation split. Each of the 400 tasks has a bare rollout and a randomized masked rollout over the 16 draft chunks. Paired-delta OLS estimates main effects, and positive top-(K_{ ext{init}}) selection retains at most ten chunks.

The released (S_0) contains ten chunks. It was recovered from frozen recorded skill contexts and did not use test outcomes for selection; because the matching historical bare-validation identity is unavailable locally, the recovery is not presented as a newly rerun Gate A. The provenance note documents that boundary.

Fast loop: guided exploration and selective internalization

Each GRPO group contains eight trajectories for one task. The runtime first draws one treatment for the whole group:

  • with probability (q=0.20), the group is completely skill-free;
  • otherwise, each active chunk is sampled once using its contribution-dependent inclusion probability (ρ_j ∈ [0.20,0.90]).

All eight siblings receive the same task, prompt, skill mask, curriculum identity, and environment protocol. They differ only through policy sampling. This group-consistent rule ensures that the group-relative advantage compares actions under one information condition. It does not use with-skill versus without-skill reward differences as GRPO advantages.

Across groups, however, the policy experiences full, partial, and empty guidance. A useful chunk can push exploration into successful regions that the bare policy rarely reaches. Within that assisted context, sibling reward variation teaches the model which sampled actions worked. Skill-free and partially masked groups require the policy to reproduce useful behavior with less support. In this way masking supplies an internalization pressure without making the prompt inconsistent inside a GRPO group.

The fast loop updates model parameters only. It cannot admit or delete chunks. Between gates, (q), (ρ), chunk text, order, and curriculum hashes remain frozen. Diagnostic retries never enter the trainer.

Dynamic filtering

Each rollout step generates 32 groups and admits 16 after filtering, for 128 training episodes. A global batch of 64 produces two optimizer updates per rollout step. Across the four released rounds, 20,480 episodes were generated and 10,240 entered the trainer. Technical failures are replenished; zero-variance groups are deprioritized but can be used to fill the batch. These details affect the denominator of training curves but do not change the group-consistent treatment rule.

Slow loop: turn unresolved failures into candidates

The slow loop starts only from a normally scored all-wrong group: all eight siblings failed. It then runs one additional trajectory with the full current skill and the same frozen model.

  • If the full-skill retry succeeds, the current bank already contains a usable procedure. This is evidence of a possible execution or internalization deficit; it does not justify a new chunk.
  • If the full-skill retry also fails, its trajectory goes to the Failure Analyst. Using the original masked group here would confuse “a needed chunk was omitted” with “the bank lacks a procedure.”

The Failure Analyst may use privileged gold information to investigate, but gold is firewalled from the deployable skill. It replays the full-skill trace, locates the earliest causal branch, proposes the smallest observable repair, and tests that repair in the environment. It returns one of three outcomes:

  • NO_PROPOSAL when the failure cannot support a reusable intervention;
  • ADD when a verified mechanism is absent from the current bank;
  • REWRITE when an active chunk addresses the right problem but has an incomplete condition, procedure, or verification rule.

An ADD needs at least one real terminal counterfactual success. A REWRITE must name its active target. At round end, the compiler merges eligible cards many-to-one, deduplicates mechanisms, and emits at most six canonical candidates. The four released rounds produced 6, 6, 6, and 4 candidates from 61, 22, 20, and 11 eligible cards.

Paired randomized validation: estimate a chunk, not a whole prompt

Every 20 rollout steps, the model checkpoint freezes. On the same 400 validation tasks, the evaluator runs a bare trajectory and the gate runs a trajectory under a randomized mask over all current and candidate versions. Model identity, task, sample, decoding settings, prompt outside the intervention, environment, and action protocol must match.

For task (i), let (B_i) be bare reward, (R_i) masked reward, and (C_i) the actual version-inclusion vector. The gate fits:

delta_i = R_i - B_i
beta_hat = argmin_beta sum_i (delta_i - C_i^T beta)^2

There is no fitted intercept, no centering of the treatment matrix, and no subtraction of a global bare mean. An all-zero randomized mask remains a valid observation because the two stochastic trajectories can still differ. The strict reward is the main outcome; each complete reward component receives its own paired fit.

Slot-wise replacement

An ordinary current or ADD slot uses absent/present states. A REWRITE creates a replacement family with mutually exclusive absent / old / new_1 / ... / new_l states. Old and new versions of the same logical chunk never co-occur in one prompt.

Selection happens in two steps:

  1. choose the largest-coefficient version inside each logical family, with exact ties favoring the old version;
  2. rank family winners and ordinary chunks together, retaining only strictly positive coefficients up to (K \le 10).

A losing rewrite version cannot consume another budget slot. A current chunk with a nonpositive or budget-excluded coefficient retires without a separate DELETE proposal. The resulting (S_{t+1}) determines the next round's inclusion probabilities.

The coefficient is a checkpoint- and mask-distribution-specific additive main-effect estimate. It is not a permanent skill-quality score, and it does not identify interactions. The gate cleanly separates skill selection from GRPO, but repeated use of the same validation split still creates selection dependence.

What this project establishes

The four-round run demonstrates a complete operational cycle: guidance assists exploration, the policy trains under randomized omission, unresolved failures produce new candidates, and chunk-level interventions revise the external bank. The trained bare policy performs substantially better than the initial model on the frozen test set, while the final skill provides a smaller additional inference-time gain.

The project does not include plain-GRPO, fixed-(S_0), no-mask, no-online-gate, alternative-(q), or repeated-training-seed controls. It therefore supports the statement “this skill-model training system produced a stronger policy in one applied shopping experiment,” but not the stronger causal claim “co-evolution is proven superior to ordinary RL.” The experimental protocol, results, and behavior analysis keep those evidence boundaries explicit.

For implementation-level contracts, see the training guide, Trace2Skill design, and paired validation specification.

← Project overviewTask, environment, and reward →