Experimental protocol and runtime

This report covers the frozen Qwen3.5-4B Iter80 experiment. It distinguishes the method proposal, the implemented contracts, and the settings actually used by this run. The preceding task, environment, and reward chapter explains the shopping problem, personal profile, simulator, tool interface, and reward construction with a complete episode; this chapter fixes the experimental conditions layered on top of that environment.

Environment identity and scoring contract

All runs use ShopSimulator single-turn personalized shopping and the same frozen persona split: 3,726 train, 400 validation, and 400 test tasks. The environment is included as an ordinary ShopSimulator/ directory rather than a separately versioned repository dependency.

The run uses environment version shopsimulator-paper-aligned-v8, observation contract shopping-observation-v7, and action protocol openai-function-tools-repair-v3. Strict reward (r_strict, also exposed as reward) is the optimization target. Success means full credit, not merely reaching a terminal purchase page. Type, attribute, option, and price components help diagnose partial matches. Field meanings, the \(R_{strict}\) versus \(R_{loose}\) choice, and length-overflow terminals are in the task, environment, and reward chapter; the implementation contract remains in the environment document.

A model turn can fail to produce a valid tool call. The runtime supplies protocol feedback and may ask for repair without executing an environment action. An executed but invalid environment action is a separate event. A purchase can terminate successfully at the protocol level while failing the task's reward criteria.

Frozen settings

Item	Released run
Policy	Qwen3.5-4B; final evaluation identity `qwen35-4b-iter80`
Schedule	Four rounds × 20 rollout steps
Siblings	Eight trajectories per group; shared mask
Generated / admitted	32 / 16 groups per rollout step
Global batch	64 episodes; two optimizer updates per rollout step
Learning rate	Constant 1 × 10⁻⁶ in both optimizer parameter groups, verified across all W&B updates
Guidance	q fixed at 0.20; contribution-mapped rho in [0.20, 0.90]
Skill budget	At most 10 active chunks; at most 6 online candidates per round
Gate	400 bare + 400 randomly masked trajectories at each checkpoint
Sampling	Temperature 0.6, top_p 1.0, thinking enabled
Limits	4,096 generated tokens per turn; 30 model steps per episode
Seeds	`send_seed=false`; matching task seed is not shared decoder randomness

The launcher defaults to four GPUs, trainer tensor parallelism 2, and rollout tensor parallelism 1. These are execution settings rather than claims of hardware-independent cost. Training uses slime and SGLang extension points; the policy runtime and general evaluator share the shopping contract. Exact current commands and topology constraints are in the training guide.

Three kinds of evaluation

Training generation follows the current policy, sampled training tasks, and changing assistance masks. It includes groups subsequently filtered out. It is useful for diagnosing training behavior but is not an independent held-out performance estimate.

Validation uses the fixed 400 tasks at the base, Iter20, Iter40, Iter60, and Iter80 checkpoints. Bare validation supports longitudinal behavior comparisons. Random-mask validation at the four trained checkpoints estimates chunk contributions and selects the next bank. These repeated selection uses matter when interpreting validation gains.

Final test compares Base free, Base + initial S₀, Iter80 free, and Iter80 + final Sₜ on the same 400 tasks, one completed rollout per task and condition. All four conditions have full coverage. Base free also contains 21 failed attempts that were subsequently retried; they are not extra test samples. Duplicate completed records are rejected, rather than silently taking the latest.

Evidence boundaries

The public analysis covers exactly four completed rounds, rollout steps 1–80. Group summaries distinguish 20,480 generated episodes from 10,240 trainer-admitted episodes. Diagnostic full-skill retries are excluded from those training sample counts.

The initial S₀ was recovered from historical frozen context, with uses_test_outcomes=false and validation_refitted=false. It is not represented as a freshly rerun Gate A satisfying every current provenance check. Online gates retain their saved paired observation tables, selected banks, and curriculum states.

For behavioral tokens, training traces save output text but not the original sampled token IDs. We re-encode that text with the official Qwen tokenizer and record its SHA-256, while marking the match to the historical tokenizer hash as unverified. Evaluation traces instead contain API completion_tokens. These two sources are labeled separately; neither is treated as a substitute for the exact token stream used by the optimizer.

Data and source traceability

Each analysis extract records input hashes. Public numeric episode and turn tables omit persona text, full conversations, and catalog bodies. Selected examples contain action categories and costs with an explicit selection rule. Rebuilding aggregate figures requires no GPU, model endpoint, W&B key, or raw trace directory. Refreshing the raw extraction requires the local experiment traces and tokenizer; refreshing W&B requires read access to those runs.

See task, environment, and reward, results, behavior, reproduction, and the original provenance notes.

实验协议与运行时

本报告覆盖冻结的 Qwen3.5-4B Iter80 实验，区分方法草案、当前实现契约与本次实际运行设置。上一章任务、环境与奖励已经用一条完整 episode 解释购物问题、personal 画像、模拟器、工具接口和奖励构造；本章固定叠加在该环境之上的实验条件。

环境身份与评分契约

所有运行均使用 ShopSimulator single-turn 个性化购物和同一份冻结 persona split：train 3,726、validation 400、test 400。本地环境以普通 ShopSimulator/ 目录随项目提供，不作为单独版本化的仓库依赖。

本次环境版本为 shopsimulator-paper-aligned-v8，observation 为 shopping-observation-v7，action protocol 为 openai-function-tools-repair-v3。优化目标是 strict reward（r_strict，也以 reward 暴露）。Success 表示获得完整奖励，并非仅成功抵达购买终局。类型、属性、规格和价格分量用于诊断部分匹配；字段含义、\(R_{strict}\) 与 \(R_{loose}\) 的取舍、以及长度超限终局见任务、环境与奖励，实现契约见环境文档。

一次模型生成可能没有合法工具调用。Runtime 可返回协议反馈，让模型修复，而不执行环境动作。已提交但被环境判无效的操作是另一种事件。购买动作在协议上成功终止，也可能没有满足任务奖励条件。

冻结设置

项目	本次运行
Policy	Qwen3.5-4B；最终评测身份 `qwen35-4b-iter80`
轮次	四轮，每轮 20 个 rollout step
Siblings	每组八条轨迹，共享 mask
生成 / 接纳	每 rollout step 为 32 / 16 groups
Global batch	64 episodes，每 rollout step 两次优化器更新
学习率	两个 optimizer parameter groups 均为固定 1 × 10⁻⁶，全部 W&B updates 已核验
指导	q 固定 0.20；rho 按贡献映射至 [0.20, 0.90]
技能预算	最多 10 个 active chunks，每轮最多 6 个在线候选
Gate	每个 checkpoint 400 bare + 400 random masked
采样	Temperature 0.6、top_p 1.0、开启 thinking
上限	每次生成 4,096 token；每 episode 30 模型步
Seed	`send_seed=false`，任务 seed 一致不代表解码共享随机数

Launcher 默认四卡，trainer TP=2、rollout TP=1。这是执行配置，不是硬件无关成本声明。训练通过 slime 与 SGLang 扩展点接入；policy runtime 与通用 evaluator 共享购物契约。当前命令与拓扑约束见训练指南。

三类评测

训练生成随策略、训练任务和辅助 mask 变化，包含随后被过滤的 groups。它适合诊断训练行为，但不是独立 held-out 性能估计。

Validation 在 Base、Iter20、Iter40、Iter60、Iter80 上使用固定 400 任务。Bare validation 支持纵向行为对比；四个训练后 checkpoint 的 random-mask validation 用于估计 chunk 贡献、选择下一版技能。解释验证集增益时必须考虑这种重复选择用途。

最终 test 比较 Base free、Base + 初始 S₀、Iter80 free、Iter80 + 最终 Sₜ。四个条件均在相同 400 任务上各有一条完成轨迹，覆盖完整。Base free 另有 21 次后来补测成功的 failed attempts，不作为额外样本。若同一任务出现重复 completed 记录，分析会拒绝而不是静默保留最新一条。

证据范围

公开分析严格覆盖四个完整轮次，rollout steps 1–80。组级汇总区分 20,480 条 generated episodes 与 10,240 条 trainer-admitted episodes，额外诊断 full-skill retries 不计入这些训练样本数。

初始 S₀ 来自历史冻结上下文恢复，uses_test_outcomes=false、validation_refitted=false。它不是一次重新运行、满足所有当前 provenance 要求的新 Gate A。在线 gates 保留逐任务配对表、selected banks 与 curriculum states。

行为长度方面，训练 trace 保存输出文本，但没有原始采样 token IDs。本次使用官方 Qwen tokenizer 对文本重新编码，记录文件 SHA-256，并明确标记无法核验其与历史 tokenizer hash 一致。评测 trace 使用 API completion_tokens。两种来源分别标注，都不替代 optimizer 实际使用的精确 token 流。

数据与来源追溯

分析摘录记录输入哈希。公开数值 episode / turn 表不含 persona 正文、完整对话或商品目录正文。案例只保留动作类别和成本，并公开选择规则。重建汇总图表不需要 GPU、模型 endpoint、W&B key 或原始 runs；刷新原始摘录才需要本地轨迹和 tokenizer，刷新 W&B 才需要 run 读取权限。

继续阅读任务、环境与奖励、结果、行为、复现和原始来源说明。