Task, environment, and reward: personalized shopping in a local simulator
ShopSimRL studies a single-turn shopping agent. Each episode starts with a purchase request, a user profile, and an empty search page. The agent must turn that information into queries, inspect local catalog results, choose a product and every required variant, and purchase within 30 model turns. A private evaluation target scores the purchase with the paper reward; episodes that never purchase receive zero. The environment is a deterministic local simulator: it does not browse a live commerce website, spend money, or contact a real seller.
What constitutes one task?
A task is more than a product name. It combines a visible shopping request with a richer user_persona, while the evaluator retains a private structured goal. This creates a personalized decision problem: the request states the immediate need, and the profile supplies evidence about preferences or context that the shorter request may omit.
| Layer | Example content | Visible to the agent? | Purpose |
|---|---|---|---|
task_instruction |
“Find a mobile food cart for dumplings, buns, oden, and fried stinky tofu, priced from 1,000 to 1,200 yuan.” | Yes | The immediate purchase request |
user_persona |
Demographics, region, transaction habits, category and attribute preferences, tags, and recent behavior | Yes | Personalization evidence |
observation / observation_state |
Current page, products, prices, attributes, options, feedback, and legal actions | Yes | The current shopping state |
| Current tool schema | search(query) or click(value) with page-specific choices |
Yes | The only action channel |
| Full instruction, target attributes, target option, and budget fields | Structured facts used to score the purchase | No | The private evaluation target |
| Gold ASIN | Original product associated with the task | No | Diagnostics only; it is not a required exact match |
In persona mode, task_instruction is the shorter instruction_simple. The runtime adds the public user_persona to the system context and removes any internal __reasoning__ field. It never exposes the full instruction, target ASIN, goal attributes, goal options, or reward calculation before purchase. Assisted conditions may also add a selected skill context; that changes guidance, not the task or page state.
What does the personal information look like?
The profile is a nested JSON record rather than a natural-language biography. Its six useful groups are:
| Profile group | Typical fields | How it can matter |
|---|---|---|
| Demographics | Age band, gender, membership and consumption tier | Weak background context; often irrelevant to a specific purchase |
| Region | Province, city, district, time zone | Geographic context when a product or use case is location-sensitive |
| Transaction features | Average order value, repurchase rate, promotion sensitivity, payment preference | Price and shopping-style evidence |
| Interest preferences | Brands, categories, materials, colors, functions, style, size ranges | Candidate ranking and attribute interpretation |
| Behavioral features | Recent searches, viewed or saved products, activity times and visit frequency | Stronger evidence of current intent |
| User tags | Short summaries such as “night-market operator” or “equipment upgrade” | High-level context, which still needs support from observable product facts |
The following is an editorial excerpt from the profile attached to Task 3538. Identifier and timestamp fields are omitted here; the actual rollout received the complete public profile.
{
"人口属性": {
"年龄段": "35-44",
"性别": "男",
"会员等级": "黄金会员",
"消费等级": "中"
},
"地区信息": {"省份": "浙江省", "城市": "杭州市"},
"交易特征": {
"平均客单价": 356.25,
"是否促销敏感": false,
"支付方式偏好": "支付宝"
},
"兴趣偏好": {
"商品属性偏好": {
"价格区间": {"最小值": 800, "最大值": 2000},
"功能": ["多炉头", "可移动", "节能", "多功能"],
"材质": ["不锈钢", "铝合金"]
},
"类目偏好": {"餐饮设备": "高"}
},
"行为特征": {
"最近14天搜索关键词": [
"配备煮面炉的流动小吃车",
"带煤气灶和三个锅的手推小吃车"
]
},
"用户标签": ["夜市经营者", "设备升级需求", "多品类小吃", "中端预算"]
}
The agent should treat profile entries as evidence, not as universal hard constraints. Explicit task requirements dominate. Empty size fields or an unrelated brand preference should not be forced into the query; recent searches for a three-pot gas cart, by contrast, are directly useful for this food-cart request.
How the environment is constructed
The environment wraps a frozen Chinese commerce snapshot in a local search engine and page state machine. The raw archive contains 23,421 products. Data preparation removes 695 products whose price-changing option axes cannot be resolved safely, leaving 22,726. The personalized subset contains 4,526 tasks, split into 3,726 train, 400 validation, and 400 test tasks. Linked records are assigned together so that duplicated or shared persona/product combinations cannot cross held-out boundaries.
Search uses a deterministic SQLite FTS5 multi-field BM25 index over title, brand, category, model, attributes, observable options, and bullet points. A stable ASIN tie-break makes repeated queries reproducible. The simulator then converts search and click actions into in-memory page transitions. HTML remains an internal compatibility mechanism; the agent receives only shopping-observation-v7, a deterministic text rendering of the canonical observation_state.
task + persona
↓
runtime prompt ── standard function call ──→ HTTP adapter + leased session
↑ ↓
agent observation ← deterministic renderer ← page state machine
↙ ↘
SQLite BM25 search product / option state
The environment version is shopsimulator-paper-aligned-v8. Its construction and frozen hashes are documented in the environment contract.
Pages, observations, and tools
The model never manipulates an arbitrary browser DOM. At every turn, the runtime derives its tools from the current structured page state.
| Page state | Public observation | Available actions |
|---|---|---|
| Search home | Search capability | search(query) |
| Search results | Query, page/rank range, up to 20 products, titles, ASINs, price ranges, brands, categories, key attributes | click(value) for a listed ASIN, pagination, or return |
| Product detail | Product facts, available variant axes, current selections, missing axes, resolved selected price when known | click(value) for an axis-qualified option or navigation |
| Purchase-ready detail | options_complete=true; every required axis has a valid selection |
The same actions plus click("buy now") |
| Terminal | Purchase result and no interactive elements | None |
Only one tool call is allowed per model turn. click(value) uses an enum generated from the current page, so the model must copy a legal value exactly. Invalid calls do not alter the page, consume a step, and return explicit feedback such as unavailable_action, already_selected, or incomplete_options. Protocol errors, such as emitting no valid tool call, are handled separately by the runtime and can trigger a repair turn without executing an environment action.
A complete example: Task 3538
This example is the frozen Iter80 skill-free validation rollout already selected in the behavior analysis as a failure-to-success case. It succeeds in four model turns.
Visible request: “查一款可移动的小吃车，我需要制作煎饺、煎包、关东煮和油炸臭豆腐，能接受的价格在1000到1200元之间。”
The profile adds relevant cues: the user is tagged as a night-market operator, strongly prefers catering equipment, wants mobile and multi-function equipment, and recently searched for a cart with a gas stove and three pots.
| Turn | Agent action | Environment response | Decision made |
|---|---|---|---|
| 1 | search(query="可移动小吃车 煤气灶 三个锅 煎锅 关东煮 油炸 1000-1200元") |
150 results; the first page exposes ranks 1–20 | Integrate explicit food and price constraints with the profile's gas/three-pot evidence |
| 2 | click(value="824100258497") |
Opens rank 5: a mobile cart for fried food, oden, buns, dumplings, stinky tofu, and a gas stove | Verify product-level type and attributes before choosing a variant |
| 3 | click(value="颜色分类=三锅小吃车煮面炉＋关东煮＋油炸") |
selected_price=1100, missing_option_axes=[], options_complete=true; “buy now” becomes available |
Select the exact axis-qualified configuration and verify its actual price |
| 4 | click(value="buy now") |
Terminal purchase, reward=1.0 |
Commit only after all option and budget checks pass |
The purchase receives r_type=1, r_att=1, r_option=1, and r_price=1, so r_strict=1 and r_success=1. The chosen item also has target_asin_match=true, but that field is diagnostic: another product can succeed if its public facts and chosen option satisfy every scored requirement.
This trajectory also shows why the environment is long-horizon despite having only two tool names. The agent must construct a discriminative query, compare returned evidence, move into the right product, use the exact option namespace, verify the selected SKU price rather than the displayed range, and delay purchase until the page exposes a valid terminal action.
How the reward is constructed
Positive task reward is calculated only after click("buy now"). Unpurchased terminals receive zero and do not get partial credit for being “almost successful.” The scalar reward used in training and primary evaluation equals r_strict. reward_detail still reports r_loose, r_success, and the components for diagnosis; it does not change the optimization target.
What each component means
The private evaluation target scores the purchased product and the selected options. The agent never sees these targets or the reward formula before purchase.
| Component | API field | What it checks |
|---|---|---|
| \(R_{type}\) | r_type |
Category path and title terms |
| \(R_{attribute}\) | r_att |
Required attribute list |
| \(R_{option}\) | r_option |
Required option values |
| \(R_{price}\) | r_price |
Task budget and selected SKU price |
Scoring:
- \(R_{type}\): 1 if at least two category nodes intersect or title-term overlap exceeds 0.2; 0 if the titles share no terms; 0.1 if overlap is below 0.1; otherwise 0.5. The frozen data has no WebShop
querytag, so that field is not scored. - \(R_{attribute}\): hits \(M_{att}\) divided by the number of required attributes \(N_{att}\). A hit may come from a fuzzy match against product attributes or a literal match in the title, bullets, or description. Score 1 when there are no required attributes.
- \(R_{option}\): matching ratio \(M_{opt}/N_{opt}\) between selected values and required values; 1 when there are no required options. The comparison uses option values, not axis names.
- \(R_{price}\): 1 when no budget is declared. If a budget exists, the score is 1 only when the verifiable selected-SKU price does not exceed the upper bound; otherwise 0. A declared budget with an unverifiable price is also 0, with
price_verifiable=false.
The budget comes from task annotation or instruction parsing; it is never inferred from the Gold product price. A displayed price range is not a substitute for the selected SKU price.
Two diagnostic fields stay outside the three reward formulas:
target_asin_match: whether the purchased ASIN equals the Gold product that generated the task. A non-Gold item can succeed if it meets every scored requirement; buying the Gold ASIN with the wrong option or an over-budget SKU cannot.price_verifiable: whether the selected combination’s transaction price was resolved by the variant-price module.
The Task 3538 purchase scores 1 on every component, so r_strict=1 and r_success=1, with target_asin_match=true. The last field only records that the original product happened to be bought; it is not required for success.
Two aggregations: \(R_{loose}\) and \(R_{strict}\)
Let \(N_{att}\) and \(N_{opt}\) be the numbers of required attributes and options, and \(M_{att}\) and \(M_{opt}\) the corresponding hit counts:
\(R_{loose}\) is WebShop-style additive coverage: type score gates the average of the remaining hits, so one failed constraint can still leave a high partial score. \(R_{strict}\) is a multiplicative bottleneck: if any component approaches 0, the total approaches 0. r_success is 1 only when all four components equal 1. A syntactically valid purchase can therefore terminate with zero or partial diagnostic reward. Buying the Gold ASIN with the wrong option can still have target_asin_match=true and r_option=0, which yields r_strict=0 while r_loose may remain around 0.75.
Why the project uses \(R_{strict}\)
The ShopSimulator paper compared the two RL objectives in this shopping setting. ShopSimulator found that training with multiplicative \(R_{strict}\) outperformed additive \(R_{loose}\) across all task scenarios, with a particularly clear advantage on attribute and option matching. The bottleneck concentrates optimization on the weakest constraint; models trained with \(R_{loose}\) tend to optimize coverage or “completing a purchase,” with less pressure on per-constraint accuracy.
This project did not rerun an \(R_{loose}\) versus \(R_{strict}\) ablation. From environment v2 onward, the API-level reward switched from r_loose to r_strict, and IntraSkill follows that choice: GRPO, the validation gate, and the public headline metric all use r_strict. r_success is the binary full-credit criterion; r_loose remains only in reward_detail.
Unpurchased terminals: action limit and length overflow
Only a purchase invokes the formulas above. Every other terminal returns the standard zero vector: r_loose, r_strict, r_success, and all four components are 0.
| Termination | When it fires | Reward | Training and evaluation |
|---|---|---|---|
purchase |
click("buy now") after options_complete=true |
Computed as above | Scored trajectory |
action_limit |
30 model turns without a purchase | All zeros | Scored policy failure |
generation_length |
A model turn hits the generation-length cap before a unique valid tool call | All zeros | Scored policy failure; no resample and no protocol repair |
This run allows at most 4,096 generated tokens per turn. When the model returns finish_reason=length without a unique legal tool call, the runtime records policy_failure.code=generation_length and immediately asks the environment to terminate for that reason. This is distinct from a protocol error: a missing tool call, a wrong number of calls, or a malformed argument consumes one turn and may be repaired on the same page, without changing the page or ending the episode. A submitted click that is not in the current enum is still sent to the environment, which returns feedback such as unavailable_action and counts the step. Length overflow is treated as the policy failing to emit an executable action within budget, not as infrastructure failure. The trajectory therefore enters the GRPO and evaluation denominators; slime may mark the corresponding Sample as truncated, but it is not removed from the scored set.
There is no finish action, voluntary abstention reward, or “no-progress” terminal. A further step on an already terminated session is an error.
The next chapter records the exact training, sampling, validation, and test settings layered on top of this environment. See the experimental protocol and the implementation-level runtime contract.
Scope and limits
- The catalog, search results, prices, and product pages come from one frozen snapshot; there is no live inventory, shipping, seller interaction, or payment.
- The current dataset lacks useful reviews and its full descriptions duplicate titles, so those legacy page actions are intentionally absent.
- The experiment uses single-turn persona tasks. The agent cannot ask the shopper a clarification question or revise the request through dialogue.
- Persona records contain noisy or irrelevant fields. The task tests whether the model can select useful evidence, not whether every profile field appears in its action.
- Search quality and the page state machine are part of the task difficulty. They are fixed across compared model conditions, but the results do not directly transfer to arbitrary commercial websites.