Turning Database Grading into RL Rewards: Publish-Teaching Brokers with ThinkingBox

That’s the observe as a lot as our first weblog submit, The Agent Said It Was Accomplished. The Database Disagreed. ThinkingBox-Teaching is now accessible on GitHub.

We constructed ThinkingBox to guage an agent’s work by evaluating the last word database state with the anticipated finish consequence. We then used the equivalent type of executable signal as a reward on teaching duties open air ThinkingBox-Bench. For Qwen3.8-27B, widespread success rose from 51.70% to 60.78%, whereas duties completed precisely on all twenty makes an try rose from 38 to 107 with 50 GRPO steps.

Decide 1. ThinkingBox turns executable checks of terminal database state into rewards for post-training brokers on duties open air ThinkingBox-Bench.

This is usually a joint weblog by Microsoft and Hugging Face, specific due to Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face and our former intern Zhuochun Li (School of Pittsburgh) for co-authoring/reviewing efforts.

We constructed ThinkingBox to catch brokers that say they’re executed once they aren’t.

Each of its 507 duties in ThinkingBox-Bench (our benchmark dataset) palms an agent an precise backend and a set of devices, lets it work, then checks what really modified throughout the database. Inside the first submit, we caught a model closing a purchaser’s ticket as resolved whereas the supplier exception behind it was nonetheless open. 9 competent-looking software program calls, one flawed database row.

Then we noticed what we had constructed. For a lot of duties, an executable take a look at can return 1 or 0 immediately from the last word database state. That is not solely a grader. This typically is a reward function.⁺

So we constructed a training pool from ThinkingBox-format duties open air the benchmark and used their executable outcomes as rewards. The catch was choosing which duties are acceptable for agent teaching: most duties are each already solved or hopeless for the model in entrance of you, and every prepare it nothing.

Subject is policy-relative: a job is hard for the current model solely until that model learns it.

We title this offline policy-relative downside. We estimated each job’s success chance as quickly as using the underside protection sooner than teaching and saved the chosen teaching pool mounted all by the run.

What occurred after teaching

We used offline policy-relative downside to assemble the teaching pool, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. To recap, we take into account each of the 507 benchmark duties 20 cases. cross@1 measures widespread success all through all 10,140 makes an try. cross@20 measures the share of duties solved on the very least as quickly as. Seen 20/20 measures the share of duties that cross every recorded strive.

Desk 1. Qwen3.8-27B outcomes sooner than and after 50 steps of GRPO on the offline-selected teaching pool.

Teaching sharply elevated repeatable success, with a small decline in at-least-once safety. The rest of the submit explains how we chosen observe duties, constructed the reward loop, and measured that tradeoff.

What does that enchancment appear to be on one job? A traveler requests a room downgrade, nevertheless agency protection requires supervisor approval first. Sooner than teaching, the agent modified the reserving and issued refunds with out approval in 15 of 20 makes an try. After RL fine-tuning, it preserved the reserving and price state in all 20.

Decide 2. One firm hotel-booking job, 20 makes an try per model. The underside Qwen3.8-27B model modified the reserving with out required supervisor approval in 15 makes an try. The RL fine-tuned model preserved reserving and price state in all 20; all anticipated information matched in 19.

Contents

1. What the benchmark found, in sixty seconds
2. Outcomes: the dependability column moved
3. The place the constructive features landed
4. What dependability costs
5. The teaching loop
6. Offline policy-relative downside
7. What earns a reward
8. Run the teaching stack
9. The place this goes subsequent

In a rush? The headline consequence’s above. The implementation is in Run the teaching stack.

What the benchmark found, in sixty seconds

ThinkingBox-Bench grades brokers on the database information they depart behind. Each of its 507 duties begins an precise backend, palms the agent a set of MCP devices and a simulated LLM particular person with private context, and on the end runs executable checks in opposition to the terminal database state. Checks are in opposition to the backend file that have to be acceptable, not the actual software program calls. This incentivizes the model to search out presumably new decision paths fairly than “rediscover” the human-backed decision, which may be suboptimal albeit acceptable.

In our first submit we ran eighteen fashions by way of that benchmark twenty cases each and reported a distinct segment that headline accuracy hides. Kimi-K3 solved 93.89% of duties on the very least as quickly as nevertheless completed solely 68 of 507 duties (13.41%) on all twenty recorded makes an try. Claude Opus 5 inverted it: narrower safety, way more consistency.

And larger or greater model did not shut it. Claude Opus 5.5 scored 0.66 share elements bigger than Claude Opus 5 on cross@1, however handed exactly the equivalent number of duties, 241, on all twenty makes an try. That purchase in headline accuracy bought zero further dependability.

If you happen to’re choosing a model for work that touches manufacturing database information, cross@1 should not be be taught alone. Seen 20/20 reveals which duties succeed on every strive, which is nearer to what manufacturing reliability requires.

The executable terminal-state verdict was moreover the teaching reward. Sooner than unpacking the teaching loop, this is what modified on the sealed benchmark.

Outcomes: the dependability column moved

After teaching Qwen3.8-27B, we evaluated the chosen step-50 checkpoint on the 507-task benchmark it had on no account expert on. The 507 duties characterize an out-of-distribution check out set: their domains, databases, and most MCP devices weren’t encountered all through RL teaching.

Desk 2. ThinkingBox-Bench cross@1 (%) by space, for chosen fashions from our first weblog’s Desk 2 and our fine-tuned model for varied domains and normal. Seen 20/20 numbers are over the entire benchmark with 507 duties. Each model is evaluated on all 507 duties for 20 repeated trials. Daring marks the column chief; underline marks the runner-up.

The headline consequence’s the observed-20/20 row in Desk 1. A 27B model that initially completed 38 (7.5%) of ThinkingBox-Bench on every strive now completes 107 (21.1%) on every strive, nearly thrice. For comparability, seen 20/20 for Kimi-K3 is 68, GPT-5.4 with 128, GPT-6 Astra has 231, and Claude Opus 5.5 leads with 241.

Frequent success and repeatability produce utterly totally different orderings. In Desk 2, we uncover that GPT-5.6 Sol scored barely bigger on cross@1 than the RL fine-tuned Qwen3.8-27B (61.91% versus 60.78%) nevertheless handed fewer duties on all 20 recorded makes an try: 82 versus 107. Claude Sonnet 4.6 scored lower on cross@1, at 59.19%, however handed virtually the equivalent number of duties every time, at 102. Neither metric alone captures the entire picture.

The seen pattern is in accordance with sharpening: repeatable success rose whereas at-least-once safety fell barely. That’s in-line with prior work. One rationalization is that chance distribution moved in the direction of trajectories that work repeatedly whereas some low-probability paths disappeared.

Whole, the model did not develop into broader. It grew to turn out to be further repeatable. The next question is the place these constructive features landed.

The place the constructive features landed

Decide 3. Space-level base scores, RLFT scores, and percentage-point changes. Inexperienced change cells current constructive features and purple cells current declines; blue shading signifies the ranking diploma inside each metric.

Cross@1 and seen 20/20 rose in all 5 domains. Cross@20 was mixed: it elevated in neobank and consulting, nevertheless declined in retail, journey, and auto insurance coverage protection. Auto insurance coverage protection had crucial breadth loss (−7.0 elements), whereas retail had crucial observed-20/20 purchase (+24.5 elements).

What dependability costs

The first submit priced consistency using primary math: divide the estimated value of the entire 20-run advertising marketing campaign by the number of duties the model handed every single time.

Worth per dependable job = estimated value of the 507 × 20 advertising marketing campaign ÷ duties passing 20/20

Making use of the equivalent value math to the fine-tuned checkpoint’s measured token utilization:

Desk 3. Estimated evaluation value and dependability for base and RL fine-tuned Qwen3.8-27B. Worth per worthwhile strive divides complete estimated advertising marketing campaign value (507 x 20) by the number of passing makes an try.

At $8.00 per dependable job, the fine-tuned model acquired right here close to GPT-5.4 ($6.80), GPT-6 Astra ($7.45), and Claude Opus 5.5 ($7.80) throughout the companion comparability.

The one-attempt view

Decide 4 prices value per worthwhile strive in opposition to cross@1.

Decide 4. The golden arrow traces Qwen3.8-27B base pareto value place to the RLFT (diamond): cross@1 rises from 51.70% to 60.78% whereas estimated value per worthwhile strive falls from $0.177 to $0.139. Ringed circles mark the Pareto value frontier.

The switch areas the fine-tuned checkpoint close to GPT-5.6 Sol, although GPT-5.6 Sol stays cheaper and barely bigger on cross@1. 4 fashions priced in our paper fall open air the plotted fluctuate.

The buck values are a comparative serving index, not invoice or cloud funds. Serving the fine-tuned checkpoint might be the equivalent as the underside model. The calculation consists of model inference value, and excludes teaching run and policy-probe value for curating the teaching dataset.

These outcomes inform us what moved. The next sections make clear the teaching loop and why selecting the suitable duties points in all probability probably the most.

The teaching loop

Decide 5. The six ranges, now in context: candidate duties feed in, the first three run as quickly as offline, the ultimate three repeat for every group throughout the teaching run. The expert checkpoint is evaluated in opposition to ThinkingBox-Bench.

Selection runs as quickly as, offline, in opposition to the underside protection model, i.e on this case Qwen3.8 27B; teaching then makes use of that mounted pool that has been curated. The dashed amber arrow in Decide 5 marks the path we did not take: re-probing downside by the teaching run.

Every rollout is a whole episode, not a textual content material completion. It should get a up to date backend, an isolated MCP session, and GPT-5 Chat as a result of the simulated particular person. Most rewards come from deterministic terminal-state checks; a minority of slender rubric requirements can invoke the configured select (GPT-5 Chat).

Teaching and evaluation ought to interpret Qwen output identically. That options reasoning extraction, software program calls, argument typing, rendering, and isolated job state. In some other case the reward measures a definite agent from the one evaluated. The shared interpretation code is in coach/core/tb_interpret.py.

For the reference run, we ran full-parameter RL fine-tuned Qwen3.8-27B on 187 duties using token-mean GRPO. Each substitute generated 18 groups of 8 rollouts, or 144 episodes, all through 24 H100s. Teaching ran synchronously with parameter and optimizer offload; AdamW used a 1e-6 learning price and nil weight decay. We ran the teaching for 50 steps.

Zero-variance groups have been modified DAPO-style all through teaching. System errors have been handled individually: a backend timeout should not be an incorrect agent finish consequence, so affected groups have been refilled fairly than scored as failures.

That explains how retained duties have been expert. The remaining question is which duties entered the teaching pool.

Offline policy-relative downside

Sooner than diving deep into the gathering of teaching duties, a disclaimer:

No ThinkingBox-Bench job was included throughout the RL teaching pool. The benchmark was reserved for evaluation.

ThinkingBox-Bench is out-of-distribution relative to the RL teaching pool: its domains, databases, insurance coverage insurance policies, and most devices weren’t encountered all through RL teaching. The two models nonetheless share the ThinkingBox framework, simulated-user gear, executable terminal-state grading, numerous frequent devices, and a similar job trend. These shared parts limit the scope of the change declare.

Why selection points comes right down to 1 arithmetic actuality about GRPO. If all eight rollouts fail, the reward vector is [0, 0, 0, 0, 0, 0, 0, 0]. If all eight succeed, it is [1, 1, 1, 1, 1, 1, 1, 1]. In each case, centering contained in the group produces no task-reward profit on account of no trajectory did greater than one different.⁺⁺

A mixed group equal to [0, 0, 0, 1, 1, 1, 1, 1] gives the excellence GRPO might be taught from. That distinction is expensive for a stateful agent: every rollout provisions an isolated backend, runs an precise software program session, and calls a separate model to play the particular person.

The question that shaped this work was as a result of this reality not solely which model or optimizer? It was: which candidate duties should enter the mounted teaching pool?

We used an offline pass-rate filter on the posterior to select the teaching pool. It estimates which duties usually tend to produce contrasting terminal rewards beneath the underside protection.

So we probe. Each candidate job is run repeatedly in opposition to the base protection model. The plain-language rule is straightforward: duties the model on a regular basis fails are unlikely to indicate it however; duties it on a regular basis passes are already found; duties it solves usually are the place outcomes can current distinction.

Estimating downside from restricted probes

Each candidate job is run n cases, producing okay successes. We model its latent success chance with Jeffreys posterior chance:

success chance | okay successes in n probes ~ Beta(okay + ½, n − okay + ½)

Decide 6. Offline policy-relative downside is estimated from repeated base-policy probes and used to retain duties extra prone to produce contrasting rewards.

We partition the unit interval into 5 buckets with boundaries at 0.08, 0.40, 0.60, and 0.90, chosen as hyperparameters for these runs. Each job is assigned to the bucket containing the very best posterior mass:

• Infrequently solved [0, 0.08): drop
• Difficult [0.08, 0.40): keep
• Intermediate [0.40, 0.60): keep
• Frequently solved [0.60, 0.90): keep for full-parameter RLFT
• Nearly always solved [0.90, 1]: drop

Further particulars are in Appendix C.4 of our ThinkingBox paper.

One job, end to complete

Decide 7. Zooming in, one job by way of all six ranges: GPT-5.4 screening, base-model probes, downside classification, a rollout group, terminal-state grading, and a GRPO substitute.

Decide 7 follows one candidate job which will doubtlessly enter the teaching pool. For the Qwen3.8 27B full-parameter pool, GPT-5.4 first served as a solvability show. The underside protection then succeeded on few of the probes e.g 3 of 8, producing a Beta(3.5, 5.5) posterior. Its largest mass, 54.0%, fell throughout the Robust band.

When sampled for teaching, the responsibility produced the mixed reward group [0, 0, 0, 0, 1, 1, 1, 1]. Worthwhile trajectories as a result of this reality acquired above-mean job profit and failed trajectories below-mean profit, giving GRPO distinction to be taught from. A particular protection model might place the equivalent job in a definite downside band.

The last word Qwen3.8-27B curated teaching pool contained 187 duties.

What earns a reward

Nothing regarding the trajectory. The reward is the responsibility’s private executable verdict.

When an episode ends, the sandbox extracts what actually modified throughout the backend and runs the responsibility’s deterministic checks in opposition to the terminal state. Any trajectory that produces the fitting end state passes. Mistaken, missing, or additional undesirable uncomfortable side effects fail. The reward is 1 or 0.

Inside the benchmark, the database grader will likely be thought-about the “examiner”: it caught brokers that reported success and left the flawed file. In teaching, the equivalent type of take a look at will likely be seen as a result of the “coach” on a definite, disjoint set of duties.

Two reward paths, one binary verdict

• Executable take a look at (the huge majority). When the required finish result’s represented in terminal state or undesirable uncomfortable side effects, a deterministic check out accepts or rejects it; no LLM select title is required.
• Resolve-assisted take a look at (a minority). When a slender requirement cannot be determined cleanly from database state, the configured LLM select options a binary rubric question utilized by the executable check out.

Run the teaching stack

We launch the data-selection, rollout, reward, and Verl integration used to educate Qwen3.8-27B in microsoft/thinkingbox-training. The repository README is the entry stage. The detailed guides are:

Sooner than you start

Examined on Linux and WSL with Python 3.12 and uv. You need NVIDIA GPUs with an appropriate CUDA/NCCL stack, an space Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for the simulated particular person and select. The validated reference configuration makes use of three nodes x 8 H100 GPUs; smaller personalized configurations depend on model measurement, context dimension, and rollout concurrency.

Most people sample job guidelines exercise routines the interface. It is not the personal 187-task teaching pool used for the reported run. Present an authorized teaching provide that’s nonetheless disjoint from most people benchmark.

Arrange

Maintain the three repositories facet by facet:

mkdir -p ~/thinkingbox-workspace
cd ~/thinkingbox-workspace

git clone https://github.com/microsoft/thinkingbox.git
git clone https://github.com/microsoft/thinkingbox-data.git
git clone --branch basic https://github.com/microsoft/thinkingbox-training

git -C thinkingbox checkout 892964e7226044e5463ad188da119df838f4bec1
git -C thinkingbox-data checkout thinkingbox-bench-v1.0

Create the validated environment

The launched stack targets Linux, Python 3.12, Verl 0.9.0, and the pinned package deal deal lock:

cd ~/thinkingbox-workspace/thinkingbox-training

uv venv --python 3.12
provide .venv/bin/activate

uv sync --frozen --extra fast --extra monitoring --extra eval-proxy
uv pip arrange -e ../thinkingbox
uv pip arrange --config-settings editable-mode=compat 
  -e ../thinkingbox-data/servers/thinkingbox_tools
uv pip arrange --config-settings editable-mode=compat 
  -e ../thinkingbox-data/servers/tb_business_ops_servers_202606

python scripts/prepare_verl.py --dest .deps/verl --install
python scripts/verify_verl_install.py
uv pip take a look at

The last word two directions should report a verified patched Verl checkout and no broken requirements. Do not run one different automated package deal deal sync afterward: it would in all probability trade the patched provide arrange with the vanilla Verl wheel.

Put collectively the teaching info

Start with the publicly supplied samples, or add your particular person licensed dataset in ThinkingBox format. Then run the repository’s preparation script:

cd ~/thinkingbox-workspace/thinkingbox-training
provide .venv/bin/activate
python info/prepare_data.py --help

The script aggregates protection probes, applies offline policy-relative downside filters and benchmark exclusions, and writes the chosen follow.yaml and check out.yaml info. docs/data-preparation.md paperwork the enter fields, loader contract, and full command.

Configure and study the launch

Adjust to docs/reference-q38-run.md to begin out Typesense and the MCP servers, configure the simulated particular person and select, set the model, task-list, checkpoint, and topology variables, and put collectively the Ray cluster. Maintain credentials in runtime secret injection fairly than devoted info or shell historic previous.

Sooner than allocating a training job, use the data’s dry-run command:

DRY_RUN=true ./scripts/run_train.sh 
  coach.total_training_steps=50 
  coach.save_freq=5

It ought to exit zero with out starting Ray or allocating GPUs. Be taught the emitted python -m verl.coach.main_ppo command and every resolved override sooner than launching the run.

Apply

The launcher configures teaching; it would not provision hosts. On the reference three-node topology, export the equivalent environment on every node, mount CHECKPOINT_DIR on the equivalent absolute path, then start Ray:

ROLE=head NODE_IP="$HEAD_IP" GPUS_PER_NODE=8 
  ./scripts/start_ray.sh


ROLE=worker NODE_IP="$WORKER_IP" 
  HEAD_ADDRESS="$HEAD_IP:6379" GPUS_PER_NODE=8 
  ./scripts/start_ray.sh

After ray standing experiences all 24 GPUs, launch as quickly as from the highest node:

export RAY_ADDRESS=auto
export GPUS=0,1,2,3,4,5,6,7

./scripts/run_train.sh 
  coach.project_name=thinkingbox-training 
  coach.experiment_name=q38-full-rlft 
  coach.total_training_steps=50 
  coach.save_freq=5

Take into account the checkpoint by way of OpenEnv

The first ThinkingBox
submit
covers Typesense, the MCP Session Proxy, and OpenEnv setup. Merge the expert
checkpoint following
docs/evaluation.md,
then serve it with vLLM.

From the OpenEnv repository root, set MERGED_MODEL to the merged model
itemizing and THINKINGBOX_DATA to the pinned benchmark checkout. The
historic checkpoint evaluation used this eight-A100 serving configuration:

vllm serve "$MERGED_MODEL" 
  --served-model-name q38-rlft 
  --host 127.0.0.1 
  --port 8001 
  --tensor-parallel-size 8 
  --dtype bfloat16 
  --max-model-len 262144 
  --gpu-memory-utilization 0.85 
  --max-num-seqs 16 
  --max-num-batched-tokens 8192 
  --language-model-only 
  --enable-chunked-prefill 
  --enable-prefix-caching 
  --reasoning-parser qwen3 
  --enable-auto-tool-choice 
  --tool-call-parser qwen3_xml

In a separate terminal, start the OpenEnv compatibility proxy for the Qwen chat template from the ThinkingBox Teaching checkout. It listens on port 8014 and forwards to vLLM on 8001:

cd ~/thinkingbox-workspace/thinkingbox-training
provide .venv/bin/activate
bash scripts/launch_sysmerge.sh

Maintain it working all through evaluation. From one different terminal, take a look at readiness:

curl --fail http://127.0.0.1:8014/readyz

Exchange the agent entry throughout the current thinkingbox.yaml; retain its MCP, simulated-user, and select configuration:

orchestrator:
  agent_model:
    type: aoai
    endpoint_url: http://127.0.0.1:8014/v1/chat/completions
    deployment: q38-rlft
    credential:
      type: api-key
      api_key: 
    is_reasoning: true
    temperature: 1.0
    top_k: 20
    top_p: 0.95
    max_completion_tokens: 8192
    timeout: 1800

Start OpenEnv as throughout the first submit, with OPENENV_TB_CONFIG pointing to this
thinkingbox.yaml and OPENENV_TB_DATASET to $THINKINGBOX_DATA. As quickly as
/ready succeeds, run the entire protocol of 507 duties × 20 makes an try:

uv run --project envs/thinkingbox_env --frozen 
  thinkingbox-eval 
  --config "$PWD/thinkingbox.yaml" 
  --base-url http://127.0.0.1:8000 
  --dataset "$THINKINGBOX_DATA" 
  --output q38-rlft-full.jsonl 
  --errors-output q38-rlft-full-errors.jsonl 
  --repeat 20 
  --coverage-profile full 
  --aggregate-output q38-rlft-full-aggregate.json 
  --message-timeout 1800

The full profile validates all 10,140 anticipated job/repetition keys sooner than
aggregation.

The place this goes subsequent

The useful part of this work should not be one Qwen3.8-27B checkpoint. It is the teaching loop and training info curation that turns executable outcomes into rewards.

If you happen to’re teaching or evaluating an agent that touches precise database information:

1. Probe sooner than you follow. Take your particular person agent job set, run the underside protection in opposition to it n cases, and take a look on the band histogram. In our experience teams usually have not inspected this distribution, and it is usually further lopsided than anticipated.
2. Check your parity. Replay the equivalent recorded model output by way of the teaching and evaluation parsers, and make sure that every extract an equivalent software program names and arguments. In some other case, teaching might reward habits that evaluation interprets another way.
3. Report every safety and repeatability. State what variety of duties cross on the very least as quickly as in okay makes an try and what variety of cross all okay. A model that always finds a solution should not be primarily one which repeats it consistently.

Further particulars will likely be found throughout the following hyperlinks:

Cite the paper

@article{li2026thinkingbox,
  title   = {One Success Will not be Reliability: Thinkingbox, a Sandbox and Benchmark for Brokers in Stateful Enterprise Workflows},
  author  = {Li, Zhuochun and Ko, Youngmin and Keramati, Ali and Kundu, Tuhin and
             Tsai, Liang-Chun and Ferri, Nicola and Milletari, Mirco and Liu, Jiaxiang and
             Lopez Pelaez, Susana Palmaz and Wang, Yuepeng and Smolyakov, Vadim and
             Jiang, Xiang and Olafsson, Kjartan and Man, Tommy},
  journal = {arXiv preprint arXiv:2608.19741},
  12 months    = {2026},
  url     = {https://arxiv.org/abs/2608.19741}
}

ThinkingBox code is MIT-licensed; the benchmark info is CDLA-Permissive-2.0.

Disclaimer: every job throughout the public benchmark is a synthetic reconstruction. The workflows and insurance coverage insurance policies are modeled on precise enterprise agent patterns; the purchasers is not going to be precise.

ThinkingBox, ThinkingBox-Bench and ThinkingBox-Teaching is constructed by the Microsoft Copilot Studio group in partnership with Toloka with collaborators from the School of Pittsburgh, Northwestern School, Columbia School and UC Irvine who interned at Microsoft. Questions are welcome throughout the suggestions half beneath or on github or by e-mail at firstnamelastname@microsoft.com.

⁺ A minority of duties moreover embrace slender binary rubric requirements for outcomes that the database cannot seize cleanly; these checks can invoke the configured LLM select.

⁺⁺ That zero-gradient draw back has an unusual batch-time restore: oversample, discard the degenerate groups, refill. The whole-parameter run did exactly that, similar to DAPO with substitute of zero-variance groups. Nonetheless refilling nonetheless pays full price for every wasted rollout sooner than throwing it away.