Beam by Reflection is out on Hugging Face! It’s the first mannequin from the brand new US lab, Reflection, and it’s open! It is a mixture-of-experts (MoEs) mannequin with 501B parameters (23B lively per token), helps textual content as enter, has a context window of 1M tokens, and is launched underneath the Apache 2.0 license.
Beam is targeted on coding and agentic use circumstances. It’s for groups that desire a frontier-scale mannequin on their very own {hardware}, infrastructure, or cloud, and need to customise it to their very own use circumstances. Beam ships with its personal devoted open supply Rust based mostly harness that delivers the most effective efficiency from the mannequin.
Beam has a local transformers implementation, in addition to downstream libraries like vLLM and SGLang. We’ve labored with the Reflection workforce to deliver you inference, SFT (supervised wonderful tuning), and RL (reinforcement studying) recipes so that you can construct with this mannequin to your wants. On high of day-0 assist in transformers, SGLang and vLLM, this submit reveals you how one can run it, how one can post-train it, and what we constructed with it.
TL;DR
- 501B-parameter MoEs LLM with 23B lively per token. 52 layers, 256 specialists per MoE layer, 8 routed per token.
- Textual content in, textual content out with context window of 1M tokens.
- Checkpoint codecs: BF16, FP8, and NVFP4
- Reasoning that may keep in context throughout turns (
preserve_thinking), device calling with a devoted grammar. The mannequin helps 5 reasoning values: “low”, “medium” (default), “excessive”, “xhigh”, and “max”.- Response template within the tokenizer.
- Speculative decoding drafter mannequin with DFlash2
- Day-0 libraries: transformers, SGLang, vLLM. Full-parameter RL with Miles.
- Ships with a Rust-based customized harness.
- Apache 2.0.
Construct with Beam
Reflection goes to offer out $10,000 Hub credit to the most effective tasks that use Beam. All it’s important to do is make a factor, and share it on the Hub:
- You can make a demo that makes use of the Beam mannequin on Areas
- You can quantize it to your favourite {hardware}
- You can generate an artificial dataset to your favourite drawback
- You can construct an RL surroundings with or for Beam
- You can benchmark Beam and share the outcomes.
- You can shock us!
When you construct one thing with Beam, then share it on the Hub for an opportunity to win $10,000. All it’s important to do is construct the area, mannequin, or dataset repo and share it on the mannequin’s repo web page. We’ll accumulate all of the hyperlinks, choose the most effective, and ship out the credit to your Hub accounts.
What’s Beam?
Beam is a text-only mannequin constructed for long-context (1M context size) reasoning and agentic work. It’s a sparse MoE mannequin, the place every token is computed by a small fraction of the community, so it runs on the velocity of a a lot smaller mannequin whereas protecting the capability of a big one.
Consideration
The decoder has 52 layers with a hidden measurement of 5,120. Most layers use sliding-window consideration over the earlier 512 tokens; each sixth layer, and the final one, attend to the total context. That provides 43 sliding-window layers and 9 world ones.
Inside the eye block (each layer varieties)
The 2 sorts of layers deal with place in another way. Sliding-window layers use RoPE, since relative place issues most inside a brief window. International layers use no positional encoding. As a substitute, every question is multiplied by a scaler that grows slowly with the variety of seen tokens, (1 + (pos + 1) / 2048) ** 0.1. It acts as a temperature: the softmax retains its precision because the context lengthens as a substitute of flattening out. The implementation calls this scalable-softmax question scaling.
Consideration makes use of grouped-query consideration with 48 question heads and eight key-value heads, with RMSNorm on queries and keys. A realized sigmoid gate, computed from the layer enter, scales each channel of each head earlier than the output projection.
Combination of Specialists
The primary 3 layers use a dense SwiGLU MLP. The opposite 49 exchange it with a Combination-of-Specialists block of 256 specialists, every a SwiGLU MLP with an intermediate measurement of two,560, or about 39M parameters. The specialists maintain 98.4% of the mannequin’s parameters.
The router follows DeepSeek-V3’s group-limited scheme. It scores all 256 specialists with a sigmoid, splits them into 8 teams of 32, retains the 4 finest teams (ranked by the sum of every group’s high two scores), then picks the highest 8 specialists inside them. Beam makes use of the identical group sizes as DeepSeek-V3 however no shared knowledgeable. Professional outputs are mixed with normalized router weights and scaled by 2.5.
Solely 8 specialists run per token, so the mannequin computes every token with about 23B parameters. All 256 specialists keep in reminiscence, so deployment wants the reminiscence of a 501B mannequin however runs on the velocity of a 23B one. The BF16 checkpoint is about 1 TB in safetensors format. The NVFP4 checkpoint shops most parameters in 4-bit precision for NVIDIA Blackwell GPUs, some parameters are stored in BF16 for accuracy. There are additionally FP8 checkpoints on the hub which are perfect for Hopper and Ampere sequence chips.
If you wish to deploy Beam your self, try the vLLM or SGLang recipes under.
Residual Stream
The residual connections are probably the most distinctive half. A regular transformer provides every sublayer output to the stream with weight 1, so the stream grows with depth.
Beam counts the enter embedding as step 1. At step okay it provides the sublayer output with weight 1/sqrt(okay) and retains the prevailing stream with weight sqrt((k-1)/okay). The squares of the 2 weights sum to 1, so the stream stays at a relentless scale from the primary layer to the final. In spite of everything 104 additions, each sublayer output and the enter embedding carry the identical weight within the closing hidden state.
Each sublayer can be normalized earlier than and after, and the ultimate logits cross via a tanh softcap at ±30, computed in FP32. The implementation fixes the place it computes in FP32: the router, the SwiGLU product, the norms, RoPE, the question scaling and the softcap.
Lengthy Context
Hybrid consideration works nicely for lengthy contexts. Every layer caches 2 x 8 KV heads x 128 dims = 2,048 values per token. If all 52 layers stored each token, the cache would develop by 208 KiB per token. Solely the 9 world layers maintain the entire sequence. The 43 sliding-window layers maintain a hard and fast 512-token window every and cease rising. So the cache grows by 36 KiB per token, roughly a sixth of the naive fee.
| Context | If all 52 layers have been world | Beam |
|---|---|---|
| 32K tokens | 6.5 GiB | 1.2 GiB |
| 128K tokens | 26 GiB | 4.6 GiB |
| Full context (1M) | 208 GiB | 36.08 GiB |
These numbers are per sequence, in BF16, with the transformers dynamic cache.
Benchmark Outcomes
Listed below are the benchmark outcomes throughout totally different precision ranges, as offered by the Reflection workforce.
| Beam (BF16) | Beam (FP8) | Beam (NVFP4) | |
|---|---|---|---|
| Agentic Coding / Terminal | |||
| DeepSWE v1.1 | 43.3% | 43.0% | 42.7% |
| SWE Bench Professional v2-Arduous | 76.5% | 78.4% | 78.6% |
| SWE Bench Professional v1 | 66.1% | 65.9% | 65.6% |
| Terminal Bench v2.1 | 80.1% | 78.9% | 78.2% |
| SWE Atlas Codebase QnA | 31.8% | 32.2% | 31.4% |
| SWEBench Multilingual | 78.3% | 78.1% | 77.5% |
| SWEBench Verified | 81.6% | 80.2% | 80.7% |
| Reasoning | |||
| AIME 2026 | 97.7% | 98.0% | 97.5% |
| HLE (no instruments) | 36.0% | 36.0% | 34.8% |
| HLE (with instruments) | 51.1% | 51.1% | 51.1% |
| SciCode | 49.2% | 48.3% | 46.9% |
| CriPT AA | 16.3% | 17.4% | 15.4% |
| GPQA Diamond | 90.6% | 90.3% | 89.3% |
| LiveCodeBench | 88.3% | 88.1% | 87.4% |
| Device Calling / Search | |||
| AutomationBench – public break up solely | 39.3 | 38.2 | 39.8% |
| MCP Atlas | 80.8% | 81.5% | 80.5% |
| Tau3 Banking | 36.1% | 36.8% | 35.4% |
| BrowseComp w/ context administration | 76.8% | 76.7% | 75.6% |
| DeepSearchQA w/ context administration | 80.5% | 79.6% | 81.1% |
| Common Capabilities | |||
| AA-LCR | 78.2% | 79.4% | 78.7% |
| LongBench v2 | 67.5% | 67.6% | 67.2% |
| IFBench | 79.6% | 80.7% | 79.2% |
| AA Omniscience (index) – public break up solely | 10.8 | 8.8 | 9.8 |
{Hardware} Necessities
Essentially the most sensible technique to serve the mannequin is serving the NVFP4 checkpoint, which retains a lot of the high quality of the bf16 checkpoint. NVFP4 shops some weights in 4 bits, and this fashion, it shrinks the 501B parameters in BF16 from 1 TB to ~280 GB. Blackwell GPUs run NVFP4 natively however GPUs older than Blackwell just like the H200 cannot. The FP8 checkpoints are appropriate for GPUs older than Blackwell.
A B200 has 180 GB of HBM3e, so two of them (360 GB) can maintain the Beam weights, with little room left for activations and the KV cache. Plan for 4 B200s (720 GB), which leaves loads of room for lengthy contexts and concurrent requests. You may also serve it on a full 8xB200 node, or on two B300s at 288 GB every, and even 8xRTX Professional 6000. If you do not have Blackwell {hardware}, use Inference Suppliers (serverless) to check Reflection.
Inference Help
Beam comes with day-0 assist in transformers, SGLang and vLLM.
Transformers
Set up the newest model of transformers.
pip set up -U "transformers>=5.20.1"
The best technique to run Beam is the text-generation pipeline:
from transformers import pipeline
pipe = pipeline(
"text-generation", mannequin="reflection/Beam-1.0-501B-A23B", device_map="auto"
)
messages = [
{"role": "user", "content": "Write a limerick about Python exceptions."}
]
output = pipe(messages, max_new_tokens=2000)
output[0]["generated_text"][-1]
The pipeline returns the entire chat. The final message is Beam’s reply, already parsed by the response template.
Going one stage decrease, use AutoModelForCausalLM and AutoTokenizer.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "reflection/Beam-1.0-501B-A23B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
mannequin = AutoModelForCausalLM.from_pretrained(
model_id, dtype="auto", device_map="auto"
)
Chat
messages = [
{"role": "system", "content": "You should only answer with a number."},
{"role": "user", "content": "What is 17 * 23?"},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
reasoning_effort="medium"
).to(mannequin.machine)
input_len = inputs["input_ids"].form[-1]
outputs = mannequin.generate(**inputs, max_new_tokens=2000)
tokenizer.parse_response(
outputs[0][input_len:], prefix=inputs["input_ids"][0]
)
Beam’s tokenizer carries a response template. parse_response reads it and turns the uncooked output right into a message dict, with the reasoning, the reply, and any device calls in separate fields. This avoids the same old regex for splitting reasoning from solutions from device calls. And the foundations ship with the tokenizer, so the parsed construction matches what Beam was educated to supply. The chat template can pre-write a part of the assistant flip, which the parser must see.
Device calling
Full tool-calling instance
def get_weather(metropolis: str, unit: str = "celsius"):
"""
Get the present climate for a metropolis.
Args:
metropolis: The town to lookup.
unit: Temperature unit, "celsius" or "fahrenheit".
"""
...
instruments = [get_weather]
messages = [
{"role": "user", "content": "I land in Paris tonight. Do I need a coat?"}
]
inputs = tokenizer.apply_chat_template(
messages,
instruments=instruments,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(mannequin.machine)
input_len = inputs["input_ids"].form[-1]
outputs = mannequin.generate(**inputs, max_new_tokens=1000)
message = tokenizer.parse_response(
outputs[0][input_len:], prefix=inputs["input_ids"][0], instruments=instruments
)
message.get("tool_calls")
Go instruments to parse_response as nicely. Beam writes tool-call arguments inline, in its personal XML-style grammar, so a naked 7 may very well be a quantity or a string. With the device schemas, the parser casts every argument to its declared sort.
Multi-GPU and multi-node
Beam’s config defines an expert-parallel plan. Professional parallelism places every knowledgeable on one machine. Consideration, the dense layers, the embeddings and the LM head are replicated on each machine. That’s 8B parameters, or 16 GB in BF16. Load the mannequin with a DistributedConfig:
import os
from transformers import AutoModelForCausalLM
from transformers.distributed import DistributedConfig
mannequin = AutoModelForCausalLM.from_pretrained(
"reflection/Beam-1.0-501B-A23B",
distributed_config=DistributedConfig(
tp_size=int(os.environ["WORLD_SIZE"]), ep_size=int(os.environ["WORLD_SIZE"]), tp_plan={}
),
)
Launch the script with torchrun throughout 16 H200 GPUs. Each then holds 78 GB of weights. With 8 GPUs every would maintain 139 GB, which leaves no room on a 141 GB H200. The variety of gadgets should divide the variety of specialists, 256.
vLLM
Beam has day-0 assist in vLLM. The command under serves the BF16 checkpoint with tensor parallelism throughout 16 GPUs (two 8x H200 nodes) and exposes an OpenAI-compatible API:
vllm serve reflection/Beam-1.0-501B-A23B --tensor-parallel-size 16 --served-model-name beam
For quicker era, add DFlash2 speculative decoding. A small drafter mannequin proposes as much as 15 tokens forward, and Beam checks them multi function ahead cross, so every step can settle for a number of tokens. The associated fee is a bit additional reminiscence for the drafter:
vllm serve reflection/Beam-1.0-501B-A23B
--tensor-parallel-size 16
--speculative-config '{ "methodology": "dflash", "mannequin": "reflection/dflash2-drafter-preview", "num_speculative_tokens": 15 }'
Each instructions begin an OpenAI-compatible server, so you’ll be able to level any OpenAI shopper at http://localhost:8000/v1. For extra serving configurations, see the proposed Beam recipe for vLLM.
SGLang
Beam additionally has day-0 assist in SGLang. We advocate serving it with DFlash2 speculative decoding, which each instructions under activate. On Blackwell, the NVFP4 checkpoint matches on 4 B300s:
python -m sglang.launch_server
--model-path reflection/Beam-1.0-501B-A23B-NVFP4
--served-model-name beam
--tp 4
--attention-backend trtllm_mha
--page-size 64
--context-length 1048576
--mem-fraction-static 0.85
--enable-unified-memory
--speculative-algorithm DFLASH
--speculative-draft-model-path reflection/Beam-1.0-501B-A23B-DFlash2
--speculative-dflash-block-size 8
--tool-call-parser beam --reasoning-parser beam
--host 0.0.0.0 --port 30000
On Hopper GPUs, use the FP8 checkpoint on 8 H200s as a substitute. Solely three flags change: the checkpoint, the tensor-parallel measurement (8 as a substitute of 4), and the eye backend (fa3 as a substitute of trtllm_mha, which wants Blackwell):
python -m sglang.launch_server
--model-path reflection/Beam-1.0-501B-A23B-FP8
--served-model-name beam
--tp 8
--attention-backend fa3
--page-size 64
--context-length 1048576
--mem-fraction-static 0.85
--enable-unified-memory
--speculative-algorithm DFLASH
--speculative-draft-model-path reflection/Beam-1.0-501B-A23B-DFlash2
--speculative-dflash-block-size 8
--tool-call-parser beam --reasoning-parser beam
--host 0.0.0.0 --port 30000
Inference Suppliers
Beam is on the market on Hugging Face Inference Suppliers, powered by Baseten, Fireworks, and Collectively, with extra to comply with quickly.
import os
from openai import OpenAI
shopper = OpenAI(
base_url="https://router.huggingface.co/v1",
api_key=os.environ["HF_TOKEN"],
)
completion = shopper.chat.completions.create(
mannequin="reflection/Beam-1.0-501B-A23B:auto",
messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(completion.selections[0].message)
Inference Endpoints
We’ve arrange a Beam recipe so that you can one click on deploy in Inference Endpoints from the catalog in FP8 or NVFP4, or set up with the hf cli like so:
hf endpoints catalog deploy --repo reflection/Beam-1.0-501B-A23B --name beam
What we examined
We’re sharing some vibe evals of Reflection Beam-501B-A23B on 5 duties, as soon as every at high and low reasoning effort via the Reflection platform. These are vibe evals, not benchmarks. When you’re planning on utilizing Beam, it is a nice technique to kick the tires and see the way it behaves on particular use circumstances, and importantly, the place it wins and loses.
Under is an summary of the prompts we ran and the output traces from the hub. The whole inputs, outputs, API utilization, process fixtures, graders and dialog traces are on this public Hugging Face bucket.
| Class | Immediate Description | Low reasoning (tokens) | Excessive reasoning (tokens) |
|---|---|---|---|
| Multi-step device calling | Fulfill ORD-742: evaluate inventory, reservations, surcharges and arrival dates; reserve the most affordable eligible warehouse. | ✅ 544 | ✅ 811 |
| Lengthy-context retrieval over a code base | Derive Click on command configuration and two invocation outcomes from three full supply recordsdata; cite six implementation symbols. | 🔴 1,613 * | ✅ 5,007 |
| Terminal process in a sandbox | Examine and restore a CSV ledger CLI; repair all bugs, protect exams, run exams and a CLI examine. | ✅ 582 | ✅ 1,548 |
| Math reasoning (AIME-style) | What number of eight-element subsets of {1, …, 16} have a sum divisible by 17? Give a justification. | 🟠 829 † | ✅ 2,809 |
| Structured output | Convert an out-of-order revision log into strict JSON: merge patches, cancellations, nulls, dates, currencies and totals. | ✅ 1,040 | ✅ 2,088 |
* Low’s long-context response received all code conduct and all six supply citations appropriate. Nevertheless, the response didn’t correspond to the parsers format, so that is partly immediate ambiguity.
† Low returned the proper reply, however omitted the requested justification. Excessive provided an accurate roots-of-unity proof, independently reviewed.
Don’t take our phrase for it. Dig into the bucket, rewrite to your use case or venture and push Beam to the restrict.
Put up-training Beam
You may wonderful tune Beam to your personal use circumstances. Right here, we share some examples in TRL and Miles to get you began on supervised fine-tuning and reinforcement studying.
SFT with TRL
The quickest technique to adapt Beam to your individual information is supervised fine-tuning with TRL’s SFTTrainer. To maintain reminiscence manageable, we practice LoRA adapters with PEFT fairly than updating all 501B parameters:
We used TRL’s SFTTrainer to adapt Beam to a easy objective: maintain its reasoning within the language requested by the person. We fine-tuned on Multilingual-Considering with LoRA, which trains a small set of adapter weights whereas protecting the bottom mannequin frozen.
The setup combines mannequin sharding at load time, knowledgeable parallelism, FlashAttention 4 and sequence packing. LoRA SFT matches on a single node with 8 B300 GPUs, reaching round 19,000 tokens per second. The corresponding H100 setup makes use of 4 nodes.
After one epoch of Multilingual-Considering, Beam reasoned within the requested language on 16 out of 16 held-out prompts, in contrast with 3 out of 16 for the bottom mannequin. This small analysis measures language adherence, fairly than the correctness of the reasoning itself.
Coaching loss (left) and imply token accuracy (proper) throughout a separate 500-step run on a reasoning subset of open-r1/Combination-of-Thought, utilizing one 8-GPU B300 node. These coaching metrics are from the official TRL instance PR.
GRPO with Miles
Reflection and RadixArk constructed a recipe that trains the mannequin with GRPO in Miles, in BF16. You may replace the total mannequin or use LoRA to coach adapters whereas protecting the bottom weights frozen. Megatron trains, SGLang generates the rollouts, and Miles publishes the brand new weights to the rollout engines after every replace. We reviewed the recipe.
| Workload | Reference configuration |
|---|---|
| Inference / eval, BF16 | 16x H200 per engine |
| Full-parameter GRPO, BF16 (Miles) | 96x H200: 64 practice, 32 rollout. Or 64x H200 colocated. |
| LoRA GRPO, BF16 (Miles) | 32x H200: 16 practice, 16 rollout. |
| Full-parameter GRPO, BF16, 128-token context (Miles) | 48x H200: 32 practice, 16 rollout. The setup Reflection and RadixArk first validated on the true checkpoint. |
The recipe runs GRPO on DAPO-Math-17K and evaluates on AIME25. Construct the picture, obtain the information, and submit the job to a Ray cluster of twelve 8x H200 nodes:
docker construct -f docker/Dockerfile.beam -t miles-beam .
python scripts/run_beam.py put together
--model-id reflection/Beam-1.0-501B-A23B
--model-dir /root/fashions/Beam
--data-dir /information/datasets
export MASTER_ADDR=10.0.0.1
export RAY_ADDRESS=http://${MASTER_ADDR}:8265
python scripts/run_beam.py execute
--model-dir /root/fashions/Beam
--data-dir /information/datasets
--output-dir /root/shared_data/beam-run
--num-rollout 100
What occurs in a rollout:
Pattern: 32 prompts are handed to the mannequin with 8 responses every. Two SGLang engines serve the rollouts on 16 GPUs every.
Rating: the boxed reply is checked in opposition to the label, with a smooth penalty for responses close to the size restrict.
Replace: one optimizer step on all 256 samples. Group-relative benefits with truncated significance sampling, no KL or entropy penalty.
Prepare in parallel: tensor parallelism 8, knowledgeable parallelism 8, information parallelism 8, with sequence parallelism and full recompute.
Optimizer: Adam at a relentless 1e-6. Gradients in BF16, grasp weights in FP32, moments in BF16. The optimizer state streams via node-local NVMe, so every coach node wants 2 TB of scratch.
Consider: AIME25 each 5 rollouts, eight samples per drawback. A checkpoint is saved each twenty.
The subsequent rollout is generated whereas the present one trains, so responses are one replace behind. Add --resume to proceed from the final checkpoint. execute-colocate runs the identical recipe on eight nodes, with each GPU coaching and serving in flip.
Prepare with LoRA
You may also run GRPO with LoRA on 4 8× H200 nodes: 16 GPUs for coaching and 16 for rollouts. The recipe trains rank-16 adapters on Beam’s consideration, dense MLP, and knowledgeable layers whereas protecting the bottom weights frozen. Solely adapter weights are synchronized with SGLang, and the optimizer state stays in GPU reminiscence, eradicating the necessity for NVMe optimizer offloading.
python scripts/run_beam_lora.py execute
--model-dir /root/fashions/Beam
--data-dir /information/datasets
--output-dir /root/shared_data/beam-run
--num-rollout 100
Construct with Beam: Demos
When you construct one thing with Beam, then share it on the Hub for an opportunity to win $10,000. All it’s important to do is construct an area, mannequin, or dataset repo and share it on the mannequin’s repo web page. To encourage you, right here are some things we constructed.
Hey Beam, construct a local native MacOS utility!
At Hugging Face we love open and native fashions that run native to their {hardware}, however constructing native apps on Mac could be arduous! Let’s see if Beam can assist us out.
We requested Beam to construct a local MacOS utility that turns speech to textual content, and conveniently drops it at your cursor. We need to use MLX to get probably the most out of our Mac Steel and the Parakeet Mannequin for excellent speech-to-text.
Construct BeamSpeak, a local macOS menu bar dictation app with a compact recording tablet and no important window or account.
Use ⌘⇧Area to start out and cease microphone recording, exhibiting elapsed time, microphone stage, and transcription standing. Run speech recognition totally regionally utilizing a Hugging Face mannequin, ideally Parakeet v3 via MLX, and maintain the mannequin loaded between recordings. Paste the transcript into the unique app if it stays centered and Accessibility permission is granted; in any other case copy it to the clipboard, preserving newer clipboard adjustments and by no means urgent Enter. Deal with permissions, cancellation, silence, errors, a two-minute recording restrict, and momentary audio cleanup. Ship supply code, a runnable BeamSpeak.app, setup directions, and verification of offline transcription, the bodily shortcut, microphone seize, and textual content supply.
We used Beam with the Pi coding harness to construct the preliminary native macOS menu bar app, operating Parakeet with MLX. The saved run consists of follow-up prompts; the ultimate supply additionally consists of later Codex repairs. The immediate above is a condensed model of the construct necessities.
You may try the agent hint of how Beam constructed this in pi on this hint. If you wish to strive the demo out, obtain the supply code or learn the setup directions.
Hey Beam, how do you scale on a number of brokers?
We needed to see whether or not Beam may make quicker progress by splitting a programming process throughout brokers. We paired Beam with the open-source Pi coding agent and in contrast three setups: 1. one agent working alone, 2. 5 persistent friends collaborating via a shared Git repository, and three. a lead agent delegating work to asynchronous subagents. The harness ran on Hugging Face jobs with inference via Reflection’s platform API.
For the duties, we used ProgramBench: brokers should recreate a program from its compiled executable and documentation, with out seeing its supply code or the evaluator’s exams. Every setup had the identical 8M API-token ceiling and two-hour solver restrict per process, shared throughout its brokers. We saved implementation snapshots and evaluated them in opposition to mounted hidden-test suites from the ProgramBench-Checks dataset.
Our score-versus-latency plot reveals the common fraction of exams handed. The time axis makes use of mounted reference charges for mannequin tokens plus measured device time, fairly than elapsed wall time. We initially ran 5 duties. The present determine reveals a post-hoc subset of jq, xsv, FIGlet, and hexyl; ripgrep was excluded throughout all three setups as a result of grading its saved artifacts was too gradual. The revealed outcomes retain all 15 authentic episodes.
You may examine the runs within the Trackio dashboard, discover the revealed consequence information, and reuse the examine harness.
Hey Beam, finetune a choice mannequin for my code base!
Resolution fashions are all the craze for the time being as a result of they’re tremendous quick and sensible for a lot of use circumstances. We began with Beam in Pi after Beam’s quota ran out to coach and consider a customized determination mannequin for developer incident triage.
We needed to see whether or not a coding agent may flip an incident-triage coverage right into a small, quick determination mannequin. The duty was to learn a developer incident and select the accountable workforce, its severity, and whether or not somebody must be pinged instantly. Utilizing Unsloth, we fine-tuned Qwen3.5-0.8B to attain these selections instantly.
First, the workflow constructed an artificial dataset containing 2.3k incidents break up by incident cluster for coaching, calibration, and testing. We educated a LoRA adapter and determination head for 3 epochs, with a 4-bit spine, on one H100 80GB on the Science Slurm cluster.
For a baseline, we prompted the unadapted mannequin with the total triage coverage and ticket, with out examples, and requested it to generate the three selections as JSON. On the identical 320 artificial check tickets, the fine-tuned determination mannequin received all three solutions appropriate on 76.25%, versus 4.69% for prompted zero-shot. Imply ticket latency fell from 1,218.06 ms to 102.58 ms, an 11.9× speedup.
If you wish to check out this experiment for your self, try the artifact bucket which accommodates the information, code, checkpoints and outcomes.
Wrapping Up
Beam is a 501B open mannequin you can run by yourself {hardware} and alter. Begin with the BF16 checkpoint, or strive one of many quants FP8, NVFP4. When you construct one thing with Beam, share it within the discussions tab.
