In a earlier publish, we launched TorchNeuron, the native PyTorch backend for AWS Trainium (the torch-neuronx bundle). With it, Transformers runs its private modeling code on Trainium, as a result of it does on one other machine, so any Transformers model on the Hub may run. Which ones really do?
The torch-neuronx model help Space options that question for essentially the most well-liked fashions on the Hub. It tells you whether or not or not a model runs unmodified on Trainium, and presents you the script to run it your self.
Seek for a model
The Space opens on the Chosen view: our private variety of fashions picked among the many many excessive fashions on the Hub by downloads and by trending ranking. Each row presents the result, the number of Trainium cores the model is sharded over, the wall time of the run (compilation included) and the peak machine memory.
To seek for a given model, swap to the Trending or Downloads view and type part of its title. These views adjust to the Hub’s private rankings, one course of at a time, whether or not or not a model was examined or not. When a model was omitted on goal, the Space tells you why (most of the time it was not prioritized, sometimes it is actually incompatible). You might as nicely resolve the variations of the Neuron compiler (neuronx-cc) and of Transformers to take a look at.
On the time of writing, solely Transformers fashions are examined, and our selection is proscribed to
text-generationandimage-text-to-textfashions. We plan to extend to diffusers, sentence transformers, and totally different transformers duties.
Run it your self
Select a model to see learn to run it. The Space lists the Neuron stack the model was examined on, the packages to place in on excessive of it, and the run.py script that ran. The script follows the recipe of the TorchNeuron announcement: it lots of the model with from_pretrained, sharded when it have to be, then generates with a static cache and torch.compile(backend="neuron").
On a Trainium2 (trn2) event with that stack, for example throughout the AWS Neuron Deep Finding out Container, arrange the packages, then run:
python3 run.py
No need for torchrun: the script launches its private torch.distributed ranks when the model is sharded.
A major run begins by compiling the model’s graphs. To skip that step, copy the graphs the similar compiler constructed all through the verify, from most of the people NEFF cache bucket. The Space presents you the command, with the exact compiler mannequin:
hf buckets sync hf://buckets/aws-neuron/torch-neuronx-neff-cache/neuronxcc- /tmp/neff_cache
What “runs” means
Each verify lots of the model on a trn2 event, sharded over the smallest number of Trainium cores that will keep its weights. Most fashions are sharded with tensor parallelism. Mixture-of-experts fashions and never utilizing a tensor-parallel plan use skilled parallelism solely. The verify then generates 32 tokens and checks them. On a single core, the tokens are in distinction with these of a CPU run of the similar checkpoint. When the model is sharded, a small resolve model checks that the textual content material is fluent English.
On the time of writing, solely fashions that match on 4 Trainium2 devices are examined, which suggests 16 cores at most. A PASS says nothing about throughput, or about longer generations.
The outcomes are a dataset
All of the items the Space displays comes from most of the people aws-neuron/torch-neuronx-model-support dataset, with one Parquet file per pair of neuronx-cc and library variations.
Each row holds a model’s consequence, the error when it failed, the variations of the complete stack and the run.py that ran. You can query it instantly:
import pandas as pd
df = pd.read_parquet("hf://datasets/aws-neuron/torch-neuronx-model-support/information")
df[df.model_id == "Qwen/Qwen3-0.6B"][["neuronx_cc", "library", "library_version", "neuron_cores", "result"]]
The dataset card presents the exact meaning of each consequence.
Subsequent time you ponder whether or not a model runs on Trainium, look it up.
