MPS in PyTorch 2.15

When you run fashions in your Mac or use MPS as your major acceleration software for computing, there’s lots to stay up for in PyTorch 2.15! FP8 arrives on MPS, language-model decoding will get quicker paths via linear layers and softmax and new Metallic kernels sort out some surprisingly costly elements of audio, video and coaching workloads.

A fast phrase about who we’re: we are the PyTorch workforce at Hugging Face. We contribute throughout PyTorch, and certainly one of our focus areas is the MPS backend, the place the aim is to make your Mac a first-class place to run and prepare PyTorch fashions. Initially of 2026, operating even small fashions on MPS typically meant working round unsupported operations, efficiency cliffs and surprising reminiscence progress. So we run actual fashions from the Hugging Face Hub throughout language, picture and video, discover the place MPS falls brief and repair it upstream in PyTorch, increasingly typically by changing MPSGraph with devoted Metallic kernels. A lot of this submit is the results of that work, carried out along with the PyTorch maintainers and neighborhood contributors.

TL;DR

• FP8 involves the Mac: retailer float8_e4m3fn values in half the area of BF16 and use scaled matrix multiplication and all different ops that are supported on CUDA/CPU!
• MoE fashions decode 5–6.5× quicker than in 2.14: Qwen3.6-35B-A3B goes from 9.4 to 61.3 tokens/s on an M3 Extremely, because of a local grouped matmul with a devoted decoding path, fewer GPU-CPU syncs and quicker top-k routing.
• Much less time per token for each LLM: single-row linear layers use specialised matrix-vector kernels, and softmax over the vocabulary is about 8× quicker.
• Extra linear algebra on the GPU: advanced LU and triangular solves, quicker small-matrix inverses, and quicker QR decomposition.
• Audio, video, and coaching get consideration too: quicker paths for convolutions, padding, dropout, interpolation backward and reductions. Wan2.2’s video VAE decode now peaks at 8 GiB of reminiscence as an alternative of 270 GiB.
• Extra management from Python: public MPS streams and extra dependable occasion timing.
• Up to date necessities: MPS now requires macOS 15 or later; PyTorch 2.15 requires Python 3.11 or later.
• A peek at 2.16: consideration absolutely on Metallic, extra MPSGraph-to-Metallic migrations and quicker LLM decoding.

What this implies for you

When you run fashions on a Mac with transformers, diffusers or instruments like ComfyUI, here is what 2.15 modifications in observe:

• A lot quicker native LLMs. Combination-of-experts fashions decode 5–6.5× quicker than in 2.14: on an M3 Extremely with torch.compile, Qwen3.6-35B-A3B goes from 9.4 to 61.3 tokens/s and LFM2.5-8B-A1B from 23.6 to 123.6 tokens/s. Dense fashions profit too, from quicker linear layers and an 8× quicker softmax for sampling. (Extra)
• Video technology that matches in reminiscence. Decoding a video with Wan2.2’s tiled VAE used to push driver reminiscence to 270 GiB; it now peaks at about 8 GiB. (Extra)
• Appropriate picture modifying with Qwen-Picture-2.1. On 2.14, the Qwen-Picture-2.1 VAE encoder silently corrupted photographs on MPS, breaking image-edit workflows (reported in ComfyUI). The identical padding change fixes it: an encode-decode spherical journey now matches CPU at 29.95 dB PSNR, up from 10.06 dB.
• FP8 weights in your Mac. FP8 tensors take half the reminiscence of BF16, and F.scaled_mm multiplies them straight on the GPU. (Extra)

Contents

Getting began

You may want an Apple silicon Mac with macOS 15 or later and Python 3.11 or later (see the two.15 compatibility matrix). In case your OS is simply too outdated or no Metallic GPU is accessible, the MPS availability examine now tells you why.

To strive the discharge candidate in a separate atmosphere:

uv pip set up torch==2.15.0 --index-url https://obtain.pytorch.org/whl/check/cpu

FP8 involves MPS

FP8 provides you one other strategy to save reminiscence in your Mac. PyTorch 2.15 provides torch.float8_e4m3fn tensors on MPS: one byte per worth, half the storage of FP16 or BF16.

You may multiply these tensors with F.scaled_mm, which helps each tensorwise and rowwise scaling. This is a small instance with one scale per tensor:

import torch
import torch.nn.purposeful as F

def quantize_fp8(x):
    restrict = torch.finfo(torch.float8_e4m3fn).max
    scale = x.abs().amax().clamp_min(1e-12) / restrict
    return (x / scale).to(torch.float8_e4m3fn), scale

x = torch.randn(1, 1024, system="mps", dtype=torch.float32)
weight = torch.randn(1024, 1024, system="mps", dtype=torch.float32)
x_fp8, x_scale = quantize_fp8(x)
w_fp8, w_scale = quantize_fp8(weight)

y = F.scaled_mm(
    x_fp8,
    w_fp8.t(),
    scale_a=x_scale,
    scale_recipe_a=F.ScalingType.TensorWise,
    scale_b=w_scale,
    scale_recipe_b=F.ScalingType.TensorWise,
    output_dtype=torch.bfloat16,
)
print(y.form, y.dtype)

Quicker language-model decoding

Each LLM has a prefill section and a decode section. Throughout decode, every new token goes via the mannequin as a single row, so the massive matrix multiplications flip into matrix-vector merchandise. PyTorch 2.15 hastens each sorts of layers that dominate this step: dense linear layers and mixture-of-experts layers.

Linear layers

MPS has had GEMV kernels for torch.mm since 2.14, however F.linear nonetheless went via MPSGraph. PyTorch 2.15 connects them to F.linear, so the frequent [1, 1, hidden_size] decoding enter makes use of them straight, with eligible bias additions fused in. Throughout 96 BF16 layer shapes from actual fashions, that is a 1.23× geometric-mean speedup. The most important wins land on small projections, such because the gates in MoE fashions, the place the outdated path left many of the reminiscence bandwidth unused. Massive layers and LM heads have been already near bandwidth-bound, in order that they keep close to 1×.

Combination-of-experts layers

A combination-of-experts layer sends completely different numbers of tokens to every knowledgeable. Grouped matmul handles these uneven teams in a single operation, and a couple of.15 provides MPS a local implementation.

In 2.14, grouped matmul copied the group offsets to the CPU and looped over the teams. The brand new Metallic kernels learn the offsets straight on the GPU, utilizing Metallic Efficiency Primitives the place out there and a separate Metallic path elsewhere. Small teams, the frequent case throughout decoding, get their very own kernel as nicely.

For instance, 64 tokens could be break up into teams of 16, 32, and 16:

import torch
import torch.nn.purposeful as F

tokens = torch.randn(64, 64, system="mps", dtype=torch.bfloat16)
weights = torch.randn(3, 128, 64, system="mps", dtype=torch.bfloat16)
ends = torch.tensor([16, 48, 64], system="mps", dtype=torch.int32)

out = F.grouped_mm(tokens, weights.transpose(-2, -1), offs=ends)
print(out.form)  

The decoding kernel exhibits up straight in token throughput. On an M3 Extremely with torch.compile, MoE fashions generate about 1.3× extra tokens per second:

Linear algebra will get a lift

Complicated-valued computations

Working with advanced tensors will get simpler. complex64 LU factorization, LU resolve and triangular resolve now run on MPS, which additionally unlocks the operations and gradients constructed on them, equivalent to advanced determinants. This is a small system solved straight on the GPU:

import torch

a = torch.randn(8, 8, system="mps", dtype=torch.complex64)
a = a + 8 * torch.eye(8, system="mps", dtype=torch.complex64)
b = torch.randn(8, 2, system="mps", dtype=torch.complex64)

lu, pivots = torch.linalg.lu_factor(a)
x = torch.linalg.lu_solve(lu, pivots, b)
print((a @ x - b).abs().max().merchandise())

Small matrices and bigger solves

Small matrices get quick kernels too. For digital camera geometry and picture transformations, batches of FP32 matrices as much as 4 × 4 are inverted in a single Metallic kernel that mixes factorization and fixing. Bigger issues get consideration as nicely: QR makes use of blocked updates, and triangular solves flip a lot of their work into environment friendly matrix multiplication.

Gated DeltaNet workloads get a devoted quick path for batches of 16 × 16, 32 × 32 and 64 × 64 FP32 triangular methods with not less than eight right-hand sides.

SVD can also be extra strong at numerical extremes. It now accurately handles very small singular values, very giant or very small enter magnitudes and foundation completion with full_matrices=True, which in flip makes pseudoinverse and least-squares outcomes extra dependable.

Convolutions, padding, and coaching

Audio fashions, video decoders and coaching loops have their very own costly operations. This launch provides a number of of them new paths via Metallic:

Softmax and reductions

Token sampling will get quicker in 2.15 as nicely. softmax and log_softmax now run on devoted Metallic kernels, and enormous vocabularies profit probably the most: an FP32 softmax over a [1, 128256] row of logits drops from 57 µs to 7 µs, about 8× quicker than the MPSGraph model. The speedup holds throughout different frequent vocabulary sizes:

A a lot smaller reminiscence footprint for video decoding

Probably the most putting outcomes on this launch comes from the constant-padding change. The VAE decode in Wan-AI/Wan2.2-TI2V-5B-Diffusers makes 8,764 padding calls over a full 28-tile decode. With padding on Metallic, peak driver reminiscence drops from about 270 GiB to eight GiB:

Extra management with MPS streams

Now you can decide which MPS stream runs your work, straight from Python. torch.mps.Stream and the torch.mps.stream context supervisor present the interface, and Tensor.record_stream lets the allocator monitor a tensor’s use on one other stream.

This is a small instance that sends work to a separate stream and synchronizes earlier than studying the outcome:

import torch

x = torch.randn(1024, system="mps")
torch.mps.synchronize()
employee = torch.mps.Stream()

with torch.mps.stream(employee):
    y = x.sq.()
    x.record_stream(employee)

del x
employee.synchronize()
print(y.cpu().form)

Extra operations work on MPS

A number of extra operations now run on MPS: torch.frexp, torch.logspace, torch.take and Tensor.put_, torch.heaviside, torch.particular.laguerre_polynomial_l and unsigned-integer comparisons. logspace would not but assist integer or boolean outputs with multiple aspect.

Consideration will get a correctness repair too: the MPS math implementation of scaled dot-product consideration now handles batch and head broadcasting accurately.

A sneak peek at PyTorch 2.16

Work on 2.16 is already nicely underway. A number of the modifications under have landed on most important and can be found within the nightlies; others are nonetheless in assessment, so particulars could change earlier than the discharge.

Quicker LLM decoding

As soon as a mannequin’s decode path now not incorporates any MPSGraph ops, the GPU solely begins on a token after the CPU has completed encoding all of its kernels. PyTorch 2.16 commits the command buffer each 16 kernels, so the GPU will get to work whereas the CPU remains to be encoding the remainder. For compiled BF16 decoding at a 2048-token context:

Consideration runs fully on Metallic

Scaled dot-product consideration is dropping MPSGraph fully: each form will run on Metallic kernels, with devoted vector kernels for decoding and Metallic Efficiency Primitives for half-precision prefill on newer macOS variations. The shapes that used to fall again to MPSGraph profit probably the most. On an M5 Professional, decoding with grouped-query consideration (32 question heads, 8 KV heads, head dim 80, 4K context) drops from 465 µs to 65 µs, and a 4K × 4K prefill with head dim 40 goes from 42 ms to six ms.

Extra MPSGraph to Metallic migrations

The transfer away from MPSGraph continues. Batch norm ahead and backward are in progress, whereas MSE loss (as much as 12× quicker on small inputs), std_mean and var_mean (as much as 4× quicker) and bincount have already landed. bincount is probably the most dramatic: when most values fall right into a single bin, counting 4 million integers used to take 6.8 seconds and now takes half a millisecond.

A smaller set up

Bundled shaders will construct for Metallic 3.1 by default, with Metallic 4 variations compiled just for the kernels that want them. That shrinks libtorch_cpu.dylib by about 25%, from 384 MiB to 287 MiB, and lays the groundwork for Metallic 4.1 shaders with quicker matmuls and native FP8 matrix multiplication.

Wrapping up

PyTorch 2.15 brings FP8 to the Mac, quicker LLM decoding for dense and MoE fashions, extra linear algebra on the GPU and a protracted listing of operations moved from MPSGraph to devoted Metallic kernels. Give the discharge candidate a strive!

What’s subsequent for us

• Extra MPS: 2.16 will run consideration fully on Metallic, and we’ll maintain shifting operations off MPSGraph.
• torch.compile: we’re engaged on eliminating graph breaks, so extra fashions compile finish to finish.
• Your concepts: what else ought to we work on in PyTorch? If one thing is gradual, lacking or damaged, on MPS or wherever else, open a problem on GitHub and tell us.

An enormous thanks to Nikita Shulga, Kurt Mohler and Aaron Gokaslan for reviewing a lot of this work, and to everybody who contributed to MPS on this launch.