TrnRT

Open-source LLM inference engine for AWS Trainium

Visit website →

About

TrnRT is an open-source inference engine for large language models on AWS Trainium. It uses NKI megakernels — one fused kernel per layer half — and achieves high throughput with up to 1M-token context. Provides an OpenAI-compatible API.

The currently supported model is GLM-5.3-Flash (320B parameters, 18B active, FP8). The engine runs two serving engines per trn2.48xlarge box, each tensor-parallel over 32 NeuronCores. It achieves 3.57M tokens per minute at $0.040 per million tokens.

The project is from RunAnywhere, backed by Y Combinator. All graphs are compiled offline on a CPU host; the serving box just loads them. The project is Apache 2.0 licensed and built to support more models.

AI-assisted draft, human-reviewed before publishing — see how we choose & review tools.

Why we picked it

The first full-featured open-source inference engine for AWS Trainium. Proves alternative hardware can compete with NVIDIA using open software. Important for any team seeking affordable AI infrastructure.