TrnRT
Open-source LLM inference engine for AWS Trainium
Visit website →About
TrnRT is an open-source inference engine for large language models on AWS Trainium. It uses NKI megakernels — one fused kernel per layer half — and achieves high throughput with up to 1M-token context. Provides an OpenAI-compatible API.
The currently supported model is GLM-5.3-Flash (320B parameters, 18B active, FP8). The engine runs two serving engines per trn2.48xlarge box, each tensor-parallel over 32 NeuronCores. It achieves 3.57M tokens per minute at $0.040 per million tokens.
The project is from RunAnywhere, backed by Y Combinator. All graphs are compiled offline on a CPU host; the serving box just loads them. The project is Apache 2.0 licensed and built to support more models.
AI-assisted draft, human-reviewed before publishing — see how we choose & review tools.
Why we picked it
The first full-featured open-source inference engine for AWS Trainium. Proves alternative hardware can compete with NVIDIA using open software. Important for any team seeking affordable AI infrastructure.