microsoft/onnxruntimePublic

What is onnxruntime?

A cross-platform, high-performance machine learning inferencing and training accelerator.

What does onnxruntime do?

ONNX Runtime is a robust engine designed to execute machine learning models across diverse hardware platforms with maximum efficiency. It accelerates inference by natively supporting models trained in frameworks like PyTorch, TensorFlow, and scikit-learn via the Open Neural Network Exchange (ONNX) format. The runtime optimizes execution through advanced graph transformations and dynamically leverages hardware-specific execution providers, such as TensorRT for NVIDIA GPUs or CoreML for Apple Silicon. Beyond inferencing, it also provides tools to significantly accelerate model training times on multi-node GPU clusters.

Who is onnxruntime for?

Machine learning engineers, AI researchers, and production deployment teams looking to optimize and standardize model inference across varied hardware ecosystems.

How do I get started with onnxruntime?

Install via package managers (e.g., pip install onnxruntime) or refer to the official documentation for language-specific bindings.

How popular is onnxruntime on GitHub?

microsoft/onnxruntime has 21,981 stars and 4,271 forks on GitHub, and gained 54 stars in the last 7 days.

What license does onnxruntime use?

microsoft/onnxruntime is released under the MIT license.

• Widely adopted

  21,981 stars

• Battle-tested

  7 years of history

• Very active

  3,125 commits in 52 weeks

• Community-driven

  ~1,004 contributors

• Well documented

  High community health score

• Permissive license

  MIT

ONNX Runtime is a robust engine designed to execute machine learning models across diverse hardware platforms with maximum efficiency. It accelerates inference by natively supporting models trained in frameworks like PyTorch, TensorFlow, and scikit-learn via the Open Neural Network Exchange (ONNX) format. The runtime optimizes execution through advanced graph transformations and dynamically leverages hardware-specific execution providers, such as TensorRT for NVIDIA GPUs or CoreML for Apple Silicon. Beyond inferencing, it also provides tools to significantly accelerate model training times on multi-node GPU clusters.

Machine learning engineers, AI researchers, and production deployment teams looking to optimize and standardize model inference across varied hardware ecosystems.

• Hardware Acceleration: Utilize specialized execution providers (like CUDA, TensorRT, or DirectML) to maximize performance on specific hardware.
• Framework Agnostic: Seamlessly run models trained in PyTorch, TensorFlow/Keras, and classical ML libraries without framework lock-in.
• Graph Optimization: Automatically apply graph transforms and optimizations to reduce execution latency and memory footprint during inference.
• Cross-Platform Execution: Deploy the exact same optimized model across Windows, Linux, macOS, Android, and iOS environments.
• Training Acceleration: Speed up transformer model training on multi-node NVIDIA GPU setups with minimal changes to existing PyTorch scripts.

Getting started: Install via package managers (e.g., pip install onnxruntime) or refer to the official documentation for language-specific bindings.

1. ONNX Runtime v1.28.3v1.28.3Oct 2, 202634 downloads

   This is a patch release on top of [v1.28.2](https://github.com/microsoft/onnxruntime/releases/tag/v1.28.2), containing memory-safety and model-validation fixes, a plugin execution fix, and packaging infrastructure updates. ## Highlights ### Bug Fixes - Fixed an out-of-bounds read in `EmbedLayerNormalization` shape inference when beta is a scalar ([#32610](https://github.com/microsoft/onnxruntime/pull/32610)). - Fixed a heap buffer overflow when reusing a packed sub-byte buffer for a full-byte tensor ([#32611](https://github.com/microsoft/onnxruntime/pull/32611)). - Fixed an out-of-bounds read when a Loop or Scan body input is also an initializer ([#32609](https://github.com/microsoft/onnxruntime/pull/32609)). - Hardened shape inference for applicable contrib operators, including GroupQueryAttention, SparseAttention, and CausalConvWithState ([#32607](https://github.com/microsoft/onnxruntime/pull/32607)). - Fixed out-of-bounds reads in ConvTransposeWithDynamicPads and `InferenceContextImpl::getInputData` ([#32679](https://github.com/microsoft/onnxruntime/pull/32679), [#32681](https://github.com/microsoft/onnxruntime/pull/32681)). - Rejected invalid graph structures: inp

2. ONNX Runtime v1.30.0v1.30.0Sep 10, 202653.6K downloads

   ONNX Runtime 1.30.0 expands generative AI inference, improves CPU and GPU performance, adds Go bindings, and strengthens runtime reliability. These notes cover changes since ONNX Runtime 1.29.1. ## Highlights - Expanded CUDA inference support with variable-length causal convolution for continuous batching, speculative decoding in paged XQA, and INT4 paged KV caches with per-channel scales ([#32168](https://github.com/microsoft/onnxruntime/pull/32168), [#32340](https://github.com/microsoft/onnxruntime/pull/32340), [#32515](https://github.com/microsoft/onnxruntime/pull/32515)). - Improved WebGPU PagedAttention, added GPT-OSS support and INT8 KV-cache block quantization, and extended convolution optimizations ([#31727](https://github.com/microsoft/onnxruntime/pull/31727), [#32277](https://github.com/microsoft/onnxruntime/pull/32277), [#32284](https://github.com/microsoft/onnxruntime/pull/32284), [#32420](https://github.com/microsoft/onnxruntime/pull/32420)). - Added fused CPU LinearAttention kernels for AVX-512, Arm64 NEON, and SVE, plus AVX2 LayerNorm/RMSNorm acceleration ([#31674](https://github.com/microsoft/onnxruntime/pull/31674), [#31973](https://github.com/microsoft/onn

3. ONNX Runtime v1.29.1v1.29.1Sep 10, 20262K downloads

   This is a patch release on top of [v1.29.0](https://github.com/microsoft/onnxruntime/releases/tag/v1.29.0), containing GroupQueryAttention capability and KV-cache layout improvements, plugin Execution Provider performance tooling updates, and targeted graph and optimizer fixes. ## GroupQueryAttention - Added bidirectional GroupQueryAttention support on CPU and CUDA through a backward-compatible `causal` attribute, with explicit handling for unsupported execution paths ([#31704](https://github.com/microsoft/onnxruntime/pull/31704)) - Added a session option and Execution Provider metadata contract for using the BNHS Value KV-cache layout, with graph transformations that preserve compatibility with the existing BNSH operator schema ([#32139](https://github.com/microsoft/onnxruntime/pull/32139)) - Added CPU support for `attention_bias` with a sliding-window KV cache, including explicit position IDs and post-eviction bias indexing ([#32302](https://github.com/microsoft/onnxruntime/pull/32302)) ## Runtime and Performance Tools - Fixed Compile API model serialization when output-model and custom initializer-location callbacks are used together, preventing duplicate graph fie

4. ONNX Runtime WebGPU Plugin EP v0.4.0plugin-ep-webgpu/v0.4.0Sep 22, 2026

   This release brings kernel performance improvements, expanded operator support, and reliability fixes to the ONNX Runtime WebGPU Plugin EP. ## Highlights ### Performance - Optimized `MatMulNBits` wide-tile execution using subgroup shuffle ([#31703](https://github.com/microsoft/onnxruntime/pull/31703)) - Supplied fused activation parameters to Conv/MatMul as uniforms, and fused eight more activations into Conv ([#32116](https://github.com/microsoft/onnxruntime/pull/32116), [#32117](https://github.com/microsoft/onnxruntime/pull/32117)) - Added fused activation support to the im2col Conv path ([#32185](https://github.com/microsoft/onnxruntime/pull/32185)) - Vectorized `Split` when every output segment is vec4-aligned ([#32251](https://github.com/microsoft/onnxruntime/pull/32251)) - Shared subgroup matrix MatMul with pointwise Conv, and pinned subgroup size to 32 for subgroup-matrix MatMul/Gemm ([#32304](https://github.com/microsoft/onnxruntime/pull/32304), [#32306](https://github.com/microsoft/onnxruntime/pull/32306)) - Selected the pooling path by occupancy rather than output size ([#32313](https://github.com/microsoft/onnxruntime/pull/32313)) - Prepacked Conv weights

5. ONNX Runtime v1.28.2v1.28.2Sep 3, 20265.3K downloads

   This is a patch release on top of [v1.28.1](https://github.com/microsoft/onnxruntime/releases/tag/v1.28.1), containing a targeted fix for Compile API model serialization. ## Highlights ### Bug Fixes - Fixed Compile API callback serialization to prevent duplicate graph nodes, inputs, outputs, and value information in emitted optimized models, including models with embedded or external initializers ([#32303](https://github.com/microsoft/onnxruntime/pull/32303)) ## Contributors Thanks to our contributor for this release! [@adrastogi](https://github.com/adrastogi) Full Changelog: [v1.28.1...v1.28.2](https://github.com/microsoft/onnxruntime/compare/v1.28.1...v1.28.2) > These release notes were drafted with assistance from GitHub Copilot.

• tensorflow/tensorflow

  An Open Source Machine Learning Framework for Everyone

  200.7K stars · C++

• f/prompts.chat

  f.k.a. Awesome ChatGPT Prompts. Share, discover, and collect prompts from the community. Free and open source — self-host for your organization with complete privacy.

  172K stars · HTML

• huggingface/transformers

  🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

  167K stars · Python

• Comfy-Org/ComfyUI

  The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

  136.1K stars · Python

• ggml-org/llama.cpp

  LLM inference in C/C++

  130.3K stars · C++

• rasbt/LLMs-from-scratch

  Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

  106K stars · Jupyter Notebook