51工具盒子Microsoft 和合作伙伴社区创建了 ONNX 作为表示机器学习模型的开放标准。 许多框架（包括 TensorFlow、PyTorch、SciKit-Learn、Keras、Chainer、MXNet、MATLAB 和 SparkML）中的模型都可以导出或转换为标准 ONNX 格式。 模型采用 ONNX 格式后，可在各种平台和设备上运行。
本文记录Python 平台 ONNX-GPU 相关内容。
简介 {#简介}
ONNX Runtime是一个跨平台的推理和训练机器学习加速器。
支持Linux和Windows平台CPU和GPU运算，对mac和手机终端也有相应支持。
| System | CPU | GPU | EPs |
|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Windows | 
| 
| 
|
| Linux | 
| 
| 
|
| Mac | 
| | |
| Android | | | 
|
| iOS | | | 
|
| WebAssembly | | | 
|
- ONNX Runtime inference
ONNX运行时推断可以实现更快的客户体验和更低的成本，支持来自深度学习框架（如PyTorch和TensorFlow/Keras）的模型，以及经典的机器学习库（如scikit learn、LightGBM、XGBoost等）。ONNX运行时与不同的硬件、驱动程序和操作系统兼容，并通过在适用的情况下利用硬件加速器以及图形优化和转换提供最佳性能。
官方文档：https://www.onnxruntime.ai/docs/#onnx-runtime-for-inferencing
ONNX Runtime training
通过对现有PyTorch培训脚本进行单行添加，ONNX Runtime培训可加快多节点NVIDIA GPU上变压器模型的模型培训时间。
官方文档：https://www.onnxruntime.ai/docs/#onnx-runtime-for-training
安装 {#安装}
- 在Python下安装，CPU版：
- 在Python下安装，GPU版：
版本要求 {#版本要求}
onnx与onnxruntime版本关系 {#onnx与onnxruntime版本关系}
官方链接：https://github.com/microsoft/onnxruntime/blob/master/docs/Versioning.md#version-matrix
| ONNX Runtime release version | ONNX release version | ONNX opset version | ONNX ML opset version | Supported ONNX IR version | Windows ML Availability | |------------------------------|----------------------|--------------------|-----------------------|---------------------------|--------------------------------------------------------------------------------------------------| | 1.8.0 | 1.8 down to 1.2 | 13 | 2 | 7 | Windows AI 1.8+ | | 1.7.0 | 1.8 down to 1.2 | 13 | 2 | 7 | Windows AI 1.7+ | | 1.6.0 | 1.8 down to 1.2 | 13 | 2 | 7 | Windows AI 1.6+ | | 1.5.3 | 1.7 down to 1.2 | 12 | 2 | 7 | Windows AI 1.5+ | | 1.5.2 | 1.7 down to 1.2 | 12 | 2 | 7 | Windows AI 1.5+ | | 1.5.1 | 1.7 down to 1.2 | 12 | 2 | 7 | Windows AI 1.5+ | | 1.4.0 | 1.7 down to 1.2 | 12 | 2 | 7 | Windows AI 1.4+ | | 1.3.1 | 1.7 down to 1.2 | 12 | 2 | 7 | Windows AI 1.4+ | | 1.3.0 | 1.7 down to 1.2 | 12 | 2 | 7 | Windows AI 1.3+ | | 1.2.0 1.1.2 1.1.1 1.1.0 | 1.6 down to 1.2 | 11 | 2 | 6 | Windows AI 1.3+ | | 1.0.0 | 1.6 down to 1.2 | 11 | 2 | 6 | Windows AI 1.3+ | | 0.5.0 | 1.5 down to 1.2 | 10 | 1 | 5 | Windows AI 1.3+ | | 0.4.0 | 1.5 down to 1.2 | 10 | 1 | 5 | Windows AI 1.3+ | | 0.3.1 0.3.0 | 1.4 down to 1.2 | 9 | 1 | 3 | Windows 10 2004+ | | 0.2.1 0.2.0 | 1.3 down to 1.2 | 8 | 1 | 3 | Windows 10 1903+ | | 0.1.5 0.1.4 | 1.3 down to 1.2 | 8 | 1 | 3 | Windows 10 1809+ |
onnxruntime 与 CUDA cuDNN版本要求 {#onnxruntime-与-CUDA-cuDNN版本要求}
| ONNX Runtime | CUDA | cuDNN | Notes | |--------------|--------|----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1.8 | 11.0.3 | 8.0.4 (Linux) 8.0.2.39 (Windows) | libcudart 11.0.221 libcufft 10.2.1.245 libcurand 10.2.1.245 libcublasLt 11.2.0.252 libcublas 11.2.0.252 libcudnn 8.0.4 libcupti.so 2020.1.1 | | 1.7 | 11.0.3 | 8.0.4 (Linux) 8.0.2.39 (Windows) | libcudart 11.0.221 libcufft 10.2.1.245 libcurand 10.2.1.245 libcublasLt 11.2.0.252 libcublas 11.2.0.252 libcudnn 8.0.4 | | 1.5-1.6 | 10.2 | 8.0.3 | CUDA 11 can be built from source | | 1.2-1.4 | 10.1 | 7.6.5 | Requires cublas10-10.2.1.243; cublas 10.1.x will not work | | 1.0-1.1 | 10.0 | 7.6.4 | CUDA versions from 9.1 up to 10.1, and cuDNN versions from 7.1 up to 7.4 should also work with Visual Studio 2017 |
参考资料 {#参考资料}
- https://github.com/microsoft/onnxruntime
- https://blog.csdn.net/zxgmlcj/article/details/103279869
- https://zhuanlan.zhihu.com/p/261307813
- https://github.com/open-mmlab/mmdetection/blob/master/docs/tutorials/pytorch2onnx.md
- https://www.jianshu.com/p/3a51f7d3357f
文章链接：
https://www.zywvvd.com/notes/study/deep-learning/deploy/onnx-gpu/onnx-gpu/
