To access the containers, visit NVIDIA MLPerf Inference.

TensorRT 8.6.0, the latest version of the NVIDIA high-performance deep learning inference SDK, is included on NVIDIA GPU Cloud. The NVIDIA AI platform (the core of the NVIDIA MLPerf submissions) is constantly enhanced with software innovations that increase performance and features, and leverage the capabilities of the latest NVIDIA products and architectures. In the MLPerf Inference v3.0 round, NVIDIA submitted results on several products, including NVIDIA H100 Tensor Core GPUs (both SXM and PCIe add-in-card form factors) based on the NVIDIA Hopper architecture, the recently announced NVIDIA L4 Tensor Core GPU powered by the NVIDIA Ada Lovelace GPU architecture, as well as the NVIDIA Jetson AGX Orin and NVIDIA Jetson Orin NX AI computers for edge AI and robotics applications.

This balance of workload breadth and depth ensures that MLPerf Inference is a valuable resource for those looking to select AI infrastructure to best meet the needs of a diverse set of deployments. It also represents a range of real-world scenarios, such as offline (batch) processing, and latency-constrained server, single-stream, and multi-stream scenarios. MLPerf Inference represents both data center and edge environments. MLPerf Inference-now in its seventh edition with v3.0-is a trusted, peer-reviewed suite of standardized inference performance tests that represents many such AI models.ĪI applications run everywhere, from the largest hyperscale data centers to compact edge devices. This means that an AI platform must be able to run the widest range of workloads and deliver excellent performance on all of them. New AI models are constantly being invented to enable new capabilities, and AI-powered applications often rely on many such models working in tandem. High-performance, accelerated AI platforms are needed to meet the demands of these applications and deliver the best user experiences. The most exciting computing applications currently rely on training and running inference on complex AI models, often in demanding, real-time deployment scenarios.

Nvidia recording