SkatterBencher AI Benchmark Leaderboard

The SkatterBencher AI Benchmark leaderboard consists of all benchmark results from SkatterBencher CPU and GPU overclocking guides. You can find more information about how the benchmark is configured below the AI Benchmark leaderboard.

AI Benchmark Leaderboard

SKUOC StrategyBenchmark Score
NVIDIA RTX A5000 LaptopWSL_CUDNN26830
NVIDIA RTX A5000 LaptopWSL_TENSORRT26767
NVIDIA RTX A5000 LaptopWN_DIRECTML26318
NVIDIA RTX A5000 LaptopWSL_DIRECTML23287
Intel Arc A770SB64_OCS416935
Intel Arc A770SB64_OCS316561
NVIDIA GeForce RTX 3050SB62_OCS216418
Intel Arc A770SB64_OCS215972
NVIDIA GeForce RTX 3050SB62_OCS115847
Intel Arc A770SB64_OCS115669
NVIDIA GeForce RTX 3050SB62_OCS315167
NVIDIA GeForce RTX 3050SB62_EVGA14556
Intel Arc A770SB64_Stock14325
NVIDIA GeForce RTX 3050SB62_Stock14209
AMD Radeon 780MSB70_OCS512738
Intel Xeon w7-3465XSB63_OCS412100
AMD Radeon 780MSB70_OCS411705
AMD Radeon 780MSB70_OCS311650
Intel Xeon w7-3465XSB63_OCS311431
AMD Radeon 780MSB70_OCS210688
AMD Radeon 780MSB70_OCS110353
AMD Radeon 780MSB70_Stock9986
AMD Ryzen Threadripper 7960XSB71_OCS49605
Intel Xeon w5-3435XSB61_OCS49591
AMD Ryzen Threadripper 7980XSB66_OCS29567
Intel Xeon w7-2495XSB59_OCS49492
AMD Ryzen Threadripper 7980XSB66_OCS29485
AMD Ryzen Threadripper 7980XSB66_OCS19477
AMD Ryzen Threadripper 7980XSB66_OCS19438
AMD Ryzen Threadripper 7980XSB66_OCS39426
Intel Xeon w7-2495XSB59_OCS29354
AMD Ryzen Threadripper 7960XSB71_OCS39350
AMD Ryzen Threadripper 7980XSB66_Stock9327
Intel Xeon w5-3435XSB61_OCS39283
AMD Ryzen Threadripper 7980XSB66_OCS49270
Intel Xeon w7-2495XSB59_OCS39033
AMD Ryzen Threadripper 7980XSB66_Stock9023
Intel Xeon w7-3465XSB63_OCS18988
AMD Ryzen Threadripper 7960XSB71_OCS28840
AMD Ryzen Threadripper 7960XSB71_OCS18798
Intel Xeon w7-3465XSB63_Stock8599
Intel Xeon w5-3435XSB61_OCS28541
AMD Ryzen Threadripper 7960XSB71_Stock8501
Intel Xeon w7-2495XSB59_OCS17988
Intel Xeon w7-2495XSB59_Stock7593
AMD Ryzen 9 7950XSB45_OCS57497
AMD Ryzen 9 7950XSB45_OCS37484
AMD Ryzen 9 7950XSB45_OCS47441
AMD Ryzen 9 7950XSB45_OCS27363
AMD Ryzen 9 7950X3DSB56_OCS27080
AMD Ryzen 9 7950XSB45_OCS16990
Intel Xeon w5-3435XSB61_OCS16981
Intel Core i9-13900KSB49_OCS36980
AMD Ryzen 9 7950X3DSB56_OCS36956
Intel Core i9-13900KSB49_OCS26954
Intel Core i9-14900KSB67_OCS36949
Intel Core i9-14900KSB67_OCS46943
Intel Core i9-13900KSB49_OCS46935
AMD Ryzen 9 7950X3DSB56_OCS16928
Intel Xeon w5-3435XSB61_Stock6800
Intel Core i9-14900KSB67_OCS26780
Intel Core i9-13900KSB49_OCS16754
AMD Ryzen 9 7900SB54_OCS56687
AMD Ryzen 9 7900SB54_OCS36657
AMD Ryzen 9 7900SB54_OCS46587
Intel Arc A380SB44_OCS46582
AMD Ryzen 9 7950XSB45_Stock6580
Intel Core i9-14900KSB67_OCS16552
Intel Core i9-13900KSSB53_OCS46526
AMD Ryzen 9 7900SB54_OCS26489
Intel Core i9-13900KSSB53_OCS36476
AMD Ryzen 9 7900XSB46_OCS46466
AMD Ryzen 9 7900XSB46_OCS36457
AMD Ryzen Threadripper 5990XSB43_OCS56426
AMD Ryzen 9 7900XSB46_OCS56407
AMD Ryzen 9 7950X3DSB56_Stock6405
Intel Arc A380SB44_OCS36382
AMD Ryzen 9 7900XSB46_OCS26359
Intel Core i7-14700KSB68_OCS56331
Intel Arc A380SB44_OCS26331
AMD Ryzen 9 7900SB54_OCS16328
Intel Core i7-14700KSB68_OCS46321
AMD Ryzen Threadripper 5990XSB43_OCS46315
AMD Ryzen 9 7900XSB46_OCS16311
AMD Ryzen Threadripper 5990XSB43_OCS3_NoSMT6252
Intel Core i9-13900KSSB53_OCS26219
Intel Core i7-14700KSB68_OCS36212
Intel Core i7-14700KSB68_OCS26211
AMD Ryzen 9 7900X3DSB58_OCS36195
Intel Core i9-13900KSB49_Stock6192
AMD Ryzen 9 7900X3DSB58_OCS26125
AMD Ryzen 9 7900X3DSB58_OCS16098
Intel Arc A380SB44_OCS16098
Intel Core i9-13900KSSB53_OCS16073
Intel Core i7-14700KSB68_OCS16006
AMD Ryzen Threadripper 5990XSB43_OCS2_NoSMT5952
AMD Ryzen 9 7900XSB46_Stock5951
AMD Ryzen Threadripper 5990XSB43_OCS1_NoSMT5950
Intel Arc A380SB44_Stock5799
Intel Core i9-13900K P-CoreSB52_OCS55773
Intel Core i7-13700KSB50_OCS45764
Intel Core i9-13900K P-CoreSB52_OCS45755
AMD Ryzen 9 7900X3DSB58_Stock5710
Intel Core i9-13900K P-CoreSB52_OCS35670
Intel Core i9-14900KSB67_Stock5663
AMD Ryzen 7 7800X3DSB60_OCS45655
Intel Core i7-13700KSB50_OCS35629
AMD Ryzen 7 7700XSB47_OCS35587
AMD Ryzen 7 7700XSB47_OCS45576
AMD Ryzen 7 7700XSB47_OCS25548
AMD Ryzen 7 7700XSB47_OCS55541
Intel Core i9-13900K P-CoreSB52_OCS25539
AMD Ryzen 7 7700XSB47_OCS65528
AMD Ryzen 7 7800X3DSB60_OCS55527
AMD Ryzen 7 7800X3DSB60_OCS25473
Intel Core i9-13900K P-CoreSB52_OCS15471
AMD Ryzen 7 7700XSB47_OCS15469
AMD Ryzen 7 7800X3DSB60_OCS35468
Intel Core i9-13900KSSB53_Stock5448
AMD Ryzen Threadripper 5990XSB43_OCS35401
Intel Core i7-14700KSB68_Stock5370
AMD Ryzen 9 7900SB54_Stock5368
AMD Ryzen 7 7800X3DSB60_OCS15363
AMD Ryzen Threadripper 3990XSB36_OCS55344
AMD Ryzen Threadripper 3990XSB36_OCS2_NoSMT5325
AMD Ryzen Threadripper 5990XSB43_OCS25227
AMD Ryzen Threadripper 3990XSB36_OCS1_NoSMT5212
AMD Ryzen 7 7700XSB47_Stock5172
AMD Ryzen Threadripper 5990XSB43_OCS15158
Intel Core i9-12900KFSB34_OCS45153
AMD Ryzen 7 7800X3DSB60_Stock5144
Intel Core i9-12900KSB30_OCS55115
Intel Core i7-13700KSB50_OCS25100
Intel Core i7-13700KSB50_OCS15062
Intel Core i9-12900KSB30_OCS25044
Intel Core i9-12900KFSB34_OCS34989
Intel Core i9-13900K P-CoreSB52_Stock4961
Intel Core i9-12900KFSB34_OCS24821
AMD Ryzen Threadripper 3990XSB36_OCS24820
Intel Core i9-12900KSB30_OCS44754
Intel Core i7-13700KSB50_Stock4727
AMD Ryzen Threadripper 3990XSB36_OCS44687
AMD Ryzen 7 8700GSB69_OCS24680
Intel Core i9-12900KSB30_OCS14650
AMD Ryzen Threadripper 3990XSB36_OCS14632
AMD Ryzen Threadripper 5990XSB43_Stock_NoSMT4594
AMD Ryzen 5 7600XSB48_OCS44575
AMD Ryzen 5 7600XSB48_OCS54573
AMD Ryzen 5 7600XSB48_OCS64570
AMD Ryzen 5 7600XSB48_OCS34562
Intel Core i9-12900KFSB34_OCS14550
AMD Ryzen 7 8700GSB69_OCS44544
AMD Ryzen 7 8700GSB69_OCS14544
AMD Ryzen 5 7600XSB48_OCS24529
Intel Core i9-12900KSB30_OCS34498
Intel Core i7-12700KSB31_OCS44494
AMD Ryzen 5 7600XSB48_OCS14491
Intel Core i7-12700KSB31_OCS34454
AMD Ryzen Threadripper 3990XSB36_OCS34418
AMD Ryzen 7 8700GSB69_OCS34407
Intel Core i5-13600KSB51_OCS34337
Intel Core i9-12900KSB30_Stock4324
Intel Core i5-13600KSB51_OCS44301
Intel Core i7-12700KSB31_OCS24285
AMD Ryzen Threadripper 5990XSB43_Stock4277
Intel Core i9-12900KFSB34_Stock4269
AMD Ryzen 5 7600XSB48_Stock4205
AMD Ryzen 3 5300GE (IGP)SB35_OCS34202
AMD Ryzen 3 5300GE (IGP)SB35_OCS44200
Intel Core i7-12700KSB31_OCS14115
AMD Ryzen 5 8600GSB72_OCS34084
AMD Ryzen 5 8600GSB72_OCS24037
AMD Ryzen Threadripper 3990XSB36_Stock_NoSMT4023
Intel Core i7-12700KSB31_Stock3939
AMD Ryzen 5 8600GSB72_OCS13837
Intel Core i5-13600KSB51_OCS23813
Intel Core i5-13600KSB51_OCS13748
AMD Ryzen 7 8700GSB69_Stock3746
Intel Core i5-12400SB37_OCS33680
Intel Core i5-12400SB37_OCS43648
AMD Ryzen Threadripper 3990XSB36_Stock3626
AMD Ryzen 7 5800X3DSB39_OCS33609
AMD Ryzen 3 5300GE (IGP)SB35_OCS23596
AMD Ryzen 5 8600GSB72_Stock3591
Intel Core i5-13600KSB51_Stock3563
AMD Ryzen 7 5800X3DSB39_OCS13480
AMD Ryzen 7 5800X3DSB39_OCS23473
AMD Ryzen 3 5300GE (IGP)SB35_OCS13470
Intel Core i5-12600KFSB32_OCS33257
AMD Ryzen 7 5800X3DSB39_Stock3196
Intel Core i5-12600KFSB32_OCS23196
Intel Core i5-12400SB37_OCS23195
Intel Core i5-12600KFSB32_OCS13130
Intel Core i9-11980HKSB65_OCS33088
Intel Core i5-12400SB37_OCS13083
Intel Core i9-11980HKSB65_OCS23079
AMD Ryzen 3 5300GE (IGP)SB35_Stock2775
Intel Core i9-11980HKSB65_OCS12771
Intel Core i9-11900HWSL_DIRECTML_ONEDNN2735
Intel Core i9-11900HWN_DIRECTML_ONEDNN2713
Intel Core i5-12600KFSB32_Stock2681
AMD Radeon Graphics (Ryzen 7000)SB55_OCS32645
AMD Radeon Graphics (Ryzen 7000)SB55_OCS22622
Intel Core i9-11980HKSB65_Stock2575
Intel Core i5-12400SB37_Stock2551
Intel Core i9-11900HWSL_DIRECTML2240
Intel UHD Graphics 770 (13th Gen)SB57_OCS42072
AMD Radeon Graphics (Ryzen 7000)SB55_OCS12057
Intel UHD Graphics 770 (13th Gen)SB57_OCS32055
Intel UHD Graphics 770 (12th Gen)SB33_OCS42046
AMD Radeon Graphics (Ryzen 7000)SB55_Stock2037
Intel UHD Graphics 770 (12th Gen)SB33_OCS31949
Intel UHD Graphics 770 (12th Gen)SB33_OCS21825
Intel UHD Graphics 770 (13th Gen)SB57_OCS21739
AMD Ryzen 3 5300GESB35_OCS31504
AMD Ryzen 3 5300GESB35_OCS41469
AMD Ryzen 3 5300GESB35_OCS21448
Intel UHD Graphics 750 (11th Gen)SB28_OCS61446
AMD Ryzen 3 5300GESB35_OCS11419
Intel UHD Graphics for 11th GenWN_DIRECTML1417
Intel UHD Graphics for 11th GenWSL_DIRECTML1396
Intel UHD Graphics 730 (12th Gen)SB38_OCS41388
Intel UHD Graphics 770 (13th Gen)SB57_OCS11387
Intel UHD Graphics 770 (13th Gen)SB57_Stock1379
Intel UHD Graphics 770 (12th Gen)SB33_OCS11361
Intel UHD Graphics 770 (12th Gen)SB33_Stock1352
Intel UHD Graphics 750 (11th Gen)SB28_OCS51264
Intel UHD Graphics 730 (12th Gen)SB38_OCS31223
AMD Ryzen 3 5300GESB35_Stock1179
Intel UHD Graphics 730 (12th Gen)SB38_OCS21116
Intel UHD Graphics 750 (11th Gen)SB28_OCS41107
Intel UHD Graphics 750 (11th Gen)SB28_OCS31098
Intel UHD Graphics 730 (12th Gen)SB38_OCS11080
Intel UHD Graphics 730 (12th Gen)SB38_Stock1072
Intel UHD Graphics 750 (11th Gen)SB28_OCS21066
Intel UHD Graphics 750 (11th Gen)SB28_Stock1064
Intel UHD Graphics 750 (11th Gen)SB28_OCS11064
Intel Core i9-11900HWN_DIRECTML892

Leaderboard Notes:

  1. OC Strategy is coded as SkatterBencher # + OC Strategy #
  2. The Core i9-13900KS benchmark scores seem off. I noticed a particular issue with the OS for that system where the score would be significantly slower than expected unless the process affinity was manually set.
  3. TensorFlow-DirectML was used for: Intel UHD Graphics 750 (11th Gen), Intel UHD Graphics 730 (12th Gen), Intel UHD Graphics 770 (12th Gen), Intel UHD Graphics 770 (13th Gen), AMD Radeon Graphics (Ryzen 7000), AMD Ryzen 3 5300GE (IGP), and Intel Arc A380.

AI Benchmark

ai benchmark installation guide

AI Benchmark Alpha is an open-source python library for evaluating AI performance of various hardware platforms, including CPUs, GPUs and TPUs. The benchmark is relying on the TensorFlow machine learning library, and is providing a lightweight solution for assessing inference and training speed for key Deep Learning models. The benchmark itself takes about 20 minutes to run and outputs three scores: inference score, training score, and AI-Score. It’s the latter we use as performance measurement.

While there are many ways to run AI Benchmark on your machine, I use an Anaconda Python environment with the latest Tensorflow optimizations.

In this blog post I describe my AI Benchmark process in detail: AI Benchmark to Measure Machine Learning Performance. Below you can find a quick summary

Installing AI Benchmark on Windows Native

  1. First, download and install Anaconda for Windows.
  2. After completing the installation, run the Anaconda Prompt.
  3. Create a new Python environment for the benchmark. Make sure to specify Python version 3.10 as the tensorflow-directml-plugin supports only versions 3.7, 3.8, 3.9, and 3.10
    • conda create -n aibench python=3.10
  4. Activate your newly created environment
    • conda activate aibench
  5. Download and install the base TensorFlow-CPU
    • pip install tensorflow-cpu
  6. Download and install the tensorflow-directml package
    • pip install tensorflow-directml-plugin
  7. Download and install the numpy 1.23 package
    • pip install numpy==1.23
  8. Download and install the ai_benchmark package
    • pip install ai_benchmark

Installing AI Benchmark on Windows Subsystem for Linux

  1. First, make sure WSL is correctly installed on your Windows PC.
    • Open PowerShell and type wsl –install
    • Follow the installation instructions
  2. If that’s the case, then open the WSL prompt.
  3. Now download the Anaconda for Linux 64-Bit (x86) installer.
  4. Next, install Anaconda
    • bash ~/Downloads/Anaconda3-2020.05-Linux-x86_64.sh
  5. Then, close and reopen Windows Subsystem for Linux
  6. Create a new Python environment for the benchmark. Make sure to specify Python version 3.10 as the tensorflow-directml-plugin supports only versions 3.7, 3.8, 3.9, and 3.10
    • conda create -n aibench python=3.10
  7. Activate your newly created environment
    • conda activate aibench
  8. Download and install the base TensorFlow-CPU
    • pip install tensorflow-cpu
  9. Download and install the tensorflow-directml package
    • pip install tensorflow-directml-plugin
  10. Download and install the numpy 1.23 package
    • pip install numpy==1.23
  11. Download and install the ai_benchmark package
    • pip install ai_benchmark

Running the AI Benchmark

Starting the AI Benchmark is the same on Windows Native and Windows Subsystem for Linux.

  • Open the Anaconda prompt
  • Activate the conda environment with AI Benchmark
    • conda activate aibench
  • Start Python
    • python
  • Import the AI Benchmark package
    • from ai_benchmark import AIBenchmark
  • Specify the benchmark parameters
    • benchmark = AIBenchmark(use_CPU=None, verbose_level=3)
      • Use_CPU=True runs the benchmark on the CPU
      • Use_CPU=None runs the benchmark on the GPU
      • Verbose_level=0 runs the test silently
      • Verbose_level=1 runs the test with short summary
      • Verbose_level=2 provides information about each run
      • Verbose_level=3 provides the tensorflow logs during the run
  • Then, lastly, start the benchmark
    • benchmark.run()

AI Benchmark Optimizations and Tricks

Since deep neural network and machine learning performance is a big selling point, companies work hard to release performance-optimizing software packages for their hardware. I tend to use these packages in my SkatterBencher guides.

OneDNN for Intel CPU Architectures

Intel oneDNN is an open-source, high-performance library designed to accelerate deep learning applications on Intel architecture CPUs. It provides optimized primitives for various deep learning operations, such as convolutions, inner products, and other key operations used in neural networks.

The oneAPI Deep Neural Network Library (oneDNN) optimizations are available in the official x86-64 TensorFlow after v2.5. The feature is off by default before v2.9, but users can enable those CPU optimizations by configuring the environment variable TF_ENABLE_ONEDNN_OPTS.

  • Windows Native: set TF_ENABLE_ONEDNN_OPTS=1
  • Windows Subsystem for Linux: export TF_ENABLE_ONEDNN_OPTS=1

Since TensorFlow v2.9, the oneAPI Deep Neural Network Library (oneDNN) optimizations are enabled by default.

ZenDNN for AMD CPU Architectures

I also came across an AMD equivalent library called ZenDNN. However, I’ve yet to try this on an AMD system, so I’ll leave you a link for now.

CuDNN or TensorRT for NVIDIA GPU Architectures

You can also rely on the NVIDIA CUDA Deep Neural Network library (cuDNN) for NVIDIA GPUs. CuDNN is a GPU-accelerated library of primitives for deep neural networks. The installation requires a different version of TensorFlow – not TensorFlow-DirectML – since TensorFlow 2.10 is no longer available on Windows Native.

The installation is pretty straightforward. After installing Anaconda for Linux on WSL, do the following

  • First, create and activate a new Anaconda environment
    • conda create -n aibenchNV
    • conda activate aibenchNV
  • Then, install the appropriate Cuda Toolkit. You can find a support matrix on NVIDIA’s website
    • conda install -c conda-forge cudatoolkit=11.8
  • Next, install the CuDNN package
    • pip install nvidia-cudnn-cu11==8.9.4.25
  • Then, install TensorFlow
    • pip install tensorflow
  • Then, install numpy 1.23
    • pip install numpy==1.23
  • Lastly, install AI Benchmark
    • pip install ai_benchmark

Instead of CuDNN, you can also consider installing the TensorRT Python library. Similarly, TensorRT is a deep-learning library powered by CUDA. TensorRT provides APIs and parsers to import trained models from all major deep learning frameworks. It then generates optimized runtime engines deployable in the data center, automotive, and embedded environments.

RocM for AMD GPU Architectures

Lastly, I also want to mention AMD’s ROCm software package. ROCm is an open-source software stack for GPU computation featuring a collection of drivers, development tools, and APIs enabling GPU programming from the low-level kernel to end-user applications.

While ROCm is fully integrated into ML frameworks such as PyTorch and TensorFlow, it’s currently unavailable on Windows Native, so I’ve yet to use ROCm optimizations for AMD graphics cards. So, just like with ZenDNN, I’ll leave you with a link to the documentation.

Multiple GPUs

These days it’s common to have multiple graphics devices in a single system. Usually, that’s the integrated graphics of the CPU and a high-performance discrete graphics card. If you want to switch between the graphics device, you can set an environment variable before starting Python.

  • Windows Native: set DML_VISIBLE_DEVICES=0,1
  • Windows Subsystem for Linux: export DML_VISIBLE_DEVICES=0,1

here the number indicates the specific device. Device 0 is the NVIDIA dGPU on the EK Flat PC, and device 1 is the Intel iGPU. By default, AI Benchmark will run the first available device. So, if I want to run on the iGPU, I’d have to set DML_VISIBLE_DEVICES=1.

DXGI_ERROR_DEVICE_REMOVED

The AI Benchmark is a pretty tough benchmark that can take a long time. Sometimes, you may run into an error called DXGI_ERROR_DEVICE_REMOVED while running the benchmark. That happens when there’s a timeout when the device takes too long to complete a workload.

You can increase the timeout with the registry entry “TdrDelay” to solve the issue. This registry entry will extend the time a software application waits for the IGP. https://docs.microsoft.com/en-us/windows-hardware/drivers/display/tdr-registry-keys

  • KeyPath: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers
  • KeyValue: TdrDelay
  • ValueType: REG_DWORD
  • ValueData: Number of seconds to delay. The default value is 2 seconds.

Subscribe To Newsletter

Archives