Experiments

5 min read

GPU

GPU Components

  • Command Processor
  • Streaming Multiprocessors (SMs) (NVIDIA) / Compute Units (CUs) (AMD)
    • Shading Units (NVIDIA CUDA Cores) / Stream Processors (AMD)
    • Special Function Units (SFUs)
    • Warp / Wavefront Schedulers
    • Registers
    • Shared Memory (NVIDIA) / Local Data Share (AMD)
    • L1 Cache
  • Internal Interconnect
  • Memory Controllers
  • Global Memory (VRAM)
  • Shared L2 Cache
  • Graphics Resources
    • Texture Mapping Units (TMUs)
    • Render Output Units (ROPs)
  • PCIe Host Interface

flowchart TB
    subgraph CPU_Die["CPU"]
        Cores["CPU Cores"]
        Fabric["Internal Interconnect"]
        IMC["Integrated Memory Controller"]
        RC["PCIe Root Complex"]

        Cores <--> Fabric
        Fabric <--> IMC
        Fabric <--> RC
    end

    DDR["DDR Memory Bus"]
    RAM["Main Memory (RAM)"]
    Link["PCIe Link (Physical Lanes)"]

    subgraph GPU["Discrete GPU"]
        PHI["PCIe Host Interface"]
        CP["Command Processor"]
        GPUFabric["GPU Internal Interconnect / Fabric"]

        subgraph SM1["SM / CU 1"]
            L1_1["L1 Cache"]
        end

        subgraph SM2["SM / CU 2"]
            L1_2["L1 Cache"]
        end

        L2["Shared L2 Cache"]
        MC["Memory Controllers"]
        MemBus["GDDR/HBM Memory Interface"]
        VRAM["Global Memory (VRAM)"]

        subgraph Graphics["Graphics Resources"]
            TMU["Texture Mapping Units (TMUs)"]
            ROP["Render Output Units (ROPs)"]
        end

        PHI <--> GPUFabric
        GPUFabric <--> CP
        GPUFabric <--> SM1
        GPUFabric <--> SM2
        GPUFabric <--> L2
        GPUFabric <--> MC
        GPUFabric <--> Graphics
        MC <--> MemBus
        MemBus <--> VRAM
    end

    IMC <--> DDR
    DDR <--> RAM

    RC <--> Link
    Link <--> PHI

CPU vs GPU

  • CPU and GPU cores are architecturally different; their core counts and clock speeds are not directly comparable
  • These are differences in strengths, not exclusive capabilities: CPUs can execute parallel work, and GPUs can handle branching logic
AspectCPUGPU
Design goalLow-latency execution for flexible, general-purpose workHigh throughput across many parallel operations
Core organizationRelatively few complex cores, typically with branch prediction, large caches, and advanced instruction executionMany execution lanes grouped into SMs (NVIDIA) or CUs (AMD)
Instruction schedulingEach core fetches and decodes its instruction stream

OS schedules software threads onto CPU cores		A command processor consumes submitted command buffers and dispatches work

SM/CU schedulers select warps or wavefronts to execute
Parallel executionSupports parallel threads and SIMD/vector operationsExecutes many threads in groups (SIMT)

Switching among groups helps keep execution units busy while others wait for data
MemoryCaches and system memory (RAM)Discrete GPUs use high-bandwidth VRAM

Integrated GPUs typically share system memory
Typical strengthsSequential tasks, branching logic, low-latency work, and coordinating the systemGraphics, matrix and vector operations, and other data-parallel workloads
Common roleRuns applications, prepares data and commands, and submits work to the GPUPerforms parallel graphics or compute work, often while the CPU continues other tasks
Performance considerationUsually avoids a separate CPU-to-GPU memory transferDiscrete GPUs may need data copied over PCIe; GPU work is most beneficial when it outweighs submission and transfer overhead

GPU Features

Clock

  • Specs do not list GPU Clock Frequency Multiplier like CPU
  • Actual GPU Clock Frequency is dynamic, measurable operating value, rather than a single fixed spec
  • Base Clock Frequency
    • Manufacturer-specified base operating frequency of the GPU core
    • Example: 1485 MHz
  • Turbo/Boost Clock Frequency
    • Rated higher frequency that the GPU can reach when conditions allow it
    • Actual operating frequency may exceed the rated boost clock
    • Example: 1665 MHz
  • Memory Clock
    • Operating frequency of the GPU’s video memory (VRAM)
    • Effective data rate is higher because GDDR memory transfers multiple bits per clock cycle
    • Example: 2001 MHz with effective data rate of 8 Gbps

Architecture

  • Processor size
    • Approx transistor size
    • Example: 12nm
  • Bandwidth: 128 GB/s
  • Bus Interface
    • Example: PCIe 3.0 x16
  • Outputs
    • 1x DVI
    • 1x HDMI 2.0
    • 1x DisplayPort 1.4a

Memory

  • aka VRAM (Video RAM)
  • Memory Size
    • Example: 4GB
  • Memory Type
    • Example: GDDR5
  • Memory Bus
    • Example: 128 bit

GPU Processing Units

  • SM Count (Streaming Multiprocessors)
    • aka Compute units
    • Number of major processing units in an NVIDIA GPU
    • Each SM contains CUDA cores, schedulers, registers, and other resources
    • Example: 14
    • Shading Units (CUDA Cores)
      • Perform arithmetic and shader operations
      • Example: 896 (64 per SM)
        • 14 SMs × 64 CUDA cores = 896
  • TMUs (Texture Mapping Units)
    • Handle texture sampling and filtering
    • Example: 56
  • ROPs (Render Output Units)
    • Handle pixel output operations, including blending and writing pixels to memory
    • Example: 32

Numerical Format Support

  • Vector
    • INT8, INT16, INT32
    • FP16, FP32, FP64
    • BF16
  • Matrix
    • INT8
    • FP8, FP16
    • BF16
      • Brain Floating point
      • Developed at Google Brain
      • 16-bit floating point with wider range than FP16
    • MXFP4, MXFP8, MXFP16
      • Micro scaling formats
    • TF32
      • NVIDIA TensorFloat-32 format
    • NVFP4
      • 4-bit NVIDIA Floating Point format

Theoretical Performance

  • See Local LLM Compute
  • Performance measure
    • FLOPS: Floating Point operations per second
    • TFLOPS: Tera FLOPS
    • GFLOPS: Giga FLOPS
    • 2:1 and 1:32 indicate relative throughput ratios compared with the FP32
  • FP16
    • 16-bit Floating point computation
    • Example: 5.967 TFLOPS (2:1)
  • FP32
    • 32-bit Floating point computation
    • Example: 2.984 TFLOPS
  • FP64
    • 64-bit Floating point computation
    • Example: 93.24 GFLOPS (1:32)

Cache

  • L1 Cache
    • Example: 64 KB (per SM)
  • L2 Cache
    • Example: 1024 KB shared

Graphics Features

  • DirectX: 12 Ultimate (12_2)
    • Microsoft graphics API, primarily used by Windows games and applications
  • OpenGL: 4.6
    • Cross-platform graphics API
  • OpenCL: 3.0
    • Cross-platform API for parallel computing on GPUs and other processors
  • Vulkan: 1.4
    • Cross-platform, low-overhead graphics and compute API
  • CUDA: 12.0
    • NVIDIA’s parallel computing platform and programming model for GPU computing