OCuLink vs PCIe for LLMs… The Result I Didn’t Expect

Summary

This video explores the capabilities and features of GMK Tech's mini PCs, specifically the Evo T1 and X2 models, highlighting their performance, portability, and expandability. The speaker compares them to other devices, like Apple’s MacBook Airs, and tests them with various LLM models to showcase GPU performance and token generation speeds. A key focus is on the Oculink port, which enables external GPU expansion, as well as the differences in performance between iGPUs and discrete GPUs. The video also delves into the advantages and trade-offs of expandability with external docks.

Takeaways

• 😀 GMK Tech is leading the way with mini PCs that offer portability and expandability, particularly through the inclusion of Oculink ports.
• 😀 The Evo T1 mini PC is a powerful machine designed for AI tasks, featuring 128GB of memory and versatile IO options like HDMI, USB 4, and dual Ethernet ports.
• 😀 Mini PCs offer a balance between portability and performance, combining desktop-level functionality with laptop-sized form factors.
• 😀 One standout feature in mini PCs is the Oculink port, which allows for expansion and external GPU connectivity, enhancing performance for AI workloads.
• 😀 The Intel Core Ultra 285H chip is used in several of the tested machines, but the focus is on comparing GPU performance for large language models (LLMs).
• 😀 There’s a noticeable difference in performance between iGPUs and discrete GPUs when running AI models, with discrete GPUs consistently outperforming iGPUs in token generation.
• 😀 The Evo X2 with an AMD Strix Halo chip shows superior performance for large language model tasks compared to the Evo T1, especially for longer prompts.
• 😀 Performance comparisons reveal that even older machines like the 2020 MacBook Air with M1 chip perform better in some medium-sized prompts than newer mini PCs like the Evo T1.
• 😀 When expanding the system with external GPUs through Oculink or PCI Gen 5 docks, performance improvements are noticeable, though the differences are not dramatic for LLM tasks.
• 😀 There’s a trade-off between versatility (e.g., custom GPU choices) and compactness (e.g., cleaner, more organized setups), with systems like the Beink GTI 15 offering flexibility but messy setups.

Q & A

What is the primary difference between mini PCs and traditional desktops or laptops?

Mini PCs sit in between desktops and laptops in terms of portability and expandability. They are portable like laptops but have more I/O ports, such as HDMI, USB 4, and Ethernet, which desktops generally have more of, offering greater flexibility for peripherals and use cases like local AI or gaming.

Why is the Oculink port significant for mini PCs?

The Oculink port is significant because it offers expandability that desktops and laptops lack. It enables users to connect external devices like GPUs or other PCIe cards, making mini PCs more versatile, especially for tasks like AI processing and gaming.

How does the GMK Tech Evo T1 compare to the MacBook Air for running AI models?

For shorter prompts, the MacBook Air (even older models like the M1) performs better than the GMK Tech Evo T1. However, for longer prompts, the GMK Tech Evo T1 surpasses the MacBook Air in terms of tokens per second, although the performance gap is less noticeable for medium-sized tasks.

What is the role of the iGPU in the testing process, and how does it compare to discrete GPUs?

The iGPU (integrated GPU) is used for testing the performance of LLMs (Large Language Models) in generating tokens. While iGPUs provide decent performance, they are generally slower than discrete GPUs, especially for large models or tasks requiring high parallel processing and bandwidth, where discrete GPUs excel.

How does the Beink GTI 15's PCI Gen 5 dock differ from the GMK Tech Evo T1's Oculink port in terms of performance?

The Beink GTI 15’s PCI Gen 5 dock offers significantly faster bandwidth (31.5 GB/s) compared to the GMK Tech Evo T1's Oculink port, which is limited to 7.9 GB/s. This results in the Beink GTI 15 offering better overall performance, especially for tasks that require high GPU throughput like gaming or rendering.

What advantages does using a PCIe external GPU offer over an Oculink external GPU?

Using a PCIe external GPU offers better performance due to higher bandwidth, with PCIe Gen 5 offering up to 31.5 GB/s, compared to Oculink's 7.9 GB/s. This makes PCIe better for tasks like gaming or AI model inference that require fast data transfer between the CPU and GPU.

How do sparse models perform differently compared to dense models in the context of iGPU versus discrete GPU?

Sparse models tend to perform better on iGPUs for shorter prompts due to their regular memory access pattern, which avoids the overhead of transferring data to an external GPU. However, for longer prompts, discrete GPUs show their superiority with faster processing times due to greater parallelism and bandwidth.

Why does the Beink machine perform better than the GMK Tech machine in some benchmarks?

The Beink machine outperforms the GMK Tech machine in certain benchmarks due to the faster PCIe Gen 5 connection, which provides higher bandwidth and better support for intensive tasks. However, the performance difference is often marginal and only noticeable with specific tasks or longer prompts.

What is the potential downside of using the GMK Tech Evo T1 with external GPUs through Oculink?

The GMK Tech Evo T1, while compact and neat, has limitations when using external GPUs through Oculink. The bandwidth is limited (7.9 GB/s), which may impact performance for GPU-heavy tasks. Additionally, Oculink is restricted to specific GPUs and power supplies, making it less flexible compared to PCIe-based systems.

How does power consumption compare between the GMK Tech Evo T1 and the Beink GTI 15 when using external GPUs?

The Beink GTI 15, with its external GPU setup, consumes more power due to the high wattage of the GPU and power supply (around 186 watts), while the GMK Tech Evo T1 consumes less. The power draw varies depending on the GPU used and the type of task being processed.