M3 max 128GB for AI running Llama2 7b 13b and 70b

Summary

This video compares the inference performance of Llama models on different devices: M3 Max with 40-core CPU, M1 Pro with 16-core CPU, and RTX 4090 on a 16-core AMD CPU. Tests are conducted for models with 7 billion, 13 billion, and 70 billion parameters. The M3 Max outperforms due to its 128GB memory, while the RTX 4090 struggles with the 70 billion model requiring over 35GB memory. The video also teases an upcoming project on cloning a grandma's voice and memories using AI.

Takeaways

• 💻 The video focuses on testing the performance of LLaMA 2 models (7 billion, 13 billion, and 70 billion parameters) on different hardware setups.
• ⚙️ Three systems are being compared: the M3 Max (40-core CPU, 128 GB memory), the M1 Pro (16 GB memory), and an RTX 4090 with a 16-core CPU and 32 GB RAM (24 GB GPU memory).
• 🚀 The M3 Max has an advantage due to its 128 GB memory, which helps when running large inference models.
• 📉 The M1 Pro performs slower, especially on larger models, and struggles with the 70-billion parameter model due to memory limitations.
• ⚡ The RTX 4090 handles the models well, especially for GPU-intensive tasks, but struggles when the model size exceeds the GPU memory limit (70 billion parameter model).
• 🖥️ When running the LLaMA 7 billion and 13 billion models, all systems perform well, with slight differences in speed. The M3 Max and RTX 4090 are closely matched, while the M1 Pro lags behind.
• 🔋 The M3 Max uses less power (65 watts) compared to the RTX 4090 (250-300 watts), making it more efficient for certain tasks.
• ❌ The 70 billion parameter model crashes the M1 Pro and runs slowly on the RTX 4090 because it offloads to the CPU due to insufficient GPU memory.
• 🏆 The M3 Max handles the 70 billion parameter model more efficiently, completing the task faster than the RTX 4090, which requires more expensive hardware (like the A6000) to manage such large models.
• 🔮 The video also teases a future project involving AI-based memory and voice cloning of the presenter's grandmother, using tools like GPT-4, 11 Labs, and Nvidia Omniverse.

Q & A

What models are being tested in the video?

The video tests Llama 2 models with 7 billion, 13 billion, and 70 billion parameters.

What hardware is used for the performance comparison?

The hardware used includes the M3 Max with 40-core CPU and 128 GB of memory, the M1 Pro with 16 GB of memory, and an AMD system with a 16-core CPU and RTX 4090 with 32 GB RAM and 24 GB of GPU memory.

Why is the M3 Max preferred over the M1 Pro for inference tasks?

The M3 Max is preferred because it offers 128 GB of memory, which is a significant improvement for running inference on large models, compared to the M1 Pro's 16 GB.

How does the performance of the RTX 4090 compare to the M3 Max for the 7 billion parameter model?

The RTX 4090 and the M3 Max perform similarly when running the 7 billion parameter model, but the 4090 finishes slightly faster due to its higher power and GPU capability.

What is O Llama and why is it used in the tests?

O Llama is an API that automatically uses GPU power when available and falls back to CPU when necessary. It's used because of its efficiency in leveraging hardware resources for inference.

How much memory does the 13 billion parameter Llama 2 model use?

The 13 billion parameter model uses around 11 GB of memory.

Why does the M1 Pro struggle with the 70 billion parameter model?

The M1 Pro only has 16 GB of memory, which is insufficient to run the 70 billion parameter model, causing the system to crash and reboot.

What happens when the 70 billion parameter model is run on the RTX 4090?

The 70 billion parameter model cannot fit into the 4090's 24 GB of memory, so it switches to running on the CPU, resulting in much slower performance.

How much memory does the 70 billion parameter model require?

The 70 billion parameter model requires around 35-37 GB of memory.

What is the key advantage of the M3 Max over the RTX 4090 for large model inference?

The M3 Max can handle large models, like the 70 billion parameter model, more efficiently due to its 128 GB of memory, while the RTX 4090 struggles with models that exceed its 24 GB of GPU memory.