Dual 3090Ti Build for 70B AI Models

Summary

In this video, the creator shares their experience upgrading their PC by adding a refurbished Nvidia 3090 TI to their existing setup to enhance their large language model (LLM) machine. They detail the process of testing the new card, transferring components into a larger case, and overcoming challenges with cable management and power supply. The video concludes with the successful integration of both GPUs, demonstrating improved performance and the potential for future upgrades.

Takeaways

• 🛒 The user purchased a refurbished 3090 TI and a 390 from Micro Center for $7.99 and $6.99 respectively.
• 💡 The user is interested in enhancing their LLM (Large Language Model) machine with an additional 3090 to run larger models.
• 📦 The refurbished graphics cards came with a power adapter and were boxed.
• 🔧 The user's current PC case is an old model from Building 19, which has been through multiple iterations of use.
• 🛠 The user is not particularly skilled or concerned with cable management, which is evident in their build.
• 🔄 The user tests the new card by swapping it with the old one and performing a quick benchmark test.
• 🔌 The user mentions that the power supply unit (PSU) is mounted at the top, which is not common in newer cases.
• 🔉 The user's build includes a unique feature: aluminum wheels similar to those on newer Apple computers.
• 🔄 The user plans to transfer all components into a larger case but first tests the new card in the current setup.
• 💻 The user encounters an issue with memory when trying to load a 70b model, which requires increasing the swap file size.
• 📈 The user successfully tests the new setup with two cards, achieving a speed of 16 tokens per second for the 70b 4bit quantized model.
• 🛠️ The user acknowledges the need for a new motherboard to support both cards at full capacity due to PCI lane limitations.

Q & A

What graphics cards were available at Micro Center for refurbishment?

Micro Center had the 3090 TI and the 390s Founders Edition available for refurbishment.

What was the price difference between the refurbished 3090 TI and the regular 390 at Micro Center?

The refurbished 3090 TI was priced at $7.99, while the regular 390 was priced at $6.99.

Why does the speaker want to add another 3090 TI to their setup?

The speaker wants to add another 3090 TI to create a more potent LLM (Large Language Model) machine, allowing them to run larger models.

What does the refurbished graphics card come with according to the script?

The refurbished graphics card comes with a power adapter.

Where did the speaker originally purchase the case they are using?

The speaker bought the case from a local store in the New England area called Building 19.

What was the original purpose of the case before it became the speaker's main PC case?

The case was originally purchased as a fully built computer called Velocity Micro from Building 19 and has gone through various iterations, including mining crypto.

Why is the speaker considering getting a new motherboard?

The speaker is considering a new motherboard because the current one may not support both new graphics cards at full x16 PCI lanes due to the processor's limitations.

What issue did the speaker encounter when trying to load the 70b model?

The speaker encountered an out-of-memory issue when trying to load the 70b model, despite having the correct amount of video RAM.

What did the speaker do to resolve the out-of-memory issue for the 70b model?

The speaker increased the size of their swap file to resolve the out-of-memory issue.

What is the estimated tokens per second speed for the model running on the dual GPU setup?

The estimated tokens per second speed for the model running on the dual GPU setup is around 16.9 tokens per second.

What is the speaker's plan for the current PC build after the test?

The speaker plans to temporarily use the current build until they get a larger motherboard and then redo the entire build with proper cable management.