Dual 3090Ti Build for 70B AI Models

Summary

In this video, the creator shares their experience upgrading their PC by adding a refurbished Nvidia 3090 TI to their existing setup to enhance their large language model (LLM) machine. They detail the process of testing the new card, transferring components into a larger case, and overcoming challenges with cable management and power supply. The video concludes with the successful integration of both GPUs, demonstrating improved performance and the potential for future upgrades.

Takeaways

• 🛒 The user purchased a refurbished 3090 TI and a 390 from Micro Center for $7.99 and $6.99 respectively.
• 💡 The user is interested in enhancing their LLM (Large Language Model) machine with an additional 3090 to run larger models.
• 📦 The refurbished graphics cards came with a power adapter and were boxed.
• 🔧 The user's current PC case is an old model from Building 19, which has been through multiple iterations of use.
• 🛠 The user is not particularly skilled or concerned with cable management, which is evident in their build.
• 🔄 The user tests the new card by swapping it with the old one and performing a quick benchmark test.
• 🔌 The user mentions that the power supply unit (PSU) is mounted at the top, which is not common in newer cases.
• 🔉 The user's build includes a unique feature: aluminum wheels similar to those on newer Apple computers.
• 🔄 The user plans to transfer all components into a larger case but first tests the new card in the current setup.
• 💻 The user encounters an issue with memory when trying to load a 70b model, which requires increasing the swap file size.
• 📈 The user successfully tests the new setup with two cards, achieving a speed of 16 tokens per second for the 70b 4bit quantized model.
• 🛠️ The user acknowledges the need for a new motherboard to support both cards at full capacity due to PCI lane limitations.

Q & A

What graphics cards were available at Micro Center for refurbishment?

Micro Center had the 3090 TI and the 390s Founders Edition available for refurbishment.

What was the price difference between the refurbished 3090 TI and the regular 390 at Micro Center?

The refurbished 3090 TI was priced at $7.99, while the regular 390 was priced at $6.99.

Why does the speaker want to add another 3090 TI to their setup?

The speaker wants to add another 3090 TI to create a more potent LLM (Large Language Model) machine, allowing them to run larger models.

What does the refurbished graphics card come with according to the script?

The refurbished graphics card comes with a power adapter.

Where did the speaker originally purchase the case they are using?

The speaker bought the case from a local store in the New England area called Building 19.

What was the original purpose of the case before it became the speaker's main PC case?

The case was originally purchased as a fully built computer called Velocity Micro from Building 19 and has gone through various iterations, including mining crypto.

Why is the speaker considering getting a new motherboard?

The speaker is considering a new motherboard because the current one may not support both new graphics cards at full x16 PCI lanes due to the processor's limitations.

What issue did the speaker encounter when trying to load the 70b model?

The speaker encountered an out-of-memory issue when trying to load the 70b model, despite having the correct amount of video RAM.

What did the speaker do to resolve the out-of-memory issue for the 70b model?

The speaker increased the size of their swap file to resolve the out-of-memory issue.

What is the estimated tokens per second speed for the model running on the dual GPU setup?

The estimated tokens per second speed for the model running on the dual GPU setup is around 16.9 tokens per second.

What is the speaker's plan for the current PC build after the test?

The speaker plans to temporarily use the current build until they get a larger motherboard and then redo the entire build with proper cable management.

Outlines

00:00

🛠️ Upgrading to a More Potent LLM Machine

The video script describes the process of upgrading a personal computer to enhance its capabilities for running large language models (LLMs). The narrator purchases a refurbished 3090 TI and a 390 Founders Edition from Micro Center, with the intention of adding the 3090 TI to their existing setup. They detail the testing of the new card, ensuring it works properly before transferring it into a larger case. The script also reflects on the history of the current PC case, which was bought as a pre-built system from Building 19 and has seen various uses over the years. The narrator acknowledges their lack of cable management skills and proceeds to swap the old card with the new one, aiming to test it thoroughly before integrating it into their main setup.

05:01

🔧 Integrating Dual GPUs in a Compact Case

This paragraph details the attempt to fit two powerful graphics cards into a single case, despite the challenges of limited space and potential airflow issues. The narrator discusses the technical aspects of ensuring the motherboard can support both GPUs, the process of physically fitting them into the case, and the concerns about the power supply unit (PSU) fitting and sagging due to its weight. They also touch on the aesthetic preference for an older, worn-out look in their builds, as opposed to modern RGB setups. After successfully installing the cards and ensuring they work together, the narrator considers the need for a larger motherboard with more PCI lanes to avoid bifurcation and improve airflow in the future.

10:04

💻 Testing and Troubleshooting the New Setup

The narrator proceeds with testing the new dual-GPU setup, initially facing issues with memory limitations when attempting to load a large model. They address this by increasing the size of their swap file, which allows for more temporary storage space. After successfully loading the model, they transfer the system onto a 512GB SSD and test the functionality of the new setup through a text generation web UI. The script highlights the increased intelligence and speed of the model compared to previous versions, noting a real-time, readable output at 16 tokens per second. The narrator also discusses the need for better cable management and plans for a future rebuild in a new case with improved design.

15:07

🔄 Reassembling and Optimizing the PC Build

In this paragraph, the narrator focuses on the reassembly of their PC after the successful testing of the dual-GPU setup. They mention the intention to use the HDMI port from the motherboard to free up the graphics cards for their primary task. The script describes the process of reassembling the case, including the challenges of cable management and the use of zip ties to organize the cables. The narrator also discusses the need for a more permanent solution involving a larger motherboard and improved airflow, acknowledging that the current setup is a temporary one. They conclude by expressing satisfaction with the outcome of their weekend project and the functionality of the upgraded system.

20:13

🎨 Final Touches and Aesthetic Considerations

The final paragraph of the script discusses the final steps in reassembling the PC case and the aesthetic considerations involved. The narrator ensures that both graphics cards are visible within the case, despite the limited space, and expresses satisfaction with the visual outcome. They mention the difficulty in seeing both cards clearly but are pleased with the overall look. The script concludes with the successful reassembly of the case, with a focus on the aesthetic appeal of the build, and the anticipation of future improvements with a new case and motherboard.

Keywords

💡Micro Center

Micro Center is a well-known American retailer specializing in computers, electronics, and related products. In the video, the narrator mentions purchasing refurbished 3090 TI and 390 graphics cards from Micro Center, indicating it as the source for the hardware upgrade for their computing setup.

💡Refurbished

Refurbished refers to products that have been previously used, returned, or repaired and then certified for sale again. In the context of the video, the narrator bought refurbished graphics cards, which are likely to be more affordable than new ones, and still suitable for their intended use in a more potent machine setup.

💡3090 TI

The 3090 TI is a high-end graphics card model, part of NVIDIA's lineup, known for its powerful performance in gaming and professional applications like machine learning and image generation. The video's theme revolves around enhancing the narrator's existing setup by adding another 3090 TI to improve computational capabilities.

💡Founder's Edition

The Founder's Edition typically refers to the first version of a product released by the original manufacturer. In the script, the narrator mentions the Founder's Edition of the 390 graphics card, which is one of the models they considered purchasing from Micro Center.

💡LLM (Large Language Model)

A Large Language Model (LLM) is a type of artificial intelligence that processes and generates human-like text based on input data. The video's narrator is interested in creating a more potent LLM machine, which is why they are upgrading their hardware with additional powerful graphics cards.

💡PCIe Lanes

PCIe Lanes refer to the data channels on a computer's motherboard that connect to peripheral devices like graphics cards. The script discusses the limitation of the current motherboard's PCIe lanes, which affects the performance of the graphics cards when used in tandem.

💡Bifurcated

In the context of PCIe lanes, bifurcated means that a single PCIe slot is split into two, effectively reducing the bandwidth available to each device. The narrator mentions the limitation of their current motherboard where the PCI lanes are bifurcated, impacting the performance of the dual graphics card setup.

💡Threadripper

Threadripper is a brand of high-end desktop processors by AMD, known for their multiple cores and threads, making them suitable for heavy multitasking and compute-intensive applications. The narrator plans to upgrade to a Threadripper processor with a new motherboard to better support their dual graphics card setup.

💡Cable Management

Cable management is the practice of organizing and routing cables in a computer case to maintain a clean and efficient build. The narrator admits that cable management is not their strong suit, as evidenced by the messy wiring in their current setup.

💡Power Supply Unit (PSU)

A Power Supply Unit, or PSU, is the component in a computer that supplies power to the system. The script mentions concerns about the PSU's size and mounting position in the case, as well as the need for additional power connectors for the new graphics cards.

💡Image Generation

Image generation refers to the process of creating visual content, often using AI or machine learning models. The video's narrator tests the new graphics card setup by running an image generation task, which is a practical application demonstrating the card's capabilities.

💡70B Model

The 70B model likely refers to a specific version of a large language model with a 70-billion-parameter configuration. The narrator attempts to download and run this model on their upgraded setup but encounters memory limitations, requiring adjustments to their system's swap file.

Highlights

Micro Center had 3090 TI refurbed for $7.99 and 390s for $6.99.

The goal is to add another 3090 to create a more potent LLM machine.

The refurbished card comes with a power adapter.

Testing the new card by removing the old one for a quick bench test.

The case was bought as a fully built computer from Building 19.

The case has been used since 2008 and has gone through many iterations.

The computer originally came with aluminum wheels similar to newer Apple computers.

Cable management is not a priority in this build.

The new card is tested for functionality before transferring to a larger case.

The open Dolly Del test is run to ensure the card is recognized.

The 70b model failed to load due to insufficient video RAM, requiring an increase in swap file size.

A 512GB SSD was used to transfer everything for the model.

The new setup achieved 16.9 tokens per second for text generation.

The build quality of the current case is questionable due to heat concerns.

A new motherboard and case are planned for a future rebuild.

The final assembly includes using the HDMI port from the motherboard.

The reassembled case showcases both graphics cards, despite the small size.

Transcripts

so Micro Center had 3090 TI refurbed as

well as the 390s the founders Edition

ones the ti was

$7.99 and the regular 390 was $6.99 I

believe I've been wanting to get another

3090 TI for

3090 to make a more potent llm

machine I currently have one

so today I'm going to be adding in

another one so that I can run

larger

llms it's what it looks like boxed up

and

refurbed comes with

the power

adapter and we'll pull the card

out

[Applause]

wonderful so before I go about

transferring this all into a new larger

case I'm just going to test the new card

by removing the old one that I have

making sure it works with some image

Generation stuff just quick bench test

this case I actually bought as a fully

built computer from a local store here

in the New England area called Building

19 it was called a velocity

micro and it was a really expensive

pre-build and for some

reason their whole stick was to get

things and sell them much cheaper at

Building 19 so I've had this case since

probably 2008 2009 it's gone through a

large amount of

iterations from mining um

crypto a long time ago to now doing llm

stuff so unfortunately today is the last

day that this case is going to be my

main PC case which is sad for

me this computer also came with these

cool aluminum wheels that you may have

seen on newer Apple computers for the

price of a whole entire computer but

these were a bit cheaper also I find it

now relevant to mention that cable

management is not my strong suit nor

something that I pay much care to so

please don't be too hard on me for that

so I'll quickly remove this card swap it

with the new one and put everything back

together

so I have the new card in now I will

simply plug in the existing wiring that

was

here and voila I'll bring it over to my

test bench area and run it just to make

sure everything's all right and then

I'll go about transferring all this into

a much larger case

also side note if the camera did Pan

down to a large mess down here on the

floor this is not how I live this is a

workshop so there is large amounts of

trash and machinery and the likes of

that so it's fired up it's running and

the card has lit up which is a good sign

so now we're in the btu environment and

I'm just going to quickly run this to

see if the card is being

recognized and it

is got our power draw vrm wonderful so

the next step is to

run the open do Dolly Del please correct

me if that's an

issue which takes a little while to run

then we'll open up the web interface so

that we can do image generation

now this is up and I'm going to test it

and of course this is not why I've

gotten a second GPU this is just a way

to quickly just test this one make sure

everything's working all

right all right let's see usually

get about that speed on the other card

so that's good and we'll go over here

we'll run this again just to see if it's

being utilized more should be around

yep and the uh I go to Micro Center brow

picture did work well let's make this

adhere a bit more all the way shall we

and we'll try this one more time as well

as this while it's

generating yep perfect

is using a lot of power when it does

this I've had them generate some

insanely large images like 4,000 by

4,000 so that picture the contrast isn't

messed up on the camera it actually

looks that messed up but so the initial

test has worked well and I will now

shove Two Cards into a

case I put both of the cards in this

case just to see what it will look like

now this motherboard is only temporarily

here I will eventually soon get a thread

Ripper with a new motherboard so that

the PCI lanes are not bifurcated

bifurcated bifurcated I don't know but

cuz this processor doesn't have enough

PCI Lanes to run these both in

x16 but considering they're going to

have to be this close anyway which I

know is probably horrible from an

airflow perspective I kind of want to

just keep them in my case that I'm

essentially bonded to until I get a new

motherboard and then just totally redo

the build probably from an engineering

standpoint is bad however I find that

sometimes we look past that to make

aesthetic decisions and I'm no

different so this case uh the power

supply is mounted up top which I noticed

today after going to Micro Center does

not seem to be the norm any longer seems

like they all get mounted on the bottom

which based on the size of these seems a

reasonable decision I am not quite sure

if the new one is going to fit in here

without sagging massively

so I'll perhaps have to support it some

way

and one two three okay those are the

wrong

screws they're all the right screws

now you can see this computer is uh aged

I know a lot of people do nice new RGB

builds and I always prefer the more worn

out and old aesthetic and this thing has

been around the

block which I like I like older worn out

things especially

guitars if you hear fan noise right now

by the way that's actually one of my

uh light bulbs I know that sounds weird

but the lighting I'm using the bulbs

have fans in them and one of them

is rather

unhappy all

right I've got

that just going to unplug everything

which I should have done that's

h

I'm just going to reuse the cables that

were

here and I will only need to add

the additional ones for the extra

graphics

card right we now have this up of course

there

1,000 it's been a wonderful

PSU never any problems with

it

now oh man this

is of

questionable uh build quality here

my jury

rigging truth be told I really wanted to

just cut up an old Power Mac G 5 case

and Frankenstein that into holding both

of these graphics cards but that

requires a large amount of aluminum

cutting which I'm not so keen on having

to

do let's check the size difference

between the old and new actually does

not seem as

substantial

as I first feared obviously

the there

so and how long is that

30 millim 40 millim not too bad I think

we might be all right so I'll move this

off to the

side

and I

will put this

back not that way not that way that

way

this is

a Perhaps it is a good idea to just use

a new

case so these don't go in

this thread

it fortunately my fears of sagging have

been quelled due to the fact that this

does have some supporting elements of

the case frame here and these tabs here

and here so the PSU won't

sag fortunately this came with screws

because I was trying to put the ones

that were in there back in and I must

not have been in my right mind uh in the

last iteration build of this computer

because they

were completely the wrong

size for p wonderful you can't see

that I would lie to you instead had

fully taken this apart so that I could

clean the dust from the fans in the

front however that would be fitting I'm

taking it apart to get the top of the

case off because that will allow me far

far easier access to the power supply to

plug everything back

in I noticed a lot of newer cases and

builds seem to have emphasis on ease of

use in terms of actually plugging

everything in and getting everything

wired and

clonin so with this

off I now

have large amount of access to put the

power supply back in really to get

everything I have everything wired up

now including the two connectors

necessary for the graphics

cards and fortunately it all worked all

right here this case actually has pretty

adequate depth for a larger power supply

including the cords protruding out so

now I'm just going to gently slam this

in and is

sagging up a large bit but it's also not

screwed into the back which gives it

some lateral support that

way please do feel free to shame me for

the wiring I'm going to clean it up a

bit before I turn it on which will just

consist of zip tying things to one of

these caddies and now I'm going to test

it real quick I don't actually 100% know

if this motherboard's going to be okay

with supporting both of these cards from

a pcie standpoint so I may have to run

back out and get a new motherboard which

I'd prefer not to have to do but if I

must I

must it's all plugged in now both cards

are on and running I just have it naked

just CU I wasn't sure it would work with

this motherboard and you can see

here and I run this both cards do pop up

so now I'm going to download a large

model in the text generation web

UI 70b 4bit

Quant and this will take a little

while about 40 gigabyte download so

yeah after a rather large amount of time

I realized that the 70b model would not

load it was saying it was out of memory

even though I had the correct amount of

video RAM I needed to increase the size

of my swap file

uh essentially what that means is say

you have like things on a desk like this

and you need more space temporarily you

can move them to shelves and then you

know move them back I think that's right

I don't know check it on chat GPT later

so this uh loaded now and I transferred

everything over to a 512 gig SSD that I

had lying around which was uh a bit

involved of a process so this is an

instruct follow model let's see if this

actually starts working chat

instruct character Gallery this

one and I will say hello

friend I spelled hello wrong let's see

what

happens oh there what can I do for you

today five tokens a second I'm not quite

sure what the speed is supposed to be

for a model of this size let me check

the Nvidia

panel to see what the utilization of the

cards is

currently

so we have 23 gigs of RAM being used on

one and then 19 gigs of the

other uh so let's just see what this

says

uh

sorry this is po to

see cool this works pretty well I think

I need to do some cleaning in terms of

the response

uh way because it's actually showing the

line Brakes in format like that so that

did 17 tokens a second 16 .9 seems

pretty good I'll run this again to see

if the cards are heating up right now

because they are in an open air

environment but this case is very small

so yeah we're going up a bit So

eventually I'll swap them out of this

case when I get a larger motherboard

with hopefully a thread

Ripper but for now this is pretty cool

it seems relatively intelligent let's

ask it one more thing

maybe tell me a

story

about a big Micro

Center trip and we'll leave this up to

its

interpretation okay so it does know

about a local Micro Center

store pretty

good this is definitely much more

intelligent than the 7B dolphin models I

was using earlier on the single card and

speed-wise it's 16 tokens a second I

think I was getting like 50 to 60 on a

single 39 DTI with a 7B uh EXL 2 model I

think that's the word so but this is

totally readable in real time you can

see it this has been a pretty fun

Endeavor so far just for

a simple

Saturday I've got two cards in there

which are both likely be being

suffocated due to the heat of one

another however this will be a temporary

solution now all that's left to do is

put this case back together and clean up

the wiring a

bit now for the grand

reassembly now I'm going to quickly just

try to tie these cables at least

together with the zip ties that were

included with the new power

supply one last thing I want to note is

that I am just going to use the HDMI

port from the motherboard instead of

from either of the graphics cards so

that they're not preoccupied with

anything other

than running what they need to be

running let's see if this is even

possible able to do this with any form

of cless

whatsoever again this will be

rebuilt in a new

case with proper Cable Management

sometime in the next few

months

and there's really not much I can do

here

Well for

now cool with it put these

back and final piece of the

puzzle will be the side which could use

a swift

ring here's some Swiffer pad ASMR

all right this needs more than I can

give it with

that this on without pinching

the

cords

309s

and it's in it's back

together kind of hard to

see both cards in there but it'll

probably look pretty cool let me turn

this L one off there we go made no

difference should look cool

on