New 130TB Storage Server ft. Linus

Summary

In this video, the host discusses the challenges of managing large video files created by high-speed Phantom cameras, which generate 16-24 GB of data per second. They highlight a past project, 'Super Slow Show,' that filled their storage capacity. To address this, the host collaborates with Linus from Linus Tech Tips to build a custom storage server. They detail the server's specs and setup process, emphasizing its reliability and large storage capacity. The server provides peace of mind and efficiency in managing and accessing extensive video files. The video concludes with gratitude to collaborators and sponsors.

Takeaways

• 📸 The video discusses the challenges of managing large files generated by high-speed cameras, which can produce between 16 to 24 gigabytes per second.
• 🎥 The Phantom cameras used in the video shoot for the 'Super Slow Show' were responsible for creating massive files, making storage a significant issue.
• 💾 The speaker previously used external drives for storage, but this method became insufficient as storage needs grew, leading to a PC upgrade with more hard drives and faster networking.
• 🇨🇦 Linus from Linus Tech Tips was suggested to build a storage server, which eventually happened, providing a solution to the storage problem.
• 🔌 The new storage system consists of 45 Western Digital drives, providing a total of 130 to 140 terabytes of raw storage, which is expected to last until the end of summer.
• 🛠 The WD Red 10 terabyte drives used in the storage system are designed for 24/7 operation and are more resilient to conditions like vibrations and read head noise compared to desktop drives.
• 🔄 The drives have a five-year warranty, indicating their reliability and expected lifespan for continuous operation.
• 💻 The speaker's previous setup involved a large PC filled with hard drives and a basic software RAID setup, which was deemed insufficient for their needs.
• 🌐 The new storage solution from 45 Drives is a high-speed, reliable network storage system that is easy to administer through a web browser, simplifying the process of managing large files.
• 🔋 The storage system is designed to handle large files efficiently, with the largest file that can be recorded directly to the storage being one terabyte, which is a limitation due to the need to leave space for file synchronization.

Q & A

What is the main topic of the video?

The video discusses the challenges of managing large files generated by high-speed cameras, specifically focusing on upgrading storage solutions.

How large are the files generated by the Phantom cameras mentioned in the video?

The files are between 16 and 24 gigabytes per second, depending on the camera used.

Why did the speaker mention Linus from Linus Tech Tips?

The speaker mentioned that many comments suggested Linus Tech Tips for building a storage server to manage the large files.

What storage solution did Linus and his team provide?

Linus and his team provided a storage server with 130 to 140 terabytes of raw storage.

What is special about NAS drives compared to desktop drives?

NAS drives are designed to operate 24/7 in suboptimal conditions with many other drives vibrating nearby, unlike desktop drives which are typically asleep most of the time.

What is the difference between RAID and Unraid in terms of data protection?

RAID stripes data across all drives, meaning if you lose two drives you might lose all the data. Unraid, however, stores data such that if you lose two drives, only the data on those two drives is lost while the rest remains safe.

How long is the warranty for the NAS drives provided?

The NAS drives have a five-year warranty.

Why is it important to have a two-drive failure resiliency setup?

It provides extra protection as hard drives tend to fail around the same time, and during the rebuild of a failed drive, the remaining drives are heavily stressed which increases the chance of another failure.

What was the humorous aspect of the naming of the storage server?

The speaker named the storage server 'slow fatty,' which led to some humorous discussion about the name.

What is the significance of the file named 'anus.txt' mentioned towards the end of the video?

The file 'anus.txt' was a test file created by the speaker during a late-night maintenance session to test file synchronization between servers.

Outlines

00:00

📷 Introduction to the Video and Storage Challenges

The video introduces the topic of managing large files generated by high-speed cameras, such as Phantoms, which produce massive files at 16-24 GB per second. The host explains the storage challenges faced during long shoots and the capacity issues encountered. Past solutions involved filling external drives and upgrading a PC with multiple hard drives and 10-gigabit networking. The host references audience suggestions to seek help from Linus of Linus Tech Tips to build a storage server.

05:01

🖥️ Collaboration with Linus and Server Setup

The host visits Linus in Canada to address the storage issue. Linus explains the solution provided by 45 Drives, Western Digital, and Unraid, resulting in a setup with 130-140 terabytes of raw storage. The conversation covers the high-speed and reliable network storage system, the custom-built enclosures, and the technical specifications, including 32 GB of RAM, an LGA 1366 server board, and a dual 10-gigabit network card. The discussion highlights the durability and 24/7 operation capability of NAS drives.

10:02

🔧 Setting Up and Configuring the New Storage Server

The host and Linus proceed with the physical setup of the server, installing hard drives, and configuring the system. Linus explains the benefits of using NAS drives designed for continuous operation and the importance of proper data protection. They decide on a name for the server, 'Slow Fatty,' and discuss RAID versus Unraid configurations, choosing a setup that allows for two drive failures without data loss. The host is guided through accessing and managing the server via a web browser.

15:03

📁 Testing the New Server and Final Thoughts

Linus and the host test the server by transferring files and demonstrating the ease of use. The host appreciates the straightforward management through the web interface and the significant storage capacity. Linus explains the limitations of file sizes in relation to drive capacity. The host plans to make the storage last until the end of the year. The video concludes with a post-setup update from the host, praising the server for simplifying file management and improving workflow efficiency. Acknowledgments are given to Linus, 45 Drives, Western Digital, and Unraid for their contributions.

Keywords

💡Large files

Large files refer to the massive data files generated by high-speed cameras, like the Phantom cameras mentioned in the video. These files are enormous because the cameras take many pictures every second, resulting in file sizes between 16 and 24 gigabytes per second. This concept is central to the video's theme, which discusses the challenges of storing and managing these large files.

💡Phantom cameras

Phantom cameras are high-speed cameras that capture a large number of frames per second, creating extremely detailed slow-motion footage. In the video, they are responsible for producing the massive files discussed, with recording sessions leading to extensive storage requirements. These cameras are essential tools for projects like the 'Super Slow Show.'

💡RAM saves

RAM saves refer to the process of saving data from the camera's RAM (Random Access Memory) to a storage device. In the video, Phantom operators perform full RAM saves frequently during shoots, contributing to the large volume of data that needs to be stored. This is a crucial step in managing high-speed video data.

💡Storage server

A storage server is a dedicated computer designed to store and manage large amounts of data. The video discusses the need for a robust storage solution to handle the extensive data generated by Phantom cameras. Linus from Linus Tech Tips helps build a custom storage server to meet these needs, offering a solution with 130 to 140 terabytes of raw storage.

💡NAS drives

NAS (Network Attached Storage) drives are hard drives specifically designed for continuous operation in network storage environments. Unlike typical desktop drives, NAS drives can operate 24/7 under suboptimal conditions. The video highlights their importance in ensuring reliable and long-term data storage for the massive files created by the Phantom cameras.

💡RAID

RAID (Redundant Array of Independent Disks) is a data storage technology that combines multiple physical disk drives into one or more logical units for redundancy or performance improvement. In the video, Linus explains why traditional software RAID setups might be inadequate for managing such large data volumes, advocating for more robust solutions like unRAID for better data protection and reliability.

💡unRAID

unRAID is a storage operating system that allows for flexible, scalable, and reliable storage solutions. Unlike traditional RAID systems, unRAID can handle drive failures more gracefully, ensuring data integrity even if multiple drives fail. The video emphasizes its use for archival storage, making it ideal for the large and numerous files generated by high-speed cameras.

💡Data redundancy

Data redundancy involves storing multiple copies of data to prevent data loss in case of hardware failure. The video discusses setting up the storage server with sufficient redundancy, using techniques like unRAID, to ensure that data remains safe even if some drives fail. This is critical for preserving the massive and valuable files created during shoots.

💡10-gigabit networking

10-gigabit networking refers to network connections capable of transmitting data at speeds up to 10 gigabits per second. In the video, upgrading the storage server with 10-gigabit networking is crucial for quickly transferring the large files generated by the Phantom cameras. This high-speed connectivity is essential for efficient data management and workflow.

💡Archival storage

Archival storage is the long-term storage of data that is infrequently accessed but must be preserved for future use. The video discusses setting up a storage server to archive the vast amounts of data from Phantom camera shoots. Archival storage ensures that important footage remains accessible and safe over extended periods, despite being stored in large volumes.

Highlights

Introduction to the video and the issue of managing large files generated by high-speed cameras.

Explanation of how Phantom cameras create massive files, with sizes ranging from 16 to 24 gigs per second.

Overview of the Super Slow Show, a 26-day shoot that produced an enormous amount of data.

Discussion on the need for a robust storage solution due to the large volume of data from Planet Slo-Mo.

Description of previous attempts to manage storage using external drives and a PC upgrade with added hard drives.

Introduction of Linus from Linus Tech Tips to help build a more efficient storage server.

Overview of the new storage setup, including 130 to 140 terabytes of raw storage capacity.

Details on the advantages of using NAS drives designed for 24/7 operation in suboptimal conditions.

Explanation of the five-year warranty on the new NAS drives.

Comparison of the new storage server setup with the previous makeshift solution using a PC.

Introduction of the server from 45 Drives with a dual ten-gigabit network card and 32 gigs of RAM.

Highlight of the server's quiet operation due to the use of high-quality Noctua fans.

Demonstration of installing the drives into the server and setting it up for use.

Discussion on the benefits of Unraid for archival storage and the setup's data protection features.

Review of the server's performance and its impact on the workflow after 90 days of use, emphasizing its convenience and reliability.

Transcripts

hello there this video is about large

files this camera is responsible for

many of them the phantoms typically only

record for a few seconds at a time

but they're taking so many pictures

every second that you end up with

absolutely massive files it's between 16

and 24 gigs per second depending on

which camera were using and when you

have a long shoot for example the super

slow show was a 26 days shoot with

phantom operators just doing full Ram

saves every time it was an extremely

expensive show just to store and since

doing planet slo-mo I am absolutely at

capacity in the past I have just been

filling external drives so a few years

ago I did a little PC upgrade where I

pretty much just opened the massive case

I already had and stuffed it full of

hard drives

absolutely rammed the thing and added 10

gig networking at the same time and a

lot of the comments while most of the

comments almost all of the comments said

you need Linus from Linus tech tips to

build you a storage server so why don't

we pop over to Canada and get it done

all right I did it now I'm here with

Linus hey guys

so you've you've seen firsthand the

files that our cameras create slightly

crazy you upstairs said you don't envy

me working with these files I don't with

that so 45 drives Western Digital and

unread basically hooked you the math out

up it's been a long day

it's like actually very late right now

so they hooked you up you're gonna have

at the end of all of this

somewhere between 130 to 140 terabytes

of raw storage that'll last you well I

think to this point I've shot maybe 250

terabytes not including my YouTube

originals probably each a hundred

terabytes so this will last what like

end of summer

okay well let's get started

I literally need it I'm I'm like at

capacity right now let's be look pay

close attention while we're building

this because clearly you're gonna be

building another one in a couple of

months here how how is expansion on

these are we filling this up so these

are your WD red 10 terabyte master eyes

so what's special about an ass drive is

unlike a desktop drive which is designed

to be basically asleep most of the time

and then you sit use your computer for a

bit yeah they'll spin up no take forever

you go hygiene or whatever yeah go and

do a hygiene yeah like brush your mouth

hygiene you've got like the hygiene all

bhai genius so these are designed to

operate 24/7 they're designed to operate

in suboptimal conditions especially with

many other drives vibrating away nearby

not just spinning but also with their

with their read heads like ticking and

clacking around so that's what's special

about them so when you say 24/7 24/7 for

how long uh typically they're rated for

anywhere from three to five years

I don't quote me on this one let me get

back to you on that thank you for asking

you're welcome they actually have a

five-year warranty five-year warranty

yep

solid do you have any idea what my

current setup is I base to just have a

big PC right and then I just got a

measuring it's not a contest well yeah

and then what I did I counted how many

harddrive holes they had which I think

is the technical term and then I shoved

them in and I just filled it up way so

there's no like data protection on them

or anything well I did a little software

raid like a raid one or something like

in Windows yeah so tell me why that's

bad Linus it's just

flaky yeah like Windows Windows raid is

not I thought that would be the case

when I set up I was like this is

just some jank I'm making a jank thing

let's put these aside for now and let's

have a look at the server your hands

yeah that would actually be great thank

you

you were bound to do something to help

with this energy uh yeah

came to Canada in all seriousness though

he did come to our studio and help us

with a fantastic at least I think it's

fantastic video so you haven't seen this

at all no oh I know nothing about this

excited so these are our Canadian bros

they're based out of Eastern Canada

company by the name of 45 drives oh and

their whole shtick slightly differently

to how I would have done that you know

it got got it done is high speed

reliable network storage actually

fabricate everything themselves their

parent company is called proto case so

they build these enclosures and they can

customize all kinds of cool stuff

including that front oh that's so cute I

love it so I actually had this link yet

so that's like the one video you've seen

right oh yeah no no I've seen a few but

that's it that's one of one of the three

yeah that's a that's a memorable one it

looks like the Harrier kicked this

I think they've actually revised this

product since they've sent this because

we said we were gonna do this collab

like six months ago or something like

that I think it's longer than that I

think the way it works fundamentally is

pretty similar to now so you've got 15

Drive bays right here all conveniently

now so just this bit as the drives

mm-hmm the rest of this is the server

that runs the thing all right so it

looks like they actually sent you a

pretty nice little configuration here

that's pretty sick so you've got 32 gigs

of RAM on an LGA 1366 server board and

this is your storage card so that's

a host bus adapter so it wires into each

one of you drives individually over here

and then this right here is a dual ten

gigabit network card damn another cool

thing is these guys use all Noctua fans

you familiar with those guys only from

your videos yeah high quality quiet so

this thing is actually quiet enough that

unlike a lot of sort of rackmount which

server type equipment you could you

could put it on a desk or under a desk

or whatever and something about you so

fire it up for the drives in yeah

excited about see if you can handle this

he's apparently putting hard drives into

a thing before so I knew someone was

coming okay yep that's it I'm gonna fill

you up gonna fill you up it was safe

until you said that this one is not

behaving just keep going come back I'll

come this video would have been longer

but Linus kept saying stuff that we

can't use we had another eight second

bleep I'm gonna try this one over here

when will it end and people really want

to tell Linus tonight one of our videos

but you really just didn't say much it

was weird all right Oh God is it heavy

now you know how I was saying like you

pick up a hard drive and you're like

that's not that heavy and then you put

15 of them in a box and you try and pick

it up and you're like oh so let's go

ahead and fire it up do you have like

your laptop here or anything cuz I can

show you I can map out on your laptop

like a spot in your browser bar for

where you go to yes great it

cool stuff he's looking for it he's been

up for like okay so do you want to do

the honors this button yeah go ahead

did you do something right when I did

that did you press it well I was going

to and then the light went off oh hi

again feel like that would have been

better if I didn't notice that you I was

trying to do it really quietly I hold on

let me see if I can do a quieter decide

yeah okay this place is too well sound

press the button VGA sweet you don't

ever have to use that oh yeah I remember

I set marques up with one of these and I

was like hey do you have a VGA cable

it's like what decade is it like I know

but server stuff still uses it yeah I

still have a few VJs tries to capture

from an arcade machine

I need a VGA for that there you go so

this is it

you access your server by typing its

name or IP into your browser bar and

boom you're configuring it so I'll do a

couple initial setup things that you'll

never have to do again or worry about

but I still need your help from them

first things first what do you want it

to be called slow Fae slow fatty I like

it is that how you spell fight

ie I thought a fatty is like like a

blunt like ie

but then a fatty with the Y is like a

fat person well this is fat storage it's

a fat amount of storage oh so you want

pH no no slow fatty like is that what

you're after

that's perfect okay so stick to that now

you have to make a very critical

decision yeah yeah well yeah it shows up

everywhere what were you expecting you

actually I'll be honest I didn't expect

it to make me laugh so soon after naming

it but their eyes I mean here does this

does this tickle you you actually type

it into your browser bar accesses yeah

that's I'm gonna regret naming of that

but I'll just remember that it was

really late when we did it sure and that

would be my excuse

do you want a hundred and forty or a

hundred and thirty terabytes of storage

I assume one hundred and thirty-four

that'll give you up to two Drive

failures at a time without losing any

data when a drive fails I'll know

immediately right now you can set it up

so it'll send you notification so with

so all that is is contingency for if two

drives fail at the exact same time not

necessarily okay so the thing about hard

drives is especially as their capacities

have gone up the odds of them failing

during a rebuild have increased because

once you get to the stage in their

lifetime where they are starting to fail

well they're all equally worn out so

once one fails and you pop in a new

drive and you go to rebuild that parity

data that's on that new drive all the

other ones are getting hit that whole

time and it can take a full day in order

to do that so if you have a second

failure during your rebuild operation

you can lose data now what's cool about

on raid because this is not raid are you

familiar with raid right yeah you're

running raid okay so this is not raid

this is unmade normal raid will actually

stripe your data across all the drives

so if you have a one drive resiliency

let's say and you lose two drives hold

on if you have one drive worth of

resiliency in a set up like this and you

lose two drives you lose whatever was on

those two drives and that's it the rest

of them are all still protected so it

comes with a speed penalty but in the

event that you had your tower up on its

side like this and you got flooded with

six inches of water you have lost six

drives worth of data and the rest is

still there got it

so that's why we recommend this for just

archival storage yeah I mean and that's

what I need cool I mean is that what you

would recommend I always set up to

parity drives okay do you want to press

it I like that you let me press all that

are you gonna turn it off before I press

it no no you done with that guy so there

you have it

we're formatted these still aren't

synchronized you'll have to do that at

home or maybe I'll just leave this

running and then we'll ship it to you

once that's done and that is a hundred

and thirty terabytes of form

storage why am i missing point zero four

of a terabyte I don't make the rules

look they're all 9.99 I don't know okay

good now this is tough for you what is

the largest file you could conceivably

dump on here you can record directly to

the mag for as long as it takes to fill

it so one terabyte would be the biggest

file I can make Wow okay that sucks

because what that's gonna do is it's

gonna leave a terabyte of space free on

every one of these drives to account for

Oh you can't split a file between drugs

okay well let's just say I won't do that

and then I would say the biggest RAM on

camera would be like 288 gigs that still

a lot better than that yeah but that's

less horrible 290 gig okay good let's do

a biggest file much smaller than that

because we don't leave the the a cave

cameras running continuously you don't

use them it's like security cameras but

yeah sorry we can easily access these

and we could even do something as crazy

as I don't know copy a file here's a

file let's put that in there oh it's my

first file is this a anus you made it

did you pre make a file called anus I

did I just have that oh you would you

like to open it oh I forget why I called

it that I think I was really late at

night and I was doing like super

maintenance and I was testing I know I

do remember the story so I was testing

file synchronization between two of our

servers as the main production server it

has a service running on it that

automatically takes any file that gets

put onto it and then replicates it to a

completely separate machine so I just

needed a lightweight file that I could

use to test what the leg was between

copying something to the main server and

it going to the other one I was in like

kind of a weird mood I was

late night giddy yeah it's not like how

I am right now so I made anus txt yeah

and are you deliberately less

professional with me like did you put

anus on like Marquez is one or Justine's

one no no that just happens to be the

folder I had open that's my My Documents

folder on this computer sure all right

whoa that's the biggest and I've ever

seen Wow should we get like slow-mo

footage of it walking wow I mean yes

flippity floppity you select the volumes

you want him out

and this anus dot txt and there it is

just like that phenomenal so that's it

that's the whole thing you actually

don't need this VGA cable or a keyboard

and mouse plugged into this thing two

cables power network you administer the

whole thing through your web browser and

actually you barely administer it at all

if you're just planning to use it for

both archival storage you just copy

stuff over to it in that's it it's

honestly a lot easier than I thought

it'd be I mean I haven't obviously

haven't dealt with it yet but it seems

not too complicated I'm gonna try and

make it last at least till the end of

this year before it's oh I gotta say

you're making me feel kind of like this

whole effort was very pointless no it

isn't I'm buying you one two months no

that's great no I'll stretch it out I'll

be do have any idea how hard it was to

get my hands on this stuff for me at

least eight months I just want shoot as

much so there you have it if you don't

see any videos over on the main slow-mo

guys channel those of you who follow

over here on the second channel are

gonna know that it was all because of me

there you go thanks for that slow Fae

20/20 at least that segment was actually

shot about three months ago I've been

holding off on posting this because I

wanted to get the server in get my files

on it and see how I've been getting on

with it it's been online for over 90

days at this point it's almost

completely full as we expected what it's

allowed me to do is have all of my

exports in the same place a ton of

phantom raw footage that I can easily

pull from if I reference an old video

and a new one I no longer have to

scramble through old external drives and

try and figure out my 10 year old

labeling system

a lot of this stuff I frequently need to

go back to is all in one place and it's

honestly wonderful it's one of those

little tweaks that just makes life so

much simpler when I'm editing all these

videos it's just peace of mind knowing I

can quickly grab something that's

already sat on my network always running

I love it

so big thanks to Linus and crew for

organizing all of that and staying up

until midnight helping me make the video

big thanks to 45 drives Western Digital

for the drives themselves and underage

hopefully you enjoyed that very techie

video make sure you subscribe to Linus

tech tips the slow mo guys and this

channel if you want the slow mo guys too

I'll see you next time