Unboxing the Tenstorrent Grayskull AI Accelerator!

Summary

In this engaging conversation, Yasmina, a fellow at T-torrent, discusses the company's innovative approach to hardware-software co-design, emphasizing the importance of accessible programming for new architectures. She introduces T-torrent's gen one hardware, the Grull dev kit, designed for developers with a focus on ease of use and community feedback. The discussion highlights the balance between high-level and low-level programming entry points, the company's commitment to open-sourcing their bare metal software stack, and the future integration with PyTorch 2.0. Yasmina shares insights on the company's goals to empower developers and grow the AI hardware ecosystem collaboratively.

Takeaways

• 🎉 Yasmina is a guest on the channel, discussing the exciting developments at Tenstorrent, a company specializing in novel hardware and software.
• 🌟 Tenstorrent focuses on hardware-software co-design, emphasizing the importance of making new hardware accessible through programming.
• 🔧 Yasmina's background in FPGAs and high-level synthesis has been instrumental in her work at Tenstorrent, contributing to the development of their novel architectures.
• 💡 The company is releasing its first generation hardware, called 'Grull', in the form of a developer kit, with two versions: E75 and E150, differing in wattage and size.
• 📦 The Grull developer kit includes everything needed to set up and start working with the hardware, targeting developers and small to medium-sized businesses.
• 💻 The developer kit is designed to be used with Linux, with Windows support planned for the future, and it comes with tools and drivers for developers to get started.
• 🛠️ Tenstorrent offers two software stacks: a high-level entry point called 'Buddha' for easy model deployment, and a low-level 'bare metal' programming model for fine-grained control.
• 💸 The E75 version of the Grull developer kit is priced at $599, while the E150 version is priced at $799, positioning them as accessible to developers and businesses.
• 🔄 Tenstorrent is committed to open-sourcing their bare metal software stack, allowing developers to see and interact with the underlying APIs and system components.
• 🔄 The company aims for forward compatibility, ensuring that anything developers create for the current generation will work with future Tenstorrent hardware.
• 🤝 Tenstorrent values community engagement and feedback, encouraging developers to test, experiment, and provide input to improve their products.

Q & A

What is the significance of Yasmina's background in FPGAs to her current role at Tenstorrent?

Yasmina's background in FPGAs, including her doctorate and experience in high-level synthesis, is significant because it provides her with a strong foundation in hardware-software co-design. This expertise is crucial for developing novel architectures that are easy and enjoyable to program, which is a key focus at Tenstorrent.

What is the main goal of Tenstorrent's hardware-software co-design approach?

The main goal of Tenstorrent's hardware-software co-design approach is to create hardware and software environments that are accessible and engaging for developers. They aim to provide easy entry points for programming, empowering developers to create innovative applications and feel excited about the potential of new hardware and software integrations.

What are the two different versions of the Tenstorrent's Grail dev kit mentioned in the transcript?

The two different versions of the Tenstorrent's Grail dev kit mentioned are the e75, which is a smaller card with 75 watts, and the e150, which is slightly larger and has a higher wattage.

How much does the Tenstorrent's Grail dev kit cost, and who is the intended audience for this product?

The Tenstorrent's Grail dev kit costs $599 for the 75-watt version and $799 for the 150-watt version. The intended audience is developers and small to medium-sized businesses interested in exploring the utility of the hardware for their models.

What is the purpose of the 'Buddha' software stack offered by Tenstorrent?

The 'Buddha' software stack is a compiler developed by Tenstorrent. It serves as a high-level entry point for developers, allowing them to easily download models from sources like Hugging Face and run them on Tenstorrent's hardware without needing to change their environment or rewrite their models.

What is the 'bare metal programming model' that Yasmina refers to, and who is it intended for?

The 'bare metal programming model' is a low-level software stack that provides developers with fine-grained control over the workloads running on Tenstorrent's hardware. It is intended for developers who want to have a deep understanding of the underlying architecture and are interested in customizing their applications at a lower level, including writing custom operators and exploring novel hardware features.

What is Tenstorrent's policy on community feedback and open sourcing their software?

Tenstorrent encourages community feedback and plans to open source their full bare metal software stack. They are interested in hearing both positive and negative feedback to improve their products. They believe in empowering developers and fostering an open collaboration environment, which includes sharing APIs, kernel compilation methods, memory allocators, and other底层细节 to allow developers to fully explore and utilize their hardware.

How does Tenstorrent ensure backward compatibility for their APIs?

Tenstorrent aims to maintain backward compatibility for their host APIs, which are designed to be intuitive and familiar to developers, mimicking existing models like OpenCL and CUDA. For kernel APIs, they value backward compatibility but also recognize the potential for significant performance and functionality improvements in next-generation architectures, which might require them to allow for some changes while ensuring that everything written for the current generation will work on future generations.

What are the differences between high-level and low-level entry points for developers in Tenstorrent's software stacks?

High-level entry points, like the Buddha software stack, offer a quick path to desired outcomes without requiring developers to change their environment or rewrite their models. They are less susceptible to low-level changes and provide a consolidated, specialized API for certain purposes. Low-level entry points, on the other hand, give developers full access to the hardware and allow for fine-grained control, enabling them to write custom kernels and explore the hardware's capabilities in depth.

How does Tenstorrent support developers in accessing their hardware, both physically and through the cloud?

null

What is the significance of Tenstorrent's decision to open source their bare metal software stack?

By open sourcing their bare metal software stack, Tenstorrent aims to foster community engagement and collaboration. This transparency allows developers to see and understand the underlying mechanisms of the software, including how kernels are compiled, memory is allocated, and runtime arguments are managed. It also ensures that anything developers write for the current hardware will be forward-compatible with future Tenstorrent hardware generations.

Outlines

00:00

🎥 Introduction and Meeting Yasmina

The video begins with an introduction to the setting, a special guest, Yasmina, is welcomed. Yasmina is a fellow at T-torrent, and the host expresses excitement about upcoming developments at T-torrent. Yasmina's background in FPGAs and high-level synthesis is highlighted, emphasizing her expertise in hardware-software code design, which is crucial for making new hardware accessible through programming. The conversation touches on the challenges of programming FPGAs and the goal of making novel architectures user-friendly for developers.

05:01

📦 Unboxing the T-torrent Hardware

The host and Yasmina proceed to unbox T-torrent's hardware, revealing two products: the Grull chip and two different cards, the e75 and e150. The e75, a 75-watt card, is the focus of the unboxing. The host describes the card's features, including its use in machine learning applications and its developer orientation. Pricing and availability for developers are discussed, along with the company's intention to make the hardware accessible and gather feedback from the developer community.

10:01

🛠️ Developer Experience and Software Stacks

The conversation shifts to the developer experience with T-torrent's hardware. Yasmina explains the two software stacks available for developers: Buddha, a high-level compiler for quickly running models, and a low-level, bare metal programming model for fine-grained control. The importance of low-level APIs for IP customers and the company's commitment to open-sourcing their software stack are emphasized. The host and Yasmina also discuss the company's approach to API compatibility and the balance between maintaining backward compatibility and leveraging new architecture capabilities.

15:02

🌐 Cloud Access and Future Roadmaps

The discussion moves to T-torrent's cloud offerings and how they serve as an easy entry point for developers. The cloud's role as an internal testing ground and its importance for customer feedback loops is highlighted. Yasmina shares insights into the company's roadmap, including plans for hardware in desktops and the desire to empower developers with both cloud and physical hardware access. The conversation also touches on the company's approach to community engagement and updates, as well as the potential for collaboration with other AI hardware startups.

20:02

🤝 Community Feedback and Next Steps

Yasmina and the host conclude the discussion by emphasizing the importance of community feedback. Yasmina outlines the process for developers to get started with the hardware, including a 'first five things' guide for both software stacks. The conversation highlights the company's commitment to open sourcing and the desire to grow collaboration with the open-source community. The host and Yasmina express optimism about the future of T-torrent and the AI hardware ecosystem.

Keywords

💡Hardware-Software Co-Design

Hardware-Software Co-Design refers to the collaborative process of designing both the hardware and software components of a system to ensure they are optimized for each other. In the context of the video, Yasmina emphasizes the importance of this approach in creating accessible entry points for developers and ensuring that new hardware is easily programmable. The video highlights how this co-design strategy is a key building block at Tenstorrent, where the focus is on making novel architectures programmer-friendly and creating a smooth development experience.

💡FPGAs

Field-Programmable Gate Arrays (FPGAs) are integrated circuits that can be programmed after manufacturing to perform specific tasks. Yasmina mentions her doctorate work in FPGAs, indicating their importance in the field of hardware development. FPGAs are noted for their flexibility and reprogrammability, which are crucial for developing novel hardware architectures and software ecosystems, as discussed in the video.

💡High-Level Synthesis

High-Level Synthesis (HLS) is a technique used in electronic design automation that allows developers to design hardware using high-level programming languages, such as C or C++. Yasmina's experience in HLS is highlighted, showcasing her expertise in translating high-level code into hardware description languages, which is essential for creating accessible programming environments for new hardware technologies.

💡Machine Learning Frameworks

Machine learning frameworks are software libraries that provide an environment for developing, training, and deploying machine learning models. The video mentions frameworks like PyTorch and TensorFlow, which are widely used in the industry. These frameworks are important for Tenstorrent as they aim to support and integrate with these tools to make their hardware more accessible and useful for machine learning developers.

💡Grull

Grull is the name given to Tenstorrent's first-generation hardware architecture, which is a developer kit aimed at machine learning and AI applications. The video discusses two versions of the Grull card, the e75 and e150, differing in wattage and presumably performance. The Grull cards are designed to be developer-friendly, allowing easy access to the hardware for testing and development purposes.

💡Bare Metal Programming

Bare metal programming refers to the process of writing software that interacts directly with the hardware, without an operating system or other abstraction layers. In the video, Yasmina talks about Tenstorrent's approach to providing developers with the ability to program at a low level, giving them fine-grained control over the hardware. This approach is contrasted with high-level entry points and is intended for developers who want to optimize their code for performance.

💡Open Sourcing

Open sourcing refers to the practice of making the source code of a software project publicly available, allowing others to view, use, modify, and distribute the code. In the context of the video, Tenstorrent plans to open source their bare metal software stack, which includes APIs and other tools, to foster community engagement and collaboration. This is seen as a way to empower developers and to ensure that the hardware remains accessible and adaptable to future needs.

💡Community Engagement

Community engagement in the context of the video refers to Tenstorrent's strategy of involving developers and users in the development process of their hardware and software. By providing access to the hardware, tools, and documentation, they aim to create a feedback loop that can help improve their products. Yasmina emphasizes the importance of community feedback and the company's commitment to being open and transparent with their development process.

💡NDA (Non-Disclosure Agreement)

A Non-Disclosure Agreement (NDA) is a legal contract that establishes a confidential relationship between parties and restricts the disclosure of certain information. In the video, it is mentioned that Tenstorrent does not require NDAs for developers working with their hardware, encouraging open communication and feedback. This approach is intended to foster a collaborative environment and to enable developers to freely share their experiences and insights with the technology.

💡Backwards Compatibility

Backwards compatibility means that a newer version of a technology or software is able to understand and work with data or code from older versions. In the video, the importance of maintaining backwards compatibility is discussed, particularly in relation to the APIs and software stacks that Tenstorrent is developing. The goal is to ensure that anything developers create for the current generation of hardware will also work with future generations, providing stability and continuity for developers.

💡Cloud Computing

Cloud computing refers to the delivery of computing services, such as storage, processing power, and databases, over the internet (the 'cloud'). In the video, Tenstorrent mentions having their own cloud infrastructure, which allows clients and developers to access and test their hardware and software without needing physical access to the devices. This cloud-based approach provides a convenient and quick way for users to experiment with and develop for Tenstorrent's hardware platforms.

Highlights

Introduction of Yasmina, a key figure at Tenstorrent, a company focusing on novel hardware and software.

Discussion on the importance of hardware-software co-design for accessible programming.

Yasmina's background in FPGAs and her contribution to Tenstorrent's high-level synthesis efforts.

The unveiling of Tenstorrent's first-generation hardware, the Grail chip, aimed at developers and enthusiasts.

Explanation of the two different cards available: the 75-watt e75 and the 150-watt e150.

Unboxing of the Grail chip and its development kit, providing insights into the physical product.

Discussion on the necessity of active cooling even at 75 watts and future optimizations.

Pricing details for the Grail chip, with the e75 at $599 and the e150 at $799.

Emphasis on the Grail chip as a developer kit, designed for hands-on experience and feedback.

Mention of the lack of Windows support, with plans to add it in the future.

Introduction to Tenstorrent's software stack, including the Buddha compiler for high-level entry points.

Explanation of the bare metal programming model for developers seeking fine-grained control over workloads.

Tenstorrent's commitment to open-source their full bare metal software stack for community engagement.

The importance of low-level APIs for IP customers and the company's dedication to providing direct access to hardware.

Tenstorrent's cloud-based hardware, offering an easy entry point for clients and developers.

The company's approach to community feedback and support, with plans for open collaboration.

The significance of having both cloud and desktop hardware options for developers, empowering them to choose their preferred environment.

Discussion on the trend of specialized software stacks for specific hardware and the benefits of such an approach.

The competitive landscape of AI hardware startups and Tenstorrent's view on collaboration over competition.

The importance of community values in software development and Tenstorrent's commitment to making the entry point convenient and enjoyable for developers.

Transcripts

[Music]

[Applause]

so hi everyone once again I'm here at 10

torrent um we're going to ignore these

for a second cuz I've got a special

guest I want you to meet this is yasmina

one of the fellows here at T torrent

welcome to the channel yasmina thanks

awesome to be here I've known yasmina

since I've been doing my visits here to

bit Jim and the family um and the first

time we met you gave me a a very secet

deep dive into some of the architecture

on the T I'm not sure it was it was very

fun and it's stuff that they definitely

want to talk about at some point in the

future um but we're here because ten T

has got some exciting stuff lined up

we're it's actually just before all that

stuff is being finalized so there's some

things we can't talk about but I wanted

to get you asmin on because she's part

of this industry that's hustling and

bustling working with new novel Hardware

Hardware software side um I mean you may

deal on the software side um how's that

been we do a lot of Hardware software

Cod design yeah so the hardware software

code design is one of the key aspects of

the software right because if you just

make new hardware but nobody can program

it what's the point like the access the

entry point is really really important

right so the hardware software codesign

is is one of the key building blocks of

what we do and it's the key thought

process that drives a lot of things here

at Thor so

what in your background brought you to

tens T oh that's a good question I I did

my doctorate in

fpgas and uh sort of you know a little

bit of the cat tool place and routee and

then uh and then most of it on the high

level synthesis side right and uh you

know sometimes in the fpj industry we

joke that the fpj is the hardest thing

to program in the

world which makes our our pain tolerance

level really high uh so you could be

needled for a long time and up with it

yeah right yes and and it gives us a lot

of this like a strong drive to make

novel architectures really fun to

program really convenient to program to

create these easy you know entry points

to hello world programs and then to

create hardware and software

environments that are fun to Tinker with

and that developers feel empowered and

excited about new Futures coming in and

sort of can just brainstorm and think

about all the co and fun apps they can

build on top it's uh I I speak with a

number of the fpga companies out there

and and it's I always say you guys need

to abstract higher and higher make it

more and more and more accessible and I

feel like we're kind of at that same

point a little bit with machine learning

right are we're dealing with lots of

these Frameworks pytorch tensorflow Onyx

that's right and support for those is

vital now that's right that's right it's

it's really interesting to think about

the different entry points what they're

for and then to think think about the

levels of abstraction and then you know

is it a really high level entry point or

is it a low level entry point and each

one of those is important but for a

different use case right so you always

want to have a high level entry point if

you don't have one like like you know

like the entry points are a matter of an

and not an or right you never want to go

I want to you know grab the developers

by the florens and force them down this

path never force a developer you will

never force a developer you will welcome

them and you will guide them and have

them enjoy all the different paths that

lead to Rome I.E Tor T stor and hardware

and uh yeah it's been really fun to

watch sort of the development of the

different Frameworks so before we go a

little bit down to the through I do want

to get into one of the reasons why we're

here so I kind of reached out and said

hey do you have any hardware for me to

unbox for us to unbox uh and they and

they said kind of um so what we have

here explain explain what we have so

there's two products here that's right

so in these boxes is a product that we

call Grull or a chip we call Grull it is

our gen one Hardware our gen one

architecture so it's the first one that

is going out to customers to developers

it's a dev kit MH and uh there are two

different cards uh e75 which is which is

a smaller card a smaller 75 cuz it's 75

Watts 75 watts and then this guy is e

E150 again again with a reference to

wattage it's it's slightly bigger and uh

yeah I'd love to show yeah it's uh we've

been told to focus on this one um all

right so so so that's the 150 let's

focus on the 75 watt um I can already

feel it's a bit it's a bit lighter than

this one it's a bit lighter than this

one right yeah yeah yeah yeah that's

right how about you do the honors oh

it's it's it's it's Christmas well it's

not Christmas for another few weeks but

it's Christmas it's Christmas early so

so so so uh when people get their hands

on these this is what it's this is what

they're going to end up with that's

right this is what so

for

it so the minute we open uh obviously

the standard stuff but because I know a

lot of people in the audience are very

familiar with the unboxing say graphics

cards sure and everybody knows what a

graphics card is for what's a a she

learning card for well here's here's the

handy everything you need to set up um

your your your your card um and we kind

of already know that this is the very

basic very basic this is the PCI version

that's right so we've got a half height

full length card um let's shift this out

of the way just for a second um I can

tell you've been using this on demos

already with clients that's right he's

got a very few fingerprints on it um but

this is a typical sort of machine

learning PCO card we would see you know

for Mass scale inference perhaps in a

data center um but this is the developer

version that's right so there's a bit

more branding that's right definitely

got the logo on side um you've got a

blower fan it's a bit hard not to notice

um yep you got to plug this guy in it'll

be fun and it'll dry out you know drown

out a little bit of noise from your

neighbors um but it's not it's not that

loud um and it it does fit into your

desktop yeah yeah but it's Cu uh even at

75 Watts you still need some amount of

active Cooling in this form factor I

mean if you put this in a dual wide

double proper height PCI card that's

right so there's there's fun things that

we are sort of exploring and optimizing

with respect to the fans and the cooling

so there'll be more fun announcements

coming down the line for that uh this is

what it looks like today and we were

very eager to get these out to get them

into the hands of developers we you know

we're we're okay with the cooling um

it's it's it's interesting like we just

want people using them we want them

going to the website downloading the

tools plugging them in trying things out

like just banging away on the hardware

it's um so the way that is going to go

for you guys who are interested in this

stuff there's um at some point they're

going to be available to signups um with

that somebody will reach out and at

least acknowledge that you're actually

truly a developer and you're actually

going to use this stuff and and then

it's going to be a case of you'll be

able to buy it can we mention the

pricing y absolutely so so uh if I if I

remember this correctly this is going to

be the 599 version that's US Dollars um

and then the 150 W version will be

$799 yeah is that right that's right um

which uh I know for a lot of um very you

know entrylevel developers may seem like

a lot of money that the the realistic

expectation you have to have is this is

essentially a dev kit that's right it's

designed for developers to get a uh

grasp with a system so there are going

to be small medium businesses who are

wanting to see if this is useful for

their models and that's the sort of

price point that it goes at the it's it

low volume typically than a massive GPU

launch exact so you have to do factor

that in for reference um uh SCI did a

board and that was

$666 uh qualcomm's dev kit for the hexan

is is about 600 so this fits in right

about that level exactly the ballpark is

there we want to make the hardware

accessible we don't want it you know to

hit your pockets too hard like we want

people to be excited to play around with

it the the the the one thing I so you

know I speak to many companies in this

space and I I keep telling them where's

the dev kit where's the dev kit

accessible make it accessible these guys

are actually doing that because it

benefits a company like ten turnor to

have you know several thousand

developers with the even with if you

even if it's high level still the

framework and and and and and or the

lower level um

optimization um but yeah we actually

have

Hardware that's right you can smell

it it's uh we're excited we're excited

to get these into the hands developers

to get feedback um it's going to be you

know Tor is very very proud to ship to

ship hardware and to have people you

know give us feedback good or bad we

want to hear it all so so let's go

through a bit what that experience is

going to be like and this won't

definitely won't stand up I didn't plan

for this it's okay there it'll be there

um so so what's that going to look like

for for for developers who get in touch

end up with a card in hand is it is just

a link to websites downloaded the yeah

so the card sort of welcomes you and

tells you where to go that's your first

entry points from there you can sort of

download the driver's tools and get the

basic setup going um and that's kind of

table Stakes so you can plug this into

your desktop and then install our

drivers and tools is it Linux only or

are you supporting Windows as well we're

not supporting Windows yet that's right

yet she said yet yes I did say yet

that's right that's right um it's it's

on the road map it's not it's not there

yet um and uh once you install the basic

tools and drivers you can sort of check

the health of the car you can see it

come up uh there there tools that will

you know give you confirmation that that

the hardware works and that your

computer recognizes what's has been

plugged in uh from there you have a

choice of going down one of two paths um

or both preferably both preferably both

it's an and not an or yeah yeah they are

looking for testers so that's right

that's right so one is a software stock

that we call Buddha and it's a compiler

um Buddha that sounds familiar it does

yeah we talked about it a lot it's on

our website um it's it's the compiler

that we've been working on and uh and

it's a it's a really fun way to get

models working out of the box right it's

You Know download model from hugging

voice uh from from hugging face uh

Buddha will compile it and it will run

it on our so that's really really fun

and uh we refer to this entry point as a

you know a high level top down entry

point because you don't have to change

your environment you don't have to

rewrite your model it's kind of push

button it works it will run the hardware

and show you what's going the other way

is a bottom up stock that we call our

bare metal programming model okay and

that one is a lot lower level right so

comes back to sort of the abstractions

and the enter points its use case is a

bit different right it requires you to

you know rewrite things in Python apis

right it's not a pie torch out of the

box experience yeah right it's for

developers that want to have fine grain

control over the workloads that they're

running on our hardware and an

alternative path to being able to write

kernels all the way down to kernels that

run on our risk cores and and drive the

heavy Math logic like custom operators

custom operators custom data movement

custom hacks custom you know custom

Explorations with Noble Ops they want to

plug into their llns like control flow

like any kind of you know fancy caching

new embedding like like all of that is

accessible to you as a developer to

Tinker with and you are never sort of

blocked by a high level of raction layer

you can go you can bypass it and you can

go directly to kernels and control the L

level Hardware in order for developers

to do that they're going to need to have

a deep understanding of the underlying

architecture that's right so there's

going to be some disclosures about the

10 six cores soon so the feedback that

we got from customers that have looked

at our bare metal software stock is you

know they would come in and they would

they would look at a sum and after a few

weeks they would say we understand

everything that's in your Hardware now

right without the need for that

discussion which is really really fun I

mean we have documentations right like

obviously you know it's a non-trivial

entry point right and um you know we

have documentation that explains the

high level view of the architecture and

the programming model you know the

two-dimensional grid of course the knock

so we do set them up with with Basics

and then what we see happen is that you

know experts will go in and they will

they will read our low-level programming

model and and you know we we say like

this layer is just a reflection a mere

image of the hardware right so what you

see there is what you get we don't try

to package it for you we don't try to

steer you this way or that way like you

know what's in the hardware it's is

what's there and then the bare metal

programming model is just a reflection

of how to directly Drive everything

that's available in the engine which is

really cool and I'm really excited about

sharing that with the community well

it's the developers you've worked with

today and the the clients the companies

the partners they've obviously been n ND

up to the hill until now that's right um

but but any developer gets hands on this

there's going to be no sort of NDA in

place and the idea is go out there go

talk play poke right tell us what's

wrong tell us what's right yes we want

that feedback we want the community

engagement so with the hardware that

we're going to be shipping we're also

going to be open sourcing our full bare

metal software stock so that means that

you get to see the apis of course but

you also get to see everything under the

hood you get to see the way that the

kernels get compiled the memory

allocator the way that the runtime

arguments get copied onto device the

kernels get dispatched like you see all

of the plumbing and functionality so

it's kind of cool everything that

somebody's going to write for one of

these cards is going to be Ford

compatible with all future tense tent

Hardware right so that's an interesting

point um maybe the right way to think

about it is that there's two aspects of

apis because that's kind of like API

compatibility s process right so there

are host apis and then there are kernel

apis right on the host side of the apis

um we looked at opencl we looked at Cuda

like we're very familiar with these

lowlevel programming models we didn't

want to reinvent the wheel there and so

we mimicked those apis to be intuitive

and kind of you know behave very very

similar and be very familiar to

developers at that level so host apis

are are you know relatively easy to ke

to keep backwards compatible I'll

probably regret that as as soon as I say

it right um but you know they're kind of

defined and they've matured to a certain

degree that you know the design space is

is not being like wildly explored in

that area and then on the Kernel API

side there is a strong desire to keep

backwards compatibility that's important

to us um however in reality you know if

you are allowed to color outside the box

with next gen architectures you can make

leaps in performance and functionality

and it's a conscious decision right to

kind of go okay I'm going to you know

this is a new architecture New Gen next

gen we're going to maintain backwards

compatibility or we're going to allow

ourselves to color outside the box and

make a leap so that's backwards

compatibility but everything will be

forwards compatible anything you write

for this gen will work on next gen

that's the goal that's the goal that's

the goal I know I know you guys have

been pretty vocal about upcoming road

maps and as the company has taken on you

know new clients and new investors some

of those things are changing um from

your pers perspective obviously it's one

thing to support this but you've still

got to think about what's coming down

the line uh how much does that change

over time what's coming down the line

world now no I mean how how you look at

it from from that sort of high and high

and low level software layer um yeah

that's a really really good question

high level entry points are less

susceptible to lowlevel changes yeah

right and developers like them because

of that right and because they give them

a quick path to a desired outcome when

they stay within that that sort of high

level programming model and we want that

and you know we've seen like to go back

to what you were saying earlier right

like like before there were a lot of

Frameworks that developers used over

time they kind of Consolidated on two

pie torch right so there was a

consolidation effort now we see a growth

in number of Frameworks high level ones

and it's interesting to knowe that you

know like developers seem to enjoy

things that are for a particular purpose

mhm and we see a lot of specialization

there like if you make a high L API that

aims to do everything under the sun you

usually end up with leaky abstractions

and developers that kind of get and

attack surface for exactly exactly so it

seems that at the high level you know

there are things that are special there

are apis that are you know specializing

towards a certain purpose Frameworks

that are specializing towards a purpose

and we are in that game as well um with

the lowlevel API we want to make sure

that developers always have access ACC

to the hardware and that nothing is

hidden from them and IP business is a

big branch of ours that's fairly

important and for a customer that's a

potential IP customer or is an IP

customer they want to know exactly

what's in there yeah right they want to

control it they want to drive it and

then sometimes they will get ideas about

oh I wish I had this feature that

feature and then they can sort of

visualize that and and and and sort of

figure out what they could want through

the low-level software yeah so that

entry point is is super important for us

um

I mean these cards here that we got in

front of us they're they're more for you

know sort of the take-home put in your

workstation but you guys have had uh

Hardware in the cloud for a little bit

as well that's right buil built its own

cloud ground ground

up no it's important for those first

clients who are getting on board to

start testing how's that been it's an

easy entry point it's super convenient

right it's you know it's SSH and off you

go to Hello World so that's been really

really fun it's the fastest way to get

people to access the hardware and try

out simple things and then also try out

complicated things we have customers who

are running on the cloud today and it's

a great test bed for us right so we make

drops to the cloud we deliver our

software to the cloud they are our first

internal customer and that feedback loop

has been really really important

customers enjoy the very quick

turnaround that we can give them okay in

terms of machine access and everything

is set up and works as they just as in

so and the Reas so the reason why why

you're perhaps not opening so some

companies open up a cloud to developers

to get one instance or whatever you guys

are going down the Hardware Route

because we're about ANS not ores about

or yeah we want both y we want both um

we have a certain amount of cloud

capacity today we utilize it almost 100%

every single new server we put in

there's there's a weight list the folks

ready to try okay and uh we want to make

sure that developers can get Hardware in

their desktop you know they can hear it

run and you know they can go on and

install all the tools and and test that

flow as well I think hardcore serious

developers they love being able to touch

the hardware and install and having

things in their own hands right as

opposed to some server distance

somewhere that something happened to it

and went down like it's you know it's

about it's about empowering vs right so

so with the cloud it's easy to do those

very FastTrack updates especially if

it's a client who's putting in you know

money um and money over time for

individual developers how are you going

to uh discuss with that and that

Community um about how updates are being

rolled out and so we have releases for

both software Stacks y um announcements

that will go uh together with that

release Cadence um the sort of

underlying tools have their own releases

as well yeah and so all of this will be

announced on our on our website yeah and

developers can uplift to latest versions

as as kind of as see fit of course on

the cloud uh this is a little bit more

behind behind the scenes fluid yeah a

bit more fluid right yeah so when this

Hardware goes out you've already got a

support staff ready to De that's right

we have uh we're you know we're a pretty

small team yeah I'll say um so on the

bare metal programming model we're we're

not a we're not a huge team we're we're

a small team of of you know very very

smart individuals super excited and

super dedicated to shipping the software

to open sourcing it and and kind of

showing it to the community that said

we're not yet at the stage of being able

to service and and sort of keep up with

large pull requests right and I think I

think that's normal like I think that's

kind of like a growing stage that that

Community mostly mostly understands um

we want to be able to develop in the

open we want to be able to show what's

there um you know and then and then we

have strong ambition to grow to the

point that that we can have strong

collaboration with open source

Community does it matter that there are

several dozen other AI Hardware startups

out there doing their own thing did you

ever think about you know competition

versus collaboration or anything El yeah

I think it helps that there are other

startups doing similar things as we Doh

um you know the goal here like we're in

the race against Nvidia and and we want

as many players in our court as possible

right of course we with them you know a

lot of our colleagues work in the other

startups we all kind of know each other

it's right it makes for for interesting

Thanksgiving dinners okay okay okay yeah

but um one family M at one company one

family member another it it's happened

it's happened my husband and I worked at

zyink and ala at the same time so it's

for okay yeah um yeah I think I think it

helps I think things that you know

activities and companies software stocks

that that grow the community and and get

it to be diverse like increase that

design space it helps we learn from each

other um and and it's a long it's a long

path ahead I I think it's going to be

really really fun and now we see

software stocks that are moving away

from just pytorch right so for us uh we

have a road map item to integrate into

pytorch 2.0 natively and to generate a

pull request and to be in the open open

source repo right so that's that's on

our road map um but we also see that

that there's a trend of software Stacks

that are are being developed

specifically for a piece of Hardware

because you can then control the apis

and allow users to do specific things

that are native to that Hardware right

without sort of forcing developers to go

behind this General one you know to rule

them all right API and ecosystem that

that leads to Leaky abstractions if you

want to do specific

things um so I think it helps anything

that grows the ecosystem is good and fun

so for somebody who ends up getting this

Hardware what's the first thing that

they should do what's the first model

that they should run just to make sure

it all works you what's what's the

Shakedown procedure what's the Shakedown

that's cool we have for both software

stocks um a landing page that takes you

through first five things Y and that's

really fun for Buddha there's a there's

a few models you can just sort of you

know like you're five clicks away and a

script away from running end to end on

the bare metal side uh there are a few

models that are optimized for

performance that you can run out of the

box and then there are few few kernels

that you can run and sort of see how

things work end to end and the stack

also comes with debug tools kernel print

uh performance uh performance we

integrated into the open source Tracy

tool so you can see performance profile

of what's Happening um so it's kind of

you know what's the level of exposure

and deep dive that you want to go in and

there's the first five things that take

you progressively down that path all the

way to running running kernels and

seeing what happens so how often do you

have to report back to Jim on what the

community is saying is that decided yet

uh it's it's it's moving from hourly to

daily yeah

yeah it's it's it's frequent he he

deeply cares about what the community is

doing yeah and and he's driving T

torrent into strong awareness of of

Community Values software development

values and ensuring that the entry point

is is really convenient and fun for for

developers right so it's good you you

you say Hardware software

co-optimization you're very you're very

much I I feel like so passionate about

the software

so it's it's it's it's good to hear it's

great that you guys are being open and

and essentially explain to everybody

what you're doing right it's I have so

many um conversations with other people

and they just want to keep it closed up

just for them and um and and and and you

guys know I've been advocating for more

Hardware the hands of people that's

right um so so this is this is a first

step long may continue thank you y for

being on the channel thanks yeah it's

really

[Laughter]

[Music]

fun