Exclusive: We tried Meta's AR glasses with Mark Zuckerberg

Summary

Orion, Meta's AR glasses prototype, offers a glimpse into the future of smart glasses. With a 70-degree field of view and ULED projectors, Orion provides an immersive experience. It features a neural wristband for gesture control, enhancing interaction. Despite its impressive tech, the software is still in early stages, and Meta plans to refine it before a commercial release.

Takeaways

• 🤖 Orion is a prototype AR glasses project from Meta, not intended for commercial sale but to showcase future hardware innovations.
• 👓 The glasses include a neural wristband for control and a wireless compute puck to improve battery life and reduce heat.
• 👀 Navigation is achieved through eye and hand tracking, voice commands, and the neural wristband which interprets neural signals into inputs.
• 📱 The neural wristband uses electromyography to recognize specific hand gestures, providing a new way to interact with AR glasses.
• 🖥 Orion offers a 70-degree field of view, wider than any AR glasses tested, providing a more immersive experience.
• 🔍 The glasses use ULED projectors and wave guide lenses to display graphics, and seven cameras and sensors to anchor virtual objects in the real world.
• 🍹 A demo showcased Meta AI's ability to identify ingredients and suggest recipes, indicating potential uses of AR in everyday tasks.
• 💻 The software is still in early stages, with demos being guided and limited in scope, showing there's room for improvement.
• 🤝 Despite the advanced technology, there were moments where the AR experience felt intrusive or disconnected from reality.
• 🚀 Orion represents Meta's vision for the future of AR glasses and the Metaverse, even if the technology isn't ready for mass-market yet.
• 🔮 Meta is using the prototype to demonstrate the potential of AR glasses and to gather feedback for future development.

Q & A

What is the significance of Meta's Orion AR glasses?

Orion is a prototype that provides a glimpse into the future of smart glasses and Meta's hardware releases. It's not a product for sale but a marker of the direction AR glasses are heading.

What are the three main components of the Orion AR glasses system?

The Orion AR glasses consist of the glasses themselves, a neural wristband for controlling them with finger movements, and a wireless compute puck that offloads app logic to improve battery life and reduce heat.

How does the neural wristband work?

The wristband uses electromyography to interpret neural signals and translate them into input for the glasses. It recognizes specific gestures like pinching fingers to select or invoke menus and swiping to scroll.

What is the field of view like with Orion AR glasses?

Orion offers a 70-degree field of view, which is wider than any AR glasses tried to date. This allows AR graphics to fill more of the user's vision.

How are graphics projected in the Orion AR glasses?

ULED projectors inside the frame beam graphics in front of the user's vision via wave guides in the lenses, which are made of silicon carbide instead of glass.

What role do the cameras and sensors in the Orion frames play?

Seven cameras and sensors in the frames sense the world around the user to anchor virtual objects in space, allowing for a realistic AR experience.

What was the user's experience with the graphics quality in the Orion AR glasses?

While the graphics quality wasn't high enough for watching movies, the user had no problem reading text on a webpage several feet away.

What was a notable demo experience with the Orion AR glasses?

A notable demo involved using Meta AI to identify ingredients for a smoothie and then showing a recipe with instructions, demonstrating the potential intersection of generative AI with AR glasses.

What challenges did the user encounter during the demo?

During the demo, an incoming video call pushed a window over a person the user was sitting across from, making it feel like the glasses were breaking reality rather than augmenting it.

Why is Meta showing Orion to the world if it's not for sale?

Meta is showing Orion to demonstrate the progress they've made in AR technology and to give a glimpse into the future of AR glasses, even though they are not yet ready for mass-market release.

What is the current status of the Orion AR glasses in terms of production and availability?

Meta has made about 1,000 pairs of Orion for internal prototyping and demos. The glasses are not available for sale and were initially intended to be a product until Meta realized the manufacturing challenges.

Outlines

00:00

🤖 First Look at Meta's Orion AR Glasses

The script introduces Orion, Meta's first pair of augmented reality (AR) glasses, which were intended to be a commercial product but are now showcased as a preview of upcoming hardware. The author describes their experience using Orion for browsing the web, playing games, and video calling via Messenger. The glasses are part of a system that includes a neural wristband for control and a wireless compute puck to manage app logic. The interface is navigated through a combination of eye and hand tracking, voice commands, and the wristband, which uses electromyography to interpret neural signals into inputs. The wristband's main gestures include selecting by pinching the index finger with the thumb, invoking the main menu with the middle finger and thumb, and scrolling by swiping the thumb against a closed palm. The script also discusses the glasses' 70-degree field of view, which is wider than any AR glasses the author has tried, and the use of ULED projectors and silicon carbide wave guides for displaying graphics. The demo included experiences simulating daily use of AR glasses, such as identifying ingredients for a smoothie and displaying a recipe.

05:01

👓 Peeking into the Future of AR Glasses with Orion

The second paragraph discusses a video call experience using Orion AR glasses, where the author notes the quality of the video and audio. It mentions the glasses' inward-facing cameras that can create a Codec avatar for the wearer during calls. The author reflects on Mark Zuckerberg's long-standing vision for AR glasses as a 'holy grail' device to replace smartphones and how Orion makes that future feel more tangible. However, the script also points out the challenges and the work that Meta still needs to do to refine the AR experience. An incident during the demo where an incoming video call obstructed the view of a person sitting across the author highlights the need for improvement. The paragraph concludes with insights into Meta's decision to showcase Orion publicly despite it not being available for sale, suggesting that Meta wants to demonstrate the progress on its AR technology and the potential of the Metaverse, even though the technology is not yet market-ready.

Keywords

💡AR glasses

AR glasses, or Augmented Reality glasses, are wearable devices that overlay digital information onto the physical world. In the video, AR glasses are the central theme, with Meta's Orion being a prototype that allows users to browse the web, play games, and make video calls, indicating their potential to revolutionize how we interact with technology.

💡Orion

Orion refers to Meta's first pair of AR glasses mentioned in the script. It represents a glimpse into future hardware and the direction smart glasses are heading. Despite originally being intended as a commercial product, Orion is now used for prototyping and demonstration, showcasing features like a wider field of view and advanced interaction methods.

💡Neural wristband

The neural wristband is a component of the Orion system that uses electromyography to interpret neural signals into inputs for the glasses. It allows for control through finger movements, enhancing the interaction with AR without direct touch. The wristband is highlighted as a 'magical' part of the technology, indicating its innovative approach to user interface.

💡Eye and hand tracking

Eye and hand tracking are methods used in Orion to navigate the interface. The script mentions using one's eyes or hands as a 'mouse', with selections made by pinching fingers together. This technology is integral to the user experience, providing a more intuitive and less intrusive way to control AR applications.

💡Field of view

Field of view (FOV) is a measure of the extent of the observable environment one can see at a given moment. The script notes that Orion's 70-degree FOV is wider than other AR glasses, allowing for a more immersive experience by reducing the disruption of AR graphics' edges in the user's vision.

💡ULED projectors

ULED projectors are used within Orion to display graphics. As described in the script, these projectors beam graphics in front of the user's vision via wave guides in the lenses, which are made of silicon carbide. This technology is crucial for the visual output of the AR experience, determining the quality and clarity of the graphics.

💡Wave guides

Wave guides in the script refer to the technology used in Orion's lenses to direct the projectors' light to the user's eyes. They play a critical role in how AR images are superimposed onto the real world, affecting both image quality and the device's form factor.

💡Generative AI

Generative AI is mentioned in the context of using Meta AI to identify ingredients for a smoothie and providing a recipe. It represents the ability of AI to create new content based on existing data. In the video, this technology is showcased as a powerful intersection with AR, suggesting future applications in personalized content creation.

💡Codec avatar

Codec avatars are virtual representations of a person used in video calls. The script discusses how the glasses have inward-facing cameras to scan the user's face and represent them as an avatar during calls. This feature is an example of how AR can enhance communication by providing a more dynamic and personalized visual interface.

💡Metaverse

The Metaverse is a collective virtual shared space, and the script positions Orion as a significant step towards realizing it. Mark Zuckerberg's vision for AR glasses as the 'holy grail' device that could replace smartphones is tied to the broader concept of the Metaverse, suggesting that devices like Orion are intended to be central to this emerging digital ecosystem.

Highlights

Orion is Meta's first pair of AR glasses, offering a glimpse into future hardware releases.

Orion was intended to be a purchasable product but is now a prototype to showcase upcoming technology.

The reviewer experienced using Orion for web browsing, gaming, and video calls via Messenger.

Orion consists of three parts: glasses, a neural wristband, and a wireless compute puck.

Navigation is done through eye and hand tracking, voice, and the neural wristband.

The neural wristband uses electromyography to interpret neural signals into inputs for the glasses.

The wristband recognizes specific gestures like pinching fingers for selection and scrolling.

Haptic feedback on the wristband confirms gesture recognition.

Orion has a 70-degree field of view, wider than any AR glasses tried to date.

ULED projectors and wave guides in the lenses project graphics.

Seven cameras and sensors anchor virtual objects in space.

The graphics quality is sufficient for reading text but not yet for watching movies.

A demo showed using Meta AI to identify ingredients and provide a smoothie recipe.

Mark Zuckerberg has long envisioned AR glasses as the future, potentially replacing smartphones.

Orion was to be sold but is now a prototype due to manufacturing challenges.

Meta is showcasing Orion to demonstrate progress in AR technology.

A demo issue showed a window overlaying a real person, indicating work still needed for perfecting AR integration.

Transcripts

(ethereal music)

- This is Orion, Meta's first pair of AR glasses.

(ethereal music)

Up top, I want to be clear,

Orion was supposed to be a product you could buy,

but it's not.

Instead, it's a peek at what Meta has coming

for hardware it's releasing over the next few years.

And in a bigger way,

it's a marker of where the smart glasses of today

are eventually headed.

I was one of the first people outside of Meta to try Orion.

For about two hours, I used the glasses to browse the web,

play games, and video call with people through Messenger.

Oh, and Mark Zuckerberg beat me in Pong.

This is gonna be the whole interview.

It's just an hour of us-

- I know, it's like we were gonna ask questions,

but instead I just-

- [Alex] You say you're undefeated?

- No, no. - Okay.

- Absolutely not.

I'd never won a game before this week.

- [Alex] Okay, oh, okay.

- I had only been defeated.

Good game. - Good game.

You have an unfair advantage, but that's all right.

- I've played a few more.

(Alex laughing)

(upbeat dance music)

- Orion exists in three parts,

the glasses themselves, a neural wristband

for controlling them with finger movements,

and a wireless compute puck

that offloads app logic from the glasses

to help improve battery and reduce heat.

The way you navigate the interface

is through eye and hand tracking, voice,

and most importantly, the neural wristband.

It's hard to compare this experience to a laptop or phone,

but in the simplest of terms,

your eyes or hands act as the mouse

and you click by pinching your fingers together.

If you've ever used a Vision Pro,

you'll get how the eye tracking works pretty quickly.

I want to focus on the wristband for a second

because it's truly one of the most magical experiences

I've had with a piece of new technology

and I think the world will be seeing it in the wild

even before the commercial successor to Orion is released.

It's about the size of a Fitbit

and uses electromyography to interpret neural signals

and translate them into the input for the glasses.

The main gestures it recognizes

are pinching your index finger with your thumb to select,

pinching your middle finger and thumb

to invoke the main menu,

and swiping up or down with your thumb

against your closed palm to scroll.

There's haptic feedback on the band

to let you know when it recognizes a gesture,

which is a helpful signal

that made me quickly comfortable with using it.

Do you like using hand tracking at all with this

or do you mostly just do the band?

- It depends on what the app is.

I mean, for some of the things,

I think it is still pretty natural

to, like, reach out and touch it and all that.

- Yeah. - But, I don't know,

I mean, the band is just gonna

keep getting better and better.

You know, right now you have to do

these little subtle gestures, but, I mean,

the plan is over the next few years

to make it so that it can just pick up

almost motionless gestures that you make.

So I think that'll be pretty wild once we get there.

- The first thing you notice

when you put Orion on is it's 70 degree field of view,

which is much wider than any pair of AR glasses

I've tried to date.

A narrow field of view means that AR graphics

don't fill much of what you're seeing,

but with Orion, I had to get up pretty close

to something before its edges started to disappear.

This made a huge difference for the overall experience.

ULED projectors inside the frame of Orion

beam graphics in front of your vision

via wave guides in the lenses,

which are made of silicon carbide and not glass.

Seven cameras and sensors in the frames

sense the world around you

to anchor virtual objects in space.

You can leave a window open, turn your head and walk away,

come back, and it's still there.

The quality of the graphics isn't at the level

that I'd want to watch a movie in the glasses,

but I had no problem reading text on a webpage

that was several feet away.

My demo consisted of experiences designed

to simulate how one may use AR glasses

throughout the day in the future.

It's clear Meta has an idea

for how people will use glasses like these,

but the software is early.

My demo was very much on rails

and guided by Meta employees in the room,

though I was able to navigate the glasses

on my own a little bit and jump in and out of apps.

The coolest part of the demo was using Meta AI

to identify ingredients for a smoothie

that were out on a table

and then have the glasses show me a recipe

to make them with instructions.

It was a peek at how generative AI can intersect

with a form factor like this in pretty powerful ways.

Do you guys know that I make smoothies, like, every morning?

Like, this is kind of getting creepy.

I'm not gonna lie.

(all laughing)

Good to see you, man.

- Yeah.

(glasses bonging)

Give me a smoothie recipe.

- [AI Assistant] Sure, let me take a look

at what ingredients you already have,

then I can put a recipe together for you.

I found several ingredients.

Let me make a recipe.

- [Alex] That's all right.

It missed the pineapple, but that's okay.

- [Mark] You don't really want that in your smoothie.

- (chuckles) Yeah. - Deep down.

- [Alex] Okay, here we go.

- Matcha banana- - Matcha banana boost.

- [AI Assistant] That incorporates most of your ingredients.

- I would do that.

I've never done matcha in a smoothie.

We also chatted with our editor-in-chief Nilay Patel.

Can you see me at all?

Like, what do you see?

- I see nothing, I see a gray screen.

- Okay. - That's what Alex looks like

on this side of it. - Okay, I see you

but I've got Verge pulled up on a pane here

and then you're just, like,

a video hanging out in the middle.

- Do you see, like, a whole square?

Am I just, like, floating, is it just my head?

- [Alex] Yeah, you're floating.

It's like a square window.

You're floating and I can, like, come up to you.

- What's the resolution like?

I just came off "The Vergecast",

I got, like, a very French headphone hair going on here.

Can you see that? - Yeah, it looks good.

- I don't know about good, it's something.

- [Alex] It's good, I mean, it's like it's, you know,

it's not like you're gonna, like, watch "Avatar",

but it's like, it's good.

Like, I can see you well.

- You sound great, I have to say.

I can't tell if that is the microphone on the glasses

or the speakers on the iPhone, but you sound great.

- [Alex] It's the glasses, yeah.

- It's weird that I can't see you.

Are they gonna add avatars to this thing?

- [Alex] Yeah, so the glasses have inward-facing cameras,

so they will scan my face and represent me

as a Codec avatar when I'm calling you.

- But they're not selling this thing?

- [Alex] Yeah, they're not.

They didn't make many, they cost a lot of money.

(Nilay laughing)

It's just like, yeah. - Amazing.

Well, tell them to send me one.

- Yeah.

Yeah, Nilay wants one.

(Nilay laughing)

(upbeat music)

Mark Zuckerberg has been talking about AR glasses

for a long, long time.

He's called them the holy grail device

that will eventually replace smartphones.

And Orion is the first pair of AR glasses

I've tried that made me feel like that future

isn't so far away that it feels like a pipe dream.

But there was a moment in my demo that showed me

just how much work Meta still has to do to get this right.

At one point I had multiple windows open to the side

while sitting across from someone at a table

and an incoming video call then pushed

one of the windows directly

over the person I was sitting across.

And at that point it felt like the glasses

weren't augmenting reality, but breaking it.

(upbeat music)

Orion was going to be a product Meta sold to the world

until a couple of years ago

when it realized it couldn't manufacture them

in a way that would even make sense

at an ultra high-end price point.

So, instead, it's made about a 1,000 of them

for internal prototyping and demos like the one I received.

So, why is Meta showing Orion

to the world if you can't buy a pair?

It's impressive, even for a prototype.

And it's not an exaggeration

to say these glasses are the end state

of Zuckerberg's big bet on the Metaverse.

After all these years and the billions of dollars

poured into making them,

I think the company wants to have something to show off,

even if AR glasses still aren't quite ready for primetime.

- I do like the visual of this looking like "Mime Wars".

- (chuckles) Yeah. - It's like-

- [Alex] Yeah.

Well, this is, like, one of the oldest

video games ever, right?

So I guess it's appropriate we're doing it on here.

- But we got some more dimensions.