Pro Interpreters vs. AI Challenge: Who Translates Faster and Better? | WIRED

Summary

In a quest to determine the efficacy of AI in translation, Wired pitted the KUDO speech translator against human interpreters in a series of tests. The evaluation focused on conveying emotions, intonation, and nuances in speeches by King Felipe of Spain, Colombian President Gustavo Petro, and El Salvadorian President Nayib Bukele. While KUDO excelled in capturing technical terms and maintaining content completeness, it struggled with conveying emotion, syntax, and grammar. Human interpreters highlighted the AI's limitations in capturing the flow of speech and the figurative language. The conclusion suggests AI translators are suitable for informal situations but not for high-stakes scenarios requiring precise language and deep understanding.

Takeaways

• 🧩 AI Interpreters vs. Humans: The script discusses a comparison between AI interpreters, specifically KUDO, and human interpreters to determine which is more effective in conveying speech nuances.
• 🌐 COVID-19 Speech Test: The first test involves a speech by King Felipe of Spain about the COVID crisis, emphasizing the importance of emotion and connection with the audience.
• 🤖 AI's Content Completeness: In the first test, KUDO performed well in terms of content and completeness, but struggled with word choice and conveying the emotional depth of the speech.
• 🎤 Spontaneity and Flow: The second test with Colombian President Gustavo Petro's speech highlights AI's difficulty with capturing the flow of speech and interpreting pauses correctly.
• 📈 Strength in Technical Terms: AI proved effective in capturing and translating technical terms and titles, as demonstrated in the second test.
• 💬 Challenges with Syntax and Grammar: The script notes that AI often made mistakes with syntax, word choice, and grammar, which can affect the accuracy of the interpretation.
• 🇸🇻 Speed and Clarity: In the third test with El Salvadorian President Nayib Bukele's speech, AI managed to keep up with the speed but had issues with clarity and accuracy.
• 🌐 Global Inflation Example: AI's interpretation of the speech about global inflation demonstrated its ability to convey complex economic concepts, albeit with some ambiguity.
• 🛑 Limitations in High-Stakes Situations: Interpreters suggest that AI should not be trusted in high-stakes situations such as courtrooms, legal systems, or important diplomatic discussions.
• 🔍 Need for Flexibility and Openness: While AI is not a replacement for human interpreters, there is a call for examining and understanding how AI can be constructively used to aid interpreters.
• 🔄 The Future of Interpretation: The script concludes that the field of interpretation should be open to new ideas and flexible in integrating AI tools where appropriate.

Q & A

What is the purpose of testing KUDO, the speech translator, against human interpreters?

The purpose is to determine which method better transmits all necessary elements for people to understand what they're hearing, including emotion, intonation, completeness, spontaneity, and speed.

What were the key aspects of King Felipe of Spain's speech that were important to focus on during the test?

The key aspects were emotion and the ability to connect with the audience, as the speech addressed the citizens of Spain during a difficult time of the COVID crisis.

How did KUDO perform in terms of content and completeness when translating King Felipe's speech?

KUDO performed well in terms of content and completeness, but struggled with word choice in some areas, leading to borderline nonsensical translations.

What specific term did KUDO struggle to translate accurately in the context of King Felipe's speech?

KUDO struggled to accurately translate the term referring to the 'emotional and physical burden' of healthcare workers as 'the great emotional charge and physics on their backs'.

What was the focus of the second test involving former senator and current Colombian president, Gustavo Petro?

The focus was on spontaneity, examining how KUDO would handle a speech with a natural flow and pauses.

How did KUDO's performance differ when translating Gustavo Petro's speech compared to the first test?

KUDO was effective at capturing terms of art but struggled with pauses, creating 'islands' of complete thoughts that disrupted the natural flow of the speech.

What were the challenges KUDO faced when interpreting President Nayib Bukele's speech?

KUDO faced challenges with syntax, word choice, and sometimes grammar in English, and occasionally chose words that were not the most accurate way a native English speaker would phrase something.

What was the main takeaway from the test involving President Nayib Bukele's speech?

The main takeaway was that while KUDO was able to capture all content, it had difficulties with the nuances of language, such as metaphorical meaning and the natural flow of speech.

In what situations would the narrator not trust the use of AI for translation?

The narrator would not trust AI for translation in high-stakes situations such as courtrooms, legal systems, medical interactions, asylum cases, or high-level diplomatic talks.

What is the narrator's view on the role of AI in the future of interpretation?

The narrator believes that AI is not a replacement for human interpreters but a tool that can be used constructively and carefully to aid interpreters, provided it is understood and respected for its capabilities.

What does the narrator suggest as an appropriate use for AI translation like KUDO?

The narrator suggests using AI translation in informal situations where there is not a high level of consequence for mistakes.

Outlines

00:00

🤖 Comparing AI and Human Interpreters

The narrator discusses the potential threat of artificial intelligence to jobs, including interpreters. Wired tested a speech translator called KUDO against human interpreters to evaluate which better conveys emotion, intonation, completeness, spontaneity, and speed. The test involved comparing translations of speeches from various public figures, focusing on different aspects like emotion and content accuracy.

05:00

🗣️ Emotional Accuracy in Translation

In the first test, Barry and KUDO were given a 2020 speech from King Felipe of Spain about the COVID-19 crisis. The AI performed well in terms of content and completeness but struggled with word choice and conveying the speech's emotional nuances. While the AI produced mostly accurate translations, it sometimes rendered phrases in ways that were borderline nonsensical, failing to capture the king's emotional intent.

10:01

🎙️ Capturing Spontaneity in Speeches

The second test involved a 2021 speech by Colombian President Gustavo Petro. The AI was effective at capturing terms of art and specific phrases, such as 'president-in-office' and 'minister of finance.' However, it struggled with spontaneity and maintaining the flow of the speech, often creating disjointed and incomplete thoughts due to the speaker's pauses and inflections.

📈 Handling Fast-Paced Prepared Speeches

In the third test, Barry and KUDO interpreted a 2021 speech by El Salvadorian President Nayib Bukele, which was delivered at a high speed. The AI excelled in capturing all the content accurately, even outperforming the human interpreter in terms of completeness. However, it often made mistakes in syntax, word choice, and grammar, choosing words that were not the most accurate or natural in English.

🔍 When to Use AI in Interpretation

The final section discusses the appropriate contexts for using AI solutions. While AI can be useful in informal situations with low consequences for mistakes, it is not reliable for high-stakes environments like courtrooms, medical interactions, or diplomatic negotiations. The importance of understanding and respecting the tool's limitations is emphasized, advocating for its careful and constructive use.

🔧 Embracing AI as a Helpful Tool

The conclusion highlights the need to understand the power of AI tools to use them effectively and benefit everyone. Emphasizing flexibility and openness to new ideas, the narrative encourages leveraging AI constructively while recognizing it is not a replacement for human professionals.

Keywords

💡Artificial Intelligence (AI)

Artificial Intelligence refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. In the context of the video, AI is being tested against human interpreters to determine its effectiveness in accurately and emotionally conveying messages. An example from the script is the use of the speech translator KUDO, which is an AI tool designed to interpret and translate spoken language.

💡Interpreters

Interpreters are professionals who convert one spoken language into another in real-time, often used in situations requiring accurate communication across language barriers. The video's theme revolves around comparing the performance of AI interpreters like KUDO with human interpreters, highlighting the importance of interpreters in conveying not just words, but also emotions and nuances.

💡Inaccurate Interpretation

Inaccurate interpretation occurs when the meaning of the original message is distorted or misrepresented during translation. The video emphasizes the potential consequences of such inaccuracies, suggesting that they can significantly impact important decisions, such as a 'nuclear deal'. The script discusses the limitations of AI in capturing the full emotional and contextual depth of human speech, which can lead to inaccuracies.

💡Emotion

Emotion refers to the affective state or feeling that accompanies and influences a person's thoughts and actions. The script highlights the importance of emotion in speech interpretation, especially in a speech by King Felipe of Spain addressing the COVID crisis. The ability to convey emotion accurately is a key factor in determining the effectiveness of interpreters, whether human or AI.

💡Intonation

Intonation is the variation in pitch and tone in a speaker's voice that can convey meaning beyond the words themselves. The video script mentions intonation as one of the necessary elements for people to understand what they're hearing, suggesting that it is a critical component of effective communication that AI interpreters must capture.

💡Completeness

Completeness in the context of translation refers to the ability to convey the entire message without omitting any parts. The script evaluates AI's performance on this aspect by comparing the translations provided by KUDO with those of human interpreters, noting that while AI captured the content well, it sometimes struggled with word choice and context.

💡Spontaneity

Spontaneity is the quality of being natural and unforced, often seen in impromptu speeches or responses. The video tests AI's ability to handle spontaneous speech through a speech by Colombian president Gustavo Petro. The script notes that AI has challenges with the natural flow of speech, particularly with pauses, which can disrupt the translation's coherence.

💡Syntax

Syntax refers to the arrangement of words and phrases to create well-formed sentences in a language. The script points out that AI, while able to capture content, often has difficulties with syntax, leading to sentences that may not be grammatically correct or idiomatic in the target language, as seen in the translation of President Nayib Bukele's speech.

💡Word Choice

Word choice involves selecting the most appropriate words to convey a particular meaning or sentiment. The video discusses how AI struggles with word choice, as seen when KUDO translated 'emotional burden' as 'great emotional charge and physics on their backs', which was nonsensical and did not accurately reflect the original meaning.

💡Metaphorical Language

Metaphorical language uses figures of speech to convey a meaning different from the literal interpretation of the words. The script suggests that AI may not fully grasp the figurative meaning of language, which is a crucial aspect of human communication, especially in high-stakes situations where the interpretation of metaphors can impact decision-making.

💡Professional Interpreter

A professional interpreter is someone who has specialized training and experience in translating spoken language in various settings, such as conferences, legal proceedings, or diplomatic negotiations. The video emphasizes the role and value of professional interpreters in processing information and delivering accurate, nuanced translations, a task that AI is not yet fully capable of performing to the same standard.

Highlights

Artificial intelligence is threatening jobs, including interpreters, highlighting the need for accurate interpretation in critical contexts.

Inaccurate interpretation can significantly impact high-stakes situations, such as nuclear deals.

Wired tested KUDO, a speech translator, against human interpreters to evaluate effectiveness in transmitting emotion, intonation, completeness, spontaneity, and speed.

The test focused on emotion using a 2020 speech from King Felipe of Spain about the COVID crisis.

Human interpreters emphasize the importance of connecting emotionally with the audience during a crisis.

KUDO performed well in content and completeness but struggled with word choice and capturing the emotional tone accurately.

The AI translation included nonsensical phrases, such as 'great emotional charge and physics on their backs,' instead of recognizing healthcare workers' emotional and physical burdens.

Test on spontaneity using a 2021 speech from Gustavo Petro revealed AI's strength in capturing specific terms but struggled with maintaining flow due to pauses.

AI effectively translated terms like 'president-in-office' and 'special drawing rights' but created disjointed speech segments due to handling pauses poorly.

Human interpreters can analyze speaker's body language and rhythm, which AI currently cannot replicate.

Test on speed using a 2021 speech from Nayib Bukele demonstrated AI's ability to retain content but struggled with syntax, word choice, and grammar.

AI translations often included ambiguous phrases, such as 'new currency issues,' which could confuse listeners.

Professional interpreters process and deliver information in a coherent and culturally appropriate manner, which AI struggles to match.

AI solutions may be suitable for informal situations with low consequences for mistakes but are not reliable for high-stakes contexts like courtrooms or diplomatic negotiations.

Understanding the limitations and potential of AI tools is essential for their constructive use in aiding human tasks.

AI cannot replace human interpreters but can be a valuable tool when used appropriately and with an understanding of its limitations.

Transcripts

- [Narrator] Artificial intelligence

has been threatening everyone's jobs,

including interpreters.

Inaccurate interpretation can be the difference

between a good or bad nuclear deal.

So Wired tested a speech translator called KUDO

versus human interpreters to see which better transmits all

of the necessary elements

for people to understand what they're hearing,

including emotion, intonation, completeness,

spontaneity and speed.

- [KUDO] Liars jobs.

- [Narrator] To test out if AI is really better

than a human.

Let's begin.

Test one.

- Here we go.

- [Narrator] We gave Barry and KUDO a 2020 speech

from King Felipe of Spain.

The speech is about the COVID crisis.

The subjects will focus on emotion.

- Because of the pandemic.

I parachuted into this not knowing anything

about the speech, when it was given.

In this speech, what matters more

is the emotion and the ability to connect

with the audience because he's speaking to the citizens

of Spain at a very difficult time

when many have suffered loss,

are going through a number of challenges.

This reminds us of very deep feelings.

At this time, many of you are fighting against the illness

and what it has done in your homes.

And the words are just the vehicle to get that across.

- [Narrator] Now let's evaluate the speech translator.

- [KUDO] Good evening, I address you on this Christmas Eve

when we are living truly exceptional circumstances

due to the pandemic.

- In terms of content and completeness,

the AI did very well.

- [KUDO] The virus has broken into our lives,

bringing suffering, sadness or fear,

altered our way of life.

- It struggled with word choice in a number of areas.

It referred to an overflow in some hospitals.

- [KUDO] And also overflow in some of our hospitals.

- Now, what that was referring to

is at that time when hospitals were overwhelmed,

when the king was recognizing the healthcare workers

for the work that they had done

and he recognized the tremendous physical

and emotional burden that they were carrying

and the AI translated that

as the great emotional charge and physics on their backs.

If you know Spanish, you can go back and figure out

how it is that the AI determined to use those words.

But obviously, that is borderline nonsensical.

If you look

at what the speech-to-speech translation produced,

it was by and large, quite accurate.

However, it's very difficult to get the sense

of what the king is seeking to portray

through the AI in its current state.

- [Narrator] Test two, spontaneity.

We gave Walter and KUDO a 2021 speech from former senator

and now current Colombian president, Gustavo Petro.

- The most powerful and nonproductive sectors

of this country, 20 billion pesos a year.

So I took the hour beforehand to look him up,

look at some of his videos.

Just listen to the level of language he employs.

Listen to his rhythms.

Listen to, you know, watch his body language.

- [Narrator] Now let's see how the speech translator did.

- [KUDO] You are, Mr. President-in-Office.

[Gustavo speaking in Spanish]

- [KUDO] In the year of 2019, this Congress decided to give

to the most powerful and unproductive sectors

of the country, 20 billion pesos a year.

- The particular strength of artificial intelligence

was in capturing those terms of art.

Artificial intelligence captured president-in-office

or sitting president,

[Walter speaking in Spanish]

minister of finance,

[Walter speaking in Spanish]

special drawing rights.

That's a consecrated term.

It's a term that's enshrined

that you can find in any number of glossaries.

That's the official translation, and I missed that.

Official caucuses as well

that the artificial translation rendered that very clearly

and it pulls those in and did so very well.

That's what I felt it was most effective at.

And where I felt it had the most shortcomings

was because of the pauses.

- [KUDO] When El Pais has a stagnant economy

and an impoverished middle class

what this tax reform brings.

[Gustavo speaking in Spanish]

- [KUDO] It is to increase taxes

on the productive company,

small and medium fundamentally.

[Gustavo speaking in Spanish]

- [KUDO] An absolute contradiction and absurdity.

- The artificial intelligence creates these little islands,

these little bubbles that read like complete thoughts

but are not because there's a flow.

There's a flow that's coming after the pause.

And thanks to an absolutely aggressive social policy,

making a gift to the bankers.

- [KUDO] And thanks to an absolutely

aggressive social policy,

how to issue money to give to bankers.

- So the pauses in this particular speech seem

to throw the artificial intelligence off.

You have the vocal inflection.

Increasing taxes on those who govern Colombia.

That you know allows you to understand

there's something else coming.

I'm not finished yet.

Unproductive capital, those who don't work

but prey on the state, on the people and on nature.

Thank you, sir.

- [Narrator] Test three.

We gave Barry and KUDO a 2021 speech

from El Salvadorian President Nayib Bukele.

The speech is highly crafted, prepared

and read from a teleprompter at a high rate of speed.

- We are going to make use of BANDESAL.

I will tell you that it was a very challenging experience

for me as a professional interpreter

to interpret that speech accurately and completely.

What you have to do is process that information

and extract the ideas to be able

to give your listener something that's usable

for them to be able to follow from that huge amount

or that torrent of words that's coming at you.

- [Narrator] Now let's evaluate the speech translator.

- Salvadoran people, good morning.

As you all know,

the world has suffered an increase in prices.

- When I was able to go back and listen

to the AI interpretation, what I noticed

throughout the entire speech

is that it did not miss any of the content.

- [KUDO] The cost of these goods increases

to match the supply of new dollars,

new euros, new yen, and therefore,

that affects world prices.

[Nayib speaking in Spanish]

- [KUDO] That is why we are experiencing a global inflation

that hits the pockets of everything, of all the people

in the world.

And El Salvador is no exception.

- I was taken aback

by the artificial intelligence's ability

to get all of the content and hold onto it.

So in terms of completeness, it was very good and better

than what I was able to do at that rate of speed.

However, it often had mistakes and challenges

with making the syntax, the word choice,

and sometimes the grammar in English,

and sometimes it would choose words

that weren't really the most accurate way,

the way a native English speaker would say something.

It talked about new currency issues.

- [KUDO] This inflation generated by the incorporation

of new currency issues, usually dollars, euros, yen.

- Which was really ambiguous

because it was talking about new infusions

of cash or currency into the monetary system.

And people could confuse issues

with problems and think, okay,

what's the problem with these new currencies?

What is happening is that the process

that the professional interpreter has been doing

for as long as professional interpreters have been working

is to process that information

and then serve it up to the listener already processed

to know this is what's being talked about.

So rather than saying new currency issues,

I would've said infusions or injections

of more cash into the monetary system.

- [Narrator] So for a profession that started

with the Nuremberg trials and continues to be used

for important meetings and negotiations today,

when is it appropriate to use an AI solution?

- I would say in informal situations, situations

where there is not a high level of consequence for mistakes.

I would not trust this in courtrooms.

I would not trust this in the legal system.

- If there is a decision that has to be made,

a medical interaction, asylum cases.

- I don't know that I would trust it

for situations that require metaphorical language,

that don't catch the figurative meaning.

- Do I want to have a high-level diplomat

or a president of my country speaking

to another president through AI to make important decisions

about the future of literally millions of lives?

No. - We need to look at it.

We need to examine it.

We need to find out how we can use this constructively

and how we can use it carefully to aid us and not be afraid

of it, because that's no use if we're not flexible.

We're eminently in a world where we need to be flexible

and we need to be open

to new ideas and new ways of doing things.

- It is not a replacement, and you have to understand that.

And if we're able to understand the power of the tool,

we can respect the tool and use it to everyone's benefit.