Talk to AI: Calling LLMs Directly from Your Phone with Twilio

Summary

In this tutorial, the process of building a phone assistant using Twilio, OpenAI, and Hono in TypeScript is demonstrated. The assistant receives calls, processes speech-to-text with Twilio, and generates responses via OpenAI's GPT-3.5. The system maintains conversation context using cookies to store and manage message history. It starts by setting up a server, handling incoming calls, gathering user speech, and responding with AI-generated text. With features like reservation assistance, the solution illustrates how to build a dynamic, stateful voice assistant that can engage users in a variety of contexts.

Takeaways

• 😀 The script demonstrates how to build a phone assistant using OpenAI's GPT and Twilio API.
• 😀 The server is built using TypeScript and the Hono library, which is set up to handle incoming requests.
• 😀 Twilio's API is used to handle incoming phone calls, where each call triggers a POST request to the server.
• 😀 The server responds to incoming calls with a basic XML response to interact with the user via voice.
• 😀 The `gather` command in Twilio is used to collect user speech, with features like speech timeout and model selection.
• 😀 The script processes the user's speech by sending it to OpenAI’s GPT model to generate a response.
• 😀 The conversation state is managed using cookies, allowing the assistant to remember prior interactions and maintain context.
• 😀 An error is handled when the user’s speech is not available, and the assistant defaults to repeating what the user said.
• 😀 The assistant responds with messages generated by the GPT model, tailored to the conversation, such as asking questions about operating hours or reservations.
• 😀 After each response, the system redirects the conversation back to the `/incoming-call` endpoint, allowing ongoing dialogue with the user.
• 😀 The script can be extended to integrate more advanced features like tools for managing reservations, connecting with calendars, or making calls to external systems.

Q & A

What is the purpose of the phone assistant being built in the script?

The phone assistant is being built to handle incoming calls using a large language model and the Twilio API. It is designed to interact with users via voice, gather input, and respond using AI-generated text, acting as a virtual assistant.

How is the server for the phone assistant set up?

The server is set up using TypeScript with the Hono library, an HTTP server framework. It is then tunneled to the internet using ngrok, allowing the phone system to communicate with the server over the web.

What role does Twilio play in this project?

Twilio provides the API for handling phone calls. When a user calls the phone number, Twilio triggers an HTTP POST request to the server, allowing it to process the call and respond with voice interactions.

How does the server respond to incoming calls from Twilio?

The server responds to incoming calls by sending an XML response that Twilio can understand, which includes commands like 'hello, how are you?' to engage the user.

What does the 'gather' command do in this context?

The 'gather' command in Twilio is used to listen for user input. It specifies parameters like the timeout duration for speech and the speech-to-text model to use, enabling the system to collect and process voice data from the user.

How is OpenAI integrated into the phone assistant system?

OpenAI is integrated by calling the OpenAI API when the user speaks. The user's speech input is passed to the GPT-3.5 Turbo model for text generation, and the AI responds with appropriate answers based on the conversation context.

Why is the conversation history stored in a cookie?

The conversation history is stored in a cookie to maintain state across multiple API calls. Since the system doesn't inherently maintain session data, using a cookie allows the assistant to remember past interactions and provide contextually relevant responses.

What kind of initial message is set when a new conversation begins?

When a new conversation begins, the initial message is a system prompt that defines the assistant's role. For example, the assistant may be set as a helpful phone assistant for a pizza restaurant, with details like business hours and reservation capabilities.

What does the '/respond' API endpoint do?

The '/respond' API endpoint processes the user's speech input, sends it to OpenAI for a response, and returns the generated message. This endpoint ensures the conversation progresses based on the AI's response, which is then relayed back to the user via Twilio.

How does the system handle errors or undefined values in the conversation flow?

The system handles errors or undefined values by using quick fixes like adding fallback solutions in the code. For example, an exclamation mark is added when certain values are undefined, ensuring the system still runs smoothly during tests.