Building AI Apps in Python with Ollama

Matt introduces viewers to developing applications with Ollama, a Python-based AI tool. He outlines the two main components of Ollama: the client, which is used for interactive command-line operations, and the service, which runs in the background and publishes the API. Matt explains the REST API endpoints, focusing on generating completions through the 'chat' and 'generate' endpoints, and the importance of understanding the underlying API before using the Python library. He demonstrates how to use the Python library to interact with Ollama, including non-streaming and streaming responses, handling images, and managing context in conversations. The video concludes with an example of connecting to a remote Ollama server and using the chat endpoint for more complex interactions. Matt invites viewers to join the Ollama community on Discord for further support.

Takeaways

• 🚀 **Introduction to Ollama**: Matt provides an introduction to developing applications with Ollama using Python, assuming prior knowledge of Ollama.
• 🔌 **API Access**: Ollama consists of a client and a service, with the service running in the background and publishing the API.
• 📚 **Documentation**: API endpoints are documented in the GitHub repo under `docs` and then `api.md`.
• 🤖 **API Capabilities**: The API allows for generating completions, managing models, pushing/pulling models, and generating embeddings.
• 🗣️ **Chat vs Generate Endpoints**: The `chat` endpoint is suitable for conversations with context management, while `generate` is for one-off requests.
• 🌐 **API Endpoint Usage**: The `generate` endpoint is used for model-specific questions and can accept images in base64 format.
• 📈 **Streaming API**: Responses are streamed as JSON blobs, which include tokens, model information, and context for continuation.
• ⏱️ **Keep Alive**: The `keep_alive` parameter determines how long a model stays in memory, with a default of 5 minutes.
• 📏 **Python Library**: The Ollama Python library simplifies working with the API, especially with streaming and non-streaming responses.
• 🔄 **Context Management**: In Python, context from one API call can be fed into the next to maintain conversation state.
• 🖼️ **Image Processing**: For multimodal models, the Python library expects images as byte objects, not base64 encoded strings.
• 🔗 **Remote Hosts**: Ollama can be set up on a remote server, and the Python library can be pointed to interact with the remote Ollama instance.

Q & A

What are the two main components of Ollama?

The two main components of Ollama are the client and the service. The client runs when you execute 'ollama run llama2' and is the REPL you interact with. The service is what 'ollama serve' starts up and typically runs in the background as a service, publishing the API.

How can I find the REST API endpoints for Ollama?

You can find the REST API endpoints for Ollama in the GitHub repository under the 'docs' folder, specifically in the 'api.md' file.

What is the difference between the 'generate' and 'chat' endpoints in Ollama's API?

Both endpoints can generate a completion, but the 'generate' endpoint is used for one-off requests where you ask a question and get an answer without maintaining a conversation. The 'chat' endpoint is more suitable for interactive conversations with the model, where managing memory and context is important.

What is the role of the 'model' parameter in the 'generate' endpoint?

The 'model' parameter in the 'generate' endpoint specifies the name of the model you want to load. If the model is already loaded and you call 'generate' with just the model name, the unload timeout will be reset to another 5 minutes.

How can I provide an image to the Ollama model using the REST API?

When working with a multimodal model like Ollama, you can use the 'images' parameter to provide an array of base64 encoded images. The model can only handle base64 encoded images, so you must perform this conversion yourself.

What does the 'stream' parameter do in the Ollama API?

The 'stream' parameter in the Ollama API controls whether the response is returned as a stream of JSON blobs or as a single value after the generation is complete. If 'stream' is set to false, you will have to wait until all tokens are generated before receiving the response.

How does the 'format' parameter affect the response in the Ollama API?

The 'format' parameter allows you to specify the format of the response, which can only be 'JSON'. Using 'format json' also implies that you should indicate in the prompt that you expect a JSON response and ideally provide an example of the schema to avoid inconsistent outputs.

What is the purpose of the 'keep_alive' parameter in the Ollama API?

The 'keep_alive' parameter determines how long the model should stay in memory after a request. The default is 5 minutes, but you can set it to any duration you like, or use -1 to keep the model in memory indefinitely.

How does the Python library simplify the use of streaming in Ollama?

The Python library simplifies the use of streaming by allowing function calls to return a single object when not streaming or a Python Generator when they are streaming. This makes it easier to switch between streaming and non-streaming modes without changing the underlying API.

What is the process to use Ollama with a remote server?

To use Ollama with a remote server, you need to set up the server with Ollama, configure it to be accessible (e.g., setting Ollama_host environment variable to 0.0.0.0 and restarting Ollama), and then in your local code, create a new Ollama client pointing to the remote host's address.

How can I contribute to the Ollama community or ask for help?

You can contribute to the Ollama community or ask for help by joining their Discord server at discord.gg/ollama. You can also provide feedback or ask questions in the comments section of their documentation or video tutorials.

Outlines

00:00

🚀 Introduction to Ollama and API Access

Matt introduces the video's purpose, which is to guide viewers on developing applications with Ollama using Python. He assumes viewers are already familiar with Ollama and its basic operations. The video focuses on accessing the Ollama API, which has two main components: the client and the service. The client is used for interactive sessions, while the service runs in the background and publishes the API. The API offers various functionalities, including generating completions, managing models, and creating embeddings. Two endpoints, 'chat' and 'generate', are highlighted for generating completions, with a choice between them depending on whether the use case involves a conversation or not. The 'generate' endpoint is detailed, including its parameters and the structure of its response.

05:06

📚 Understanding API Parameters and Python Library Usage

The paragraph delves into the specifics of the 'generate' endpoint's parameters, such as 'model', 'prompt', 'images', and 'stream'. It explains the importance of the 'context' in continuing conversations with the model and how to use it in subsequent API calls. The paragraph also discusses additional parameters like 'options', 'system', 'template', 'raw', and 'keep_alive'. It then transitions to the 'chat' endpoint, which is similar to 'generate' but uses 'messages' instead of individual parameters. The paragraph concludes with an introduction to the Ollama Python library, which simplifies the process of switching between streaming and non-streaming responses. Practical examples of using the library are provided, including generating responses, handling context, and describing images using the Python module.

10:07

🌐 Remote Ollama Setup and Advanced Usage

This paragraph showcases how to use Ollama with a remote server. Matt demonstrates setting up a remote Ollama API on a Linux box using tools like tailscale for network configuration. He details the process of changing the Ollama_host environment variable and restarting the Ollama service to point to the remote host. The paragraph also includes an example of how to modify the local code to interact with the remote Ollama instance. Finally, Matt encourages viewers to explore the provided code repository for more examples and to reach out for clarification or to join the Ollama community on Discord.

Keywords

Ollama

Ollama is the main subject of the video, which is a system or platform for developing applications using Python. It is assumed that the viewer has a basic understanding of it. In the context of the video, Ollama is used to access an API for various operations like generating completions, managing models, and handling multimodal inputs.

API

API, or Application Programming Interface, is a set of rules and protocols that allows different software applications to communicate with each other. In the video, the focus is on how to access and use the Ollama API, which has two main components: a client and a service, each serving different functionalities.

Client

In the context of the Ollama system, the client is the component that runs when the command 'ollama run llama2' is executed. It provides a Read-Eval-Print Loop (REPL) for interactive use, allowing developers to work with the system directly.

Service

The service component in Ollama is what is started with the 'ollama serve' command. Unlike the client, the service operates in the background as a daemon, publishing the API endpoints that other applications can interact with.

REPL

REPL stands for Read-Eval-Print Loop, which is an interactive programming environment where users can type in commands and immediately see their results. In the video, the client provides a REPL for working with Ollama.

Endpoints

Endpoints in the context of the Ollama API are specific URLs that provide particular services or functionalities. The video discusses various endpoints like 'generate' and 'chat', each serving different purposes such as generating completions or managing conversations with the model.

Streaming API

A streaming API is a type of API that allows data to be transmitted in a continuous flow, rather than in a single response. In the video, it is mentioned that most of the endpoints in the Ollama API respond as a streaming API, sending a 'stream' of JSON blobs with each piece of the response.

Multimodal Model

A multimodal model is a type of machine learning model that can process and understand multiple types of data, such as text and images. In the video, the use of the 'images' parameter in the API is discussed, which is relevant when working with a multimodal model like Llava.

Python Library

The Python library mentioned in the video is a set of Python modules that simplify the interaction with the Ollama API. It allows for easier switching between streaming and non-streaming responses and provides a more Pythonic way to work with the Ollama system.

Context

In the context of the Ollama API, context refers to the information that is remembered from one interaction to another, which is particularly useful for maintaining a conversation with the model. The video explains how to use the context from one 'generate' call to influence the next.

Keep Alive

The 'keep alive' parameter in the Ollama API determines how long a model should stay in memory after its last use. The default is 5 minutes, but it can be set to any custom duration or kept in memory indefinitely with a value of -1. This is important for managing the lifecycle of models in the system.

Highlights

Matt introduces the development of applications with Ollama using Python.

Assumption that the audience knows what Ollama is and how to work with it.

Introduction to Ollama available for those who need to get up to speed.

Explanation of how to access the Ollama API with two main components: client and service.

The client is the REPL that runs with 'ollama run llama2'.

The service is what 'ollama serve' starts up and runs in the background.

The service publishes the API with REST API endpoints documented on GitHub.

Different functionalities of the API, including model management and embeddings generation.

Two endpoints for generating completions: 'chat' and 'generate', with different use cases.

The 'generate' endpoint is suitable for one-off requests without conversation.

The 'chat' endpoint is more convenient for managing memory and context in a conversation.

Parameters for the 'generate' endpoint, including 'model', 'prompt', and 'images'.

The response is a stream of JSON blobs, each with a token and other information.

Option to disable streaming and receive a single value after generation is complete.

The 'format' parameter allows for specifying the output format, with JSON being an option.

The Python library simplifies the switch between streaming and non-streaming responses.

Using the Python library with `pip install ollama` to interact with Ollama.

Examples provided in the Python library for using the 'generate' and 'chat' endpoints.

Demonstration of describing an image using the Python library with a bytes object instead of a base64 encoded string.

Setting up a remote Ollama server and connecting to it from a local machine.

Invitation to join the Ollama community on Discord for further discussions.