Cloud Scheduler For AI Models With Golang and Remix?

Summary

In this video, the creator shares their exciting new project: a platform to schedule and run AI models on cloud GPUs. Using GoLang for the backend with PocketBase and Remix for the frontend, the platform automatically selects the most cost-effective cloud providers based on GPU requirements. The system simplifies AI model deployment by handling scheduling, job management, and callback responses. Users can seamlessly integrate various models like image generation or text-to-speech into their applications. The project is aimed at AI developers, and the creator invites viewers to join the community, collaborate, and follow along with the development.

Takeaways

• 😀 The speaker is developing a platform that allows users to schedule AI models in the cloud and run them on the most cost-efficient GPUs.
• 😀 The platform will automate model scheduling and selection based on factors like GPU availability and memory requirements.
• 😀 The project combines a Go backend (using Goang and PocketBase) and a React Remix frontend for the application’s structure.
• 😀 The platform will support various AI models, including image generation, text-to-speech, and more, integrated via API calls.
• 😀 The speaker is aiming for an interface similar to Hugging Face and Replicate AI, where users can run models and access results via APIs.
• 😀 Scheduling and job management will be handled by the backend, with the platform determining the best provider (e.g., Runpod) based on the model's needs.
• 😀 Providers will be selected based on the best prices, availability, and GPU memory requirements for the models being run.
• 😀 The system will include job tracking, with status updates (e.g., completed, failed) and callback URLs to notify users of results.
• 😀 Images and other results will be hosted on Cloudflare R2 or similar storage, with URLs returned to the users for further use.
• 😀 The speaker is encouraging the community to join the development process via Discord and to follow the project’s progress through YouTube devlogs and streams.

Q & A

What is the main idea behind the platform being developed in the video?

The platform aims to schedule and run AI models in the cloud on specific GPU providers. It will optimize for cost by selecting the most efficient and affordable GPUs available for each model request.

Why is GoLang chosen for the backend of this platform?

GoLang (Go) is chosen for its efficiency in handling complex backend operations, such as managing user APIs, job scheduling, and ensuring high performance, especially in a system that involves many API calls and processing tasks.

What are the challenges mentioned regarding GPU availability?

GPUs are hard to find, especially given high demand and pricing challenges. The platform aims to address this by checking GPU availability from different providers and ensuring that the most suitable GPUs are selected based on the model's requirements.

What is the role of Remix in the platform's frontend?

Remix is chosen as the frontend framework for the platform due to its simplicity and compatibility with React. It provides an efficient way to build and manage the user interface of the platform.

How does the system determine which GPU provider to use for running a model?

The system checks multiple GPU providers for availability, compares their pricing, and selects the one that best fits the model's specific requirements in terms of GPU power, memory, and pricing.

What is the process for running a model on the platform?

A user makes a request via the frontend, which triggers the backend to create a job. The system then selects an appropriate GPU provider, runs the model, and sends the results back to the user through a callback URL, where the user can retrieve the generated output.

What types of AI models does the platform support?

The platform is designed to support a wide range of AI models, including text-to-image, image-to-text, image-to-video, and text-to-speech models, among others. Users can select models to run via a playground interface.

What is the purpose of the callback URL in the system?

The callback URL allows the platform to notify the user once the job is complete, updating the job status (e.g., completed, failed), and providing a direct link to the generated content, such as an image hosted on Cloudflare R2 or a similar cloud storage solution.

How is the platform planning to handle cloud storage for generated content?

The platform will use Cloudflare R2 (or similar cloud storage services like S3) to host the generated content, ensuring that users can easily retrieve their results through a URL after the job is processed.

What makes this project different from existing platforms like Hugging Face and Replicate AI?

While similar to Hugging Face and Replicate AI, the platform aims to add unique features, such as optimizing GPU selection for cost-efficiency, automating the scheduling process, and focusing on ease of use through the combination of GoLang backend and Remix frontend. It also aims to provide better collaboration features and may be open-sourced.