Realtime Search Using n8n, BrightData and Google AI Search

Summary

This video demonstrates a hackathon project: a Code Researcher application that leverages NAT and Bright Data to search the internet for medical and pharmaceutical research topics. The system uses an LLM to generate summarized research reports by gathering clinical trial data and literature. Built with Next.js and Airtable, the application validates research questions, triggers automated workflows, and integrates multiple Bright Data agents for comprehensive searches. The video walks through the workflow, the AI agents, and the architecture, showing how research queries are processed, results are compiled, and the final report is generated, offering an efficient, AI-powered research solution.

Takeaways

• 😀 The project is a Code Researcher application built using N8N and Bright Data for a hackathon.
• 😀 The application searches the internet for research topics and uses an LLM to generate summarized reports.
• 😀 There are two N8N workflows: a main workflow and a sub-workflow acting as an AI agent.
• 😀 Bright Data is used in three different AI tools to gather data from Google AI search.
• 😀 Users submit research questions, which are validated to ensure they are medical or pharmaceutical topics.
• 😀 The workflow retrieves clinical trial and literature data, optionally gathering additional information if needed.
• 😀 Airtable is used to store research questions, IDs, status, and results.
• 😀 The system monitors Bright Data snapshot status and waits until the data is ready before generating the summary.
• 😀 Two separate system prompts are used: one for clinical trials and one for literature research.
• 😀 The frontend is built with Next.js, and the repository contains both the Next.js code and JSON exports of the N8N workflows.
• 😀 The AI-generated report combines multiple sources and presents a summarized research output for the user.
• 😀 Optional searches for additional safety data are incorporated, demonstrating flexibility in data retrieval.

Q & A

What is the main purpose of the project demonstrated in the video?

The project is a code researcher application designed to search the internet for a given research topic, gather relevant information, and use a large language model (LLM) to generate a summarized research report.

Which tools and technologies were used in building this application?

The application uses NAT, Bright Data, Next.js for the front end, Airtable to store questions and results, and LLMs for processing and summarizing research.

How does the application handle new research questions submitted by users?

When a new research question is submitted, a new entry is created in Airtable with the research description, ID, and status set to 'processing.' A webhook is then triggered to start the workflow for gathering research data.

What types of research are allowed in this application?

The application restricts research questions to medical and pharmaceutical topics. If a submitted question is outside these domains, it is marked as invalid and the user is notified.

How does the application gather information from the internet?

The application uses Bright Data's API to perform a Google AI search and optionally gathers additional data. Two agents are responsible for retrieving clinical trial data and literature information, respectively.

What is the role of Bright Data in the workflow?

Bright Data is used as an AI tool to perform web scraping and Google AI searches, creating snapshots of relevant information. The workflow monitors the status of these snapshots and processes them once they are ready.

How are the search results processed and summarized?

The application uses LLMs to parse the gathered data. System prompts are used to differentiate between clinical trial data and literature information, and the outputs are combined into a summarized research report.

What happens if additional information is needed during the workflow?

An optional agent is available to gather more data if required. The AI decides whether it needs more information or if the existing summary is sufficient.

How does the application ensure that invalid research questions are handled?

The submitted question is validated by checking if it belongs to the allowed medical or pharmaceutical domain. If it fails validation, the Airtable entry is updated to mark the question as invalid, preventing further processing.

What is the final output of the application once the workflow is complete?

Once the workflow completes, the summarized research report is updated in the Airtable results column and can be accessed via the application's front-end interface.

Can you explain the overall architecture of the application?

The architecture includes a Next.js front end, Airtable database to store research questions and results, Bright Data API for web scraping and Google AI searches, and LLMs for processing and summarizing the gathered data. Webhooks trigger workflows and agents manage specific data retrieval tasks.