Part 8/8: ML Based Web App Firewall : Testing the IPS in Real Time

Summary

In this informative video, Devashesh demonstrates how to deploy and test a machine learning model for a Web Application Intrusion Prevention System (IPS) using the Pikered library. The model is integrated with a proxy server to intercept HTTP requests in real-time, analyzing them to determine if they are malicious. Viewers witness a live test using Firefox, where the model successfully detects SQL injection attacks, highlighting the real-time capabilities of the IPS. Devashesh acknowledges the need for further refinement in feature extraction and promises ongoing improvements to enhance the model's precision and accuracy.

Takeaways

• 😀 The video is a tutorial by Devashesh on deploying a machine learning model using the Piker server library.
• 🛡️ The model being discussed is an Intrusion Prevention System (IPS) designed to detect malicious HTTP requests in real-time.
• 💡 The process involves creating a proxy server that integrates with the machine learning model to intercept and analyze HTTP requests.
• 🔍 The model extracts features from the HTTP requests to determine if they are 'good' or 'bad' in nature.
• 📈 The video demonstrates using a Jupyter notebook to set up the environment and apply a K-means clustering model with two clusters.
• 📚 It references a previous dataset saved in 'data.csv' for training the model.
• 🌐 The testing is done using a Firefox web browser configured to send all requests through the proxy server.
• 🔬 The model is tested against a dummy website, 'demo.testfire.net', which is a known vulnerable web application.
• 🚀 The video shows real-time feature extraction and model execution on HTTP requests sent by the browser.
• 🛑 The model successfully identifies some SQL injection payloads as malicious, printing 'intrusion detected'.
• 🔄 The presenter acknowledges the need for further work on feature extraction to improve the IPS's overall quality and accuracy.

Q & A

What is the main topic of the video?

The main topic of the video is deploying and testing a machine learning model for a web application intrusion prevention system (IPS) in real-time using a proxy server.

What library was mentioned for deploying the model in the last video?

The library mentioned for deploying the model in the last video is 'pikered'.

What is the purpose of the proxy server in this context?

The purpose of the proxy server in this context is to intercept HTTP requests and integrate with the machine learning model to determine whether the requests are good or bad in nature.

What tool is the presenter using to demonstrate the real-time feature extraction from HTTP requests?

The presenter is using a Jupyter notebook to demonstrate the real-time feature extraction from HTTP requests.

What is the method used for training the model in the script?

The method used for training the model is K-means clustering, with the number of clusters set to 2.

What is the data source for training the model mentioned in the script?

The data source for training the model is a dataset saved in 'data.csv'.

How is the Firefox web browser configured in the demonstration?

The Firefox web browser is configured to send all requests through the proxy server created in the Jupyter notebook.

What website is used for testing the IPS in the video?

The website used for testing the IPS is 'demo.testfire.net', a known vulnerable web application.

What type of payloads are used to test the IPS for detecting bad requests?

SQL injection payloads taken from the internet are used to test the IPS for detecting bad requests.

What is the presenter's plan for improving the IPS after the demonstration?

The presenter plans to continue working on the feature extraction from the training data and tuning the clustering model to make it more precise and accurate.

How does the presenter conclude the video?

The presenter concludes the video by asking viewers to stay subscribed for updates on the IPS development and improvement.

Outlines

00:00

🛡️ Deploying and Testing an IPS with Pikered Library

In this segment, Devashesh introduces the process of deploying a machine learning model using the Pikered library, which he demonstrated in a previous video. The focus now shifts to testing the Intrusion Prevention System (IPS) in real-time. A proxy server is set up to integrate with the machine learning model, which will intercept HTTP requests to determine if they are benign or malicious. The video demonstrates the use of a Jupyter notebook to extract features from HTTP requests in real-time and apply a K-means clustering model with two clusters to classify the requests. The setup includes configuring a Firefox web browser to send all requests through the proxy server for testing against a dummy website known for vulnerabilities.

05:02

🔍 Real-time Feature Extraction and IPS Testing

This paragraph delves into the real-time feature extraction from HTTP requests sent by a browser, as demonstrated in the video. Initially, only benign requests are sent to gather data points. Subsequently, the presenter tests the IPS by sending SQL injection payloads, which are malicious requests, to observe the system's response. The IPS successfully identifies some of the payloads as intrusions, printing 'intrusion detected', while others are missed. The presenter acknowledges the need for further improvement in feature extraction and tuning the clustering model for better precision and accuracy. The video concludes with a call to action for viewers to subscribe for more content on the channel and an overview of the ongoing development and improvement of the IPS.

Keywords

💡Deploy

To 'deploy' in the context of software or models refers to the process of making them available for use in a live environment. In the video, the presenter discusses deploying a machine learning model using the 'pikered' library, which is a simple process that requires just a few lines of code. This is a key step in transitioning a model from development to practical application.

💡Pikerved

Pikerved is a library mentioned in the video that facilitates the deployment of machine learning models. It is implied to be a user-friendly tool that simplifies the process of making a model operational in a production environment. The script does not provide further details about pikerved, but its mention suggests its importance in model deployment.

💡Proxy Server

A 'proxy server' is an intermediary server that sits between a client and a destination server, handling requests and forwarding responses. In the video, the presenter creates a proxy server to integrate with a machine learning model, which intercepts HTTP requests to check their nature—whether they are 'good' or 'bad'. This is central to the video's demonstration of a real-time intrusion detection system.

💡HTTP Request

An 'HTTP request' is a message sent from a client to a server to request access to a resource. In the context of the video, the presenter discusses extracting features from HTTP requests in real-time to feed into a machine learning model for analysis. The script mentions that the proxy server checks the nature of these requests, which is integral to the functioning of the intrusion prevention system (IPS).

💡Feature Extraction

Feature extraction is the process of pulling out the characteristics or features from raw data that are relevant for a particular task, such as machine learning. In the video, the presenter mentions a function responsible for extracting features from HTTP requests in real-time, which is a crucial step for the machine learning model to analyze and classify the requests.

💡Machine Learning Model

A 'machine learning model' is a system that uses algorithms to learn from data and make predictions or decisions without being explicitly programmed. The video script describes using a model to analyze HTTP requests and determine if they are potentially harmful. The model's accuracy and efficiency are central to the effectiveness of the IPS being demonstrated.

💡K-Means Clustering

K-Means clustering is a type of unsupervised machine learning algorithm used for grouping similar data points into clusters. The video script mentions applying a K-Means clustering model to classify the nature of HTTP requests. The number of clusters is set to 2, presumably to differentiate between 'good' and 'bad' requests.

💡Intrusion Prevention System (IPS)

An 'Intrusion Prevention System' is a cybersecurity mechanism aimed at detecting and preventing malicious activities. In the video, the presenter demonstrates an IPS that uses a machine learning model to analyze HTTP requests in real-time and identify potential threats, such as SQL injection payloads.

💡SQL Injection

SQL injection is a type of cyber attack where an attacker inserts malicious SQL code into a web application's input fields to manipulate the database. The video script includes examples where the presenter tests the IPS by sending SQL injection payloads to see if the system can detect them as 'bad' requests.

💡Firefox Web Browser

The 'Firefox web browser' is a popular internet browser used for accessing the web. In the video, the presenter configures Firefox to send all requests through the proxy server created in the notebook. This setup is used to test the IPS by observing how the machine learning model processes and evaluates the requests.

💡Real-Time

The term 'real-time' refers to the processing or analysis of data as it is being collected or received, without significant delay. The video script emphasizes the real-time capabilities of the IPS, showcasing how the machine learning model can instantly analyze incoming HTTP requests and detect intrusions.

Highlights

Introduction to deploying machine learning models using the Piker library, emphasizing its simplicity with just a few lines of code.

Explanation of testing the IPS in real-time with a proxy server integrated with a machine learning model.

Description of the proxy server's role in intercepting HTTP requests to check their nature.

Use of Jupiter notebook for step-by-step explanation of the process.

Feature extraction from HTTP requests in real-time for analysis by the IPS.

Training the model using a dataset from a previous video, setting up the environment for the model.

Application of a K-means clustering model with a specified number of clusters.

Creation of a simple proxy server in Python for testing the IPS.

Testing the IPS against a dummy website known for vulnerabilities.

Configuration of Firefox to send requests through the proxy server for testing.

Real-time demonstration of the IPS detecting bad HTTP requests using a machine learning model.

Use of SQL injection payloads to test the IPS's ability to detect malicious requests.

Demonstration of the IPS successfully identifying a bad request with an intrusion detected message.

Testing additional SQL injection payloads to evaluate the IPS's detection capabilities.

Acknowledgment of the need for further work on feature extraction to improve the IPS's quality.

Commitment to continue working on and improving the IPS, with updates to be shared.

Encouragement for viewers to stay subscribed for more content on the channel.

Transcripts

[Music]

hello everyone my name is devashesh and

i welcome you all to this video

so in the last video we have seen how

can we deploy our model using pikered

library

and we have seen it how easy it is to do

that

it literally took two to three lines to

deploy our created model using bikerate

now it is time to test our ips

in the real time so this is exactly what

we are going to do here

you must be pretty familiar with this

particular this particular diagram

uh we'll be creating a proxy server and

the proxy server will be integrated with

the machine learning model that we have

created in the last few videos

and whenever we are actually sending any

request to our server

our proxy is going to intercept the http

request

and it is going to check whether this

uh the request http request is good in

nature or bad in nature

and so let's do it

as usual i'm going to use the jupiter

notebook

and i have already actually created this

notebook so i'm going to you know

go ahead and explain to you line by line

or step by step

so basically since we have to extract

the features from the http request

in real time uh this code actually you

know responsible for doing that

so we are going to extract some of the

features from the http request that

our ips receives in real time so this

function is responsible for that

so we'll just go ahead and do that and

after that

we are what we are doing this this is

where we are actually our

we are actually training our model

so basically we are reading the old data

set that we have seen in our previous

video

uh which is saved in you know all

data.csv

and we are reading that data we are

actually you know setting up the

environment for the model and we are

actually

applying k means clustering model and

this is actually the k

means object and the number of cluster

we want to create

is what we have seen before is 2 so

let's

execute it

now it is created now what we have to do

we have to actually you know

create a simple proxy server uh using

python

that we have already created and this is

the class responsible for you know

creating this proxy server uh so let me

quickly explain

to you guys you know how we are going to

test it so ah this is the this is

firefox web browser and

we are going to test our ips against a

dummy website

demo.testfire.net as you can see it is a

known vulnerable web application

we can actually search several stuffs in

here

and so now we have to configure our

firefox to you know

send all the request through the proxy

server

that we have we are going to you know

create

in this notebook so let's go ahead and

do that

manual proxy one two seven zero zero one

uh and eight zero zero okay

so now let's open our jupiter notebook

and

let's execute it

it is actually now

it is not showing this interrupt

yeah as you can see it is it is

listening to

the local interface 8080 so now whatever

request we are going to send

it has to uh go through this proxy

server and it the machine learning model

that we have developed is going to apply

on the

all the request that we have uh you know

we are going to send

uh so let's you know split the screen in

two parts so that we can see that in

real time

and this is our so let's

so as you can see our ips or the proxy

is listening to port 8080 and we have

configured firefox to send all the http

requests through

this 8.0 port so whenever firefox is

going to send in a request

ah through 8080 our machine learning

model

is going to you know execute on the http

request

and we are going to find out whether the

request

is it is bad in nature or not

so that's the plan so if you just you

know

it's going to you know show us some of

the inputs here so if we just press

5.

six it is going to actually it is on the

real time it is extracting all the

features

uh from the http request that the

browser is sending and it is actually

printing it here

uh so now actually we are sending all

the

good requests so that is why you know

that is the you know

that is the data point that we are

getting for this request so now

let's uh send some you know bad

bad request to the server so we are

going to use some

sql injection payloads

taken from the internet and we'll see if

it is able to detect any of this

so let's use this one

if we just paste it here

it it has missed it

so let's use this one

as you can see it has printed that

intrusion is detected

so it is successfully our machine

learning model is successfully able to

uh identify that the request that we are

sending is actually bad in nature so

let's try out some other

you know other sql injection payload and

see if it is able to catch

it was actually missed

let's see if it is able to catch this

one or not

it's also missed

so let's take this one

it is able to catch that you know uh

this

request that our browser is sending is

bad in nature and it has

printed intrusion detected uh so as i

have

you know said in my previous video i

need to work more on

you know this feature extraction from

the

the training data uh i have not much

spent much time to do that actually so

obviously the

this uh overall quality of this ips is

going to improve

so before i do that actually i just

wanted to show you

uh how real time our ips web application

intrusion prevention system works uh so

i'll continue to work on that

and probably you know keep you guys

posted how am you know

trying to improve this uh ips that you

know i have developed

and how i am actually tuning this

clustering model

to kind of you know make it more precise

and accurate

so i hope you have enjoyed this video uh

so if you enjoy the kind of content i

upload on this channel i'd request you

to

uh stay subscribed to this channel so

that's all i wanted to discuss in

today's video i'll see in the next video

bye

[Music]

you