How to Get Google To Index Your Site

Summary

This video offers a comprehensive guide for content creators struggling with low traffic due to unindexed articles. It explains the phases of Google indexing, the importance of sitemaps, and how to use Rank Math, a WordPress SEO plugin, to connect with Google services and submit sitemaps efficiently. The video also covers methods to check indexing status, troubleshoot common issues, and provides bonus tips for improving indexing chances, including creating unique content and avoiding duplicate pages.

Takeaways

• 😀 Content creators often face the frustration of their articles not getting indexed by Google, leading to no traffic.
• 🔍 Google's indexing process involves two main phases: crawling and indexing, where Google finds and processes pages to understand their content.
• 📈 To expedite indexing, use tools like Rank Math, a WordPress SEO plugin, which simplifies the process of connecting your site to Google services and submitting sitemaps.
• 🛠️ Ensure your website is set up correctly for indexing by checking settings like 'Discouraged search engine' in WordPress Reading settings and 'Robots Meta' in Rank Math.
• 🔗 Utilize sitemaps to help Google discover new and updated pages on your website, which is crucial for indexing.
• 🔄 Google Search Console's 'URL Inspection' tool can be used to check if a page is indexed and to request indexing if it's not.
• 🚀 'Instant Indexing' and 'IndexNow' are advanced methods for time-sensitive content to get indexed quickly.
• ❌ Duplicate content can deter Google from indexing your pages frequently, so consolidate similar content or use canonical URLs to avoid this.
• 🔒 Be cautious of 'Crawler Traps' like excessive filtered URLs that can confuse search engines and prevent proper indexing.
• 📝 Creating unique, high-quality content that directly answers specific queries is key to getting indexed and ranking well on search engines.

Q & A

What is the first step in getting a website's content indexed by Google?

The first step in getting a website's content indexed by Google is the discovery or crawling phase, where Google finds the site either by following a link from a known page or through sitemaps provided by site owners.

How does Google decide whether to index a page after it has been crawled?

After crawling a page, Google analyzes the non-text and text content, visual layout, and other elements like title, tags, and alt attributes to understand the page's purpose, meaning, and context, which helps decide if it should be indexed.

What is the role of Rank Math in helping content get indexed?

Rank Math is a WordPress SEO plugin that assists in getting content indexed by simplifying tasks like connecting a site to Google Search Console, submitting sitemaps, and providing tools to check and improve indexing.

How can one check if a page is indexed on Google?

To check if a page is indexed on Google, one can use the URL Inspection tool in Google Search Console, search for the targeted keyword to see if the content ranks within the first ten pages, or use the 'site:' search operator followed by the page URL.

What is the Instant Indexing feature and how can it be used?

Instant Indexing is a feature for time-sensitive content that allows for immediate indexing by Google. It can be used by installing the 'Instant Indexing for Google' plugin by Rank Math and submitting URLs for indexing through its interface.

What is 'IndexNow' and how does it relate to indexing content on search engines?

'IndexNow' is an initiative by Microsoft and Yandex to help search engines crawl and index sites more efficiently. Google is testing this method, and Rank Math is prepared to support it, potentially allowing for faster indexing of content.

What could prevent a website's content from being indexed by Google?

Several factors could prevent content from being indexed, including duplicate content, site settings that discourage search engines, issues with robots.txt, and the presence of crawler traps such as excessive filter pages.

How can internal links help in getting a page indexed?

Internal links from already indexed and ranked pages to new pages can help search engines discover and index the new pages more effectively, as search engines are more likely to follow links from known pages.

What is the importance of unique and high-quality content in the indexing process?

Unique and high-quality content that directly answers specific queries is more likely to be indexed by Google because search engines prioritize content that provides value to users, helping them solve problems or answer questions.

How can a sitemap help in getting a website's content indexed?

A sitemap provides a list of pages for search engines to crawl, making it easier for them to discover new or updated content. Submitting a sitemap through tools like Rank Math or Google Search Console can expedite the indexing process.

What should one do if they find errors or warnings in Google Search Console related to indexing?

If errors or warnings are found in Google Search Console, one should address them by checking the specific issues reported, such as fixing blocked pages, removing duplicate content, or adjusting meta tags, and then resubmitting the pages for indexing.

Outlines

00:00

📈 Boosting Google Indexing with Rank Math

This paragraph introduces the challenges content creators face when their articles don't attract traffic due to lack of indexing by search engines. The speaker, Jack from Rank Math, a WordPress SEO plugin, promises to discuss strategies to expedite Google indexing. The video will cover how to check if a page is indexed, common hindrances to indexing, and bonus tips to enhance indexing chances. Jack also encourages viewers to subscribe for more SEO and business insights.

05:03

🕸️ Understanding Google Indexing Phases

The paragraph delves into the mechanics of Google indexing, detailing the discovery or crawling phase where Googlebots identify new or updated web pages through links or Sitemaps. It explains the indexing phase, where Google processes the content for understanding its purpose and context. The focus is on making content accessible for indexing rather than on ranking, which is a subsequent phase. Jack also provides a link to another video for content optimization techniques.

10:06

🔌 Practical Steps for Google Indexing

Jack outlines practical methods to index pages on Google, emphasizing the importance of submitting a sitemap. He guides viewers on how to connect their site to Google Search Console and submit a sitemap manually or through the Rank Math plugin. The paragraph also covers setting up Rank Math, connecting Google services, and the benefits of using the plugin, such as automatic sitemap submission and privacy assurances.

15:08

🔄 Ensuring Google's Awareness of Your Site

This section describes how to submit a sitemap to Google Search Console and use Rank Math's features like 'Ping search engines' and 'Instant Indexing' to notify Google of updates. Jack also explains how to use Google's URL Inspection tool to check if a page is indexed and to request indexing if it's not. He mentions 'IndexNow' as another method to improve search engine efficiency.

20:09

🚫 Identifying and Fixing Indexing Issues

The paragraph discusses how to troubleshoot indexing problems by checking Google Search Console for errors or warnings and understanding the reasons behind them. It advises on ensuring content uniqueness, using canonical tags, and checking settings like 'noindex' tags and robots.txt files. Jack also introduces Rank Math's 'Index Status' tab for easier indexing issue resolution.

💡 Bonus Tips for Effective Indexing

In the concluding paragraph, Jack shares bonus tips for improving indexing, such as leveraging internal links from indexed pages, producing unique and high-quality content, and avoiding 'Crawler Traps' by managing URLs generated by filters. He reiterates the importance of these strategies and invites viewers to engage with the content by liking, commenting, and subscribing.

Keywords

💡Content creator

A content creator is an individual or entity responsible for producing written, visual, or auditory material to engage an audience. In the context of the video, content creators spend significant time crafting articles, hoping for visibility and traffic. The frustration arises when despite their efforts, the content fails to attract visitors, which is a common scenario the video aims to address by discussing indexing strategies.

💡Google indexing

Google indexing refers to the process by which Google discovers, crawls, and adds web pages to its search engine index. The video emphasizes the critical nature of indexing, as it is the first step for any online content to become discoverable on Google. Without indexing, content remains invisible to search engine users, which is a primary concern for content creators渴望 their work to be found.

💡Rank Math

Rank Math is a WordPress SEO plugin mentioned in the video as a tool to enhance search engine optimization. It aids in making websites more searchable and competitive by handling various SEO tasks, including but not limited to indexing assistance. The video uses Rank Math as a practical example to demonstrate how to connect a website with Google services and submit a sitemap, thereby improving the chances of content being indexed.

💡Crawling

Crawling is the mechanism by which search engines like Googlebot visit websites and pages to scan and add new or updated content to their databases. The video script explains that effective crawling is essential for new content to be considered for indexing. It forms part of the broader discussion on how content gets noticed by search engines.

💡Sitemaps

A sitemap is a file where all of a website's important pages and content are listed, allowing search engine crawlers to discover and index them more efficiently. The video script provides a step-by-step guide on how to submit a sitemap through the Rank Math plugin, highlighting its importance in the indexing process.

💡Google Search Console

Google Search Console is a free service offered by Google that helps website owners monitor, maintain, and troubleshoot their website's performance in Google Search results. The video script describes how to connect a website with Google Search Console through Rank Math and use it to submit sitemaps and check indexing status.

💡Index Status

Index Status is a feature within Rank Math that provides insights into the indexing status of a website's pages. The video mentions this feature as a valuable tool for Pro users to monitor and manage their website's indexing directly from the WordPress dashboard, without needing to access Google Search Console.

💡Duplicate content

Duplicate content refers to substantial similarity between pieces of content, which can lead to search engines devaluing or not indexing such content. The video script advises on handling duplicate content, either by consolidating it into a single page or using canonical tags to indicate the preferred version to index, which is crucial for maintaining a website's SEO health.

💡Canonical tags

Canonical tags are HTML elements that help tell search engines the preferred version of a web page to index when there are multiple similar or duplicate pages. The video script explains the importance of canonical tags in preventing search engines from indexing duplicate content, ensuring that the primary version of a page is the one that gets indexed.

💡Robots.txt

A robots.txt file is a standard used to communicate with web crawlers and other web robots; it helps manage which parts of a website should not be accessed or indexed. The video script addresses how a robots.txt file can inadvertently block indexing and provides guidance on configuring it properly to allow crawlers access.

💡Crawler Traps

Crawler Traps refer to situations where a website's structure or content can confuse or excessively burden search engine crawlers, leading them to ignore or fail to index pages. The video script uses the example of product filtering on e-commerce sites, which can generate numerous pages that are not useful for indexing, illustrating the concept of crawler traps.

Highlights

Content creators face the frustration of no traffic despite publishing quality content.

A common issue is realizing that the article is not indexed by search engines.

Understanding the two phases of Google indexing: discovery or crawling, and indexing.

Google uses Googlebots to crawl billions of web pages daily.

Google analyzes various page elements to decide if it should be indexed.

Submitting a sitemap is crucial for getting pages indexed.

Using Rank Math plugin simplifies the process of connecting to Google Search Console and submitting a sitemap.

Instant Indexing for Google plugin by Rank Math is useful for time-sensitive content.

IndexNow initiative helps search engines crawl sites more efficiently, including Google.

Checking Google Search Console's 'Coverage' for indexing status and errors.

Using 'URL Inspection' tool to check if a page is indexed and request indexing.

Ensuring content uniqueness to avoid duplicate content issues with search engines.

Checking and managing 'robots.txt' to avoid blocking search engine crawlers.

Using internal links from indexed pages to new content to improve indexing chances.

Producing high-quality, unique content that directly answers specific queries.

Avoiding 'Crawler Traps' by managing filtered pages and using 'robots.txt' effectively.

Final tips include leveraging internal links, producing unique content, and managing 'robots.txt'.

Transcripts

The most frustrating thing for a content

creator is to spend hours and days writing your content...

You publish your article thinking that it is the best out there....

Then days become months and you realize

that there is no traffic coming to your article.

You type in the targeted search term and your article is nowhere to be found.

And later you realize your article is not even indexed.

So in this video, we'll be talking about the things that you could do to get Google

to index your site much faster with the help of Rank Math,

how to check if a page on your website is

indexed, the things that could prevent your website from getting indexed,

and do stick with me to the end, because I'm throwing some bonus tips

to help improve the chances of getting your pages indexed.

So let's go.

Hey, it's Jack here from Rank Math, the WordPress SEO plugin that gives your

business the best chance of beating your competitors on search engines.

We put out a lot of videos to help you grow your search traffic,

so consider subscribing to our channel to get SEO and business knowledge

delivered straight to your YouTube notification.

Now, before everything else, we've got to understand the concept

of getting pages indexed on Google, and it consists of two phases.

The first phase is the discovery or

crawling phase, where your site will be found by Google.

The Web is a jungle of information, and although Google has sorted and indexed

billions of pages, there is still a constant flow of new

pages entering the web, and known pages being updated or deleted.

Google has to constantly seek for new and updated content to keep itself

relevant, so it uses robots or spiders called Googlebots to fetch or crawl

billions of pages on the web every single day.

It does that by either following a link from a known page to a new page, or

crawling a list of pages provided by site owners called Sitemaps.

Once Google discovers a page URL, it crawls the page to find out what's on it.

It analyzes the non-text and text content as well as the overall visual layout

to decide if a page should be sent for indexing.

Once the page is found and crawled, the indexing phase will begin,

where the crawled page will be processed in order to understand the purpose,

meaning, context, intent and everything about the page, which includes the title

tags, text content, alt attributes, images, videos and more, so that Google knows how

to sort and rank your content on different search terms.

That extends out to the third phase, which is serving and ranking your content.

But that's not our focus in this video.

Our focus is to get your content indexed.

So, that's the simple concept of Google indexing.

If you want to know how to optimize your content for better rankings,

you can check out this video right here where we will show you some cool methods

to optimize your content for better search ranking.

Now, let me walk you through some practical

ways to get your page indexed on Google and some other search engines.

As you know, there are millions of pages entering the web every day,

so your best chance of getting noticed by Google is to submit your sitemap.

It sounds complicated, but it's actually quite easy.

I'll walk you through step by step.

Now, you can go through the manual way of connecting your site

to Google Search Console, and submit your sitemap manually to it.

But, let me show you a much simpler way to do it through our Rank Math plugin.

On your WordPress dashboard.

Go to Plugins and Install Plugins you want

to make sure that the Rank Math plugin is installed.

If you haven't done so,

you can click on "Add New" and on the search bar search for "Rank Math".

You want to install and activate this plugin.

Once you've done that, on the Sidebar,

click on Rank Math and you'll be prompted to connect your account.

If you have an existing Rank Math account, you can connect it by clicking on this,

but if you don't, you can click on this and register for a free account.

Once you've done that, you'll be prompted with the Setup wizard.

I recommend that you select the "Advanced

Mode" because there'll be a lot more modules that will be available to you

if you select this option. You can go through the setup at your pace,

but what's important is this step where you will connect Rank Math to your Google

services such as Search Console and Analytics.

With this step, you don't need to go through the steps

to verify your site with Google Search Console.

All you need to do is to sign in to your

Google account where you want to set up your Google Search Console.

Let me give you an example.

I've logged into this account and I have not set up any properties or linked any

website to it, so I want to log in with this account.

Now in this step,

Rank Math is asking for permission for these, and it is important that you check all

these boxes because that way, Rank Math can do its job to give you all the benefits

as promised. You don't have to worry about

Rank Math seeing your analytics data like the amount of traffic coming to your site,

it's referring pages, and all those other site data,

because we simply don't store your data on our server. We just access it,

draw out all the information, and store it on your server.

We don't even look! That's how important your Privacy is to us.

So, be rest assured that we have no control over your data.

Let's continue.

Once you've connected your Google account, give it a while and you will notice

that your current website is now added to your Google Search Console.

There is no need for you to manually add and verify your website.

That's one of the big benefits for going through that step.

Now, you will also have this option

to enable the "Index Status" tab which is very useful for Rank Math Pro users,

as all the indexing data from Google Search Console will be

analyzed, sorted and displayed directly on your website.

I will show you what I mean in a while.

You can also set up your Google Analytics

account and your AdSense account if you are a Rank Math Pro user,

but that will be a topic for another video. As we go to the next step,

this is the most important step.

You have got to keep this toggled

"ON" and make sure that all the post types and assets that you wish to be included

in the sitemaps for indexation are selected. And these are the two most

important steps to complete to get your sitemap ready.

Now, as you return to your dashboard, the final step is for you to submit your

sitemap to Google Search Console, and you can find your sitemap by going

to the "Sitemap Settings" and you will see the link to your Sitemap here.

If you visit the link you will see your post, page, and Category sitemaps.

Click into each of them, and you will see

a list of all the URLs that you have in the Sitemap.

This makes it easy for Google and other

search engines to discover all the links on your site.

So, back to your site.

You want to right click and copy this link

and you want to go to Google Search Console.

If you don't know how to reach this page, you can just search for it on Google.

If not, we have left a link to this page in the description.

Let's click on "Start now".

Once you have logged into the Google

account that is connected with Rank Math, and you click on the left hamburger menu,

click on "Select Property" and you will see your website listed here.

That's because Rank Math has added

and verified your website with Google Search Console.

Let's select our site and you will see this is a brand new website.

To submit a sitemap,

click on "Sitemaps" and for some of you you will already see Rank Math

submitting the sitemap for you

automatically, so you don't have to submit your sitemap manually.

But just in case you do not see this, you want to place the sitemap link here,

and make sure to keep just the URL slug, and you want to hit submit.

Once you have submitted successfully,

Google will periodically process your sitemap and look for changes,

which means Google now knows your site exists and they will be sending their

robots to visit your site on the schedule that they have set.

And, that's how you submit your sitemap to Google Search Console.

Now, in case you are an anxious person like

me, who wants Google to know every movement on my website?

You can always resubmit your sitemap here, but it is going to take you a lot

of time, and you can't focus on what is important.

So, we have added a feature in Rank Math to Ping search engines.

In other words, give search engines a nudge periodically

and tell them that there are changes to our sitemap.

You can find this setting on the "Sitemap

Setting" and right at the bottom you will see this here.

You want to keep this setting "ON".

If there are individual pages you want

Google to take notice of immediately, or as soon as possible, you can copy your page

URL and paste it on the URL Inspection tool and hit "Enter" or "Return". Here's where

you know if your page is indexed on Google or not.

If it is not indexed, like this example, you can click on "Request Indexing"

and Google will add your page to its crawling queue.

You'll probably need to give it a couple of days before testing the page URL.

Again, you don't need to re-request for Indexing after you've submitted

because it will not give you any advantage in terms of crawling.

If the page is indexed, you will see something like this.

And, if you have made changes to this page and you want Google to notice the changes

as soon as possible, click on "Request Indexing" again.

That's how simple it is.

Two additional methods I want to show you

to make sure that search engines notice your pages.

The first is something called "Instant Indexing".

This is super useful for time sensitive

content such as news coverage, job posting or live streaming.

To use this, you need to install and activate

a standalone plugin called "Instant Indexing for Google" by Rank Math.

And once you have done that, under "Rank Math" and "Instant Indexing" under the "Console",

you can add all the URLs of your new articles, the updated articles,

or the URLs that are removed from your website, and you want to hit "Send to API".

You can also configure this to be done automatically.

If you want more information about setting up Instant Indexing for Google,

you can check out the video popping up at the top right.

The other method is something called "IndexNow", which is an initiative by Microsoft

and Yandex to help search engines crawl sites more efficiently.

Google is currently testing this method as

well, and Rank Math is very well prepared for it.

You can visit this page to find out more about it.

The most straightforward method I've shown you is to go to Google Search console

and paste your page link in the URL Inspection Tool.

The other method is just to search

for the targeted keyword on Google, and if your content is ranked,

maybe it is not on the first page, but it happens to be on the first ten pages.

It means that your page is indexed.

Just type in "site:" and you want to place your page URL right after it without any

spaces, and hit enter. If your site is not indexed, nothing will show here, but if it

is indexed, you will see a search result like this.

You can check your Google Search Console

under "Coverage" and see if a particular URL is having an "Error",

"Valid with warning", if it is "Valid" or if it is "Excluded".

If there is an error, you want to know what error it is and you want to fix it.

The same goes with the "Valid with warning".

And, with "Valid" it means that these pages are indexed on Google, and with "Excluded"

these are the reasons that these pages here are excluded from indexing.

Check if that particular URL is part of this group.

Especially the "Alternate page with proper canonical tag".

Sometimes your URL or your page title is too similar to the ones that are being

indexed and Google has automatically placed a canonical tag for you.

In that case, firstly, you need to make sure

that the content of that page is not similar to the ones that are indexed.

Then, you may want to change up your URL slug

and title and resubmit that page for indexing.

The key here is to ensure that your

content is unique from the other pages of your website.

Do you remember at the start of the video,

we have toggled this option "ON" on a setup wizard?

With that, we now have this "Index Status" tab under "Analytics".

This is very useful for Rank Math Pro users

because you don't need to access Google Search Console and we have made every

detail of your website much more easily accessible straight from this tab.

Troubleshooting Your site is now much easier.

It will show you the number of pages and its percentage that is submitted

and indexed, and all the other indexing issues found on your website. And, we have

consolidated all the data into this useful graph which makes it easy for you

to identify and troubleshoot the URLs on your site.

You can use this filter feature to sort

those pages that are not indexed by Google, and you can identify specifically why

by clicking on the results and it will tell you all the details about this URL.

For this case, this URL is blocked by robots.txt,

and Google could not fetch this page properly.

If this is not intended,

you now know where the issue is. And for the URLs that have errors,

it shows that this page is blocked by a meta tag and the server returned

an error page when Google attempted to crawl the page.

Another very useful data is if the user

determined Canonical URL is the same as the Google determined Canonical URL.

If they are different, it means that Google views your canonical

URL differently and you may need to visit those pages and make sure that they are

not duplicated pages. For "URLs that are unknown to Google",

We can take action to submit these URLs

to Google using one of the methods we have discussed,

if it is true that you want these pages

to be indexed. You may also encounter issues such as "Excluded by 'noindex' tag",

which means you have submitted a page for indexing,

but maybe there is a 'noindex' meta tag

instructing Google not to index this page. Anyway,

these are all the issues that are commonly seen on Google search console.

If you want to learn more about each

of these issues, you can visit this support document

from Google that shows you what they are and how to fix them.

If you want to check this page out, we have left a link in the description.

Now, we have one more trick up our sleeves

to see if your particular page is indexed on search engines.

If you're logged into your site and you

visit one of your pages, you will see the WordPress taskbar at the top.

Towards the right,

you will see "Rank Math SEO". Hover to it, and you will see a list of options here.

I don't want to go too in-depth into what each of these options are, so you can visit

this link to see what other awesome features are in this quick action menu.

What we need is to check if this page is

indexed, so you want to hover to "External tools".

Click on "Google Cache" and if the page is

cached and indexed, you will see this message at the top of your page.

It shows the date when this page is cached.

And, if you have made changes to this page after this page is cached, and you want

Google to take notice of it, you may want to send this URL

for re-indexing. But, if it is not cached, you will see this error.

These are all the common ways to discover if your page is indexed.

Of course there are many other methods, but what's the point.

Right?

Now, let me ask you a question...

Imagine I am your close friend

and for some reason you'll visit me every day at my house to collect something useful

from me that will help you with your business.

But every time you visit me,

instead of giving you something useful, I always present you with something

that you don't like to the core, for example, I always betrayed your trust.

My question to you is, would you visit me more? Or, would you visit me less?

Or, would you not visit me at all?

The same thing that is going to happen with Google.

It doesn't like duplicate content to the core, so will it visit your site often,

if you always serve duplicate content to them?

I'm pretty sure you know the answer, but the truth is, they will visit your site

less often, so your site will be crawled less frequently.

If you are a victim to this, maybe you want to consolidate all

the similar content and information into one single page and remove the rest

of the pages with a 301 redirect to the main page.

Or, if you think your duplicate content, for example,

your service pages with different currencies and prices are still relevant

to your business, and you can't delete them,

then you want to add a canonical URL to other duplicated pages pointing to one

page so that Google will index that page and would not index the rest of the pages.

Also, as what I discussed earlier about "Alternate page with proper canonical tag"

when visiting your Google search console,

you can use that method too. Other things that could stop Google from indexing your

pages probably come from the settings on your site.

It's like Google knocking on your door and you put up a sign saying "We are closed!".

So, to prevent that from happening, first,

on your WordPress dashboard, under "Settings" and "Reading", you want to make sure

that this "Discouraged search engine" option is unchecked.

If it is checked,

you are basically putting a sign saying we are not open for business.

So, make sure that this is unchecked then,

if you have our Rank Math plugin installed,

you want to go to "Titles and Meta" and under "Global Meta" you want to make

sure that the "Robots Meta" is set to "Index".

This will ensure that all the pages,

posts, or custom post types, will be set to the "Index" option by default.

And, just to make sure that all the post

types are set with the index option, visit "Posts" and under the "Post Robots Meta",

if this is turned off,

it will follow the Global meta we have set earlier, which is to index.

But, if this is turned "ON" with "No Index",

make sure that it is set with the "Index" option.

The same goes for the other post types and in case there is this particular page

or post that you have done everything to get it indexed,

but it is still not indexed after a couple of weeks or months,

you want to go to the individual page under Rank Math, Select "Advanced" and you

want to make sure that the "Robots meta" is set to "Index" for this post.

robots.txt

is also another cause of the problem. By default,

if you use Rank Math, under "General Settings", you will see the "Edit robots.txt"

option.

If there is no conflicting robots.txt

file on your file manager of your

website, we will add a code like this for you.

I don't want to go too deep into this as

it may become technical, but as long as you do not see a code like

this, that disallows your entire site to be crawled, you are fine.

You can check this page out as we have

laid out some useful information to help you understand robots.txt

Now, if you see this message where it says "Contents are locked because a robots.txt

file is present in the root folder."

This means that in the file manager of your website, you have an existing robots.txt file.

You can visit the contents of this file if your site has the code I've mentioned,

but if you want Rank Math to manage your robots.txt file,

just delete the existing file in your file manager.

Head back to your Rank Math and refresh

this page and you will see the default configurations from Rank Math.

If you don't see this, you can visit this page.

We have left the link in the description as well.

Copy this set of codes, head back to your website and paste it here.

All you need to do is to replace this

sitemap link accordingly, which in this case it is this,

and you can save changes.

As promised. Since you are still tuned in,

I want to share with you some cool bonus tips.

Do you remember at the start of the video

I mentioned that Google will follow links on pages that they already know?

You need to leverage on that!

What I mean is "Internal links".

Some of your pages are probably ranked on search engines and are getting traffic,

so if your new page is something that is relevant to your existing page that is

indexed and ranked, you may want to add an internal link

on the page that is indexed, pointing to the page that is not indexed.

This is the best way to get search engines

to index your new page on top of those methods that I've shared.

Another tip is just to produce unique

and high quality content that answers a specific query.

Remember not to go off topic and always ask yourself if every sentence on your

content helps with answering the targeted topic.

If not, don't add them to your content.

Google is always in search of the best

content out there, so to stand out from the rest,

you got to write the best, to your knowledge and experience.

And, one more thing I want to highlight,

which is very common to ecommerce or listing sites, is that it takes resources

for Google or other search engines to crawl your site.

So, if they find too many irrelevant URLs

that are generated due to the filters for pricing, sizes, or whatever,

search robots may not know which page or pages you want them to index, and they may

end up giving up indexing any of your pages.

Just imagine you sell a T-shirt in sizes such as "Extra Small", "Medium", "Large",

"Extra Large", Extra Extra Large", and you have eight different colors.

So, for this T-shirt itself it generates 48 different pages.

Not forgetting if you're sorting in price

and popularity, it would generate even more filter pages.

This is what we called "Crawler Traps".

Search engines wouldn't know what pages

to crawl and index, so we recommend adding something like this to your robots.txt file

to prevent search robots from crawling filtered pages.

To recap,

these are all the methods of getting your

website indexed, and these are the methods to know if your site is indexed.

Also, these are the probable reasons why your site is not indexed,

especially when it comes to duplicate content,

you want to delete those duplicated pages with a 301 redirect to just one page,

or if you think you can't delete them for your use case,

add a canonical URL for all those duplicated pages pointing to one main page.

That's it! We hope that you are now an expert

in getting your content indexed on Google and other search engines.

If you find this video helpful,

can you do us a little favor and smash that thumbs up button?

It means a lot to me and the Rank Math team.

If you have any questions,

feel free to leave them down below, we'll do our best to help you out as much as we

can, and don't forget to subscribe to our channel for more SEO and business

knowledge delivered straight to our YouTube notification.

This is Jack from Rank Math,

I'll see you in the next video.