Google is a major traffic source for most websites. To get that traffic your website needs to be crawled, indexed and ranked. Then it will appear in search results for related queries. In this SEO guide we will learn how to get crawled and indexed by Google faster.
Most of the time we create a new website or write great content and expect visitors to come to our website. We wait a couple hours or days and nobody gets to our website. Spending so much time and effort creating content and not getting any visitors is frustrating.
To fix this problem there are several tactics to get indexed. Faster your website gets indexed faster it will appear in search results and get visitors to your website.
For brand new websites to get indexed there should be at least some content. Preferably it should be well written articles. If not it can be regular content pages, about us page, homepage describing your services etc.
I have experience indexing and ranking single page websites as well. Main point is it should be a well written page that has some value to visitors.
Table of Contents
How does a search engine index websites?
Google organizes the whole internet in order to find searched information. It performs 3 main actions in order to show your website to interested searchers.
Lets see what each step does:
- Crawl – Google spiders crawl the internet following links and find new pages to check.
- Index – Google reads pages that were collected in the crawl stage.
- Rank – Using various factors and algorithms Google ranks pages to determine its position in related search results.
Our focus is to index your website and your new content pages fast.
What is needed for google to index a website?
There are 3 things a website needs for getting indexed.
- Content – google indexes pages that have some content. Preferable it should be well formatted text optimized for search engines. Text with heading and subheadings, images, video, links to other pages, properly optimized page title and description etc.
- Sitemap – submitted to Google informing about new content and updates.
- Links – pointing to your web page tells google to crawl it for indexing.
For indexing to work we also need to make sure that your website is crawled.
Links play double role in search engine optimization of your website. They will help to crawl (initially to find your new content faster) and to rank pages. More relevant links are important ranking factor for your pages.
Submit your website to Google
Google has built a tool for submitting websites to their search engine. It is called Google Search Console. Search console will show you keywords, your ranking pages, indexed pages, performance of your website and additional search data measured by search engines.
Navigate to https://search.google.com/search-console/about
Click the “Start now” button and login with your google account. Create one if you do not have any google account. Usually all users with android phones already have a google account.
After you register to Google Search Console you can add your website to google.
Click on the property drop-down on the left top corner and at the bottom of the list click the “+ Add Property” button as shown in screenshot.
You will be presented with 2 options. Choose the “URL prefix” option because it is much easier and in some cases can be automated. Enter the address of your website and click the “Continue” button.
Now you will have several options to verify that you are the owner of the website.
- If you have already added “Google Analytics” or “Google Tag Manager” code to your website then choose those options. I use analytics on all my web properties so it is easier to verify with it. You do not need to add any code to your website if you already added Analytics. Select the “Google Analytics” option and click the “Verify” button. Your website will be automatically verified in Google Search Console.
- HTML file option if you want to access your server with FTP or file manager.
- Domain name provider option allows you to verify by adding additional DNS records to your registrar.
- For websites using WordPress it is also easy to verify with the “HTML tag” option. Select this option and copy the HTML code as shown in the screenshot.
If you choose to verify with an HTML tag then use the Rank Math SEO plugin for WordPress to verify your website. Rank Math SEO also provides content optimization for focus keywords and tracking SEO performance for your website. I use it on my websites.
How to verify your website with Google Search Console using WordPress?
- Copy “HTML tag” verification code from Google Search Console.
- Navigate to “Rank Math” → “Settings” → “Webmaster tools” from your WordPress dashboard.
- Paste verification code and save changes.
- Go to Google Search Console and click the “Verify” button. Your website will be verified.
After verification it will take some time to populate your website with data collected by google search engine.
Now we can add a sitemap to it.
Generate XML sitemap
Sitemap is used to inform search engines about your new and updated articles. These are your main content pages that you want to be shown in google for qualifying searches.
WordPress has a built-in feature to generate sitemap. Navigate to
yourwebsite.com/sitemap.xml and you will see it.
I am using Rank Math SEO so I can control which content should be listed in the sitemap. For example I do not want to index forum pages, tags, attachments or other custom content types that I use in WordPress. It also allows
noindex content which is not adding value to my website or duplicates existing content.
You can also add images to your post and page sitemap.
Depending on your WordPress setup there can be alternative locations for sitemap.
What are alternative sitemap URLs?
yoursite.com/sitemap.xmlis the default sitemap location. In some cases it will redirect to one of the following URLs.
yoursite.com/wp-sitemap.xmlis generated by WordPress when no custom plugins are used.
yoursite.com/sitemap_index.xmlis generated by some SEO plugins for WordPress. Rank Math SEO also uses this URL for sitemap.
Why use SEO plugins for sitemap?
WordPress default sitemap has some URL that should not be indexed by Google. Following URL types are not advised to index for better SEO.
- Pages protected by user login like shopping cart, checkout page, logged in account page etc.
- Pages that may appear as duplicate content. For example tag, category, author pages. They have latest posts that may appear in multiple places. In order to prioritize your articles it is better to not index article listing pages that show the same content multiple times.
- Forum, bbPress, buddy press pages that have user generated content. These pages are not controlled by you and content appearing in them may harm your overall website authority.
- Attachment pages that have only one image which is used in your articles. These pages do not have any SEO value alone. They discovered better and ranked inside your articles.
For removing not needed URLs from your sitemap it is better to use custom SEO plugins. I use RankMath for this purpose. Additionally you can use
noindex individual pages in case they are indexed by google by mistake following internal or external links. It will help to remove login protected pages, duplicate content, user generated content with no value from google index.
Add xml sitemap to Google Search Console
After tuning your sitemap xml submit it to google search console.
Navigate to “Sitemaps” in Google search console and type
sitemap_index.xml in “Add a new sitemap” field. Then click the “Submit” button.
I also added an RSS feed as a sitemap to this field. So type
feed/ into “Add a new sitemap” field to add RSS feed. It does not add any new URLs to google but will serve as an alternative to xml sitemap.
RSS feed will add only the last 10 posts to google while sitemap will add all post and page URLs.
Request indexing a page inside URL inspection tool
Google Search Console has a URL inspection feature. You can view the index status of any URL on your blog.
How to check the index status of a URL?
To check index status of a URL we need to inspect the URL in Google Search Console. Use following steps to inspect URL:
- Navigate to “URL inspection” in Google Search Console.
- Paste URL of page you want to check and press enter.
- New page will show you the current index status. You will see a green check mark saying “URL is on Google” when your page is indexed.
How to add URL to google crawler tool?
Use following steps to add a URL to Google crawler tool (request indexing):
- First inspect the URL that you want to index.
- URL status page will open. Click the “Request Indexing” button if your page is not indexed or if it is changed.
- Google will check if the Live URL can be indexed and record your request.
How to check if crawling and indexing allowed for URLs?
- Inspect the URL that you want to check.
- When the report opens, check “Crawl allowed?” and “Indexing allowed?” status.
When crawling or indexing is not allowed then the report will show the reason why it is not allowed.
For example like this:
- No: ‘noindex’ detected in ‘robots’ meta tag
- No: blocked by robots.txt
When blocked with a
noindex meta tag, fix it using WordPress SEO plugin.
If blocked with
robots.txt file then edit and remove blocking directive from that file.
Create robots.txt file
Robots file is designed to inform search engines like Google, Bing etc. what pages to index and what should not be indexed at all. Every website should have a robots.txt file to control which content indexation. Check your robots file navigating to
Robots file will help you block hidden content from search engines. For example login protected pages, login protected downloadable files like pdf, zip etc.
Check these directives that can be used to block protected content from showing in search results. Most helpful rule can be disallowing protected pdf pages if you provide them for your subscribers or paid members only.
robots.txt file looks like this.
It blocks access to admin (
/wp-admin/) and internal resource (
You can add sitemap url to robots.txt file as well. Robots file will be read by all search engine robots and they will discover your XML sitemap file from there.
Creating static robots.txt file and placing it in the root folder is always preferred. Because it will not use any PHP code and will not use server resources. Create it with a text editor and upload to your root folder with a file manager or FTP program.
If you are unable to upload it then you can use RankMath to manage your robots.txt file. Related settings located in “Rank Math” → “General settings” → “Edit robots.txt” page. Edit your robots file and save changes.
Having a sitemap will technically inform google that you have new or updated content. It will not provide any value in terms of ranking. For ranking and getting indexed faster, link to your content within your website and from external websites.
I assume that you follow my SEO guide for WordPress and always link to your new content from old articles.
Get external links to your website
External links can boost your ranking and get your content indexed faster as well. You can use features of external websites like social networks to link to your new content.
Create social profiles for your website
For brand new websites I always recommend creating profiles on various social networks. Current popular social network sites that can bring traffic are: Facebook, Twitter, Instagram, Pinterest, Telegram, Reddit, Quora, Telegram, YouTube, LinkedIn etc.
Create a profile on each of them if appropriate to your website content. They will not only help you get indexed faster but will bring traffic from your followers when you use those platforms. They will also allow you to grow your fan base and make sharing your content easier.
Create video introduction for your website
YouTube is the second most popular search engine in the world. Google also shows results from YouTube if they are related to user search results.
If you want your website to get indexed faster than you can create an introduction video of your website and post it on YouTube. Link to your website can be added in the description of your video. You can also add a link to your website from the YouTube channel info page. Linking to your other social network profiles can be added to the channel info page as well.
You can create video content for tutorials, review articles as well.
After uploading video to YouTube embed it to your related page on your website. This will bring more traffic to your content and help it to rank higher.
I have a video of tutorial for my simulator software here which constantly brings some traffic to my website.
Get links from related blogs, forums, community websites
Links from blogs and forums also will help to index your new content. Use it with caution. Do not spam forums with links to your website. Forum users and google will not like it. Provide value in the forum by solving their problem and link to your relevant content for learning more.
Ask your friends to introduce you in their blogs
Getting a link from a relevant blog to your brand new blog is like gold. You can ask your friends with blogs to introduce your blog to their readers. This can be done like a regular post or you can offer to write some guest post and link from there to your blog.
Getting links from popular blogs can be hard. Asking your friend to link may be easier because they are your friends. But it is not guaranteed that they will link.
Link from your existing website
You can also link to your new website from your old one. This is possible if you already have a website. Users following your blog are familiar with your content and can be interested in your other websites as well.
Places where you can add link from your existing website so link will be relevant:
- You can write an article about your new website.
- Add it to projects or tools page with some introduction.
- Mention your new website in your about page.
I do not suggest adding a site-wide link if it is not closely related to your existing website niche. Because the footer link will not be relevant to all your pages. We know that links from relevant pages pass more value.
You can add site-wide links in the footer only if your websites niche complete each other. For example if you have one website for plugins, other for themes then you can link to each other from the website footer.
Is Google indexing free?
Google crawls and indexes websites for free. You can follow instructions on this page to add a sitemap of your website to Google Search Console. There might be some third party services that offer to add your website to Google. They will charge you for their time and service. You can do it yourself for free or instruct your staff to do it by sharing this guide with them.
How to get google to index my site faster?
For indexing faster you need to complete following steps:
- Submit sitemap and RSS feed to Google Search Console.
- Add a couple internal links from old content to your new page.
- Add links from social networks, messaging apps, YouTube etc. Share your content in all relevant social media that you have registered.
Completing all these steps allows your new content to be discovered faster. This will lead to crawl and indexing faster as well. When you have several links pointing to new content you are increasing the probability of getting discovered by google crawler. Faster your new content is discovered means faster crawling and indexing for your new content.
How long does it take for Google to index a site?
Usually it takes from a couple hours up to a couple weeks for Google to index a page. Indexing speed varies for each website. It is measured by the update frequency of a website in the past couple months.
For example new content on classifieds ads websites that has 100 new listings daily will be indexed in a couple hours. Content on a blog that has 1 new post each month will be indexed in a week. Blog with 2 new posts per week may be indexed in a couple days.
Conclusion for getting indexed by Google
Getting indexed by search engines is an important part of getting traffic. Only indexed content can qualify for showing up in Google search results. So far Google is a traffic source that brings more than 50% of all visitors to a website for free. For this reason help search engines to index your website fast and increase traffic.
I explained how google and other search engines work. Add your website to Google search console, add your sitemap and RSS feed there. Add links to your new content and your content will be indexed in a short time. Keep updating your blog with more relevant content and your pages will get higher in search results and bring more traffic.
Also make sure your important content is indexable and content with low value is not indexed. Control what should be indexed with SEO plugins for WordPress.
Remember to monitor indexed pages report in Search Console for big drops in number of indexed pages. Fix corresponding issues with page index recovery guide.
Google search console is also used to view rich snippets and find posts with low organic CTR for improvement.
Search console with so many useful features has some shortcomings as well. For example it will not show featured snippets report or opportunities that can be used to target for getting featured snippets. I hope that they will add it in future but until then you have to use paid services like Ahrefs and Semrush.