Author: Adriana Stein
If you own, work on, or market a website, you need to understand the basics of crawling and indexing.
Why?
Well think of it like this:
If you can’t see, access, or find something, how can you buy it?
The same goes for websites.
If your content isn’t crawled and indexed, Google can’t see it and it certainly won’t show it when someone searches for it.
Essentially, if people can’t find your website, they’re going to have a much tougher time buying from you.
So when Google can find you, customers can find you. In fact, making your website “discoverable” is an essential element of SEO success.
When your content is crawled and indexed, you have a chance to show search engines like Google that your content is relevant and valuable to searchers, which increases the likelihood of it showing up in search results (i.e., ranking).
The more pages you rank, the more organic traffic you stand to gain, which often correlates to leads and conversions, meaning more money in your pocket.
So let’s now take a look at how crawling, indexing, and ranking are three essential elements of good SEO.
Table of contents:
The basics of website ranking
Ranking refers to a website’s position in the search engine results pages (SERPs). This can be achieved through optimizing various on-page and off-page factors. While attracting and converting visitors is one of the primary goals behind a successful SEO strategy, ranking highly for relevant searches is typically the vehicle that drives these goals.
To be clear, ranking is the final thing that search engines do to deliver search results (crawling and indexing occurring first; more on that below), but for the sake of contextualizing the process, we’ll start with the goal in mind.
Although there are some clear-cut actions to help improve rankings, it’s a nuanced process. There are hundreds of potential factors that search engine algorithms analyze to rank websites based on relevance and value to specific queries (the terms people use Google to search for).
Simply put, focus on relevant content for your audience
If you’re reading this article, it’s likely that you’re just starting out and have a smaller website. And you probably read the above paragraphs and are thinking, “Where do I even begin? How can I manage to create content that adheres to such a variety of ranking factors?”
Well, it’s actually rather simple.
Here’s my best advice as someone who’s done SEO for years for a wide variety of websites:
01. Create content that’s relevant to your audience
02. Targets keywords (queries) that your audience searches for
03. Provide a good user experience
Despite all the noise and SEO guidance you’ve probably already run across, that’s really what all websites should focus on.
When you stick to the three elements listed above—no matter the algorithm update, changes with AI, or whatever SEO trend may come—high-quality, helpful, relevant content is simply the most effective way to rank your pages and generate results for your brand or business.
Keyword research can help you identify content ideas
One of the best ways to figure out what works for you is to research and explore different keywords (keywords being, for the sake of this article, synonymous with “search terms” and “queries”) relevant to your target audience.
On a fundamental level, you can refer to your experience with your audience as a starting point by pursuing low/zero search volume keywords if you know that's what your audience is looking for. But it’s also important to be realistic and remember that content production should also focus on targeting keywords with search volume related to your products, services, or information within your niche.
Once you’ve got an idea of your target search terms, you can then analyze the search engine results pages (SERPs) to see what types of content currently rank. That’ll give you a good idea of what you may need to create (but always try to improve on competitors rather than mimicking them—after all, you can’t win a race from behind).
A simplified example of keyword research in action
To provide a quick example of this, let’s say I used Ahrefs free SEO tools to research the keyword free social media template. This generates the following keyword ideas for the US market:
You can see that free social media template has a search volume of 450 (as shown above), meaning that the tool estimates that there are roughly that many searches for this keyword per month. For the sake of this example, that’s perfectly fine, but remember, search volume isn’t everything.
Next, I go to Google to search our keyword, free social media template, which shows the following results for the US market:
Next, I might review each listing to assess what they have in common and what sets them apart. Identifying these elements can allow you to exploit gaps left by the competition and create content that meets your audience’s search intent.
Of course, there’s more to a successful SEO/content strategy than keyword research and search analysis—it’s also important to follow content optimization best practices. I won’t go into that topic in this article, but Wix has a helpful overview here.
How web crawling works for search engines
Now that we know what the goal is, let’s take a more granular look at the initial steps that search engines take to help us get there.
For any website to rank and appear in search results, it needs to be crawled and indexed by search engines. This is the process I mentioned in the intro of this article that makes your pages “discoverable.” We can tell from the above example that Freepik, HubSpot, and Hootsuite are “indexed” (which also means they had to be “crawled” before getting indexed) because they show up in Google search results. This is how they are able to show in the SERPs and rank for relevant keywords (in this case, free social media template).
Let’s now focus specifically on the “crawling” aspect.
Website crawling is the action that search engines perform in order to comb through websites and discover new ones.
To refer back to the previous example, if you search for free social media template, you’re likely looking for a downloadable PDF template for social media, which is precisely what we see reflected in the SERPs (as Google does its best to always show the most relevant and helpful results).
Who, or what, crawls a website?
When it comes to Google Search, website crawling is done through Googlebot, a software that reviews web pages by following hyperlinks on these web pages within known URLs to continually expand its reach.
This first step in the process of delivering search results is all about discovery. Google (and other search engines) will crawl both new and previously crawled content so that the most relevant, up-to-date content is available in the search results. The search engines then store what they find in an index in order to create a network of interconnected information.
Your pages must be crawlable for them to appear in search results. However, not all pages are meant to be crawled (for example, a “thank you” page or a site that’s still under construction) so search engines do follow certain directives from site owners to omit them from crawling.
How indexing works for search engines
Web indexing is the process of taking information that was found during a website crawl and transferring it into a database (like the Google Search index) where it is stored for later retrieval in order to generate relevant search results.
As someone who wants to improve their search visibility, it’s important to understand the relationship between crawling, indexing, and ranking. Here’s a helpful visual from SISTRIX:
Overall, as long as your website is properly set up (which it generally will be because most website creation tools allow search engines to crawl pages by default), it will be crawled and indexed.
How often do search engines crawl and index web pages?
One important thing to remember when it comes to crawling and indexing is that it takes time—it’s not always realistic to expect your pages to show up in relevant search results right after you publish them. If you consider how many websites are out there in the world, then it makes sense that search engines are constantly trying to keep up.
That’s why there’s no concrete answer on how long it takes. Depending on how big and active your website is, search engines can crawl your pages between every few days to every month.
Once a crawl starts, it can spread out over a few weeks. If you want to try to speed things up, you can try manually submitting your content for crawling and indexing in Google Search Console (GSC), but this doesn’t guarantee it’ll happen overnight.
Here’s a rule of thumb: The larger your website is, the longer it will take to crawl. Factors like crawl budget can also affect how often and quickly your website is crawled and indexed, but this generally only applies to large websites.
Checking crawling and indexing statuses for your pages
Once you’ve chosen your target keywords, created content, and are trying to rank your web page, the next step is to check that your page has been indexed and crawled so your target audience can find it on relevant SERPs. You can do this by checking Google Search Console (GSC), probably the most amazing free SEO tool there is.
Start by typing your page’s URL into the GSC search bar (at the top of the dashboard). Next, you’ll see a report that looks something like this:
In the above image, you can see that the page is indexed because there is a green check mark. So Google is able to crawl, index, and show it on relevant SERPs. If this wasn’t the case, the report will tell you so and give you some reasons as to why the URL has issues.
As a quick summary, your web page may not be indexed by Google due to the following reasons:
The newly launched site or web page has not yet been crawled by Google (which means you may need to keep waiting)
The page design hinders Google’s ability to efficiently crawl the content
Google encountered an error while crawling the page
Googlebot cannot access the page for crawling purposes (possibly because it has been marked “noindex”)
If you’d like to dig deeper into the nuances of using GSC, here’s a quick video from Google Search Central that explains how the above process works.
Note: Wix users can access Google Search Console data—including index status—from within their dashboard via the Wix Site Inspection tool.
You can’t rank if you’re not first crawled and indexed
If there’s one key takeaway you from this article, it should be this:
If your site isn’t crawled and indexed, it won’t show up in the search results. And without that, you’re losing out on potential traffic and conversions.
Of course, there’s so much more to SEO that needs to be taken into consideration over the long run. But for now, focus on creating useful content and ensuring your target audience can actually find and read it.
Originally from the US and now living in Germany, Adriana Stein is the CEO and founder of the marketing agency AS Marketing. She leads a team of multi-language SEO experts who develop holistic international marketing strategies for global companies.