Understanding how search engines crawl and index content in 2023

Robots.txt files are located in the root directory of your site and provide guidance on how search engines should crawl a site. Basic robots directives allow you to prevent search engines from crawling certain sections of your site, as well determining how fast they crawl the site. More specific rules can be created to apply to specific web crawlers.

Different search engines treat robots’ rules in different ways and some may choose to ignore them. To see how Google treats robots.txt files you can refer to their guidelines.

The robots directives are a signal to instruct search engines on how a page should be indexed without preventing it from being crawled. This is most commonly used to instruct a search engine not to index a particular page in the form of a no-index tag.

This can be implemented in two different ways, via the source code using the meta tags or using the X-Robots-Tag in the HTTP header.

A canonical tag is an HTML tag that defines the ‘true’ version of a page, helping search engines understand which page to show in its index. This is just a signal, however, and search engines like Google may choose to ignore it. This means they will still crawl the page, but they will just be less likely to index it.

The hreflang tag is used to inform search engines of the country and language of your site. Search engines won’t inherently determine if a site written in English is intended for the US market or vice versa. In the case where your site exists in multiple countries with similar content, hreflang tags can help define which users should be sent to which variation and prevent duplicate content issues.

An XML sitemap is essentially a document listing all the important pages of your site that you would like search engines to crawl. This helps search engines find your pages faster and means your site doesn’t rely as heavily on a strong internal linking structure.
The limit to a sitemap is 50,000 URLs, so for larger sites these are often used together with separate sitemaps for blog content, product pages and support content. It is recommended to reference all XML sitemaps in the robots.txt file and to submit them via Google Webmaster tools.

Google provides tools to help their spiders understand your website. One of these tools is the ‘URL inspection tool’ in Google Search Console, which shows the status of the page and whether it has been indexed. If it’s not indexed, you can request Google to do so. This gives Google a clear signal about your desire to have the page appear in results and will prompt them to recrawl it.
It is recommended to use this sparingly. It is designed to be used in cases where a problem has occurred removing a page from index, and the issue needs to be resolved quickly.

Many sites will serve slightly different content by appending different parameters to a URL. This is commonly used with faceted navigation and search filters. Google will aim to identify the representative URL on its own, but if there are specific parameters that need to be excluded, they can block them directly in the Google Search Console. This prevents the Googlebot from crawling them altogether.

HTTP authentication blocks content behind a log-in portal, which will stop both users and search engines from being able to access it. This is useful in a number of scenarios including building a staging website to test what a page will look like once live or adding sensitive information that you only want approved users to see.

Now we’ve covered the crawling and indexing, why not explore the rest of our technical SEO hub?