If you’ve been learning SEO for a while, you may have heard the terms crawling and indexing. They come up a lot whenever experts speak about SEO, which only proves their importance. First thing first: robots.txt is the file that resides in your root directory of a website. The robots.txt file gives instructions to the search engine crawlers as which page they can crawl and index during the crawling and indexing process. The first thing they do is look for the robots.txt file. After checking the content of the file, then creates the list of URLs they can crawl and index of the website.

If your website doesn’t have a robots.txt file then, search engine crawler who will visit your website will automatically assume that all the pages whether they are working or not can be crawled and indexed.

5 things robots.txt improves the situation SEO execution –

• Tells google to  find pages to crawl and not to crawl.
• Prevents duplicate content to appear in SERP resluts
• Helps in mainting the privacy of the site.
• Prevents search engines from indexing the hidden/private content of the website.
• Prevents from overloading of crawls.

Search engines work through three primary functions:

1. Crawling: Scour the Internet for content, looking over the code/content for each URL they find.
2. Indexing: Store and organize the content found during the crawling process. Once a page is in the index, it’s in the running to be displayed as a result to relevant queries.
3. Ranking: Provide the pieces of content that will best answer a searcher’s query, which means that results are ordered by most relevant to least relevant.

Crawling is the discovery process in which search engines send out a team of robots (known as crawlers or spiders) to find new and updated content. Content can vary — it could be a webpage, an image, a video, a PDF, etc. — but regardless of the format, content is discovered by links. If you used Google Search Console or the “site:domain.com” advanced search operator and found that some of your important pages are missing from the index and/or some of your unimportant pages have been mistakenly indexed, there are some optimizations you can implement to better direct Googlebot how you want your web content crawled. Telling search engines how to crawl your site can give you better control of what ends up in the index.