Search engines are complex computer programs.
Before they even allow you to type a query and search the web, they have to do a lot of preparation work so that when you click “Search”, you are presented with a set of precise and quality results that answer your question or query.
What does ‘preparation work’ include? Three main stages. The first stage is the process of discovering the information, the second stage is organizing the information, and the third stage is ranking.
Search engines have a number of computer programs called web crawlers (thus the word Crawling), that are responsible for finding information that is publicly available on the Internet.
To simplify a complicated process, it’s enough for you to know that the job of these software crawlers (also known as search engine spiders), is to scan the Internet and find the servers (also known as webservers) hosting websites.
They create a list of all the web servers to crawl, the number of websites hosted by each server, and then start work.
They visit each website and by using different techniques, they try to find out how many pages they have, whether it is text content, images, videos, or any other format (CSS, HTML, javascript, etc).
When visiting a website, besides taking note of the number of pages they also follow any links (either pointing to pages within the site or to external websites), and thus they discover more and more pages.
They do this continuously and they also keep track of changes made to a website so that they know when new pages are added or deleted, when links are updated, etc.
If you take into account that there are more than 130 trillion individual pages on the Internet today and on average thousands of new pages are published on a daily basis, you can imagine that this is a lot of work.
Why care about the crawling process?
Your first concern when optimizing your website for search engines is to ensure that they can access it correctly otherwise if they cannot ‘read’ your website, you shouldn’t expect much in terms of high rankings or search engine traffic.
As explained above, crawlers have a lot of work to do and you should try and make their job easier.
