Indexing: The knowledge gathered from the crawlers is arranged and stored in a large databases identified as an index. They can be utilized by search engines to search out not long ago updated material together with to index it making sure that end users can find it when they do a search. Crawlin