While optimizing a website for search engines, prime focus is on developing ‘search engine friendly’ content. But, what does the term search engine friendly mean? It means that crawlers should be able to scan and index your website easily and organize it in relevance to keywords. But, how does a spider understand whether a webpage should be crawled or not? Sitemap and Robots.txt give directives for search engines and hence, they can be effectively used for SEO. What is Robots.txt? Robots.txt is a plain text file that is uploaded to the root directory of a website. Bots or spiders first look fo...
7406 Hits