20 February 2009

How does a robots.txt file help improve your SEO campaign?

. 20 February 2009 .

How does a robots.txt file help improve your SEO campaign? Is it needed? Is a search engine ranking you for EVERYTHING on your site?

A robots.txt file (inserted on the root of your website) is a file that prevents search engines from crawling your entire website. You may want to prevent certain parts of your website from being exposed and searchable to the world. Additionally, as it relates to SEO, several folders and files may not be related to the website’s targeted keywords.

Without a robots.txt file, a search engines assumes it can crawl your entire site for content.

A robots.txt file can be created with a simple notepad editor. Following the standard format, you can choose what files and folders to exclude. You can additionally use a robots.txt file to tell search-engine spiders were your sitemap is located.

You may find that there are some online robots.txt generators that will do the job for you. You can google ‘robots.txt generator’ to find one. After generating your robots.txt file, you would need to save this file and upload, via ftp or your host control panel to the root of your website’s folder.
Simple example of a robots.txt file:

User-Agent: *

Disallow: /images/

Disallow: /directory/file.html

As always, it is recommended that may want to check with some experienced help on how to setup your file correctly. SEO(search engine optimization ) is a gimmick if not applied correctly. Every file, directory, and image of your website needs to be carefully reviewed for content in order to determine exclusions.


prioSathi.com said...

Good post.


peterK said...

Great and very useful article. Great recommendation. Cheers for sharing.

Anonymous said...


Mac said...

There are several link building packages out in the internet so choose the most suitable for you. Everything in the SEO world is dynamic, methods might work today but we can never vouch its effectiveness on the next day. You need to hire a good company that offers and has vast
knowledge of SEO/SEM.

SEO Boston

:)) ;)) ;;) :D ;) :p :(( :) :( :X =(( :-o :-/ :-* :| 8-} :)] ~x( :-t b-( :-L x( =))

Post a Comment

SEO TIPS is proudly powered by Blogger.com | Template by Friendly81blog