Tell the Robot what to do!


What is a Robots.txt file...?

Search engines generally use robot crawlers to locate searchable pages on Web sites and Intranets (robots are also called crawlers, spiders, gatherers or harvesters). These robots, which use the same requests and responses as Web browsers, read pages and follow links on those pages to locate every page on the specified servers. Only robots which comply with the Robots Exclusion Standard will read and obey the commands in this file. Robots will read this file on each visit, so that pages or areas of sites can be made public or private at any time by changing the content of robots.txt before resubmitting to the search engines.


...And why do I need a Robots.txt file?

With the Robots.TXT Generator from SEODIY you create the code which will actually allow the robot to scan specific pages of your Web site or disallow the robot to scan certain others.

There are a number of reasons to invite a spider to scan your pages:
- You have a Web page with many products and you are frequently adding new merchandise.
- You would want the spider to scan your pages more often to reflect your inventory.
- You are a News or Newsletter providing web site and need the spider each day to scan your pages.
- You have special events coming up and want your Web audience know about it.

There are a number of other reasons where you may wish to exclude spiders from some or all of your pages:
- You are building the site, or certain pages, and do not want the unfinished work to appear in search engines
- You have information that is of no interest to anyone but those it is intended for.
- Most people will have some directories they would prefer were not crawled! Example: cgi-bin.
- You would like to exclude some robots altogether, for example those collecting e-mail addresses.
- If you are using a Web site sitting in a frame you do not want a robot to scan parts of your frame.

...And how Can I Create A Robots.txt File?

Our SEODY Robots.txt Generator will create the Robots.txt file for you. As a SEODIY subscriber you will be able to use our automated Robots.txt tool as well as all other Web site promotion tools available.