Introduction to Web Robots, ROBOTS.TXT, and RoboGen Search engines such as Excite and AltaVista use web spiders, also known as robots, to create the indexes for their search databases.  These robots transverse HTML trees by loading pages and following hyperlinks, and they report the text and/or meta-tag information to create search indexes.  ROBOTS.TXT, a file that spiders look in for information on how the site is to be cataloged.  It is a ASCII text file that sits in the document root of the server.  It defines what documents and/or directories that confirming spiders are forbidden to index. The robot exclusion protocol was introduced by Martijn Koster in 1994 to deal with problems that had been arising due to the increasing popularity of the internet and the toll web spiders were having on system resources.  Some of the problems were caused by robots rapid-firing requests, that is loading pages in rapid succession.  Other problems such as robots indexing information deep in directory trees, temporary information, and even accessing cgi-scripts.  The robot exclusion protocol was quickly adopted by webmasters and web robot makers as a way to organize and control the indexing process. Since then, the size of the Internet has increased dramatically and millions of people are using it.  The number of web robots crawling the web is greater than before and it is more important than ever for all web sites to have a properly created and maintained ROBOTS.TXT file. With RoboGen you create robot exclusion files by selecting All Robots or a specific user-agent and adding documents and/or directories by entering the path names manually or by selecting them using FTP.  Once all the restrictions and directives are set you can save the robots.txt file to your hard drive or upload it directly to your server. It is important to remember that robot exclusion files are not a security measure.  Some robots will simply ignore the file and others may purposely load the documents that the files marks as disallowed.  This means that robot exclusion files are really only useful for controlling what appears in search engines. RoboGen Product Overview RoboGen is a visual editor for Robot Exclusion Files.  It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made searchable by the general web public.  RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable. For more information on RoboGen and robot exclusion files, see  Introduction to Web Robots, ROBOTS.TXT, and RoboGen. Feature Overview Create Robot Exclusion Files by selecting documents and directories. Log into FTP servers and even upload robots.txt from RoboGen. Manage information for multiple servers. Database of over 180 know user-agents and 10 major search engines. Ability to edit the robot database and add additional user-agents. RoboGen recognizes common document roots when logging into FTP servers and suggests that they be set as the document root. For a detailed overview of RoboGen's features, see the RoboGen Feature Comparison Chart. Pricing Base Cost: $24.95 for a single-license. Shipping and Handling: Additional $5.75 for CD-ROM delivery. $0.00 for Download Only delivery.