 |
|
ROBOTStxt |
|
|
|
| |
|
ROBOTStxt is an editor for robots.txt. Robots.txt files are used to direct bots which pages to fetch from a website.
More information regarding robots.txt can be found on the wikipedia article.
|
Features:
- Edit Robots.txt
- View Robots.txt
- Open Robots.txt from WWW
|
How To:
This application is a very simple application to edit or view robots.txt files.
You can either open an existing robots.txt file (either from a remote location on the www, or from your local disk).
You can then start adding entries. The most important entries to add are the User-agent which identifies a user agent (use * to match all) and any Disallow paths.
To delete an entry simply right click on it and click on delete. A sample robots.txt file can be found over here. Please note that robots.txt file should
reside in the root directory of your website and should be named robots.txt for the bot to find it.
|
| |
Main Window:
|
Requirements:
- Windows 2000, Windows XP, Windows 2003, Windows Vista, Windows 2008, Windows 7.
- Microsoft .NET Framework 2.0 Download
|
| |
Any ideas, comments or feature requests contact me on email ian@iannet.org
Ian www.iannet.org
|
Copyright © iannet 2010
|
