# Tells scanning robots where they are and are not welcome. # # Last modified Aug. 11, 2005 by J. Jeffry Howbert. # # # User-agent: can also specify by name; "*" is for everyone # Disallow: disallow if this matches first part of requested path # # For now disallow all access to cgi, but allow access to rest of pages. # Can modify this as needed to allow certain crawlers. # User-agent: * Disallow: /cgi-bin/