# GeektecK's web crawler direction file (robots.txt).
#
# Indexing this site is not a problem, but please be reasonable in the number
# of requests your crawler makes at one time or it may be permanently blocked.
#
# USAGE:
# Allow: (nonstandard extension) Possibly counteracts following 'Disallow'
#   directives, but not previous ones (the first rule match wins).
# Crawl-delay: (nonstandard extension) Suggests the number of seconds to wait
#   between successive requests to the same server.
# Disallow: Suggests directories that should not be indexed. Hacker bots will
#   specifically look in these directories.
# Request-rate: n/s : (nonstandard extension) Maximum rate of n pages every 
#   s seconds.
# User-agent: How the crawler identifies itself. * is all crawlers.
# Visit-time: xxxx-yyyy : (nonstandard extension) Only visit between xxxx and
#   yyyy UTC (GMT). 1:00 AM to 5:00 AM CST is 0700-1100.
#
# Last edited: 03/10/2009 -Pointman
# ----------------

User-agent: *
Crawl-delay: 10
Disallow: /cgi-bin/
Disallow: /disclaimer/
Disallow: /download/
Disallow: /ebay/
Allow: /external_host/smokeysmuskieshop
Disallow: /external_host/
Disallow: /img/
Disallow: /image/
Disallow: /images/
Disallow: /olofson/
Disallow: /resource/
Disallow: /scripts/
Disallow: /style/
Request-rate: 1/10
Visit-time: 0700-1100