The search robot from this site will respect the following
"robots.txt" features when indexing web sites not hosted here:
- Disallow - No wildcards or "regex" expressions.
The companion directive "Allow:" is not supported.
- Crawl-Delay - Recognized but exceeding its maximum value sets
that value. Use this if the robot crawls too quickly.
- Sitemap - Relative URL's are accepted.
- X-Robots-Tag Header - This method is not supported by the search
robot. Resources on this server may have this header generated
when fetched by other systems.
- Robots HTML Meta-tag - Only "(no)index" and
"(no)follow" are supported. The "(no)archive",
"(no)snippet", and "(no)cache" extensions are
ignored (but may be supported by other search engines).
- REL=NOFOLLOW HTML hyperlink tag - Currently not supported.