Overview
DTibot is the official name given to the collective web crawlers we have and will place in service.
If you are visiting this page it's probably because DTibot accessed your website.
Please be assured we make every effort to limit our use of your bandwidth.
Our web crawler is and will be used in connection with current and future services.
Web Crawler Objectives
Our current objectives:
DTi Site Search Beta Testing.
Build our experience in navigating our web crawlers across the Internet.
Support
We understand your bandwidth and server time are valuable. It is important for us that our
bot be welcomed on the internet as a well-behaved crawler.
If you notice any unusual activity from DTibot please report it to: support@dorn-technologies.com
Access
If you have a problem or concern about DTibot we would prefer to have the chance to
address it but if you need to block DTibot we do respect the robots.txt exclusion list.
To block DTibot from some parts of your site you can use the following example:
User-agent: DTibot
Disallow: /logs/
Disallow: /cgi-bin/
In this example, /logs/ and /cgi-bin/ are directories that will be blocked to DTibot
and will not be crawled. Other parts of your site will still be crawled.
To block DTibot from your entire web site you can use this:
User-agent: DTibot
Disallow: /
- Prev
- Next >>
