DTIbot is the official name given to the collective web crawlers we have and will place in service.
If you are visiting this page because DTIbot accessed your web site, please be assured we make every effort
to limit our use of your bandwidth.
Our web crawler is and will be used in connection with current and future services.
Webcrawler
Our current objectives:
Run live testing of DTI Site Search.
Build our experience in navigating our web crawlers across the Internet.
Support
We understand your bandwidth and server time are valuable. It is important for us that our bot be welcomed on the internet as a well-behaved crawler. If you notice any unusual activity from DTIbot please report it to:
Access
If you have a problem or concern about DTIbot we would prefer to have the chance to address it but if you need to block DTIbot we do respect the robots.txt exclusion list.
To block DTIbot from some parts of your site you can use the following example:
In this example, /logs/ and /cgi-bin/ are directories that will be blocked to DTIbot and will not be crawled. Other parts of your site will still be crawled.
To block DTIbot from your entire web site you can use this:
User-agent: DTIbot
Disallow: /
Site Index
If DTIbot has crawled your site and you would like to view what has been indexed, visit our search page and enter "site:yourdomain" as your search parameters.
We periodically purge our index in preparation for the next round of testing, so search results for your site may return empty.
Sitemap.xml
DTIbot supports the sitemap protocol.
Note: The sitemap feature at this time is being implemented by DTIbot at random.
You can specify the location of your Sitemap using a robots.txt file.
To do this, add the following line:
