1. standard for robot exclusion Web> A proposal to try to prevent the havoc wreaked by many of the early World-Wide Web robots when they retrieved documents too rapidly or retrieved documents that had side effects such as voting. The proposed standard for robot exclusion offers a solution to these problems. http://info. webcrawler. com/mak/projects/robots/norobots. html
standard for robot exclusion |
