"spider"

Definition: spider: An automated program that reads webpages from a website, then follows the hypertext links to other pages. If the Internet is a "web", then a spider is something that follows the strands of the web. Key point: A website can use the file "robots.txt" to give hints to spiders what they should, or should not, index. A big problem with websites is that spiders are really good at finding webpages, even those that website operators don't care to be exposed. However, users can still find these pages due to hits from search engines. Website operators can therefore "hide" pages by listing them in "robots.txt". However, hackers will therefore read "robots.txt" in order to find webpages that website operators want hidden. Example: Spammers use spiders to sift through web pages looking for e-mail addresses. For example, if you have a link that looks like (A HREF="mailto:spexamp@reckoning.robertgraham.com")me(/A) then the spam spider will find the address and funnel spam to you. A partial defense against this is to URL-encode your e-mail address, which hides it from most spam spiders, but works in most browsers. See the page at http:// www.robertgraham.com/tools/mailtoencoder.html for an example. Contrast: A spider pulls information inward; a worm pushes itself outward to other systems. A spider is a type of 'bot, rather than infectious malware like viruses, trojans, or worm. From Hacking-Lexicon

* Linux/Unix/Computing Glossary