This article is about the internet bot. For the search engine, see WebCrawler. "Web spider" redirects here. Not to be confused with Spider web. "Spiderbot" redirects here. For the video game, see Arac (video game). Architecture of a Web crawler A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and