Crawlers, Bots and Spiders

Search engine bots, also known as robots, crawlers or spiders, are automated software tools used by popular search engines like Yahoo, Google, Excite, MSN or AltaVista to locate and index web pages from one end of the Internet to the other—and to follow links from those web pages to find new pages. When you take a look at your server logs or site traffic analytics, you'll notice a few odd names used by search engine spiders, like Zyborg, Slurp or MantraAgent. Depending on which kind of site traffic reports you receive, you might find crawlers listed in the "Agents" portion of your statistics.

If you have an e-commerce or service oriented website, you’ll want to pay special attention to site reports showing how often various search engine’s spiders are hitting specific pages on your site. Improving your site’s rankings will depend on knowing what the spiders like and what they don’t and then feeding them more of what they like.

Six Things Aurora IT Wants You to Know About Search Engine Spiders:

• Bots, crawlers and spiders will only be able to see (read) the text on your website
• Most search crawlers won’t be able to read text generated by JavaScript
• Framed web pages are particularly difficult for search engine spiders to cache
• Spiders are programmed to register keyword density
• Cross link throughout your website so spiders will be able to probe your entire site efficiently
• Don't include hidden content, huge lists of keywords or the like. Most spiders are “smart” enough to know when you're trying to fool them and search engines will give your site a lower ranking as a result