Search engine crawlers and indexing programs are basically software programs.
These programs are extraordinarily powerful. They crawl hundreds of billions of web pages, analyze the content of all these pages, and analyze the way all these pages link to each other. Then they organize this into a series of databases that can respond to a user search query with a highly tuned set of results in a few tenths of a second.
This is an amazing accomplishment, but it has its limitations.
Software is very mechanical, and it can understand only portions of most web pages. The search engine crawler analyzes the raw HTML form of a web page. If you want to see what this looks like, you can do so by using your browser to view the source.
Once you view the source, you will be presented with the exact code for the website that the web server sent to your browser. This is what the search engine crawler sees (the search engine also sees the HTTP headers for the page). The crawler will ignore a lot of what is in the code.
Click here to continue reading..
