I've been interested for some time in how the big search engines use metadata. Of course, we all "know" that they are wary of using it, because of the potential of spamming. But I wanted to put this to the test.
I decided that the best way would be to seed one of my pages with metadata that contained terms that were not misleading, but would be sufficiently unusual to be identified easily in a search. The page that I chose was Maturin's Medicine and my first test was based on Description metadata:
<meta NAME="Description" CONTENT="An index compiled by Webboe to medical terms in the Aubrey Maturin books">
If the search engines picked this up, I should be able to look for "webboe". The results are listed below. What I hadn't expected was that some of the search engines would use this information in displaying the search results.
| Search Engine | Indexed | Displayed |
|---|---|---|
| MSN | Y | Y |
| AltaVista | Y | Y |
| Hotbot | Y | Y |
| Alltheweb | N | Y |
| N | N | |
| Lycos | N | N |
| Teoma | N | N |
| Wisenut | N | N |
| Gigablast | N | N |
Following this test, I tried another metadata tag: keywords. Using the same page, I inserted the following META tag (again with valid but unusual terms):
<meta name="Keywords" content="marthambles; basra">
This was less successful. After six weeks, none of the search engines listed above has used this data to index the page.
Update - 25 October 2003
I've noticed two things today - MSN / Hotbot has indexed the contents of the Keywords tag described above, and it has stopped using the contents of the Description tag in the display of the search results. Instead, it is using a variation on the text that appears at the beginning of the page.
Update - 10 April 2004
I did a search on the new Yahoo! facility. Based on my sample, it seems that it indexes words in the Description and Keywords tags. And MSN has gone back to using the contents of the Description tag in the search results list.
I'd be happy to discuss any aspect of this. Feel free to contact me at kwebb@alianet.alia.org.au
Kerry Webb
Canberra, Australia
October 2003