RSS feed for this blog

Robots Exclusion Protocol leaves ACAP dead in the water

By Paul Watson on .

In my first post on this blog I wrote (critically) about ACAP – a thoroughly wrong-headed attempt by some publishers to enforce stringent limitations on the way search engines index the content that publishers make public on their websites.

Today ACAP is completely dead in the water.

Google, Yahoo and Microsoft (see those links for details) today jointly announced their backing for the existing Robots Exclusion Protocol (REP) which comprises robots.txt, the Sitemap protocol, and individual page meta elements.

The people behind ACAP are probably still claiming that they have the backing of the world’s 4th largest search engine Exalead, but in the words of Bill Hicks “Yeah, maybe, but you know what, after the first 3 largest armies search engines, there’s a REAL big fucking drop-off.”

The decision by the three big search engines to back the existing REP standard—and to clarify exactly how they implement it—is a great example of these three competitors working together to the benefit of both website owners and searchers.

UPDATE – 4th June 2008: I just heard from a colleague that the whole ACAP debacle could have been avoided.  ACAP was primarily conceived as a way to convey rights/permissions metadata when feeding data from one partner organisation to another (for example, from a publisher to Amazon).

For some unknown reason the people behind ACAP decided to try to roll it out as a website technology.

This was obviously a huge strategic error, and it backs up my belief that the people behind this technology just don’t get the web. As a protocol for communicating permissions information from a publisher to Amazon or Google Books (not Google Search!) in a data feed it’s probably fine.  But ACAP has no place on the web.

What prompted the people behind ACAP to try to force it onto the web is unimaginable.  This ill-conceived idea was doomed from the start, especially when combined with their secretiveness (they have a forum on their site, but it’s hidden from view and they only give out logins to selected partners) and their attitude when replying to the tidal wave of criticism they received from bloggers.