Comments on: Indexing Search Results: Why Google Is Going Against Its Own Advice

By: David Lindop

David Lindop — Tue, 06 May 2008 08:00:40 +0000

I don’t quite get the logic there Chris, but I think I’m going to run some specifity tests of my own with different robots.txt, meta instructions, in-site SERPs and inline nofollows.

Man, do I know how to party!

By: Chris Dimmock

Chris Dimmock — Tue, 06 May 2008 06:05:42 +0000

Ciaran,
You can’t use a robots.txt AND a meta noindex together.

If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page (so it never reads the meta noindex).

If you allow a page via robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.

If you have both - the robots.txt trumps the meta noindex - as the bot won’t parse/ visit the page, to be able to read the meta noindex…. so you get a ‘thin’ result…..

i.e. exactly what you are seeing here - 105,000 thin results

Robots.txt = thin result
Meta noindex = no thin result - nothing
Use both = thin result

By: Ciaran

Ciaran — Mon, 05 May 2008 22:03:27 +0000

You’re right (obviously!) but why should they have to?
Google says “don’t let us find search results” and then finds a way to do just that.

Err……

By: Joost de Valk

Joost de Valk — Mon, 05 May 2008 21:50:58 +0000

btw the fix is pretty simple, change the search query from a get to a post, and it’s all done

By: Ciaran

Ciaran — Mon, 05 May 2008 21:40:30 +0000

Joost - it does seem like we (although I don’t work there any more) were rather naive to believe that following Google’s guidelines would mean that Google did what we asked.

I have to disagree with your second statement though; they’re not being linked to - they don’t exist unless you use the search box. And this is something that Google has only just started doing, and goes completely against their own guidelines.

By: Joost de Valk

Joost de Valk — Mon, 05 May 2008 21:10:16 +0000

The solution is pretty darn easy: get rid of that robots.txt line… Doing both is not double protection, it just shows that you don’t know how it works Google has ALWAYS indexed links that it wasn’t allowed to spider, for various reasons, and though I don’t agree with them doing that either, it’s how things work.

And even though you’re throwing the blame at Google now, I would seriously wonder how come 105,000 INTERNAL search result pages on that site are being linked to, apparently without a nofollow condom, thus loosing loads of link juice… What benefit is it to ANYONE to link to 105,000 internal SERPs?

By: David Mihm

David Mihm — Mon, 05 May 2008 20:15:20 +0000

Brent, get your mind out of the gutter!
Ciaran, nice find & well-written piece.

By: Brent D. Payne

Brent D. Payne — Mon, 05 May 2008 16:59:17 +0000

What a huge black hat opportunity. More or less, what Google’s Webmaster Forum is stating (linked to above from Ann Smarty) is that if you disallow a page it can still be indexed based off of the inbound anchor text and relevance of the pages pointing to that page. Googlebot is not allowed to view the page at all (and in this example can’t determine it has a noindex tag). If Google can’t view the page at all then you could theoretically rank really well on something that has absolutely nothing to do with the page. Imagine the possibilities. Linkbait widgets that have anchor text of xyz that point to an abc page that is disallowed and thus Googlebot has no clue that the page is about abc versus xyz.

I miss the good ole days . . . too bad my hat has to be white now. This could’ve been a lot of fun.

Brent

By: Ciaran

Ciaran — Mon, 05 May 2008 16:56:45 +0000

Ok Ann, I’ve been thinking some more about this.

The search results pages in question are not being indexed because they have been linked to from another site (at least according to a quick look at Yahoo Site Explorer); they are being indexed because Google has decided to index the deep web. Therefore any semantics about crawling, indexing or anything like that is just so much rubbish - they’re indexed because Google wants them to be, and nothing the site owner says is likely to make the slightest bit of difference.

By: David Lindop

David Lindop — Mon, 05 May 2008 16:34:42 +0000

Cheers for the link Ann, that’s a good way of putting it.