Google Hates Wiki

Google giveth, and Google taketh away.

Right now GoogleLovesWiki, pretty much. See:

See GoogleSearch for a c2 Wiki search experiment performed 2005-04-25.

As of June 2005, it appears that Google doesn't appear to love Wiki very much any more. Pages that used to rank highly in Google's index no longer seem to. I don't know if this is due to some new change in how PageRank is determined, a change in c2's characteristics that have resulted in it being identified as a LinkFarm, or a failure (for some reason) for the Google 'bot to index c2, or what. Yahoo also seems to not rank c2 highly anymore; search.msn.com still ranks pages here highly.

Jul05 GoogleLovesWikiNot

I can no longer do C2 research using Google. Lots of pages are not found. :(

But YahooLovesWiki.

Google, Yahoo, and MSN Search tried 2005-08-27:

Either our c2 Wiki is broken or GoogleSearch has come up with a Wiki-unfriendly algorithm for indexing pages.

Go to Google and search, for example, for "site:c2.com microsoft slave". When there's a space between 'microsoft' and 'slave', Google should turn up the Wiki page whose title is 'Microsoft Slave'. In fact, I bet that page should show up first in the list. It doesn't show up at all. All that show up are a few "Signatures" pages, and signature pages are really quite useless from a Google-user standpoint. You'll also notice that all the entries are "Supplemental Results." That means they're coming from old indexes Google has lying around, not from Google's most recent attempt at indexing Wiki. If you look at each "Cached" page you'll see the date that each useless signature page was indexed. They're all at least a few months old instead of just a few weeks old. I don't think I've spotted anything newer than March 2005 in the GoogleCache.

So try searching for "site:c2.com microsoftslave". Don't you suppose you should see all the pages where dl signed himself as MicrosoftSlave? You don't. The first page in the list is--guess what?--the MicrosoftSlave page. That's nice, but there's no text summary and no cache. I guess it means the page was found but not indexed, and who knows when it was found. The search also turns up a few useless "Signatures" pages, a few old "Changes in ..." pages, and our "Long Pages" script. This useless smattering of extra results are all "Supplemental Results," i.e., they're old.

Dogpile (which also reports results from Yahoo, MSN Search, and Ask Jeeves) barely provides much of interest, either. MSN Search seems to have a fair amount of success, and its results are from just a couple weeks ago.

Maybe Google has been up to something in the past few months. On the other hand, the new meta robots HTML tag we've been inserting in pages might be causing some search spiders not to index our site frequently with useful results. That at least would explain why the MicrosoftSlave page was found by Google but not indexed. A meta tag is only a tag. There's no guarantee any given SearchEngine will pay attention to it. Maybe MSN Search ignores it.

I'm in favor of eliminating the "meta" tricks for a couple months and see how Google responds. We haven't been blasted by spam in some months now, anyway.

-- ElizabethWiethoff

Google searched 2005-08-28:

I'm going to go ahead and be vain here. Do a GoogleSearch for "elizabethwiethoff site:c2.com". Besides some ancient "Changes In ..." pages and "Signatures" pages, Google turns up little of interest (OrientedVsOrientated, JonThoroddsen, ScAza?, AnandJain), and all results are "Supplemental Results", i.e., old. Now try searching "elizabethwiethoff site:bookshelved.org". Google turns up 27 interesting, useful pages, none from its "Supplemental Results." That means the pages were indexed recently, and checking the GoogleCache you can see they were indexed within a couple weeks.

If Google changed its algorithm for indexing wikis, I believe its BookShelved results would be just as pitiful as its c2 wiki results. Therefore, I think it's c2 wiki that's broken. Our problem might be our "meta" tricks. Or maybe our IP blocks block out Google!

-- ElizabethWiethoff

Good point. BTW I've noticed the kind of problem you've been discussing for quite a while now; it's really cramped my style on searching c2. -- DougMerritt

2005-08-28: I haven't spotted anything from our Wiki in the GoogleCache more recent than March 10, 2005. I think Wiki's been broken somehow--maybe blocking Google's spider 'bots (and Yahoo's)--since shortly after March 10. I'm sending a note to Ward... -- Eliz

It must be the wiki itself. The c2-site alone is well indexed and has a good PageRank (I mean the index.html and the html pages linked from there). But all Wiki-pages (e.g., http://c2.com/cgi/wiki?PortlandPatternRepository) have PageRank zero. Can it be, that

-- .gz

An interesting page to read at Google: http://www.google.com/intl/en/webmasters/index.html

One possibility is that c2.com, during a spam attack, managed to get on google's naughty list; Ward might inquire at google if c2 is indeed so.

I was at Google on August 29, 2005, talking to a friend. He said the Google spiders always run around as fast as they can. Another friend (not a Google person) said that you're supposed use the RobotsDotTxt file to limit the speed. In any case, there's nothing about speed in http://c2.com/robots.txt. But the Wiki server has a few roadblocks to flag and stop speeders. (See the MoreAboutWikiAccess pages, particularly WikiAccessDenied.) Maybe some months ago the Googlebot started running too fast for Ward's roadblocks and he inadvertently added it to the banned IP list. -- ElizabethWiethoff

I notice that the Yahoo search results of Wiki are as pathetic as Google's. I bet the Yahoo bot was inadvertently banned as well.

Yahoo tells you how to use RobotsDotTxt to slow down the crawl rate: http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html You specify a "Crawl-delay" for the search spider user agent. This crawl delay directive in robots.txt was invented by MSNSearch (EmbraceAndExtend!) and is honored by msnbot, Slurp (Yahoo), and Teoma (Ask Jeeves). Google, as far as I can tell, does not honor it.

Results of an interesting study done in the spring of 2005. The major search spiders behave quite differently with RobotsDotTxt: http://rss.publish101.com/articles/pages/slurp/teoma/site/search/Mike-Banks-Valentine/112391/

-- ElizabethWiethoff

GoogleHatesWiki may have something to do with procedures Ward have applied to his server since Feb05. It creates lots of difficulties in various wiki related tasks that I participate.

Example: MakeRoomForAllViewpoints difficulty

I scan a lot of articles / news items. The ones I think could be of interest to multiple persons are logged in a C2 page. Sometimes, I come across a viewpoint that may be different from the ones expressed and could be useful. Without a Google search, it becomes much more difficult for me to track down where I saw the first link. This is especially true for a large topic like WebServices that have many pages. -- DavidLiu

GoogleHatesWiki was first noticed by me (in C2 pages terms) in May05. I mentioned it to Ward around then but I did not get any feedback. My experiences in Apr05 appeared fine. DM said he noticed the same maybe a bit earlier. I do think a sharp deterioration occurred in May because I use Google search on C2 many times a day.

In Aug05, other C2 members start complaining about it. So I made another note on Wards home page. Hope something positive come out of this.

I too noticed it quite a while before people started talking about it in August, although I don't recall quite when. Probably most people assumed it was something temporary... for a long time... before speaking up. Multiple people have by now mentioned that they notified Ward.

Problem is, Ward is notoriously so busy with multiple things that it takes him typically a minimum of months (sometimes years, if a solution suitable to his philosophy and aesthetics is not obvious) to find the time to do things to c2 even when he's indicated they are high priority for him.

So I think we must be prepared to exercise patience. -- DougMerritt

That's funny. It never occurred to me that Ward might be at fault. I assumed that it was an algorithm change on the part of Google (I distinctly recall them intending to limit the page ranking of blogs) and flamed them accordingly (though nicely). Oh well, I hate the way robots.txt is abused anyways; I don't think I'm willing to take any excuse for a site not being archived. -- RK

Aside from side issues, yeah, I agree. I think it must be a bug on Google's part, even if it turns out that Ward made some kind of Google-flagging-meta-tag mistake. To be fair, questions of meta tags and robots.txt have in fact been critical arguments in court cases that have set a certain amount of legal precedence, so I think Google should indeed be concerned - but I would still think that Google merely screwed up, here. -- Doug

Well, the page HEAD says ROBOTS NOINDEX, and from my understanding that means "robots, go away!". -- ClaesWallin

See DelayedIndexing for the rationale for that.

Perhaps there is a bug in Google such that once shooed away it never comes back even if the no-index flag goes away.

Google hates "Ward's Wiki" only. 

 See: http://www.google.com/search?q=GoogleHatesWiki&hl=en&lr=&filter=0
So, the moral of this little debacle is: If you want what you have to say to show up on the net, put it on another wiki! ;)

If we want to fix the Google problem, then all we have to do is create a mirror of this wiki with links back to original, eh?

According to this tool, there is no problem:

http://www.123promotion.co.uk/tools/googlebanned.php

Perhaps it's not that GoogleHatesWiki, but that WikiHatesGoogle?. A few months ago I did a Google search that successfully coughed up a WardsWiki page based on the page title, but instead of page content the abstract contained an error message generated by WardsWiki. The error message (which unfortunately I can no longer find) suggested to me that one of Ward's anti-abuse measures was being triggered by the Google spiders' high volume of hits within a short period of time. -- DaveVoorhis

I suspect that GoogleHatesWiki will eventually kill the c2 wiki, or at least stagnate it. People looking for something usually use Google first.

I think that both too much publicity and too little publicity could kill Wiki. Too much publicity brings in lots of trolls (probably the reason google is no longer indexing Wiki). If there is too little publicity there won't be enough people to replace those who leave for whatever reason. But google isn't the only way to bring people to Wiki. -- MichaelSparks

I avoid Microsoft anything and this doesn't fix the Google/Yahoo problem, but at least MSN Search (http://search.msn.com/) still indexes this wiki. You could try it for searching the Wiki. -- Eliz

The page creation rate has been dropping - sometimes below zero - since 30,000 pages. See WikiAtFortyThousand. I suppose the instances of negative growth may be from the RA MindWipe and moving some stuff to TheAdjunct. The slow-down in positive growth might be due to lack of Google & Yahoo, might be due to lack of RA spewing pages right and left, might be due to people leaving in disgust during the 'bot wars, take a guess... FWIW, I found Wiki a couple years ago from googling (with Google) for "functional programming". I was also led here from googling for "design patterns". These searches were about a couple weeks apart, and I don't remember which I did first. My point is, I wouldn't be here were it not for Google (and the fact that Wiki contains loads of interesting pages). -- ElizabethWiethoff

"This is G o o g l e's cache of http://c2.com/cgi/wiki as retrieved on Mar 25, 2006 09:14:23 GMT."

The only pages found were: FrontPage and http:wikiRss. I believe the problem started when DelayedIndexing was implemented.

There might be useful information to be had from https://www.google.com/webmasters/sitemaps/sitestatus . I made a brief attempt to get something useful, but it (justifiably) requires proof that I have control over the piece of the site I'm requesting information on. Apparently I'm not special enough to do that :p. --WilliamUnderwood

June 2007, and Google still hates Wiki. As a result, this site is pretty much in torpor.

Million-dollar companies have similar problems:

http://www.forbes.com/home/technology/2007/04/29/sanar-google-skyfacet-tech-cx_ag_0430googhell.html

I think that the problem is the "?" in the URLs. I've been told at work that google does not index URLs with question marks anymore.

October 2007 and GoogleLovesWiki! Vanity search for site:c2.com elizabethwiethoff turns up 113 real pages cached on 2007-10-07.

I've noticed more C2 hits also of late. Apparently Google tweaked a knob somewhere.

Inspection of Wiki's HTML source reveals the use of a GoogleAnalytics? urchinTracker() JavaScript function. Perhaps Ward added this when he added the EditText PNG a few months ago. I imagine signing up for GoogleAnalytics? forces Wiki to get indexed no matter what the algorithm is inclined to do otherwise. -- Eliz

Clever, that Ward.

If Google indexes WardsWiki through proxy hacks, the original site and the hack sites are considered duplicate content and the duplicate content gets dropped from the index. See "Google Proxy Hacking: How A Third Party Can Remove Your Site From Google SERPs" (http://www.seofaststart.com/blog/google-proxy-hacking).

For exercise, try searching for dougmerritt earlemartin (sorry, guys). On 2007-10-17, sixteen pages show up in the Google search results. Many of them are legit returns from wikis where they both hang out. A couple pages are from the Simplewebs (see WhatIsSimplewebs) copy of WardsWiki:

Most of the rest are from obvious proxies: One of the sites appears to be a legit attempt at archiving, but to play nice the archive really shouldn't get indexed: -- ElizabethWiethoff

2008 October and GoogleHatesWiki again.

Hmmm. Fickle, like an Earth female.

2008 November 5

Eight single facts and a question, viewed from a far point of view

-- BobTait?.

CategoryWiki

EditText of this page (last edited December 19, 2013) or FindPage with title or text search