The Sorry State of Blog Search Engines

Maybe there's no money it. Maybe there's no love to be gained from it from bloggers. But blog search is in a pitiful state right now. There's room for someone to come along and innovate.

Now you can argue that blog search doesn't matter as much these days. Twitter and Friendfeed both have strong search capes. The main Google engine scoops up blogs as fast as Elvis ate peanut butter and banana sandwiches. In addition, paid tools like Radian6 do a very nice job. But, darn it, I grew accustomed to having good blog search FOR FREE. And now it appears to be gone.

Here's a rundown of the contenders and what's currently right/wrong with each. Also, I put each site through its paces by tracking links to my recent post on WIkipedia Mobile.

Technorati - Link searches pull up both blog rolls and mentions. Please give me a choice. Technorati seems to be more focused on rolling up bloggers into ad networks rather than on core search. They are leading right now in results, however. (Links to my post: 18. RSS feeds? Busted)

Google Blog Search - For awhile, Google gave me everything I wanted. Fast results on both keywords and link searches. But then a few weeks ago the plumbing broke. Now it's practically useless because it alerts me anytime a blogger who has me on their blogroll updates his/her site. (Links to my post: 9)

Twingly - They offer spam free results. And they look promising so far. But when you put Twingly through its paces, it is very weak on results. (Links to my post: 4)

Sphere - No longer a blog search engine. Off my list.

IceRocket - This might be the true dark horse. It's slowly been improving and it's got a spiffy new design. Note how each link shows you other posts that are referencing it. Smart. Mark Cuban may have a hit here after all. However, IceRocket Trends is currently down. (Links to my post: 14)

BlogPulse: I stopped using this site circa 2006. It has some nice features, but it's not working for me now. (Links to my post: 5)

Ask.com: I always forget about these guys. Their blog search engine is pretty good. Unfortunately, they include my own site in the search results. (Links to my post: 9. Bloglines, owned by Ask, turns up the same)

Bloggers, what are you using? Leave your thoughts in the comments. Are there any I missed? I am giving IceRocket a serious look again because they seem to be catching up.

Comments

[burtonator]

There are new and interesting search engines around the corner Steve. You just haven't seen them yet ;)

We're licensing to a number of them with Spinn3r.

It lowers the bar a bit to have your entire crawler outsourced. You can basically shave $500k-$1M in operating expenses off your capex.

Also, some of the newer tech that's going to be shipping in 2009, like flash, should make the numbers around shipping blog search a lot easier for smaller startups to grasp.

Kevin

[Blake Rhodes]

Thanks for the nice link Steve. Trends will be back shortly, we are working on a few servers at the moment. Hope all is well, keep up the amazing work!

Blake RhodesIceRocket.com

[rodica]

Lately, I monitor keywords through twitter and then click on to discover blogs. It implies several steps, but it's working pretty well so far.

On my wish list: some comprehensive system that can make google alerts & google blog search play nicely with each other. Or something that eliminates false positives by cross referencing against several systems.

I'm eager to give IceRocket a shot, thanks for the heads-up.

[Richard McInnis]

Thanks for the mention Steve. A number of our clients have come to us sharing this same frustration along with the inability to have a number of media types available in one place (Twitter, Blogs, Video, Forums etc..). Maybe we'll see it in 2009

All the best,

RichardRadian6

[Wayan]

I used to use Technorati too, but its tag-based sorting is not optimal when we now tag for internal pages vs. links to Technorati. Ice Rocket is okay, but there are too many duplicate posts and splogs listed. Which has us back at Google Blog Search, about the best we have.

My killer blog search app: would only index via feeds, and only from sites linked from reputable sites or bookmarked by known persons - no random web crawling. Even better if I could make it only show results from sites I link to/in my rss reader, or in those of people in my network (pick a social site).

[Blake Rhodes of IceRocket.com]

Anyone can email me directly with any issues they have with IceRocket. My email is rhodes@icerocket.com . We would love to hear suggestions.

Wayan- if you email me the link of what you see duplicates of, I will make sure we get it resolved. We battle with spam all the time, doing the best we can and have made huge progress over the past year with it,

Thanks,

Blake RhodesIceRocket.com

[Wayan]

Looking at Ice Rocket's results again, the splogs are all copies of Engadget's feed. My site http://www.olpcnews.com gets on Engadget every so often and the number of identical posts are my personal splog barometer. Google Blog Search & Alerts are also infested with these Engadget copies, so Ice Rocket isn't particularly worse off.

In fact, I may take back what I said earlier. Ice Rocket's results are good. And the visual presentation is so much better than Technorati. You may have a convert.

Good work & thanks Steve.

[Connie Bensen]

Hi Steve,Have you seen the Freemium version of Techrigy SM2? We have the Pro tool that's in the same range as Radian6. Although I think that Techrigy SM2 offers more analytics.

Our Freemium version searches everything - blogs, forums, microblogs, wikis, etc. I'd be glad to demo it for you. Here's an overview of SM2 http://blog.techrigy.com/?p=134

And I look forward to meeting you in Germany at Next09! I see that we're both speaking there.

Connie@Techrigy.comFreemium at http://sm2.techrigy.com

[Ian Kallen]

Steve,The RSS feed breakage was apparently a casualty of a recent front end update. That should be fixed soon. Yes, we do have a focus on creating monetization opportunities for bloggers but we're also updating our crawlers (including accurate distinction between blogrool and post links) as well as other systems to serve bloggers better.-IanTechnorati

[dominic]

Steve,

I never understood how the solutions you listed could be called "blog search engines". To me they are keyword based "post search".

A few keywords cannot grasp the reality of a community. In a community people communicate not only to be understood but to be differentiated, noticed by the ones who're in.

When I'm looking for a blog, I'm looking for a "place", a person, not a keyword. Conversations from the "marketing" community. Not a random post from someone that happen to use "dell hell" or "age conversation" as a keywords in his last post.

Maybe that's why there is not much emphasis around that. Generic search engines like Google and Yahoo do a good enough job at searching posts.

[Steve Rubel]

Connie, I haven't seen this but I will check it out. Thanks and I lookforward to seeing you again too.

[Internet Strategist]

I use the Zuula blog search every day and the bloggers I've shared it with are using it too. I even wrote about how to use it to reach out to other Bloggers in a post at http://www.growmap.com/blogging/.

What I use blog search for is to find the latest AND most relevant post on a particular subject. If this is not what you want a blog search engine to do could you better define the challenge?

[Max Kennerly]

I have an utterly massive number of RSS subscriptions, through which I search using Google Reader.

Works nicely if you make it big enough, including lots of aggregate feeds compartmentalized in folders that you frequently "Mark all as read."

[Adam from PageRush]

Steve, I'm really glad you did this write up on the state of blog search. I have been working on a mashup that takes a list of keyword phrases and does a meta-search against Twitter and Google Blog Search. It is pretty effective at finding posts to comment on. I am going to play with adding some of the services you mentioned (at least those that have APIs) and see if I get even better results.

[Ouriel Ohayon]

Steve, take a look at Wikio.com which is a blog and news search and agregation service that is growing nicely

[Paul Stamatiou]

Honestly, I haven't used a blog search engine in quite some time. Regarding Google Blog Search.. I have the same issue with alerts and blog rolls. I've been getting quite a few now.

I generally just use Google - It doesn't matter to me if what I'm looking for is on a blog or not, I just want the best solution or whatever I happen to be searching for. However, I have been using Twitter search more often. For example I just used it a few hours ago because my gmail was down and I was wondering if it was just me. Low and behold a twitter search yielded lots of helpful "no gmail!!! - 8 minutes ago" type results that were reassuring.

[Ari Herzog]

As much as I hate Google Blog Search for the same reasons you do (though more for their Alerts), the results are still valid. I've toyed with Icerocket now and then, but it never gives me comparable results. Technorati... I dunno. I like their results but the interface is clunky.

[Martin Källström]

Hi Steven, there's definitely money to be made in blog search. Twingly is approaching break-even, we'll be profitable some time during 2009. Our focus is primarily European, I can understand that you find our coverage poor in the US. We're working on it. :)

Anyway, thanks for mentioning us. Hope we can meet your expectations going forward.

Regards,Martin KällströmCEO Twingly

[Richard Stacy]

Technorati still seems the most comprehensive - but the RSS issue is a real problem - hope it is fixed soon. From a monitoring perspective it seems to me the best solution is to use as many services as possible and stick them through a filter. I use PostRank - does anyone know of a better filter?

[Brendan]

I still use Google Blog Search - it still gives me good results - but I have noticed weirdness recently, such as when searches throughout the past month seemed to have been re-set a few months back (perhaps this is while they were replumbing?).

I find it frustrating however that Google Blog Search won't play ball with Yahoo Pipes. To my mind what really makes a blog search useful - or anything online, for that matter - is RSS which you can then work into monitoring solutions, to find new blogs as a counterpoint simply to monitoring the ones you know about.

I recently came across Quantcast which is not restricted to blogs but offers a comprehensive breakdown of audience etc. Not played with it much yet but an interesting addition to the mix.

[Steve Dodd]

Hi, just to pipe in, Sysomos already pulls all content types together in a single dashboard and provides rich analytics including sentiment scoring (that is fully auditable), concept / context extraction and geo-demographic profiling. We also have a free version at www.blogscope.net.Steve Doddwww.sysomos.com

[Richard Stacy]

Like the popularity curve function - but why no RSS feeds (or am I missing something)?

[wapers]

We like www.BlogCatalog.comOn Technorati we can hardly find ourselves back.

Explorer leads on www.LEADSExplorer.com

[Josh]

Well you know what they say; a bad workman blames his tools. Only joking, sadly you are right; blog hunting can be very arduous. If anyone knows of a search engine that also sorts by location it would be VERY helpful. Our UK blogs easily get lost in the wealth of US content.

[David Mastio]

Every section of blognetnews.com has its own search engine populated based on the feeds of blogs that have been visited and verified by a human to not be spam and to have been around for at least 90 days.

Right now we cover news and politics in each of the 50 states (for instance, http://www.blognetnews.com/indiana)

We're building out a few hundred U.S. cities (http://www.blognetnews.com/richmondVA)

And we also have topics (http://www.blognetnews.com/environment)

Each site also has its own search widget and we'll be rolling out a universal search page at blognetnews.com/search next month.