VoIP Business and Virtual PBX
Business solutions

Why crowdsourced computing benchmarks are the future

Cloud computing and open source software have freed IT practitioners from so much legacy vendor baggage over the past few years. Isn’t it time to free them from inane benchmark boasting, too? A crowdsourced platform where users share their real-world performance experiences could help.

Performance benchmarks for computing systems might make for good television - there's always plenty of heated debate, talk of world records and  at times even a little ethical drama - but eventually there's not a lot of substance.

However, cloud computing and the advent of popular open source software just as Hadoop and NoSQL databases could change the way we do benchmarks. With relatively little cost and effort, anyone can conduct their own tests to see how their specific applications and configurations run on their specific infrastructure. Throw in a platform to share these results, and you have crowdsourced performance benchmarks free from vendor hype and the vacuum-like conditions of standardized tests.

Ideally, it ends working a lot like the crowdsourced medical platforms I've come across lately, PatientsLikeMe and the forthcoming Lucine Biotechnology. To put it more exactly than rely on claims from drug companies or even doctors whose knowledge is limited to published innovation, users share their own real-world experiences with drugs, symptoms and side effects, and learn from others like them what they might expect.

In cloud computing, too, providers have spent the past few years arguing against the idea of cloud servers as commodities by claiming their systems offer the best performance. There have been plenty of boasts and quite a few attempts to benchmark cloud system and network performance With that much of the cloud market all in all up for grabs, we're not but done hearing about whose cloud is the biggest and the fastest.

Use cases and test results that demonstrate the value of crowdsourcing in-the-wild performance metrics are everywhere on corporate innovation blogs across the web. Before this month, for instance, Medialets discussed how it tested its Hadoop workload and found that Cloudera in fact outperformed the supposedly faster MapR for that job. This week, video-transcoding service Zencoder shared some interesting results when it compared Amazon Web Services' highest-powered cloud instances to Google Compute Engine's best.

Cloud computing and open source software have freed IT practitioners from so much legacy vendor baggage over the past few years. Isn't it time to free them from inane benchmark boasting, too?

Research journalist since 2003

Derrick has been a research journalist since 2003 and has been covering cloud computing, big data and other emerging IT trends for GigaOM since 2009. He has written the words "cloud" and "Hadoop" possibly more than any other person on the planet. Derrick lives in Las Vegas and has a...

More information: Gigaom