Google Launches Open-Source, Cross-Cloud Benchmarking Tool

Google today launched PerfKit, an open-source cloud-benchmarking tool that, in Google’s words, is an “effort to define a canonical set of benchmarks to measure and compare cloud offerings.” The PerfKit tools currently support Google’s own Compute Engine, Amazon’s AWS and Microsoft’s Azure clouds. Google says it has worked on this project with over 30 researchers, companies and customers, including ARM, Canonical, Cisco, Intel, Microsoft, Rackspace and Red Hat.
As Google points out in today’s announcement, evaluating the performance of different cloud services isn’t easy. There are a few companies like CloudHarmony that offer cloud-performance reports (and tools like New Relic obviously let you monitor your existing installs), but none of these take your specific needs into account, and it’s often unclear how those tests were performed.

Once installed, PerfKit runs about 20 benchmark tests that look at everything from raw CPU performance to more complex database and network benchmarks. Google notes that the tool also tests the end-to-end time to provision new resources in the cloud, something that most other benchmarks can’t take into account. To compare the results, the team also wrote a visualization tool (the PerfKit Explorer, which sounds a bit like a “Fifty Shades of Grey”-branded bondage starter set).
It’s cool to see Google launch an open-source benchmarking tool like this. The company has long said that it isn’t afraid to pit its cloud performance against its competitors’. It’ll be interesting to see the first cross-cloud benchmarks from this tool, though, to see where it really stands.