Mesa: How Google Handles Big Data

As a guitarist, the first thing that comes to mind when I hear the word Mesa is Mesa Boogie amplifiers. They delivery one of the most bone rattling level of sound and tone of anything in the industry. This time though, I’m talking about an entirely different Mesa: Google Mesa.

Mesa – The High Level View

Google’s Mesa is a data warehousing environment which powers much of the Google ecosystem. Through their long experience with internet advertising. In order to capture and render content at high speeds, with low latency, they needed to do something that hadn’t been done before.

…Mesa handles petabytes of data, processes millions of row updates per second, and serves billions of queries that fetch trillions of rows per day. Mesa is geo-replicated across multiple datacenters and provides consistent and repeatable query answers at low latency, even when an entire datacenter fails.

If you want to go deep on Mesa, you can do so with this excellent research PDF that documents the overall concept, right down to the deep inner workings of the magic behind Mesa. If you want to go a little lighter on your information gathering, this great write up by Venture Beat talks about this exciting product.

What’s Next for Mesa?

With more information coming out about the platform, the real question comes as to whether this may make it into their Google cloud platforms as a data warehousing service. Even in the event that this doesn’t appear as a consumable service, the intelligence that powers this product is one that opens the door to many other opportunities.

More than the speed and content delivery, I focused on the geo-replication features that allow for the data to be available in multiple regions, fully protected. This type of redundancy and protection has been available for quite some time in object storage environments, but databases and data warehouse environments are not always conducive to redundancy. They are meant to be performance oriented.

Traditional data warehousing platforms also require that content is pulled out for analysis, referred to as cubes. These cubes are granular pieces of the overall environment which are set aside for higher speed, lower latency access on a subset of the data. Google requires real-time access to both update and query the Mesa system which makes it an impressive platform.

Can Anyone Compete? Or Would They?

One of the hopes is that Google will share in its exciting platform. Because the product runs on generic hardware, it is an ideal fit to open source and take the product beyond the Google data centers. That being said, it is difficult to say what Google will choose to do and how they may monetize the platform.

The big question is whether there will be anyone who can innovate to target Mesa with a competitive product. Many tools came together to create the product such as MapReduce, BigTable, and more. Proprietary products internally also drive the heart of Mesa which is really the culmination of the perfect storm of tools and techniques.

Whether you will see the value of Mesa for your organization or not, there is no denying that this is the kind of innovation that will continue to come and drive the ever referenced “internet of everything” which is already here in many ways, but always expanding.