Google’s Top Five Data Center Secrets (That Are Still Secret)

Steven Levy is the first reporter to ever set foot in a Google data center. And in telling the tale, he has a wonderful way of making you feel like you were there too.

“A sign outside the floor dictates that no one can enter without hearing protection, either salmon-colored earplugs that dispensers spit out like trail mix or panda-bear earmuffs like the ones worn by airline ground crews,” he writes. “We grab the plugs. Kava holds his hand up to a security scanner and opens the heavy door. Then we slip into a thunderdome of data….”

But as Levy points out near the end of his piece, Google still views its data center empire as one of its most important advantages over the online competition, and it’s still determined to keep the latest technology hidden from rivals. After its data center technology matures, Google may share bits and pieces with the world at large, but it tends to keep its latest tech to itself. “We try to be as open as possible — without giving up our competitive advantage,” Google infrastructure boss Urs Hölzle told us this summer. “We will communicate the idea, but not the implementation.”

Over the past few years, we’ve learned a lot about the Googlenet, widely regarded as the most advanced operation on the web. But there’s still a lot we don’t know. Most notably, the company is tight-lipped about the networking tech used inside its worldwide data center empire. It says only so much about efforts to build its own servers. And like most of the big net players — including Facebook — it won’t say how many servers it operates worldwide.

It’s worth seeking answers to these questions because Google is very much a bellwether for the rest of the tech world. Its data center creations are typically well ahead of the status quo, and when they’re revealed, they tend to trickle down across the rest of the industry.

Today, the big question marks hang over Google’s networking setup. This past fall, the company acknowledged that it’s designing custom networking gear for moving information between its data centers, taking advantage of an open source protocol called OpenFlow. But this is separate from the networking tech used to move info within its computing facilities. When we met with Hölzle this summer, he declined to discuss such things.

It has long been an open secret that Google designs all its own networking gear, and according to a former Google engineer named JR Rivers, the company has been doing this sort of thing since 2005. This past February, photos of a mystery networking device called the Pluto Switch turned up on an obscure web forum after it was apparently delivered to the wrong place, and though this seems to be a cost-saving Google switch that dates back about three years, the company is still mum on the device.

Some have speculated that Google is using a new breed of optical networking switch in its data centers, but this seems unlikely. Optical switches have traditionally been much too expensive for sending data across the short hops between server racks, and it’s still going to take some technical work to get them to the point where they can replace the copper switches that sit on top of server racks in data centers today.

It’s also unclear whether Google is using OpenFlow inside the data center. Presumably, this is a big part of the company’s internal operation. But Google may also employ some sort of virtual networking scheme along the lines of the technology offered by Nicira, a company recently acquired by VMware in a deal valued at $1.26 billion.

Whereas OpenFlow is a means of remotely managing network hardware, virtual networking lets you build complex networks using nothing but software, making it even easier to adjust the flow of data inside the data center. On the wide area network that moves info between Google data centers, the company uses a network controller built in tandem with Nicira.

Nowadays, Google is rather forthright about the fact that it designs its own servers, and in 2009, it revealed some designs to the world at large. But these have likely evolved over the past three years, and it’s unclear how much. A company spokesperson tells us that at least some of Google’s latest server designs can be seen in this video.

As of last month, we do know that Google equips some of its servers with GPS devices and atomic clocks to help drive a worldwide database known as Spanner. Yes, GPS devices and atomic clocks. Last week, at a systems engineering conference in Los Angeles, Google engineer Jeff Dean told us these devices were relatively inexpensive, indicating they were no more than a few thousand dollars apiece — if that.

No, we still don’t know many servers Google is running. Levy argues that this number doesn’t really matter. “Wandering the cold aisles of Lenoir, I realize that the magic number, if it is even obtainable, is basically meaningless,” he writes. “Today’s machines, with multicore processors and other advances, have many times the power and utility of earlier versions. A single Google server circa 2012 may be the equivalent of 20 servers from a previous generation. In any case, Google thinks in terms of clusters—huge numbers of machines that act together to provide a service or run an application.” And he’s right. But on another level, it does matter.

Because Google designs its own servers — and because it now run cloud services that other companies can use in lieu of servers — it’s driving a massive shift in the worldwide hardware market. According to Intel bigwig Diane Bryant, Google is now Intel’s fifth largest server chip customer, a clear sign that Google is now one of the world’s largest hardware makers — and that it’s siphoning business from the likes of Dell and HP.

Apparently, Google contracts with outside manufacturers when it comes time to actually build its machines. This has encouraged many other companies — includes such names as Facebook and Goldman Sachs — to go straight to “original design manufacturers” in Asia for their own hardware. But this summer, Hölzle told us that Google doesn’t necessarily use Asian ODMs for all its gear. Presumably, the company is using a contract manufacturer located in Canada or Mexico or perhaps South America. Celestica, a manufacturer based in Canada, comes to mind.

How many data centers does Google operate? It owns six in the U.S. and three in Europe, and four more are under construction (three in Asia and one in South America). But the company declines to say how many other “colocation” data centers it uses, sharing data center space with other outfits. When it does co-lo, you can bet it does so discreetly.

The other big point to be made is that Google is in the process of overhauling its infrastructure. And that’s a secret too.