Apple’s online services like iMessage, iCloud, and iTunes are used hundreds of thousands of times per second, and almost never go down.
But because of Apple’s famous penchant for discretion, many of the impressive technical hacks that enable these massive infrastructure systems remain trade secrets in Cupertino and Apple data centres worldwide.
Jason Forrester is pulling back the curtain. He used to be global data center manager at Apple, according to Barb Darrow at Fortune, and he’s leading a new startup, SnapRoute, staffed with ex-Apple engineers.
SnapRoute was born out of Apple’s secretive “Project McQueen” — an ambitious effort to bring all Apple’s cloud services in-house, according to Recode.
In a blog post on SnapRoute’s website, Forrester relays a few interesting anecdotes about what it’s like to work for Apple:
He reveals how nascent Apple’s infrastructure was when he signed on:
“Then I landed a job at Apple, where I had a ringside seat on one of the challenging network expansions in history. When I arrived in 2011, Apple had two data centres, mostly handling internal traffic and doling out songs and apps from the iTunes music store. By the time we left, Apple had several more centres stuffed with an incredible amount of network devices to handle billions of Siri and Map queries, iMessages, and cloud services.”
He explains about how even a small misstep would be noticed worldwide:
“It was thrilling — and terrifying. When even one per cent of Apple’s traffic gets stalled, it’s front-page news.”
He says there’s no way that Apple could’ve used off-the-shelf data center servers and components, so it built its own:
“We were dealing with problems our vendors had never contemplated, much less figured out. We began exploring radically new approaches, including a handful of supposedly open-sourced solutions so we could dive into the guts of our network ourselves — say, to look directly at the data coming off network processors. As much as we wanted these technologies to work, they didn’t.”
He reveals that Apple has several internal networking breakthroughs it’s keeping to itself:
“So we developed some of our own, including a provisioning tool for upgrading the software on thousands of switches without taking the network offline. If you haven’t heard, Apple likes to keep such internal accomplishments to itself, so I can’t share the results. Let’s just say we were able to accomplish in minutes what would have taken hours, days, or even weeks.”
And finally, he explains why a well-compensated expert like himself might decide to quit Apple — it’s the secrecy. (There’s also a lot of internal drama among Apple’s cloud teams):
“Slowly, our desire to share our ideas with the world began to overshadow the thrill and pride for working for Apple. My team and I left in 2015. Truth be told, I spent a few days crying on the couch. My mood only improved when we began to test our ideas with potential customers.”