Have you ever wondered how much traffic Twitter handles in a given day, or what software sits behind the curtain of the popular service? A recent presentation reveals some of the answers. Twitter’s incredible growth becomes obvious when you compare the recent numbers to those announced at Chirp.
On September 9th, Twitter’s university recruiting team stopped by UC Berkeley to talk about the company and what it does. The slides from platform engineer Raffi Krikorian’s talk, Twitter by the Numbers, are now online, and they disclose some fascinating technical details about the social media giant’s operations.
Twitter serves over 70 million tweets per day, totaling over 12GB of tweet text alone. Many of those messages are delivered to client apps and web sites through the Twitter API to the tune of six billion API calls per day (double what was announced at Chirp in April), or about 70,000 API calls per second. All told, the service generates 8TB of data every day, which is eight times more than the New York Stock Exchange.
Pop quiz, engineers: Your web service needs to deliver real-time message traffic to an asymmetric digraph of over 150 million users. What database do you use? WHAT DATABASE DO YOU USE?
- Shoot the hostage
- Oracle
- MySQL
- Write your own database
If you’re Twitter, the correct answer is #4: Create your own database software, call it FlockDB, and release it on github. (By the way, if you picked option 1, maybe software engineering isn’t the right career choice for you.)
FlockDB is just one of the home-grown, high-performance software systems Twitter uses to support its tremendous growth. Others include:
- hosebird, a “near real-time†streaming API back-end (instead of REST, which is only “pseudo real-timeâ€); and
- snowflake (also on github), a network service to generate unique IDs at high scale (MySQL couldn’t keep up, and was a single point of failure).
With a stated goal of supporting “half the world and all its devices,†Twitter faces many engineering challenges. This peek under the hood (full slides are embedded below) shows that they’re aware of the potential problems, and are working hard to steer clear of the fail whale.