Do you want to know more about how big websites like Twitter, Facebook, Hotmail and others handle the technical challenges of dealing with massive amounts of visitors?
Well, you’re in luck, because many of those sites and services have engineering and/or developer blogs that share plenty of information about the challenges they have to deal with and the tools they use. This is an insider’s view that you usually can’t get anywhere else, giving us a unique view of what’s going on behind the scenes of some of the world’s largest web services.
Yes, you’ll have to be a tech geek to find them interesting, so consider yourself warned. 😉 But we here at Pingdom are geeks, and we are really into web tech so we love these kinds of blogs. They are blogs that web developers and sysadmins can learn from and be inspired by.
Let’s get right to it.
Twitter’s engineering blog
Twitter’s engineering blog gives insight into what makes Twitter tick and what technical solutions they are using to solve their scalability challenges.
URL: http://engineering.twitter.com/
A few recent posts:
- Hadoop at Twitter
- Introducing Gizzard, a framework for creating distributed datastores
- The Anatomy of a Whale
Facebook’s engineering notes
Want to learn more about the technical ins and outs of the world’s largest social network? Look no further than Facebook’s engineering page.
URL: http://www.facebook.com/Engineering#!/Engineering?v=app_2347471856
A few recent posts:
Digg’s technology blog
Whether you use Digg or not this is interesting. Self-described as “a peek under the Digg hood for API developers, partners and technology geeks.”
URL: http://about.digg.com/blog/technology
A few recent posts:
- Saying Yes to NoSQL; Going Steady with Cassandra
- Testing at Digg
- Database Capabilities in a High-Volume Environment
Inside Windows Live
A blog by the engineering team behind Hotmail, Messenger and Windows Live. It has a large variety of articles with behind-the-scenes information and data analysis.
URL: http://windowsteamblog.com/blogs/windowslive/default.aspx
A few recent posts:
- Hotmail tips the scales
- Are you a piler, filer, or purger?
- Keeping the Messenger service running—on a massive scale
Flickr’s developer blog
A blog by Flickr’s development team that dives into the nuts and bolts of the photo-sharing site and working with the Flickr API.
URL: http://code.flickr.com/blog/
A few recent posts:
- Using, Abusing and Scaling MySQL at Flickr
- A Chinese puzzle: Unicode and EXIF metadata parsing
- Language Detection: A Witch’s Brew?
Meebo’s developer blog
Interesting developer-oriented articles about the inner workings of Meebo’s instant messaging service.
URL: http://devblog.meebo.com/
A few recent posts:
- Performance Case Study: The Meebo Bar (Part 1 of 3)
- Keeping your passwords safe, part 3
- Dev tips and tricks
Amazon Web Services blog(s)
You have Amazon’s official AWS developer blog which has plenty of announcements and great advice for using Amazon EC2, S3, and so on, but the real nugget of gold is the insightful All Things Distributed, a blog written by Amazon’s CTO, Werner Vogels. So, with apologies to the official AWS blog, that’s the one we’ll list here below.
URL: http://www.allthingsdistributed.com/
A few recent posts:
- Choosing Consistency
- 82 Billion Objects in Amazon S3
- Good Advice on Keeping Your Database Simple and Fast
There’s more, much more…
What we find interesting with these blogs is that they give unique insights into the nitty-gritty technical details of running sites that are much, much larger and more demanding than your average site.
There are a bunch of other sites you might also like to check out depending on your interests. The behind-the-scenes articles may be a bit sparser on these, but they still come up with the occasional gem. Here are some to get you started.
- Google Code blog
- Google App Engine blog
- Yahoo’s developer network blog
- Signal vs. Noise, the blog of 37signals, makers of Basecamp and other web apps.
- Wikimedia’s techblog (the people behind Wikipedia)
Then on top of these you can add all those great blogs that don’t actually belong to a company/service but still offer great articles about how the big sites are run. A great example is the High Scalability which is a blog dedicated to how sites deal with scalability and performance issues.
We’re sure we’ve missed several excellent blogs, so if you know of other services whose engineers or developers blog about their technology, let us know in the comments. We can build a great list here!
