Digg has a problem. That problem is that the more users Digg gets, an increasing amount of the stories that reach the Digg front page are bound to be unavailable, brought down by the storm of visitors from Digg.
If Digg keeps growing it will automatically kill almost any site that reaches the front page.
Digg can’t keep using the same model they are now. It simply doesn’t scale. It will reach a point where it has so many users that the amount of traffic that comes from being on the front page of Digg is more than any normal site can handle. (You could argue that this is already happening.) As it grows, Digg will be knocking out larger and larger sites.
We (i.e. Pingdom) monitor the uptime and response time of a lot of websites, and there are numerous occasions where we have seen pages become either very slow or completely unavailable in connection with hitting the front page of Digg. We perform more than 7 million tests per day, so we have a significant amount of data to look at. The problem tends to be the initial spike of traffic when a news item first hits the front page.
A visual example
As you can see we divided sites into three main categories:
- Sites on shared hosting accounts. Can handle a normal amount of traffic. The majority of sites on the internet are hosted this way, and a lot of these will have serious problems (i.e. crash) if they end up on the Digg front page.
- Sites on a dedicated server. Can handle a lot of traffic, but can, and often do, run into trouble during the initial peak of traffic from the Digg front page.
- Sites on multi-server setups. Handle anything you throw at them, but are basically only used by media portals, large news sites (think CNN.com) and large enterprises. Digg isn’t a problem here.
Sites with multi-server setups only make up a tiny part of the content available on the internet. Isn’t one of the main points of Digg to show niche content and find news before the big media finds it? This content is often driven by people who earn no or very little money and host their sites on modest setups.
Mirroring content isn’t the answer (legally speaking)
The so-called Digg effect is so common that it has given rise to a separate mirror service (Duggmirror) that makes copies of downed sites. The legality of this practice is highly dubious from a copyright standpoint, and isn’t run by Digg. There is even a statement by Digg-founder Kevin Rose from 2006 regarding the caching of content, which is quite interesting in this context:
“Digg should never take away traffic or cache content, should always push to site creator.”
Summary
The conclusion here has to be that if Digg wants to keep growing, and we assume they want to considering all the attention they have received from VCs, they need to somehow change their model. If they just keep going the following will happen:
- Most of the websites on the front page of Digg will be unavailable, which means that…
- The quality of Digg will suffer and it will be much less useful.
- Some sites may not even WANT to be on Digg. It may simply be more trouble than its worth.
Perhaps the Digg crew is working on some solution to this problem. We don’t know. What do you think?
A small note on Digg’s users and visitor numbers
According to an article by John Graham-Cumming, Digg currently has about 2.7 million registered users, which is almost three times as much compared to a year ago. But of course, the majority of visitors to Digg will not be registered users, but simply people there to find interesting news. According to data from comScore, the Digg website had about 12 million unique visitors worldwide in December.
As an example, last time we had a post on the front page of Digg, the dedicated server for this blog couldn’t keep up with the initial burst of visitors from Digg. This is kind of ironic considering we’re an uptime monitoring company… 🙂
