Digg and the Power Laws of Silicon Valley in 2006
It's 2006 and there is a lot of controversy about the power wielded at 'user generated content' aggregators like Digg, Reddit and Netscape 2.0.
In June 2006 I flew to San Francisco for my third US trip. I arrived at SFO just after midday on Monday, June 19, after another sleepless twelve-hour flight from New Zealand. Mike Arrington was hosting me at his place in Atherton again, for which I was grateful. I’d also asked if he could pick me up from the airport, and he’d asked an assistant to calendar it. But when the time came, he wasn’t there to greet me. I waited around with my luggage for about an hour and a half, every now and then ringing Mike’s phone from an airport payphone. I kept getting his voice mail.
Eventually, I took matters into my own hands and hired a car from Budget Rent a Car. Despite the name, it was expensive. But I realized that I should’ve done this in the first place. I needed to be more independent and not rely on Mike or others, so hiring a car was a good start.
This was my first time driving in America and as I drove out of SFO, I was terrified of crashing. I kept reminding myself to drive on the opposite side of the road, while also trying to work out which exit to take. The Californian sun glared into my eyes, and I had to squint to concentrate. Fortunately, it wasn’t peak-hour traffic, and somehow, I managed to get on the motorway heading to San Jose. Soon I was feeling comfortable behind the wheel, and the familiar thwack-thwack-thawck of the tires hitting the 101 calmed me down. It was a gorgeous summer day, and it felt great to be in Silicon Valley again.
This time at the TechCrunch ranch, I noticed that Mike was busier than ever—his reason for not picking me up at the airport. I understood that I wasn’t high on his list of priorities, so it didn’t bother me. We both agreed it was better that I had a rental car to get around.
TechCrunch had exploded in popularity by this point, and he was devoting a lot of time to it—in addition to the time he spent on edgeio, his startup with Keith Teare. Technorati now had TechCrunch ranked the eleventh most popular blog in the world, just a year after he’d started it, and the Feedburner RSS badge on Mike’s site was nearly at one hundred thousand subscribers. I had yet to crack the Technorati Top 100 (although I would by the end of the year), and I had around thirty thousand RSS subscribers.
Digg vs. Netscape 2.0
Despite Mike being so busy, we managed to collaborate on a story involving Digg, the trendy tech-news aggregator that had recently eclipsed Slashdot in popularity. Its founder, Kevin Rose, was in his late twenties and was one of the faces of Web 2.0—due in large part to his good looks. He had dark hair worn long at the front, full lips, and brown puppy-dog eyes. Add to that his lean profile and raffish stubble, and he looked like a cross between a young Steve Jobs and a singer in a boy band. He’d been on the cover of magazines, too; only instead of Rolling Stone, it was BusinessWeek and Inc.
Just before departing for California, I’d written about the relaunch of a famous internet brand: Netscape. AOL had put Jason Calacanis in charge of the brand, after acquiring his Weblogs, Inc. business the previous year. Calacanis had decided to turn Netscape into a user-voted news aggregator that bore a striking resemblance to Digg. Some bloggers—Mike included—thought this might be a “Digg killer” due to the considerable built-in audience the Netscape portal still enjoyed at the time. I noted in my piece that “Netscape gets a surprising 811 million monthly page views,” compared to Digg’s 200 million.
The key difference was that Digg was entirely controlled by the “wisdom of the crowd” (its users), whereas Calacanis had added editors (whom he called “anchors”) to select which stories would feature at the top of the page. As Calacanis himself put it, he wanted “an editorial voice to balance the hive mind.” I’d written that this approach was “inherently less democratic, even if it ends up being more effective.” I added, “I can’t see any current Digg users defecting, but perhaps Digg should hurry up with those new features it’s been promising!”
In response to my post, Digg’s CEO, Jay Adelson, emailed me. Naturally, he was skeptical about what AOL was trying to do with Netscape. “We will never have a small group of people provide oversight,” he told me. Digg was about to launch its latest version, which they had nicknamed Digg 3.0. I suggested we record an interview with Jay and Kevin for Mike’s new TalkCrunch podcast, meaning that Read/WriteWeb and TechCrunch would share the story. Digg’s PR people readily agreed, so we arranged for them to call in to Mike’s house via VoIP (Voice over Internet Protocol) on Wednesday evening.
A Lesson in Power Laws
The main story was that Digg was expanding beyond tech news. It was adding five new categories: Science, World & Business, Entertainment, Video, and Gaming. This seems unremarkable now, when Reddit (today’s Digg) has thousands of categories. But in mid-2006, Digg was purely a tech-news aggregator, so this was an important move for them, and more generally, it was another signal that Web 2.0 was moving beyond its geeky origins. If ordinary people began to use Digg, then it could become a mainstream portal—just like Netscape.
Reddit at this time was also focused on tech; it had just seven “subreddits,” all of them very geeky. Reddit had been founded the previous year, but was much less popular than Digg in 2006, so it was Digg’s game to win.
I remember being nervous during the Digg interview, since it was being recorded and I wasn’t comfortable doing podcasts at that time. Mike was sitting at his desk during the call, and I’d pulled up a chair to sit beside him. My voice sounds small and shaky on the recording, partly because Mike and I were on speaker phone and the VoIP connection wasn’t great. Rose, in contrast, was a natural. As well as looking like Steve Jobs, when he spoke he exuded a similar charm and confidence. According to Rose, they’d always had bigger plans than just competing with Slashdot. “We knew our game plan all along, and that we were going to be expanding outside of just technology news,” he said during the call.
At the time of the interview, Digg was already a key driver of traffic to RWW. I would continue to obsess about Digg for my own business and also wrote articles examining its impact on the wider blogosphere. In particular, just how democratic was it, actually? That July I noted that “a page on digg.com called Top Diggers shows that a select group of digg users are highly influential.” Rose was right at the top of that list; of the 120 stories he’d submitted with his user profile, a staggering 119 made it to the home page—guaranteeing heavy traffic for those sites.
Later, I discovered that nearly 30 percent of front-page stories on Digg were contributed by just ten people. This wasn’t too surprising, since many networks—internet or otherwise—have a similar “power law” dynamic. In my post about it, in early August 2006, I concluded that having an A-list of users was “no different to the blogging world [or] the Web in general.”
The power-law dynamic didn’t just apply to users—it also applied to content creators, in that a disproportionate amount of Digg front pages went to a select group of blogs. I knew of at least one tech blog that was able to charge high advertising rates thanks to the traffic Digg sent them. I was also aware that TechCrunch got dugg a lot more than RWW.
The power-law dynamic at Digg was much discussed in the blogosphere over 2006. Most of the controversy was based on the theory that the top users and the top sites must be gaming the system (and indeed, there was evidence to support this). For a site that promoted “non-hierarchical editorial control” by its users, this was a problem.
In September, Rose announced an “algorithm update” aimed at improving “digging diversity.” He added, “Users that follow a gaming pattern will have less promotion weight.” I reported soon after on RWW that Rose’s post “caused an uproar amongst Digg’s top users, who feel they have been accused of gaming.”
Lamb Dinner
I met Kevin Rose in person for the first time in February 2010, when he had been brought over to New Zealand to present at a local conference called Webstock. I took the opportunity to buy him and his friend Daniel Burka a lamb dinner at Pravda, a popular Wellington bistro.
Kevin had lost none of his boy band looks since 2006, although his face was thinner and his hair slightly shorter at the front. He stylishly wore a red and blue checked shirt over black jeans, and his manner was friendly—but also standoffish, in that self-assured way that popular kids usually have. He knew people wanted to get close to him, so his natural instinct was to keep them at bay.
I made a few wisecracks about all the traffic Digg had sent our way over the years, but he didn’t engage, and he also gave nothing away about the internal issues the company was facing at this time. My girlfriend sensibly steered the conversation back to vacation recommendations. Disappointed that I couldn’t talk shop, I offered up some tourist advice instead: “You must try the Cloudy Bay oysters.”
I followed up by email with Kevin later that week, wishing him well for the rest of his trip and that “we’ll catch up perhaps at SXSW” the following month. He wrote back, “We tried the oysters the next day—soooo awesome, thanks for the tip! See you at SXSW, and thanks again for dinner!” Like any shy kid spoken to by the most popular boy in class, I was glad to be the recipient of his enthusiasm. But we never did meet at South by Southwest. In fact, we never had any communication again.
Needless to say, Digg’s relationship with its top users continued to fluctuate wildly. Eventually, after a disastrous 4.0 upgrade in August 2010, its power users migrated once and for all to Reddit.
Lead image: Digg CEO Kevin Rose at the Digg 3.0 launch party, June 2006; photo by Scott Beale / Laughing Squid
This post is part of my serialized book, Bubble Blog: From Outsider to Insider in Silicon Valley's Web 2.0 Revolution. View table of contents.
Next up: 016. The Core Values of Blogging
You're reading Cybercultural, an internet history newsletter. Subscribe for free, or purchase a premium subscription. Your support for this indie publication would be greatly appreciated.