Earlier this week, the content discovery tool Shareaholic published a report claiming that the web traffic Reddit refers to other sites is rapidly declining. A chart showing the dip accompanied the report. Shortly thereafter, the report appeared on several news outlets, including The Atlantic and Business Insider, which recommended that publishers and search engine optimization experts start looking at web sites other than Reddit for referral traffic.
Normally I’d ignore a report like this because I don’t care much about SEO, but the analysis in Shareaholic’s report was so bad–and the chart so misleading–that I had to write a response.
Shareaholic’s claim rests on their comparison of two cherry-picked data points: Reddit referral traffic at December 2012 and December 2013. (That’s odd, given that Shareaholic has been running since 2008. Why didn’t the company show its data since 2008? Or at least 2010?) Shareaholic compares referral traffic between those two points and notes that it’s gone down by 33%, then speculates that Reddit’s referral traffic is crashing.
One of the things we learn in high school math class is that it takes at least three points to make a trend, not two. In fact, if Shareaholic had posted this report in June 2013, it could’ve told the opposite story: Reddit’s referral traffic is skyrocketing (up 66%)! That’s a sign right there that the methods Shareaholic is using are flawed.
If we want to talk about trends, we need to consider all of the data. Shareaholic shows us a simple line plot connecting the dots, but doesn’t even try to show us what trend is actually going on. Can you easily tell if there’s really a significant downward trend going on there? That’s the sign of a poor data visualization.
When we take a closer look at the data Shareaholic presented, we can draw a trend line through those points and notice that there isn’t much of a trend going on at all: The line is almost flat, going down just 0.007% every month.
Of course, this trend line assumes that Reddit referral traffic will stay in constant decline, which we know isn’t the case. Just look at what happened in June 2013. There’s a lot more to Reddit’s referral traffic trend than meets the eye, and Shareaholic’s oversimplified analysis doesn’t even come close to explaining it.
Even worse, if we drop the December 2013 point entirely and plot the trend again, the line becomes even flatter (-0.004% per month). If a trend relies so heavily on one data point, then it isn’t much of a trend at all. There simply isn’t enough data here for Shareaholic to make any kind of claim about Reddit’s traffic referral trends–especially the audacious claims made in the news.
When it comes to sorting out the signal from the noise, this is pure noise that Shareaholic is trying to make look like a signal.
It’s even strange that Shareaholic presented this data in terms of fraction of traffic referred from Reddit. For example, they say: “During December 2012, sites saw 0.33% of their overall traffic come from Reddit.” Why not report the raw traffic numbers instead of a fraction?
It could very well be that even though the fraction of traffic from Reddit went down, the total amount of traffic from Reddit went up.
To drive this point home: Even though 50% seems less than 75%, if we’re comparing 75% of 200 visitors (= 150 visitors) vs. 50% of 1,000 visitors (= 500 visitors), clearly the 50% represents more traffic overall. In other words, even if there was a declining fraction of traffic from Reddit, it could simply be because the websites Shareaholic tracks are receiving more traffic overall.
With that in mind, we can’t even be confident that the Shareaholic referral traffic data actually represents a decline in overall referral traffic from Reddit, as the company so boldly claimed (and publishers rehashed).
Before publishing this article, I wrote to the Shareaholic team to explain the faults in its analysis. The company’s response:
Ultimately, our goal is to provide data [that] marketers can act upon, not data that they’d sit around in a room pondering about for hours on end.
It seems that Shareaholic doesn’t even care if its analysis is correct, as long as it offers “data that marketers can act upon.” Are you sure you want your marketers acting upon data that’s as unreliable as this?