Facebook Is To Blame For A Wave Of 'Dark Traffic' Swamping The Web, But It's Working On A Fix

So-called “dark traffic” is a huge problem for publishers and website owners.

It’s useful to be able to look at your analytics, but when a huge proportion of referral traffic is listed as “direct” (and aside from a homepage, it’s very unlikely users are typing full URLs into their browser to go directly to a page) it’s difficult to know where to focus to boost your audience.

Now new research from the analytics firm Chartbeat, as well as confirmation from major publishers, shows that Facebook’s mobile apps are largely responsible for the swathes of dark traffic being directed toward websites. We already knew Facebook was the principal source of social referral traffic for most digital publishers. But Facebook has been underselling itself.

Fortunately for publishers and website owners, it appears that Facebook is working on a fix.

The term “dark social” was coined by Alexis C. Madrigal, a senior editor at The Atlantic, who found that more than half of the website’s referral traffic was seemingly untraceable. Dark traffic refers mostly to when links have been shared via an online chat, email, or app rather than through a browser or a specific social app through which referrals can be easily tracked. Chartbeat has found in some cases dark social account for 65% of a website’s traffic, averaging at about a third across its network. About 10% to 15% of The Guardian’s traffic is dark traffic.

Most analytics platforms (Google Analytics, Adobe Omniture, Chartbeat) have a huge section in their reports called “direct,” which counts all those apps that users use to click on links “direct to a site.” Facebook is one of them — alongside sources like Reddit, Gmail, and IM apps. Twitter, on the other hand, has a special “t.co” URL shortener, which means Twitter referrals are always easily trackable.

The problem with Facebook not using a shortener like this is that a huge proportion of its users access the site only via the mobile app. We know 703 million people visit the site via their mobiles daily, and presumably the majority of those do so via the app. That’s a massive wave of referral traffic going unidentified.

Chartbeat, which from this week began tracking Facebook users more effectively using a different method (more on that later,) has published a blog post that clearly marks out the Facebook/dark social impact.

Last month the analytics company began to look at time series data for specific articles to identify patterns in traffic and correlate that with the “dark social” number.

Take a look at the chart below. The dark traffic spikes at exactly the same time the article began going big on Reddit, and then again when it started picking up pace on Facebook.

Madrigal also wrote in a newblog post published this week that he too conducted a test earlier this year. He set up an Atlantic post “deep in the sand of time” so only people who were part of the test would find it. He then posted a link to the story on his Facebook page and told his friends only to click to it if they were using the mobile app. He then looked at the referrers (of which he knew 100% were coming from Facebook). A smattering of people showed up on his Adobe Omniture analytics report as coming from Facebook.com, but the rest showed up as “type/bookmarked.”

For the most part, those that had been clicking on their mobiles had gone untraced:

Figuring that those numbers would be tightly correlated with the overall number of Facebook mobile visitors, Madrigal compared Facebook mobile referrals with dark social.

The correlation was pretty clear.

As a result of its tests, Chartbeat has started looking more closely at the “user agent,” a tag of code users leave when they visit a website which identifies the type of browser they use, their operating system and usually reveals some information about where they originated from. Most analytics systems don’t dig deeply into this type of information.

Some publishers have developed their own methods to dig further into their website analytics to uncover the “user agent,” which reveals Facebook as the source of the direct referral traffic. Both The Guardian and BuzzFeed said on Twitter last night they have been digging further into user agent data for a while now. But that takes both time and expertise that most publishers don’t have.

Malcolm Coles, Trinity Mirror (owner of UK newspapers The Daily Mirror and The People) digital director (who is soon to take up the same role at the Telegraph Media Group, owner of The Daily Telegraph,) told Business Insider that he has been told by analytics companies that Facebook is “finally” fixing the problem.

A separate tweet from digital analytics company Parse.ly has also suggested Facebook has a fix in the latest updates to its mobile apps. Business Insider contacted Facebook for further details, but the company said it had nothing to share.

“The main effect has been that people have underestimated the impact Facebook has on their traffic and that people have maybe been overestimating the impact of Twitter,” Coles said.

But he adds that until every app owner — Reddit, Whatsapp email browsers and so on — does the same thing by creating bespoke automatic URL shorteners, the problem of “dark social” won’t go away any time soon.

NOW WATCH: Briefing videos

Business Insider Emails & Alerts

Site highlights each day to your inbox.

Follow Business Insider Australia on Facebook, Twitter, LinkedIn, and Instagram.