Why Net Neutrality Demands Total Transparency

Illustration for article titled Why Net Neutrality Demands Total Transparency

The net neutrality fight is moving in new directions, and quickly. Today FCC Chair Tom Wheeler announced that the FCC would press forward with new "Open Internet" rules, undeterred by last month's court decision striking down most of the old ones. Last week, Comcast and Time Warner Cable announced plans to merge. The merger would create the largest Internet Service Provider (ISP) in the nation, with five times the subscribers of its closest competitor. With only one or two broadband providers available in most parts of the country, prices may soar while the quality of services plummets. A lack of competition raises serious concerns that huge ISPs will be able to favor particular sites and services.

Wheeler's announcement today included a strong commitment to promoting the piece of the Open Internet rules that did survive judicial review: transparency. Unfortunately, even "transparency" is tougher to enforce than many might think. That's because so much of our connectivity depends on essentially secret "peering" agreements between providers.

To understand peering, let's back up to last week: news of the Comcast-Time Warner Cable merger broke while EFF was investigating a claim that Verizon was discriminating against traffic from Netflix and Amazon Web Services. A blogger in Texas named David Raphael reported that his home Internet speed to Netflix was dramatically slowed down. He tested his connection, and it turned out that indeed his Verizon FiOS Internet connection was operating thousands of kilobytes per second slower than normal—when connecting to certain sites, but not others.


Dave's problems made some nervous that Verizon had already begun to slow down connections to disfavored websites. In light of last month's court decision, many feared ISPs would act like the door was wide open for them to discriminate against traffic on their networks. As broadband markets appear to consolidate, those who care about an open, innovative Internet are rightly worried.

Verizon denied that it was discriminating against Netflix or other types of traffic, but we've seen ISPs dissemble on this point before. In 2007, when Comcast was caught intermittently blocking users' ability to exchange data using the BitTorrent peer-to-peer file-sharing protocol, the company made numerous false claims about its network interference before finally admitting its bad behavior and halting the disruptions.

Mapping connections

What makes the issue even more complicated is that we don't have a common understanding of what it means to discriminate, or provide unequal treatment to some traffic. For example, traffic could be slowed in practice because of infrastructural problems that are different from "intentional" throttling. Or ISPs could argue that systematically slowing traffic to certain destinations wasn't actually discrimination because they hadn't programmed their equipment to make it slow.


To try to unpack Dave's complaints, EFF put out a call to our tech-volunteer mailing list and asked other Verizon FiOS users around the country to test their connection speeds. The results were somewhat inconsistent. Most showed a relatively quick connection to Amazon Web Services, even in the evenings, but a few slow speeds were reported. Was Verizon only slowing down connection speeds in some parts of the country? Was Netflix targeted? Was it a technical malfunction?

These inconsistencies seem to match other people's experience: the slowdown, if any, affected only certain FiOS users, only some of the time. This teaches one key lesson: we need a regular, widespread, and methodical system for testing infrastructure to help understand the size and shape of connectivity problems and slowdowns when they develop on the Internet, so that accusations of discrimination or blocking don't have to be based on anecdote alone.


Of course, slowdowns that affect only certain regions or that are noticeable only some of the time are frustrating, and could be genuine indications of a larger problem. Even if ISPs like Verizon aren't explicitly blocking or slowing traffic, there could be deeper, more systematic issues that affect connections to certain destinations but not others.

This points to a conceptually subtle but important part of the problem: the Internet is a "network of networks," and data traveling from point to point may pass through the equipment of Comcast, Level 3, AT&T, Verizon, Cogent, and many other companies. The interconnections between these systems are almost unnoticeable to end users when the Internet is working smoothly, but they can be brought to the foreground when particular links become overloaded with congestion caused by rising traffic levels.


And that brings us to the issue of "peering." Connections between web service providers, websites, and ISPs depend on agreements to exchange Internet traffic with each other, or "peering" links. Operators of backbone and web services make peering agreements with ISPs about how to exchange Internet traffic so that data can be carried efficiently from one part of the Internet to another.

Peering relationships are crucially important to the fabric of the Internet, but today they're often hidden behind a veil of trade secrecy and business strategy. And providers often butt heads in ways that interfere with and shape Internet availability in discriminatory, non-neutral ways. Simply put, peering disputes are a serious threat to the future of our open Internet.


Peering disputes

Here's a case in point: in 2010 Comcast and Level 3, one of the largest content delivery networks (CDN) and web service providers in the world, had a major disagreement over peering. Netflix, a company whose Internet video streaming accounts for nearly one third of the America's download traffic, had just signed a contract with Level 3 to be their transit provider. Comcast said that the new traffic coming across their infrastructure from Level 3 was not within the scope of the two companies' original peering agreement. Comcast refused to build new ports to handle the increase in traffic from Level 3, claiming that they weren't responsible for building new network infrastructure needed to deliver a major influx of streaming video to users.


Comcast's solution: charge Level 3 to deliver content to Comcast subscribers. That meant Comcast was trying to charge users to connect to the Internet and charge data centers to connect to users—a doubly profitable solution. The companies have since struck a cost-sharing deal, the finer details of which are spelled out in secretive peering agreements.

More recently, in 2013, there was a peering dispute between Verizon and Cogent. Cogent is one of the nation's largest Internet providers, particularly providing infrastructure to operators of web sites and services. Cogent and Verizon peer with each other in locations around the country, and when the ports that they use to connect to each other become full of traffic, Verizon will typically have to add more ports to keep the data flowing as smoothly as before.


But with video streaming on the rise, Verizon claimed that Cogent had to share the cost in building network infrastructure to handle the tremendous uptick in traffic. Cogent refused to pay, and Verizon refused to provide Cogent with any more ports. As a result, many Verizon customers had serious trouble connecting to websites that rely on Cogent for Internet connectivity.

And back in 2008, a peering dispute between Cogent and Sprint became so bad that the two companies stopped exchanging traffic entirely. At the time, Sprint customers could not access data from companies that rely on Cogent. The Internet was partitioned—divided into separate parts that couldn't directly communicate with one another. Even though neither company may have wanted this outcome, the results are as bad for Internet users as if traffic to particular destinations had been blocked deliberately.


Peering disputes often seem to boil down to which company should have to pay for the upgrades in network architecture required to connect users to innovative new services. Should it be the content providers, the backbone companies that own data links, or residential ISPs? Increasingly, big residential ISPs like Verizon are asking for money from other players who want to send more data to end users—data that must go through the ISPs to reach their customers.

Discrimination can happen over any connection

Some journalists and network engineers have claimed that network operators may intentionally delay or avoid upgrades to garner more leverage in peering negotiations. That is, company A may know that the capacity of their connection to company B is inadequate and that the connection is frequently congested—but not fix the problem because it wants operator B to pay up in exchange for better connectivity.


Simply put, non-neutral behavior isn't only a matter of the relationship between ISPs, websites, and users. And it isn't only a matter of blocking traffic or forbidding users from reaching a certain site or from using certain software. It could be a matter of infrastructure fights that make some parts of the Internet dramatically faster and more reliable to reach than others. The details of the relationships and connections between ISPs and all other web services are extremely important to any long-term vision of network neutrality. The problem is that peering contracts are negotiated behind closed doors and often contain non-disclosure agreements, making it difficult to locate the source of a slowed down connection unless one of the companies involved chooses to take its case to the public.

This is why it was remarkably difficult to find the cause of Dave's slow Internet. It's hard to tell whether his bad Internet connectivity is an infrastructure problem or if ISPs are engaging in deliberate discriminatory, anti-competitive behavior. Companies are likely to disagree about what 'nondiscriminatory' peering would mean, since no two Internet networks are exactly alike, but these inevitable debates should not prevent us from articulating the goal that Internet interconnections need to be offered fairly to all.


And even when the reason for bad connectivity is identified, companies sometimes try to deny or shift the responsibility to build the infrastructure needed to meet the public's ever-increasing data demands. ISPs may argue that refusing to maintain network architecture isn't technically the same as interfering with network traffic. But refusing to improve infrastructure can be a form of discrimination. Keep in mind that if the Comcast and Time Warner Cable's merger comes to fruition, Comcast will have a tremendous amount of leverage in closed-door peering disputes.

So let's shine some light on these secret contracts. Peering agreements need to be brought to public view. And peering arrangements should be offered on nondiscriminatory terms, so that particular networks—and with them, particular Internet sites and services—aren't favored with access to fast connectivity on terms that aren't available to their competitors.


Brining transparency to the network of networks will be complicated, but it's not impossible. Unfettered, ubiquitous, and high-speed access to the Internet is far too important to be negotiated away in obscurity.

This article is reproduced from Electronic Frontier Foundation under Creative Commons license. Image by INCH.


Share This Story

Get our `newsletter`


So here's the deal:

1) Netflix buys some IP space, forgets some important configuration step, and drops some servers on it.

2) An engineer at an ISP sees a ton of traffic from a lot of people on a specific port to or from a specific destination using some sort of toolset designed to monitor patterns. It comes across on a specific IP address that the system doesn't reverse trace to something reasonable. The natural and reasonable assumption is that it is some new virus or malware destination (because this happens ALL the time), so the engineer throttles traffic way back because it is wasting bandwidth. This is part of the network engineer's job but they generally don't go back and review things.

3) A user notices that the IP address they are connecting to is serving content more slowly than usual. It turns out to be Netflix or some major provider that is constantly buying new IP space. The user calls their ISP. The ISP denies that it is throttling Netflix because they don't want that sort of public backlash. Customer service personnel are trained monkeys who are not allowed to think for themselves outside of the script they are reading. But regardless of whether or not the script covers "throttling", they are trained to listen but ignore the customer. Also, the network engineers and customer service rarely speak to each other. As a result, there is a massive disconnect.

4) After about the 1,000th call into the ISP of various customers complaining about the same thing, the customer service center finally notices a trend and pushes the issue upstream.

5) The message becomes one of those really long e-mail chains before it finally gets back to the network engineer.

6) The network engineer reads the e-mail chain, checks out the traffic map for some of the affected users, recognizes that they throttled the IP address for Netflix by accident, and undoes the damage.

7) Three months later the entire process repeats.

The process is a lot less exciting when you realize that the problem is that companies like Netflix don't assign new IP addresses proper reverse lookup information before they go into production and that there are huge, inappropriate disconnects within ISPs because the network engineers don't have to field customer calls. In a case like this, it is entirely likely that Netflix is at fault for not configuring their systems correctly, not necessarily the ISP that's doing the throttling of probably bad characters - throttling malware keeps spam from botnets at bay. Network engineers are busy so they don't have time to do much with "misbehaving" IPs chugging excessive amounts of bandwidth except either block or throttle them. However, network engineers should be required to, at least one week per year, sit at the phones in customer service and field real calls from customers.