AT&T's Internal Plans To Fix Their Network
AT&T was calling me to set up an interview with their CTO, but all I could hear was garbled noise on my AT&T iPhone. "I can't really hear you!" I shouted, as if volume would clear the channel. It's always been like this, in my home in San Francisco. While the howls of iPhone 3G reception issues get louder and louder, I've always wondered if it was the network's fault, as some Swedish scientists and journalists have recently suggested. Maybe it's just new AT&T customers making the bulk of the noise. From my experience, the phone isn't blameless, but the network is a major part of the issue. Continuing the call on an land line, I said I'd be glad to meet with John Donovan. To be perfectly honest, on a certain level, I didn't really want to ask questions. All I wanted to do was get the guy in front of me and berate him for his network's voice quality and reception, relatively slow 3G rollout and coverage. AT&T's been last in all those metrics for years (according to JD Powers) and they were still raking in the bucks as the nation's largest carrier. But after meeting him, I'm certain John Donovan has the intent the Old AT&T didn't. And a detailed plan on how to make "More Bars in More Places" less like a joke and more like a promise. In fact, Donovan surprised me when he said, "We want to be number one in all those metrics" — That's a lot of big talk when I can't even make a call from my own house right now. Here's the outline of the plan, which Donovan provided later — as well as some straight talk from an AT&T engineer on whether or not the plans will work.
•Cell site splitting. We're deploying about 1,500 new cell sites this year. This enhances service quality in two ways – we're expanding the geographic reach of the network, and in some places, we're adding cell sites in existing territory to improve coverage and capacity. •We monitor the usage for both data and voice on each and every cell site sector, combine that with our forecasts and customer feedback to target specific locations where we need to augment capacity via further cell splits. •TDMA turndown. Earlier this year we turned down our TDMA/Analog networks. That allowed us to free up key spectrum to redeploy into the UMTS/HSPA network. Not only does that give us the opportunity to increase overall capacity on UMTS/HSPA, it enables us to deploy UMTS/HSPA at 850 Mhz (vs 1900 Mhz). This 850 spectrum provides optimum in-building coverage. We're in the midst of turning up this spectrum on the UMTS-HSPA network across the country. •Short measurement intervals. We're deploying capabilities to measure network performance in much shorter intervals. This capability will be especially important to maximize service quality during major events. For example, during a Final Four or World Series game, traffic on the network will spike during time outs, or when a great play, bad call, or something else big happens. In normal circumstances, we might measure traffic and performance a few times per hour, but in these situations, we'll monitor every few minutes to ensure maximum performance during the spikes that occur in real time. •Wireless backhaul. We're leveraging our U-verse and metro Ethernet fiber deployments to enhance wireless backhaul connections in many areas. We're moving more and more of our wireless backhaul onto the 40 Gbps AT&T backbone network. So … the investments we're making to maximize service quality on the wired side also benefit our wireless customers. •Drive testing. AT&T technicians and other third-party vendors will drive-test its network nearly 30 million miles in the next year to improve the network's coverage and quality. •Hundreds of technicians from third-party testing companies, infrastructure vendors and AT&T technicians use specially designed vehicles to travel throughout the country and test the signal strength and coverage of AT&T's ALLOVER Network. •Along with drive-testing its own network, AT&T drive-tests competitors' networks to ensure that its coverage and quality are equal to or better than other companies providing service in the area. •The drive-test results help to prioritize where the company invests in new cell sites and equipment that enhances the network quality and coverage.
While a great deal of the document above is fairly obvious, there are many roadblocks to executing the plan and improving the network. Donovan's interview revealed some details of the plan above, but some engineers within the ranks gave me a great deal of insight, too. A large part of the problem, Donovan said, is that people would complain, and yet, by all of their measurements, the user should have had full bars in the place and at the time they reported the poor coverage. Hence the need for better tools and more frequent sampling, instead of several times per hour, they'd do it every few minutes during congested periods. A great deal of that testing is done using network tools, but drive testing will help, and AT&T also tests their competitors' networks for comparison. Adding more towers in a place is not simple. It's a local affair, requiring navigation of local building codes and politics. Somewhere like SF makes that hard, but the hilly terrain wouldn't help the situation either. And while 30 million miles of driving sounds like a good idea, the engineers I talked to insisted that drive testing is really just a final check once you've got enough towers in place. "Save the money on drive testing and build sites or improve sites we already have...[by] buying t-1s to increase capacity." He also commented that adding 1500 towers alone isn't enough to solve the problem. Turning down analog networks sounds like a winning strategy, as long as you don't mind grandma's cell call quality being degraded. The activation of the 850MHz band will also enable better indoor reception, which is going to be critical in expanding data/voice quality as user counts go up. Donovan also said that they'd shift their allocation of bandwidth towards data from voice, which makes sense. Analog aside, the EDGE legacy is taking up resources on the towers which are shared with 3G. As one of the engineers said, having UMTS and GSM use the same antennas causes "interference and performance. It's like putting a splitter on a garden hose the flow is still there but volume is cut in half on each side." (This is where Verizon and Sprint have an advantage.) And as Wired has realized, 3G range being more limited, AT&T can't actually blanket a city by using the same tower locations as their EDGE counterparts. But my engineer friend also said, "We went from 2.5g experts to 3g novices." They don't have the training or experience to find or fix issues as well as they do on EDGE networks. Many in the field are also lacking the expensive test gear for UMTS to find bad channels and interference. Why didn't AT&T make these investments in the first place, while Starbucks and T-Mobile worked on Wi-Fi hotspots and Sprint/Verizon went 3G ahead of the curve? Money. And while AT&T's financially conservative strategies in the past have limited expansion, there's no reason they couldn't also do so in the future. I asked Donovan if caution was the overriding strategy behind waiting to match Sprint's initial 3G rollout, he replied, "I'd like to say we're deliberate. " He added that initially meeting the voice quality and data rates of Sprint's 3G network would have been both technically and financially impossible, despite the customer benefit. (One only needs to look at Sprint's financial weakness now to appreciate the wisdom of his point.) He also pointed out that by waiting, they got to leapfrog the limitations of Sprint's EVDO networks, referring to the extended data rates their network will eventually run at, at a better value. "The most astute thing you can do is be as late as possible and as fast as possible. Because it's going to cost you more if you do it too early, and if you do it too late, you don't get the features you want." When AT&T's LTE networks do make the jump past Sprint, from 1.7Mbit to 7Mbit to 11Mbit to 20Mbit, their bottle neck will move to their backend infrastructure. To counteract that, they'll depend on their extensive wired and fiber backhauls the company has. But, in areas where the telco is owned by AT&T, AT&T wireless still has to buy lines from themselves and the budget isn't there. According to at least one engineer, "We still operate at the field as two completely different companies. While at the top they see "ONE" we see many." Will AT&T succeed at having the best call quality, coverage and reception? Who knows. But at least here, we have their plans on record and can hold them to the goal. After all, they're the biggest carrier — they've got a responsibility to all of us to make their network the best, no matter what the cost. I just want to be able to hear the other person on the end of the line.