When the New Horizons probe glitched and went into safe mode last weekend, the situation was worse than anyone outside the lab even knew. For hours, the lab heard not a peep from their probe, uncertain if it even still existed. When it finally called home, it was the first-ever test of the backup computer.
New Horizons team members Priya Dharmavaram and Tim Miralles monitoring telemetry from the operations center in the Applied Physics Laboratory following a course-correction maneuver on June 14, 2015 a retroactively far less-stressful day than July 4. Image credit: NASA
The New Horizons probe calls home to the Applied Physics Laboratory, a lab whose slogan “Critical Contributions to Critical Challenges” was sorely tested on July 4th. Joel Achenbach interviewed the science team on the behind-the-scenes details. At first, all the team knew was that their much-anticipated, very expensive probe had just vanished from all communication. They had no way of knowing if it was temporarily offline, permanently lost or even utterly destroyed. When they finally found it transmitting on an entirely different frequency, it was with an untested, untrusted backup computer they’d hoped to never use. From there, Achenbach describes the recovery process:
The APL team had to reconfigure New Horizons the way you would rouse a drunk on a Sunday morning to get him ready for church. This required many commands, everything made slower by the nine-hour round-trip communication challenge across the 3 billion miles of space. [Mission Operations manager Alice] Bowman slept on her office floor a second night on Sunday.
One key decision: Return control of the spacecraft to the main computer. They trusted the main computer, knew its quirks, had tested it repeatedly — unlike the backup computer. Their plan didn’t require them to ask the main computer to do the kind of heavy-duty work that caused the glitch Saturday.
With the Pluto flyby looming and staying in safe mode meant the probe would be in a stabilizing spin that made scientific measurements impossible, they couldn’t delay making the decision. Their choice to switch the main computer paid off, and the probe was back online with its main computer without missing any mission-critical science objectives (although it did miss a few photos that will be “teaching moments” in the approach animations). New Horizons is currently partway through the Pluto Flyby phase of its mission, snapping photos and collecting measurements at such a prodigious rate it will take over a year to fully transmit all the data back to Earth.
Troubleshooting the misbehaving probe is an impressive enough task considering the multi-hour communications ping time, but for it to all depend on a live-fire test of the probe rebooting and switching to a never-used backup computer is even more impressive. I knew safe mode wasn’t a good thing to have happen in the days leading up to a critical flyby, but I’d somehow imagined the team had known their probe was still out there right away, not that they had to rely on blind faith that their backups would function properly when unexpectedly called into action. Colour me even more grateful that our little probe is not just back online, but doing science at such a rapid pace that it’s hard to keep up with the newest, greatest, most detailed shots of the dwarf planet and its moons.