Here's what they said caused it in the White House briefing:
Sec Mullin: "What it appears to be right now is, as Sec Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology. And that new equipment and new technology, it was overwhelming the older system. Which is why Sec Duffy made it very clear, the patchwork doesn't work. We need one complete system."
Because if it doesn't work, they call it out for bids
again, select their cronies and get kickbacks. If it does
work, then they've missed an opportunity for grift.
Duffy also kept saying they were working as quickly as they possibly could trying to rollout the new system. Not exactly reassuring.
I think the problem with a lot of the push to "modernize" is we forget the legacy system has been real-world tested for 40 years now. It's the new stuff that needs to prove itself in terms of stability.
Not really. The problem with keeping old technology is that you need people who have knowledge of how to maintain the old systems. Ignoring the cynical view that many of those people were likely fired, even if they were kept on, they will eventually retire. There's no amount of training that can replace that institutional knowledge.
If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.
Sure, but the purpose of the tech policy office is to
argue why old systems (which have fixed maintenance costs)
should be replaced with new systems which goose the
economy when the federal government pays 10x for what they
should be paying.
I can't tell if you're being ironic or not. Sure, "one complete system", and I want a pony.
I watched Sec Duffy's presentation, and it was appalling, but par for the course with this administration: in the middle of a major nationwide outage, he tried to blame someone else.
Even if Duffy got everything he wished for, how do you think they would upgrade a system as old and complicated as ATC. They would have to do it piecemeal anyway. This was a classic upgrade that broke shit. The problem was with the new upgrade, not the old shit that worked.
I was just presenting the full quote from the Secretary of Homeland Security explaining what went wrong. I wasn't commenting on it. I added quotation marks just now to make that more clear.
This is only one of many recent “lip sores” indicating the a herpes virus that is all over US critical infrastructure.
Over a decade ago I was an intelligence officer and the alarm bells were going off and it was exceptionally clear to the National security community that US infrastructure is barely holding on.
At the press conference (current, live), they are blaming old hardware that is failing. They’re in an endless loop of repeatedly saying that it’s Congress’s fault for lack of funding and that this is the result of old hardware. I just gave up on watching because they aren’t saying anything informative.
Not only was there a lack of useful information (other than the fact that the grounding was lifted), it was an astounding demonstration of poor leadership during a crisis. No ownership, no focus on the problem/root cause/next steps.
Usually I refuse to give any detailed postmortem right after an incident is mitigated/resolved until I have had a chance to collect my thoughts and make sure I have had a chance to write a clear problem/root cause/next steps. The communication that goes out with the mitigation signal is often short and vague and the detailed postmortem follows a day or so later.
Let's be clear: it was bunch of idiot political hacks being unbelievably obsequious to Trump. So embarrassing for the country. I don't trust that the ask for $50B is anything more than grift for their friends. Was the grounding a purposefull overreaction to increase odds of getting money?
I don't doubt that the system needs upgrades, but these people should not be involved.
I'm pretty sure saying that Congress needs to give them the money to replace the ancient hardware is putting ownership directly where it belongs and identifying the problem, the root cause, and the next steps.
That if the ancient hardware was working and just the new software they tried to put on it did not worked with that hardware - it still means they messed up.
I didn’t say it isn’t. Just that they aren’t really giving any details and more importantly, they aren’t giving info for what people who are stranded should be doing right now. (And now that I’ve settled down a bit, it’s odd that if it’s old hardware failing, how is it suddenly not failing? Was it just some systems, did they have replacements, or if it’s what the other commenters wrote, then did they just turn off the AI updates to stop overloading the hardware? Which would mean that hardware is fine. It’s just underprovisioned.)
Not with the budget the FAA has to apply to the problem.
Yes, you have a point. But the importance of this system could get circuit boards manufactured to old drawings. Specific 'electronics boxes' replaced with new ones designed to replace the antiquated hardware.
FAA has been doing this for decades. A single electronics box, or even a couple should not drop the system. Maybe for the time it takes to remove and replace.
After hearing the press conference "some new stuff was plugged in and the legacy system could not handle it" or alternatively 'a mistake was made that just so happens involved non today equipment, but we want more money'
I'm being a little bit tongue in cheek, I'm sorry your wife is stranded.
I expect it to be a combination of extremely poorly funded and sidelined IT infrastructure trying to use a bloated AI system from a hype company. Which results in everything breaking and poor schmucks already at the end of their rope working to fix something they barely understand.
Because everyone [in the specific field of technical professional operating systems like this] is using AI and not everyone is smart about how they use AI?
System worked mostly fine for decades, then in the middle of integrating a new AI system it completely goes down.
It doesn't take a genius to see the problem is the new AI system. Whether or it's the "AI" part of the system isn't really relevant system this new system was vibe-coded by AI and built around the "AI" features.
Given that the old hardware has been around a long time without a problem like this, it seems very odd to blame the old thing for the new problem. My money is on a poorly planned transition, one where they didn't take the old hardware seriously enough.
It's certainly possible - but for whatever reason this administration (with the help of that guy from the robot taxi company) seems to not like air traffic controllers and has been trying to find ways around them.
It's possible this is the standard "privatize a government function and skim the profits" move but some of this stinks of union busting and trying to reduce the amount of influence air traffic controllers have whenever the organized criminals that run our government want to have a shutdown but keep their private jets flying.
Suffice it to say - there's no reason to trust any of these people when it comes to taking care of critical government functions.
Ah sorry for the misleading writing. My brain has been fried trying to work while also trying to dig up any useful info to help my wife who is stranded at the airport figure out what to do. I usually can last longer watching press conferences but this one was… just painful. Maybe someone else can give an update if they actually talk about what’s going on.
They were also insisting its not a problem with the new upgrades they were installing, but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
Not sure if this was the the exact quote, but I see AP wrote, "introduction of some new technology into an old air traffic control system overwhelmed it." Despite that, the whole conference, on loop, they were insisting that the problem is entirely the old hardware and this is why they need billions to overhaul it and rebuild from scratch.
>but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
The secretary also mentioned that the new system should be brought altogether, and not piece by piece. Which would lend credence to an incompatibility between both systems.
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system"
I assume that this was a boring, routine update that went awry.
But my completely unfounded and more conspiratorial theory, based on zero knowledge of air traffic control systems, is that to better accommodate SMART, L3Harris was strong-armed into to rolling out a hastily written patch on a Friday, which accidentally took down the system.
The press should be asking why this could be the case when the GOP has held congress and the white house for two years ? Just what was the party in charge thinking ?
I think the supplemental appropriation that the FAA got as part of the OBB is probably too big to be spent.
Unless you just hand it all to a contractor, it takes time and oversight to spend billions. Legally and practically.
"It's the safest system in the world, I mean, uh, except that it's horribly outdated with switches from the 80s and radars from the 70s, so we really need 50 billion dollars from congress STAT or we're in big trouble, but it's completely safe. Also Congress needs to give Trump the Builder this money _now_. We couldn't get it in the past couple of years even though we controlled the entire government. I mean, sure, we got 12 billion, but that's not enough to unfuck what Joe Biden did. Your loved ones are completely safe by the way, but we can't promise this won't happen again."
We need a despot that will ignore all bureaucratic bloat and get things done, quite frankly. Otherwise this country is done for. Spiraling into the trash can.
Transportation secretary saying that their networking routers were being overloaded by increased communications traffic. But its not a cyberattack or software failure.
They just said in the press conference that it couldn't possibly be related to SMART, since the systems were disconnected, but the timing and lack of a concrete answer as to the cause or likely cause, combined with asking for money from congress (which currently has a republican majority...) for "more modern infrastructure" has me doubting their version of events
I acknowledge this is borderline conspiracy. Sec Mullen's statement is that FAA was quote "installing new equipment, new technology and that the new equipment new technology was overwhelming the system"
Transportation secretary Duffy says that the smart system is unrelated. That the smart system is a dashboard for displaying all the data from across the air traffic system. My questions is obviously where is the smart system getting its data from...
I suspect FAA pushed for the new AI system, didnt take into account the increased traffic demand to feed the system. Aging infra isnt sufficient to handle the increased demand. Now Duffy is trying to deflect blame unto someone else.
I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was...
Should be an easy answer. We saw increased traffic from X and we've implemented Y to ensure it doesn't happen again.
Instead it was vague hand waving around old hardware and funding.
Even if they don't have all the details organized, what was done to fix the problem? How was the 12b spent thus far and what where does the additional 50b number come from?
Or maybe don't start with demanding money. What happened? How was it fixed? What is being done to ensure it doesn't happen again? Then mention this shit needs fixed and we'll be working with Congress for additional funding.
It was interesting that Mullen mention new technology contributing at one point. Sounds like someone made a mistake somewhere and this press conference was purposefully vague or they really don't understand what happened yet (doubtful).
Plenty of critical systems operate reliably on old tech as long as you work within the limitations.
This press conference had maybe 30s of actual info and the rest was this antiquated system is fucked even though it's been working for years and the only way to move forward is $50b. :shrug:
maybe, what is confirmed is that they were trying to upgrade some physical TDM switches with IP routers (as they have been doing for a while), and the "new systems overwhelmed the old ones".
It is possible that this overwhelming was caused by a software or configuration error, no doubt.
I think that if that's the case, rather than failing upwards and going for a full simultaneous upgrade with a 50B congress bill, the brakes should be stepped on a little bit, continue the partial migration as long as there are no issues. Or at least wait until the start of a 4year administration to go for a rehaul, so that you can stay around if things go fubar.
Unclear what happened but they have old equipment. Only Duffy can fix it, he needs $50B from Congress to do it and they can't fix it overnight. They can't guarantee it won't happen again.
So they were doing a network upgrade and putting in new equipment. This new equipment was misconfigured and caused the existing working system to stop working. And now they are saying in the press briefing that this is a problem of the existing working system which they call the "legacy system" and are using this as a political move to call for the congress the regime no longer controls to provide funding to the FAA/DHS. Framing the manufactured issue they caused with the new system as if it was congress' problem instead of their own breakage they caused and are deflecting from. Demanding an emergency funding bill now despite ignoring it in all the years they were in control.
It’s possible the interfaces to the legacy system aren’t documented well and/or it could be incompetence on behalf of the network engineers working this
There is an actual bill for $30B that's been stalled for months in the Senate called the Modern Skies Act, and there is absolutely widespread antiquation within the US air infrastructure which this bill seeks to address. So he's not just making this up. He's talking about exactly that bill albeit with all the Trump BS injected into it.
Modern Skies Act is for $30B, not $50B. A separate $12.5B from "Big Beautiful" is already being spent on ATC modernization, and only $10B of MSA is toward ATC
Sure, but blaming Democrats when they’ve controlled Congress and the presidency for the past two years and their highest priority has been cutting funding via DOGE is pretty disingenuous
Their press conference was horrendous but it seemed he was just blaming Congress overall and not Democrats specifically (since it is Republicans actually holding it up)
Assuming I'm looking at the right bill (S. 5572, the Modern Skies Act), "stalled for months" seems a tad overstated considering it was introduced Sep 25th 2026, just three days before Congress went on recess for the entire month of October (elections). The Senate wasn't back in session until Nov 9th, so the bill has been pending for a grand total of seven in-session days (even emergency legislation would rarely pass that quickly).
The Senate will be in session for four more days next week before the Thanksgiving holiday, and then back for the first three weeks of December before the 119th Congress adjourns for good on Dec 18th. So basically, after two full years with control of both branches of Congress and the WH, this bill was introduced on the very last week of the regular Congress calendar. After which the lame duck Congress would only meet for about four weeks until adjournment.
Not really stalled, more like procrastinated until days before the election (when everyone is scared to vote on anything), hoping that the lame duck Congress (notoriously unproductive period generally) would force it through in a rush?
Doesn't seem like a great strategy if they actually wanted it to pass (especially since a budget CR is needed before Dec 11 which will dominate members' attention in the remaining time).
Not much info yet. Even ops.group [1], which is for people who do flight planning, has nothing yet.
LAX hasn't restarted yet. Departures are either "Cancelled" or "Delayed".
All information about this is from PR teams. Nothing from reporters who cover airports. Nobody from airport management, airline operations, FAA operations people, or somebody who just got off shift at a control tower. People who might actually know something.
There was a briefing at the White House, and it was all cabinet secretaries. "Instead of reassuring an anxious public, and providing concrete information, this briefing appears to be intended for President Trump’s approval" wrote a New York Times reporter.
FAA policy prohibits anybody but PR people talking to the public. The details of that policy are not public.[2]
Doesn’t it feel wrong to anyone else that there’s a single point of failure for all fights in the US? I can understand if there are regional zones and the East coast gets grounded. But everything at once??
Is it a bad idea? Absolutely. But also it's to be expected. People like to have a single point of control. You can't do that without introducing single points of failure.
I wish more was being done on partition tolerance. Consistency is over rated.
"we" being him (ie. not the reporter) because he's part of the executive branch; that's the limit of the authority of the executive order, after all...
They jumped from 3 airports to nationwide in one step, and are now framing it as the existing hardware not being compatible with their new software, rather than the other way around.
Ahh the great Kubernetes cluster in the sky finally finished reconciling and got the new nodes up and running. I bet this time it only took 3 faxes and 9 different signatures from heads of the redundant departments of redundancy.
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system".
So, it was a careless update, probably the one deploying their new 875 millions AI tool.
And no journalist at the conference picked that up.
Sec Mullin: "What it appears to be right now is, as Sec Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology. And that new equipment and new technology, it was overwhelming the older system. Which is why Sec Duffy made it very clear, the patchwork doesn't work. We need one complete system."
https://static.wingify.com/gcp/uploads/abtasty/test-in-produ...
I think the problem with a lot of the push to "modernize" is we forget the legacy system has been real-world tested for 40 years now. It's the new stuff that needs to prove itself in terms of stability.
If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.
I watched Sec Duffy's presentation, and it was appalling, but par for the course with this administration: in the middle of a major nationwide outage, he tried to blame someone else.
Even if Duffy got everything he wished for, how do you think they would upgrade a system as old and complicated as ATC. They would have to do it piecemeal anyway. This was a classic upgrade that broke shit. The problem was with the new upgrade, not the old shit that worked.
I agree with your view.
Over a decade ago I was an intelligence officer and the alarm bells were going off and it was exceptionally clear to the National security community that US infrastructure is barely holding on.
Clapper in 2013:
https://nsarchive.gwu.edu/document/21491-document-90
Dan Coates in 2018:
https://www.euronews.com/2018/07/13/us-intel-chief-warns-of-...
Tip of the iceberg folks. Hold on to your butts…
I don't doubt that the system needs upgrades, but these people should not be involved.
What am I missing?
As a postmortem, once the dust has settled, is the time to ask for more funding/support
FAA has been doing this for decades. A single electronics box, or even a couple should not drop the system. Maybe for the time it takes to remove and replace.
After hearing the press conference "some new stuff was plugged in and the legacy system could not handle it" or alternatively 'a mistake was made that just so happens involved non today equipment, but we want more money'
I expect it to be a combination of extremely poorly funded and sidelined IT infrastructure trying to use a bloated AI system from a hype company. Which results in everything breaking and poor schmucks already at the end of their rope working to fix something they barely understand.
It doesn't take a genius to see the problem is the new AI system. Whether or it's the "AI" part of the system isn't really relevant system this new system was vibe-coded by AI and built around the "AI" features.
Do you really need the government to solve that problem for you? Just book another flight.
It's possible this is the standard "privatize a government function and skim the profits" move but some of this stinks of union busting and trying to reduce the amount of influence air traffic controllers have whenever the organized criminals that run our government want to have a shutdown but keep their private jets flying.
Suffice it to say - there's no reason to trust any of these people when it comes to taking care of critical government functions.
> old system give us money
> probably not a cyberattack btw
> not SMART related
> we've opened the airspace but this is gonna be slow
just ended actually as i was typing this, that's pretty much it
They didn't.
Not sure if this was the the exact quote, but I see AP wrote, "introduction of some new technology into an old air traffic control system overwhelmed it." Despite that, the whole conference, on loop, they were insisting that the problem is entirely the old hardware and this is why they need billions to overhaul it and rebuild from scratch.
The secretary also mentioned that the new system should be brought altogether, and not piece by piece. Which would lend credence to an incompatibility between both systems.
(https://news.ycombinator.com/item?id=50492957)
But my completely unfounded and more conspiratorial theory, based on zero knowledge of air traffic control systems, is that to better accommodate SMART, L3Harris was strong-armed into to rolling out a hastily written patch on a Friday, which accidentally took down the system.
Speculation on my part, they just rolled out their SMART air traffic routing system. https://www.flyingmag.com/faa-smart-ai-now-active-at-22-artc...
Bet the increased strain on data polling from this system is the culprit for increased traffic.
Transportation secretary Duffy says that the smart system is unrelated. That the smart system is a dashboard for displaying all the data from across the air traffic system. My questions is obviously where is the smart system getting its data from...
I suspect FAA pushed for the new AI system, didnt take into account the increased traffic demand to feed the system. Aging infra isnt sufficient to handle the increased demand. Now Duffy is trying to deflect blame unto someone else.
Should be an easy answer. We saw increased traffic from X and we've implemented Y to ensure it doesn't happen again.
Instead it was vague hand waving around old hardware and funding.
Even if they don't have all the details organized, what was done to fix the problem? How was the 12b spent thus far and what where does the additional 50b number come from?
Or maybe don't start with demanding money. What happened? How was it fixed? What is being done to ensure it doesn't happen again? Then mention this shit needs fixed and we'll be working with Congress for additional funding.
It was interesting that Mullen mention new technology contributing at one point. Sounds like someone made a mistake somewhere and this press conference was purposefully vague or they really don't understand what happened yet (doubtful).
Plenty of critical systems operate reliably on old tech as long as you work within the limitations.
This press conference had maybe 30s of actual info and the rest was this antiquated system is fucked even though it's been working for years and the only way to move forward is $50b. :shrug:
It is possible that this overwhelming was caused by a software or configuration error, no doubt.
I think that if that's the case, rather than failing upwards and going for a full simultaneous upgrade with a 50B congress bill, the brakes should be stepped on a little bit, continue the partial migration as long as there are no issues. Or at least wait until the start of a 4year administration to go for a rehaul, so that you can stay around if things go fubar.
Unclear what happened but they have old equipment. Only Duffy can fix it, he needs $50B from Congress to do it and they can't fix it overnight. They can't guarantee it won't happen again.
https://www.transportation.gov/sites/dot.gov/files/2026-09/M...
The Senate will be in session for four more days next week before the Thanksgiving holiday, and then back for the first three weeks of December before the 119th Congress adjourns for good on Dec 18th. So basically, after two full years with control of both branches of Congress and the WH, this bill was introduced on the very last week of the regular Congress calendar. After which the lame duck Congress would only meet for about four weeks until adjournment.
Not really stalled, more like procrastinated until days before the election (when everyone is scared to vote on anything), hoping that the lame duck Congress (notoriously unproductive period generally) would force it through in a rush?
Doesn't seem like a great strategy if they actually wanted it to pass (especially since a budget CR is needed before Dec 11 which will dominate members' attention in the remaining time).
LAX hasn't restarted yet. Departures are either "Cancelled" or "Delayed".
All information about this is from PR teams. Nothing from reporters who cover airports. Nobody from airport management, airline operations, FAA operations people, or somebody who just got off shift at a control tower. People who might actually know something. There was a briefing at the White House, and it was all cabinet secretaries. "Instead of reassuring an anxious public, and providing concrete information, this briefing appears to be intended for President Trump’s approval" wrote a New York Times reporter.
FAA policy prohibits anybody but PR people talking to the public. The details of that policy are not public.[2]
[1] https://www.ops.group/blog
[2] https://www.faa.gov/regulations_policies/orders_notices/inde...
BTW, Uber surge pricing felt absurd in this situation.
EVENT TIME: 14/0116 - 14/0130 DESTINATION AIRPORT:ALL FACILITIES INCLUDED: ALL EXPECT UPDATE:0400Z REASON: REMARKS: THE FAA COMMAND CENTER NATIONWIDE GROUND STOP HAS BEEN CANCELLED. EFFECTIVE TIME:
140119-140200 SIGNATURE:
26/11/14 01:19
I wish more was being done on partition tolerance. Consistency is over rated.
Unrelated I only fly ocunus because I don't want a blue haired pilot.
So it was the AI?
Who could have seen this coming?
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system".
So, it was a careless update, probably the one deploying their new 875 millions AI tool.
And no journalist at the conference picked that up.