For those interested, there was a 2009 outage of the FAA Telecommunications Infrastructure that at a surface level seems like it could be a similar failure mode. At the very least, it is an interesting read on how the FAAs systems operate, since a lot of the discussion so far has been wild speculation from people with little knowledge on how the national airspace system actually operates: https://www.oig.dot.gov/library-item/29670
Of particular note is this section:
Risks of FTI Outages for Existing Operational Services
As of December 2009, Harris reported that there are a total of 20,982 services operating on the FTI network. While FAA has transitioned all 1,808 IP services supporting FAA flight plans to the new FTI fiber optic network, additional risks will be introduced when the Agency begins transitioning other FAA services, such as En Route Automation Modernization (ERAM) and NextGen platforms, which also require IP technology to operate. Moreover, there are 19,174 existing services that may be transitioned to the new FTI fiber optic network. According to FAA, transitioning these remaining services is important because FTI now connects older FAA systems that provide safety-critical voice and surveillance radar information by utilizing Time Division Multiplex (TDM) technology, instead of IP services.
TBH TDM is far superior and it’s sad that the world discarded it. Packet switching is the worst thing to ever happen to phone calls. Kids today have no idea how good it used to be (albeit at $0.50/min).
Here's what they said caused it in the White House briefing:
Sec Mullin: "What it appears to be right now is, as Sec Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology. And that new equipment and new technology, it was overwhelming the older system. Which is why Sec Duffy made it very clear, the patchwork doesn't work. We need one complete system."
Duffy also kept saying they were working as quickly as they possibly could trying to rollout the new system. Not exactly reassuring.
I think the problem with a lot of the push to "modernize" is we forget the legacy system has been real-world tested for 40 years now. It's the new stuff that needs to prove itself in terms of stability.
We have a rule in my dev shop: if a piece of code (or hardware) has been working without fault in production, you do not touch it. It doesn't matter if it's old, if you don't like the tests, if it's in a programming language you despise; if there haven't been any problems in production, you don't touch it.
And then you have the common reality: "a piece of code has been working in production with known faults & known costs". And you have to weigh up costs and risks.
Sounds like the new system effectively performed a DoS on the old system. They probably don't have a development environment where they can hook the two systems up. Even if they do, it might actually have been an out-of-the-way component that failed first, impacting another, and so on. If they have a test environment with the two systems, a complex failure like that would unlikely to be found (and would explain why it took hours to resolve).
As someone who works under a different umbrella in the federal government, how do you propose testing something so archaic and complex without replicating an exact replica of a 40+ YO technology?
I've actually worked a system where the test env was scavenged old eq set up to mirror exactly the ancient prod env. It's possible, it just takes a lot of work. Scaling it might be touchy though.
Not only it takes lot of work, but to have the replica connect to the prod in some way to mirror identical behavior will possibly mean adjusting ATO, which is not a simple process in itself.
Emulate, computing is cheap. In an application as important as air traffic control it seems like it wold be worth it to _always_ have a working copy of the original system specifically for testing changes.
Not aware of specific system, but critical infrastructure systems have a quality assurance system that replicates production albeit not perfectly. You may replicate the core system but not all its interfaces, you might replicate it but scaled down etc. There are ways for solid tests to fail. For systems with downtime, a test on production is useful iif you have solid rollback procedures. There’s lots of art in the operation of these systems.
Let's set that aside and instead ask why tf they rolled out anything on a Friday? Or was it that they rolled it out on Tuesday but failed to notice it was close to overload state until it actually overloaded on Friday?
Because if it doesn't work, they call it out for bids
again, select their cronies and get kickbacks. If it does
work, then they've missed an opportunity for grift.
They also kept noting that the new IP based routers were not showing any inconsistency. To me that says that they were bridging IP traffic over the legacy systems, probably a protocol with retransmits like TCP, and a storm of retransmits was overloading the legacy systems, and older components using a less durable protocol were drowned out and dropped packets.
The parts to maintain the legacy copper infrastructure are no longer manufactured and they are cannibalizing used parts for spares.
Not to mention the expertise for these systems have mostly either retired or in some cases passed away.
My grandparents still have a landline and the number of outages in the last 12 months has been insane. Some were in excess of a week.
Many such cases.
Sometimes the telcos have been shrugging their shoulders and refusing to perform repairs. This is common after a car accident takes out a pole, cabinet, etc. They accelerate the retirement to right now and force the customers onto new technologies.
Not really. The problem with keeping old technology is that you need people who have knowledge of how to maintain the old systems. Ignoring the cynical view that many of those people were likely fired, even if they were kept on, they will eventually retire. There's no amount of training that can replace that institutional knowledge.
If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.
Sure, but the purpose of the tech policy office is to
argue why old systems (which have fixed maintenance costs)
should be replaced with new systems which goose the
economy when the federal government pays 10x for what they
should be paying.
This is only one of many recent “lip sores” indicating the a herpes virus that is all over US critical infrastructure.
Over a decade ago I was an intelligence officer and the alarm bells were going off and it was exceptionally clear to the National security community that US infrastructure is barely holding on.
I can't tell if you're being ironic or not. Sure, "one complete system", and I want a pony.
I watched Sec Duffy's presentation, and it was appalling, but par for the course with this administration: in the middle of a major nationwide outage, he tried to blame someone else.
Even if Duffy got everything he wished for, how do you think they would upgrade a system as old and complicated as ATC. They would have to do it piecemeal anyway. This was a classic upgrade that broke shit. The problem was with the new upgrade, not the old shit that worked.
I was just presenting the full quote from the Secretary of Homeland Security explaining what went wrong. I wasn't commenting on it. I added quotation marks just now to make that more clear.
Since the threads keep getting closed about Hawaii DoT confusing everyone by claiming SpaceX debris was contributing to their shutdown despite the Starship launch being in Texas, successfully reaching orbit, and having a booster land in the Atlantic:
Exploded safely as planned during an off shore divert from the tower catch attempt, after making a controlled hover in mid air slightly above the water.
Notably, the problem was described as "routers overloaded with data". When directly questioned if this could be a cyber attack or rogue ai agent the Sec/Homeland responded "there's nothing completely certain now."
Or a non-technical person is relating the information as they understand it to a non-technical press corps, so everything is generic and hand-wavy. Honestly, have you never dealt with non-technical C-suite staff?
At the press conference (current, live), they are blaming old hardware that is failing. They’re in an endless loop of repeatedly saying that it’s Congress’s fault for lack of funding and that this is the result of old hardware. I just gave up on watching because they aren’t saying anything informative.
Not only was there a lack of useful information (other than the fact that the grounding was lifted), it was an astounding demonstration of poor leadership during a crisis. No ownership, no focus on the problem/root cause/next steps.
Usually I refuse to give any detailed postmortem right after an incident is mitigated/resolved until I have had a chance to collect my thoughts and make sure I can write a clear problem/root cause/next steps. The communication that goes out with the mitigation signal is often short and vague and the detailed postmortem follows a day or so later.
Let's be clear: it was a bunch of idiot political hacks being unbelievably obsequious to Trump. So embarrassing for the country. I don't trust that the ask for $50B is anything more than grift for their friends. Was the grounding a purposeful overreaction to increase odds of getting money?
I don't doubt that the system needs upgrades, but these people should not be involved.
I thought you were being a jerk, but no this dude has an Associate in Applied Science in plumbing from the Oklahoma State University Institute of Technology. Huh.
I'm pretty sure saying that Congress needs to give them the money to replace the ancient hardware is putting ownership directly where it belongs and identifying the problem, the root cause, and the next steps.
That if the ancient hardware was working and just the new software they tried to put on it did not worked with that hardware - it still means they messed up.
There's still time for that. The mayhem is ongoing and will be felt for days.
I hope by Monday a few "bosses" will say thanks to all the staff who ran overtime to deal with desperate grounded passengers, do all the rebooking snd schedule changes.
Ah sorry for the misleading writing. My brain has been fried trying to work while also trying to dig up any useful info to help my wife who is stranded at the airport figure out what to do. I usually can last longer watching press conferences but this one was… just painful. Maybe someone else can give an update if they actually talk about what’s going on.
I didn’t say it isn’t. Just that they aren’t really giving any details and more importantly, they aren’t giving info for what people who are stranded should be doing right now. (And now that I’ve settled down a bit, it’s odd that if it’s old hardware failing, how is it suddenly not failing? Was it just some systems, did they have replacements, or if it’s what the other commenters wrote, then did they just turn off the AI updates to stop overloading the hardware? Which would mean that hardware is fine. It’s just underprovisioned.)
Not with the budget the FAA has to apply to the problem.
Yes, you have a point. But the importance of this system could get circuit boards manufactured to old drawings. Specific 'electronics boxes' replaced with new ones designed to replace the antiquated hardware.
FAA has been doing this for decades. A single electronics box, or even a couple should not drop the system. Maybe for the time it takes to remove and replace.
After hearing the press conference "some new stuff was plugged in and the legacy system could not handle it" or alternatively 'a mistake was made that just so happens involved non today equipment, but we want more money'
I'm being a little bit tongue in cheek, I'm sorry your wife is stranded.
I expect it to be a combination of extremely poorly funded and sidelined IT infrastructure trying to use a bloated AI system from a hype company. Which results in everything breaking and poor schmucks already at the end of their rope working to fix something they barely understand.
Because everyone [in the specific field of technical professional operating systems like this] is using AI and not everyone is smart about how they use AI?
System worked mostly fine for decades, then in the middle of integrating a new AI system it completely goes down.
It doesn't take a genius to see the problem is the new AI system. Whether or it's the "AI" part of the system isn't really relevant system this new system was vibe-coded by AI and built around the "AI" features.
There is zero evidence of this at all. The stuff going around on social media has zero correlation to the system in question. Unless you have other evidence others are not privy to?
> I expect it to be a combination of extremely poorly funded
Highly doubt this. Unless you mean sort of like education in many areas where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
> where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
Yeah of course I mean this, I mean the guy who had to serial connect to whatever shit show was running and was sitting there trying to parse error logs.
Given that the old hardware has been around a long time without a problem like this, it seems very odd to blame the old thing for the new problem. My money is on a poorly planned transition, one where they didn't take the old hardware seriously enough.
It's certainly possible - but for whatever reason this administration (with the help of that guy from the robot taxi company) seems to not like air traffic controllers and has been trying to find ways around them.
It's possible this is the standard "privatize a government function and skim the profits" move but some of this stinks of union busting and trying to reduce the amount of influence air traffic controllers have whenever the organized criminals that run our government want to have a shutdown but keep their private jets flying.
Suffice it to say - there's no reason to trust any of these people when it comes to taking care of critical government functions.
They were also insisting its not a problem with the new upgrades they were installing, but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
Not sure if this was the the exact quote, but I see AP wrote, "introduction of some new technology into an old air traffic control system overwhelmed it." Despite that, the whole conference, on loop, they were insisting that the problem is entirely the old hardware and this is why they need billions to overhaul it and rebuild from scratch.
>but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
The secretary also mentioned that the new system should be brought altogether, and not piece by piece. Which would lend credence to an incompatibility between both systems.
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system"
I assume that this was a boring, routine update that went awry.
But my completely unfounded and more conspiratorial theory, based on zero knowledge of air traffic control systems, is that to better accommodate SMART, L3Harris was strong-armed into to rolling out a hastily written patch on a Friday, which accidentally took down the system.
They are trying desperately to not name the AI system that got turned on yesterday that they commissioned.
Is it more likely that decades old hardware failed coincidentally or that the first rollout of a new AI tech went wrong within a number of hours because the test environment didn't have sufficiently scaled load testing?
The press should be asking why this could be the case when the GOP has held congress and the white house for two years ? Just what was the party in charge thinking ?
I think the supplemental appropriation that the FAA got as part of the OBB is probably too big to be spent.
Unless you just hand it all to a contractor, it takes time and oversight to spend billions. Legally and practically.
"It's the safest system in the world, I mean, uh, except that it's horribly outdated with switches from the 80s and radars from the 70s, so we really need 50 billion dollars from congress STAT or we're in big trouble, but it's completely safe. Also Congress needs to give Trump the Builder this money _now_. We couldn't get it in the past couple of years even though we controlled the entire government. I mean, sure, we got 12 billion, but that's not enough to unfuck what Joe Biden did. Your loved ones are completely safe by the way, but we can't promise this won't happen again."
We need a despot that will ignore all bureaucratic bloat and get things done, quite frankly. Otherwise this country is done for. Spiraling into the trash can.
Transportation secretary saying that their networking routers were being overloaded by increased communications traffic. But its not a cyberattack or software failure.
They just said in the press conference that it couldn't possibly be related to SMART, since the systems were disconnected, but the timing and lack of a concrete answer as to the cause or likely cause, combined with asking for money from congress (which currently has a republican majority...) for "more modern infrastructure" has me doubting their version of events
I acknowledge this is borderline conspiracy. Sec Mullen's statement is that FAA was quote "installing new equipment, new technology and that the new equipment new technology was overwhelming the system"
Transportation secretary Duffy says that the smart system is unrelated. That the smart system is a dashboard for displaying all the data from across the air traffic system. My questions is obviously where is the smart system getting its data from...
I suspect FAA pushed for the new AI system, didnt take into account the increased traffic demand to feed the system. Aging infra isnt sufficient to handle the increased demand. Now Duffy is trying to deflect blame unto someone else.
I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was...
Should be an easy answer. We saw increased traffic from X and we've implemented Y to ensure it doesn't happen again.
Instead it was vague hand waving around old hardware and funding.
Even if they don't have all the details organized, what was done to fix the problem? How was the 12b spent thus far and what where does the additional 50b number come from?
Or maybe don't start with demanding money. What happened? How was it fixed? What is being done to ensure it doesn't happen again? Then mention this shit needs fixed and we'll be working with Congress for additional funding.
It was interesting that Mullen mention new technology contributing at one point. Sounds like someone made a mistake somewhere and this press conference was purposefully vague or they really don't understand what happened yet (doubtful).
Plenty of critical systems operate reliably on old tech as long as you work within the limitations.
This press conference had maybe 30s of actual info and the rest was this antiquated system is fucked even though it's been working for years and the only way to move forward is $50b. :shrug:
> I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was
They did hurriedly mention it was "new patches" in the midst of their deranged ranting and finger-pointing (at...their own party, a point Duffy was visibly taken aback by when told by a reporter). When they vehemently insisted it was 'not SMART' one has to presume they meant that the hasty execution of their haphazard AI rollout was performed unintelligently.
maybe, what is confirmed is that they were trying to upgrade some physical TDM switches with IP routers (as they have been doing for a while), and the "new systems overwhelmed the old ones".
It is possible that this overwhelming was caused by a software or configuration error, no doubt.
I think that if that's the case, rather than failing upwards and going for a full simultaneous upgrade with a 50B congress bill, the brakes should be stepped on a little bit, continue the partial migration as long as there are no issues. Or at least wait until the start of a 4year administration to go for a rehaul, so that you can stay around if things go fubar.
Unclear what happened but they have old equipment. Only Duffy can fix it, he needs $50B from Congress to do it and they can't fix it overnight. They can't guarantee it won't happen again.
Not much info yet. Even ops.group [1], which is for people who do flight planning, has nothing yet.
LAX hasn't restarted yet. Departures are either "Cancelled" or "Delayed".
All information about this is from PR teams. Nothing from reporters who cover airports. Nobody from airport management, airline operations, FAA operations people, or somebody who just got off shift at a control tower. People who might actually know something.
There was a briefing at the White House, and it was all cabinet secretaries. "Instead of reassuring an anxious public, and providing concrete information, this briefing appears to be intended for President Trump’s approval" wrote a New York Times reporter.
FAA policy prohibits anybody but PR people talking to the public. The details of that policy are not public.[2]
So they were doing a network upgrade and putting in new equipment. This new equipment was misconfigured and caused the existing working system to stop working. And now they are saying in the press briefing that this is a problem of the existing working system which they call the "legacy system" and are using this as a political move to call for the congress the regime no longer controls to provide funding to the FAA/DHS. Framing the manufactured issue they caused with the new system as if it was congress' problem instead of their own breakage they caused and are deflecting from. Demanding an emergency funding bill now despite ignoring it in all the years they were in control.
It’s possible the interfaces to the legacy system aren’t documented well and/or it could be incompetence on behalf of the network engineers working this
Maybe things are moving fast today and they didn't have time to understand and communicate accurately, but at some point soon I think we deserve a more in depth explanation.
There is an actual bill for $30B that's been stalled for months in the Senate called the Modern Skies Act, and there is absolutely widespread antiquation within the US air infrastructure which this bill seeks to address. So he's not just making this up. He's talking about exactly that bill albeit with all the Trump BS injected into it.
Modern Skies Act is for $30B, not $50B. A separate $12.5B from "Big Beautiful" is already being spent on ATC modernization, and only $10B of MSA is toward ATC
Sure, but blaming Democrats when they’ve controlled Congress and the presidency for the past two years and their highest priority has been cutting funding via DOGE is pretty disingenuous
Their press conference was horrendous but it seemed he was just blaming Congress overall and not Democrats specifically (since it is Republicans actually holding it up)
Assuming I'm looking at the right bill (S. 5572, the Modern Skies Act), "stalled for months" seems a tad overstated considering it was introduced Sep 25th 2026, just three days before Congress went on recess for the entire month of October (elections). The Senate wasn't back in session until Nov 9th, so the bill has been pending for a grand total of seven in-session days (even emergency legislation would rarely pass that quickly).
The Senate will be in session for four more days next week before the Thanksgiving holiday, and then back for the first three weeks of December before the 119th Congress adjourns for good on Dec 18th. So basically, after two full years with control of both branches of Congress and the WH, this bill was introduced on the very last week of the regular Congress calendar. After which the lame duck Congress would only meet for about four weeks until adjournment.
Not really stalled, more like procrastinated until days before the election (when everyone is scared to vote on anything), hoping that the lame duck Congress (notoriously unproductive period generally) would force it through in a rush?
Doesn't seem like a great strategy if they actually wanted it to pass. Especially since a budget CR is needed before Dec 11 which will dominate members' attention in the remaining time (plus the permitting bill BAAJA).
Doesn’t it feel wrong to anyone else that there’s a single point of failure for all fights in the US? I can understand if there are regional zones and the East coast gets grounded. But everything at once??
Given that many flights are between regions, it makes sense to me that you would need a nationwide system. Even a flight within a region may depend on a plane or crew coming in from another one.
Having it degrade into regional systems would be a nice property, but risks making a complex system more complex. Put all of your eggs in one basket and then make sure it's a really good basket.
It's a miracle that US air traffic control isn't having incidents like this more often. Admiral Cloudberg has a great write-up [1] that touches on some of the systemic problems in American aviation, oh boy it's a mess.
I guess if you consider safety the top priority, you might let a local failure pause the whole system. So I’m not sure quite how much to criticize this point
Is it a bad idea? Absolutely. But also it's to be expected. People like to have a single point of control. You can't do that without introducing single points of failure.
I wish more was being done on partition tolerance. Consistency is over rated.
I took the opportunity to ask Fable what the 50% over/under on my 7am flight being delayed tomorrow, and it used 33% of my four hour window to come up with an answer of 15 minutes. Somehow, I doubt it.
"we" being him (ie. not the reporter) because he's part of the executive branch; that's the limit of the authority of the executive order, after all...
Ahh the great Kubernetes cluster in the sky finally finished reconciling and got the new nodes up and running. I bet this time it only took 3 faxes and 9 different signatures from heads of the redundant departments of redundancy.
They jumped from 3 airports to nationwide in one step, and are now framing it as the existing hardware not being compatible with their new software, rather than the other way around.
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system".
So, it was a careless update, probably the one deploying their new 875 millions AI tool.
And no journalist at the conference picked that up.
For those interested, there was a 2009 outage of the FAA Telecommunications Infrastructure that at a surface level seems like it could be a similar failure mode. At the very least, it is an interesting read on how the FAAs systems operate, since a lot of the discussion so far has been wild speculation from people with little knowledge on how the national airspace system actually operates: https://www.oig.dot.gov/library-item/29670
Of particular note is this section: Risks of FTI Outages for Existing Operational Services As of December 2009, Harris reported that there are a total of 20,982 services operating on the FTI network. While FAA has transitioned all 1,808 IP services supporting FAA flight plans to the new FTI fiber optic network, additional risks will be introduced when the Agency begins transitioning other FAA services, such as En Route Automation Modernization (ERAM) and NextGen platforms, which also require IP technology to operate. Moreover, there are 19,174 existing services that may be transitioned to the new FTI fiber optic network. According to FAA, transitioning these remaining services is important because FTI now connects older FAA systems that provide safety-critical voice and surveillance radar information by utilizing Time Division Multiplex (TDM) technology, instead of IP services.
TBH TDM is far superior and it’s sad that the world discarded it. Packet switching is the worst thing to ever happen to phone calls. Kids today have no idea how good it used to be (albeit at $0.50/min).
I mean the fidelity of modern mics/converters/codecs in devices now is much better, but ya the tradeoff of VoIP is latency/jitter/dropouts etc
Here's what they said caused it in the White House briefing:
Sec Mullin: "What it appears to be right now is, as Sec Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology. And that new equipment and new technology, it was overwhelming the older system. Which is why Sec Duffy made it very clear, the patchwork doesn't work. We need one complete system."
Duffy also kept saying they were working as quickly as they possibly could trying to rollout the new system. Not exactly reassuring.
I think the problem with a lot of the push to "modernize" is we forget the legacy system has been real-world tested for 40 years now. It's the new stuff that needs to prove itself in terms of stability.
We have a rule in my dev shop: if a piece of code (or hardware) has been working without fault in production, you do not touch it. It doesn't matter if it's old, if you don't like the tests, if it's in a programming language you despise; if there haven't been any problems in production, you don't touch it.
This is exactly the way we operated Crystal Reports. "If it works, don't touch it"
What about legitimate issues?
That's the exception they noted ;)
> if a piece of code (or hardware) has been working without fault in production
And then you have the common reality: "a piece of code has been working in production with known faults & known costs". And you have to weigh up costs and risks.
Especially on a Friday. The 13th doubly so.
Taken to the logical extreme, a magic doohickey that nobody dares to touch, and everyone prays that it finally fails on someone else's watch.
If something can't be turned back on after it gets turned off, that's just a latent, undetected fault.
I don't understand why they did not test this before deploying on production. How can we sure that when they develop a whole new system, it will work?
Sounds like the new system effectively performed a DoS on the old system. They probably don't have a development environment where they can hook the two systems up. Even if they do, it might actually have been an out-of-the-way component that failed first, impacting another, and so on. If they have a test environment with the two systems, a complex failure like that would unlikely to be found (and would explain why it took hours to resolve).
As someone who works under a different umbrella in the federal government, how do you propose testing something so archaic and complex without replicating an exact replica of a 40+ YO technology?
I've actually worked a system where the test env was scavenged old eq set up to mirror exactly the ancient prod env. It's possible, it just takes a lot of work. Scaling it might be touchy though.
And it sounds like the fault conditions only occur when you do scale up.
> It's possible, it just takes a lot of work
Not only it takes lot of work, but to have the replica connect to the prod in some way to mirror identical behavior will possibly mean adjusting ATO, which is not a simple process in itself.
Emulate, computing is cheap. In an application as important as air traffic control it seems like it wold be worth it to _always_ have a working copy of the original system specifically for testing changes.
Not aware of specific system, but critical infrastructure systems have a quality assurance system that replicates production albeit not perfectly. You may replicate the core system but not all its interfaces, you might replicate it but scaled down etc. There are ways for solid tests to fail. For systems with downtime, a test on production is useful iif you have solid rollback procedures. There’s lots of art in the operation of these systems.
You think an error that brings down _national_ networks is something you can emulate?
Thank you, you get it.
Its possible, just not probable with who you can hire for gov work.
Let's set that aside and instead ask why tf they rolled out anything on a Friday? Or was it that they rolled it out on Tuesday but failed to notice it was close to overload state until it actually overloaded on Friday?
Or it was Friday traffic that overloaded it.
> I don't always test my code; but when I do, I do it in production.
https://static.wingify.com/gcp/uploads/abtasty/test-in-produ...
I’m sure it worked on his machine.
This admin can't be trusted with lining a pool. Why is their incompetence still a surprise to you after all this time?
Even if they were confident the new components would work, why deploy at 3 PM Eastern on a Friday?
They also kept noting that the new IP based routers were not showing any inconsistency. To me that says that they were bridging IP traffic over the legacy systems, probably a protocol with retransmits like TCP, and a storm of retransmits was overloading the legacy systems, and older components using a less durable protocol were drowned out and dropped packets.
Put someone competent & accountable in charge and then let’s talk.
https://www.faa.gov/air_traffic/technology/cinp/vsrprogram maybe?
Seems kinda nuts to replace "obsolete" switched telephone circuit technology that's incredibly reliable and zero latency with VoIP.
It is obsolete.
The parts to maintain the legacy copper infrastructure are no longer manufactured and they are cannibalizing used parts for spares.
Not to mention the expertise for these systems have mostly either retired or in some cases passed away.
My grandparents still have a landline and the number of outages in the last 12 months has been insane. Some were in excess of a week.
Many such cases.
Sometimes the telcos have been shrugging their shoulders and refusing to perform repairs. This is common after a car accident takes out a pole, cabinet, etc. They accelerate the retirement to right now and force the customers onto new technologies.
Not really. The problem with keeping old technology is that you need people who have knowledge of how to maintain the old systems. Ignoring the cynical view that many of those people were likely fired, even if they were kept on, they will eventually retire. There's no amount of training that can replace that institutional knowledge.
If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.
> Ignoring the cynical view that many of those people were likely fired
In your mind maybe. Most are dead or retired.
New techs have been trained on splicing fiber and VoIP over the last two decades.
Nobody knows about vacuum tubes anymore either, that doesn't mean they were fired and had their pensions taken away.
... no it doesn't?
reminds me of the fact that (for some reason) it costs thousands of bucks to replace or install a stop sign.....even in rural FL
Mullin doesn't actually understand anything he said.
how could this possibly not have been SMART? were they really doing two simultaneous rollouts?
Sounds a bit like Chernobyl. People doing stupid stuff to keep “the party” happy.
This is only one of many recent “lip sores” indicating the a herpes virus that is all over US critical infrastructure.
Over a decade ago I was an intelligence officer and the alarm bells were going off and it was exceptionally clear to the National security community that US infrastructure is barely holding on.
Clapper in 2013:
https://nsarchive.gwu.edu/document/21491-document-90
Dan Coates in 2018:
https://www.euronews.com/2018/07/13/us-intel-chief-warns-of-...
Tip of the iceberg folks. Hold on to your butts…
But somehow it's this administration's fault.
I can't tell if you're being ironic or not. Sure, "one complete system", and I want a pony.
I watched Sec Duffy's presentation, and it was appalling, but par for the course with this administration: in the middle of a major nationwide outage, he tried to blame someone else.
Even if Duffy got everything he wished for, how do you think they would upgrade a system as old and complicated as ATC. They would have to do it piecemeal anyway. This was a classic upgrade that broke shit. The problem was with the new upgrade, not the old shit that worked.
I was just presenting the full quote from the Secretary of Homeland Security explaining what went wrong. I wasn't commenting on it. I added quotation marks just now to make that more clear.
I agree with your view.
Oh, my apologies, I had originally read the last 2 sentences in your voice. Thanks for the update.
Since the threads keep getting closed about Hawaii DoT confusing everyone by claiming SpaceX debris was contributing to their shutdown despite the Starship launch being in Texas, successfully reaching orbit, and having a booster land in the Atlantic:
Hawaii was the early/late splashdown site and pre-cleared: https://www.cbsnews.com/news/spacex-launches-starship-flight...
SpaceX released the early return around 4 p.m. Pacific: https://x.com/TrackingTheSB/status/2121392533418786992
HDOT phrased it badly, but there was a window today where Starship could have aborted over the west coast in the middle of this shutdown.
um.. you meant booster EXPLODED in the Gulf of AMERICA? Land my aSS.
Looked like it was a soft touchdown until engines shut off at the "catch" height (simulated) and then they activated the TFS.
“Exploded at a specific time”
Exploded safely as planned during an off shore divert from the tower catch attempt, after making a controlled hover in mid air slightly above the water.
The reason field is empty, but the website footer provides enough information to understand.
And yet, it seems to have been a careless update to have broken everything
probably updating winzip
They tried to register it
Shoulda used winrar.
Press conference https://www.youtube.com/live/M1WhY3qcJjE?si=KUFvgHO-EepCi1v6
Notably, the problem was described as "routers overloaded with data". When directly questioned if this could be a cyber attack or rogue ai agent the Sec/Homeland responded "there's nothing completely certain now."
The way they are being so evasive makes me think they did a pretty big screw up.
Or a non-technical person is relating the information as they understand it to a non-technical press corps, so everything is generic and hand-wavy. Honestly, have you never dealt with non-technical C-suite staff?
Do we have any indication of what went wrong? The incident report will be very interesting.
At the press conference (current, live), they are blaming old hardware that is failing. They’re in an endless loop of repeatedly saying that it’s Congress’s fault for lack of funding and that this is the result of old hardware. I just gave up on watching because they aren’t saying anything informative.
Not only was there a lack of useful information (other than the fact that the grounding was lifted), it was an astounding demonstration of poor leadership during a crisis. No ownership, no focus on the problem/root cause/next steps.
Usually I refuse to give any detailed postmortem right after an incident is mitigated/resolved until I have had a chance to collect my thoughts and make sure I can write a clear problem/root cause/next steps. The communication that goes out with the mitigation signal is often short and vague and the detailed postmortem follows a day or so later.
Sec Duffy's presentation was just embarrassing. Blame anybody else except for the people in charge.
Let's be clear: it was a bunch of idiot political hacks being unbelievably obsequious to Trump. So embarrassing for the country. I don't trust that the ask for $50B is anything more than grift for their friends. Was the grounding a purposeful overreaction to increase odds of getting money?
I don't doubt that the system needs upgrades, but these people should not be involved.
Markwayne the plumber correcting a reporter for saying AI instead of SI instead of answering the question was utterly surreal.
This is one of the reasons for the "SI" mandate.
I thought you were being a jerk, but no this dude has an Associate in Applied Science in plumbing from the Oklahoma State University Institute of Technology. Huh.
Don’t even get me started on the one nobody gave a shit about on Real World: Boston standing at a podium and demanding $50 billion.
I'm pretty sure saying that Congress needs to give them the money to replace the ancient hardware is putting ownership directly where it belongs and identifying the problem, the root cause, and the next steps.
What am I missing?
That if the ancient hardware was working and just the new software they tried to put on it did not worked with that hardware - it still means they messed up.
In the moment of crisis, they own the problem of making the system work with what they have.
As a postmortem, once the dust has settled, is the time to ask for more funding/support
You forgot to blame Trump. What are you, new here?
Have they even thanked the pilots, air traffic cobteollers and thousands of airport personnel for handling the mahyhem?
There's still time for that. The mayhem is ongoing and will be felt for days. I hope by Monday a few "bosses" will say thanks to all the staff who ran overtime to deal with desperate grounded passengers, do all the rebooking snd schedule changes.
No, but they made sure to thank the Dear Leader several times each.
i misread this as saying you were in attendance of the conference and was hype that the press uses hackernews
Ah sorry for the misleading writing. My brain has been fried trying to work while also trying to dig up any useful info to help my wife who is stranded at the airport figure out what to do. I usually can last longer watching press conferences but this one was… just painful. Maybe someone else can give an update if they actually talk about what’s going on.
ive also been watching it, pretty much repeating the same stuff
> old system give us money
> probably not a cyberattack btw
> not SMART related
> we've opened the airspace but this is gonna be slow
just ended actually as i was typing this, that's pretty much it
> Maybe someone else can give an update if they actually talk about what’s going on.
They didn't.
https://www.youtube.com/live/Mh2q-y0wIu8
Having worked in Government for a spell. Are we actually sure that old hardware isn't the problem?
I didn’t say it isn’t. Just that they aren’t really giving any details and more importantly, they aren’t giving info for what people who are stranded should be doing right now. (And now that I’ve settled down a bit, it’s odd that if it’s old hardware failing, how is it suddenly not failing? Was it just some systems, did they have replacements, or if it’s what the other commenters wrote, then did they just turn off the AI updates to stop overloading the hardware? Which would mean that hardware is fine. It’s just underprovisioned.)
Hardware can be fixed. Even if it is old. Spares exist for a reason. Maintenance staff exists for a reason.
Up to a point. If your equipment is decades old, spare parts and expertise is a quickly diminishing resource.
Not with the budget the FAA has to apply to the problem. Yes, you have a point. But the importance of this system could get circuit boards manufactured to old drawings. Specific 'electronics boxes' replaced with new ones designed to replace the antiquated hardware.
FAA has been doing this for decades. A single electronics box, or even a couple should not drop the system. Maybe for the time it takes to remove and replace.
After hearing the press conference "some new stuff was plugged in and the legacy system could not handle it" or alternatively 'a mistake was made that just so happens involved non today equipment, but we want more money'
I'm being a little bit tongue in cheek, I'm sorry your wife is stranded.
I expect it to be a combination of extremely poorly funded and sidelined IT infrastructure trying to use a bloated AI system from a hype company. Which results in everything breaking and poor schmucks already at the end of their rope working to fix something they barely understand.
It sounds like it has nothing to do with AI at all. Why does everyone keep jumping to AI when there's no indication this has anything to do with AI?
Because everyone [in the specific field of technical professional operating systems like this] is using AI and not everyone is smart about how they use AI?
AI is the name of the witch that everybody is so eager to burn
System worked mostly fine for decades, then in the middle of integrating a new AI system it completely goes down.
It doesn't take a genius to see the problem is the new AI system. Whether or it's the "AI" part of the system isn't really relevant system this new system was vibe-coded by AI and built around the "AI" features.
There is zero evidence of this at all. The stuff going around on social media has zero correlation to the system in question. Unless you have other evidence others are not privy to?
> I expect it to be a combination of extremely poorly funded
Highly doubt this. Unless you mean sort of like education in many areas where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
> where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
Yeah of course I mean this, I mean the guy who had to serial connect to whatever shit show was running and was sitting there trying to parse error logs.
> they aren’t giving info for what people who are stranded should be doing right now.
Do you really need the government to solve that problem for you? Just book another flight.
Good luck "just booking another flight" when half a million people are trying to do exactly the same thing at the same time as you.
Well, sure, that goes without saying. You just don’t need the government to tell you what to do; you can work it out yourself.
Given that the old hardware has been around a long time without a problem like this, it seems very odd to blame the old thing for the new problem. My money is on a poorly planned transition, one where they didn't take the old hardware seriously enough.
It's certainly possible - but for whatever reason this administration (with the help of that guy from the robot taxi company) seems to not like air traffic controllers and has been trying to find ways around them.
It's possible this is the standard "privatize a government function and skim the profits" move but some of this stinks of union busting and trying to reduce the amount of influence air traffic controllers have whenever the organized criminals that run our government want to have a shutdown but keep their private jets flying.
Suffice it to say - there's no reason to trust any of these people when it comes to taking care of critical government functions.
They were also insisting its not a problem with the new upgrades they were installing, but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
Not sure if this was the the exact quote, but I see AP wrote, "introduction of some new technology into an old air traffic control system overwhelmed it." Despite that, the whole conference, on loop, they were insisting that the problem is entirely the old hardware and this is why they need billions to overhaul it and rebuild from scratch.
>but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
The secretary also mentioned that the new system should be brought altogether, and not piece by piece. Which would lend credence to an incompatibility between both systems.
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system"
(https://news.ycombinator.com/item?id=50492957)
I assume that this was a boring, routine update that went awry.
But my completely unfounded and more conspiratorial theory, based on zero knowledge of air traffic control systems, is that to better accommodate SMART, L3Harris was strong-armed into to rolling out a hastily written patch on a Friday, which accidentally took down the system.
They are trying desperately to not name the AI system that got turned on yesterday that they commissioned.
Is it more likely that decades old hardware failed coincidentally or that the first rollout of a new AI tech went wrong within a number of hours because the test environment didn't have sufficiently scaled load testing?
The press should be asking why this could be the case when the GOP has held congress and the white house for two years ? Just what was the party in charge thinking ?
I'm used to bizarre press conferences from this administration but this is something else. Just baffling.
They have no idea, they stood up there and lied and played the blame game crying about funding. Completely shameful.
I think the supplemental appropriation that the FAA got as part of the OBB is probably too big to be spent. Unless you just hand it all to a contractor, it takes time and oversight to spend billions. Legally and practically.
"It's the safest system in the world, I mean, uh, except that it's horribly outdated with switches from the 80s and radars from the 70s, so we really need 50 billion dollars from congress STAT or we're in big trouble, but it's completely safe. Also Congress needs to give Trump the Builder this money _now_. We couldn't get it in the past couple of years even though we controlled the entire government. I mean, sure, we got 12 billion, but that's not enough to unfuck what Joe Biden did. Your loved ones are completely safe by the way, but we can't promise this won't happen again."
and the journalists haven't yet picked up the actual reason that they revealed among the lines
We need a despot that will ignore all bureaucratic bloat and get things done, quite frankly. Otherwise this country is done for. Spiraling into the trash can.
That’s what Trump was supposed to be. Turns out it’s way less efficient than having someone who’s good at working with other people.
Transportation secretary saying that their networking routers were being overloaded by increased communications traffic. But its not a cyberattack or software failure.
Speculation on my part, they just rolled out their SMART air traffic routing system. https://www.flyingmag.com/faa-smart-ai-now-active-at-22-artc...
Bet the increased strain on data polling from this system is the culprit for increased traffic.
They just said in the press conference that it couldn't possibly be related to SMART, since the systems were disconnected, but the timing and lack of a concrete answer as to the cause or likely cause, combined with asking for money from congress (which currently has a republican majority...) for "more modern infrastructure" has me doubting their version of events
I acknowledge this is borderline conspiracy. Sec Mullen's statement is that FAA was quote "installing new equipment, new technology and that the new equipment new technology was overwhelming the system"
Transportation secretary Duffy says that the smart system is unrelated. That the smart system is a dashboard for displaying all the data from across the air traffic system. My questions is obviously where is the smart system getting its data from...
I suspect FAA pushed for the new AI system, didnt take into account the increased traffic demand to feed the system. Aging infra isnt sufficient to handle the increased demand. Now Duffy is trying to deflect blame unto someone else.
I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was...
Should be an easy answer. We saw increased traffic from X and we've implemented Y to ensure it doesn't happen again.
Instead it was vague hand waving around old hardware and funding.
Even if they don't have all the details organized, what was done to fix the problem? How was the 12b spent thus far and what where does the additional 50b number come from?
Or maybe don't start with demanding money. What happened? How was it fixed? What is being done to ensure it doesn't happen again? Then mention this shit needs fixed and we'll be working with Congress for additional funding.
It was interesting that Mullen mention new technology contributing at one point. Sounds like someone made a mistake somewhere and this press conference was purposefully vague or they really don't understand what happened yet (doubtful).
Plenty of critical systems operate reliably on old tech as long as you work within the limitations.
This press conference had maybe 30s of actual info and the rest was this antiquated system is fucked even though it's been working for years and the only way to move forward is $50b. :shrug:
> I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was
They did hurriedly mention it was "new patches" in the midst of their deranged ranting and finger-pointing (at...their own party, a point Duffy was visibly taken aback by when told by a reporter). When they vehemently insisted it was 'not SMART' one has to presume they meant that the hasty execution of their haphazard AI rollout was performed unintelligently.
Sounds like a software failure to me.
maybe, what is confirmed is that they were trying to upgrade some physical TDM switches with IP routers (as they have been doing for a while), and the "new systems overwhelmed the old ones".
It is possible that this overwhelming was caused by a software or configuration error, no doubt.
I think that if that's the case, rather than failing upwards and going for a full simultaneous upgrade with a 50B congress bill, the brakes should be stepped on a little bit, continue the partial migration as long as there are no issues. Or at least wait until the start of a 4year administration to go for a rehaul, so that you can stay around if things go fubar.
Was this rolled out on a Friday?
Summary from the press conference:
Unclear what happened but they have old equipment. Only Duffy can fix it, he needs $50B from Congress to do it and they can't fix it overnight. They can't guarantee it won't happen again.
Only Donald can fix it. Pay attention.
Yesterday's Headline: "The FAA Is Rolling Out Its AI Air Traffic Tool Nationwide" https://gizmodo.com/the-faa-is-rolling-out-its-ai-air-traffi...
Not much info yet. Even ops.group [1], which is for people who do flight planning, has nothing yet.
LAX hasn't restarted yet. Departures are either "Cancelled" or "Delayed".
All information about this is from PR teams. Nothing from reporters who cover airports. Nobody from airport management, airline operations, FAA operations people, or somebody who just got off shift at a control tower. People who might actually know something. There was a briefing at the White House, and it was all cabinet secretaries. "Instead of reassuring an anxious public, and providing concrete information, this briefing appears to be intended for President Trump’s approval" wrote a New York Times reporter.
FAA policy prohibits anybody but PR people talking to the public. The details of that policy are not public.[2]
[1] https://www.ops.group/blog
[2] https://www.faa.gov/regulations_policies/orders_notices/inde...
you’ll never get a proper retro for this, it’s too political
So they were doing a network upgrade and putting in new equipment. This new equipment was misconfigured and caused the existing working system to stop working. And now they are saying in the press briefing that this is a problem of the existing working system which they call the "legacy system" and are using this as a political move to call for the congress the regime no longer controls to provide funding to the FAA/DHS. Framing the manufactured issue they caused with the new system as if it was congress' problem instead of their own breakage they caused and are deflecting from. Demanding an emergency funding bill now despite ignoring it in all the years they were in control.
It’s possible the interfaces to the legacy system aren’t documented well and/or it could be incompetence on behalf of the network engineers working this
Agreed, but why not just say that?
Maybe things are moving fast today and they didn't have time to understand and communicate accurately, but at some point soon I think we deserve a more in depth explanation.
Waiting for the RCCA
There is an actual bill for $30B that's been stalled for months in the Senate called the Modern Skies Act, and there is absolutely widespread antiquation within the US air infrastructure which this bill seeks to address. So he's not just making this up. He's talking about exactly that bill albeit with all the Trump BS injected into it.
Modern Skies Act is for $30B, not $50B. A separate $12.5B from "Big Beautiful" is already being spent on ATC modernization, and only $10B of MSA is toward ATC
https://www.transportation.gov/sites/dot.gov/files/2026-09/M...
Thanks, I corrected it, 3's and 5's get mixed up for me due to my synesthesia giving them the same color
The One Big Beautiful Bill Act cracks me up as even the naming was shortsighted and defective.
Mentioning how they previously received $15B but nothing improved - so why will any further money allocated actually end up where it's meant to go?
Sure, but blaming Democrats when they’ve controlled Congress and the presidency for the past two years and their highest priority has been cutting funding via DOGE is pretty disingenuous
Their press conference was horrendous but it seemed he was just blaming Congress overall and not Democrats specifically (since it is Republicans actually holding it up)
Assuming I'm looking at the right bill (S. 5572, the Modern Skies Act), "stalled for months" seems a tad overstated considering it was introduced Sep 25th 2026, just three days before Congress went on recess for the entire month of October (elections). The Senate wasn't back in session until Nov 9th, so the bill has been pending for a grand total of seven in-session days (even emergency legislation would rarely pass that quickly).
The Senate will be in session for four more days next week before the Thanksgiving holiday, and then back for the first three weeks of December before the 119th Congress adjourns for good on Dec 18th. So basically, after two full years with control of both branches of Congress and the WH, this bill was introduced on the very last week of the regular Congress calendar. After which the lame duck Congress would only meet for about four weeks until adjournment.
Not really stalled, more like procrastinated until days before the election (when everyone is scared to vote on anything), hoping that the lame duck Congress (notoriously unproductive period generally) would force it through in a rush?
Doesn't seem like a great strategy if they actually wanted it to pass. Especially since a budget CR is needed before Dec 11 which will dominate members' attention in the remaining time (plus the permitting bill BAAJA).
In the press briefing (https://www.nytimes.com/live/2026/11/13/us/ground-stop-natio...) it was just stated that new "patches" "overwhelmed" the "legacy system". Sounds to me like the new SMART AI rollout went haywire.
Soon enough, you'll be hearing calls to privatize the FAA for "efficiency" due to "failures" like this.
The beast is starving and thrashing about
A botched update, allegedly.
L3Harris Update
my buddy used to work at L3Harris. It sends shivers down his spine whenever i mention it.
my buddy used to work at L3Harris. It sends shivers down his spine whenever i mention it. He's gonna love when I dog him on this one.
The sceptics will say Trump caused the outage so the FAA get approved for the $50B they claim to need to replace the air traffic control system.
That would imply there was a plan. A series of predictable events baffle this administration, so assuming a nefarious plan would seem extreme.
As friend from Mastodon has suggested: "So this would be a great night to play atc(6) from the BSD Games collection on NetBSD"
https://tilde.zone/@sirwumpus/117611840050306362
ATCSCC Advisory ATCSCC ADVZY 005 DCC 11/14/2026 CNX NATIONWIDE GROUND STOP MESSAGE:
EVENT TIME: 14/0116 - 14/0130 DESTINATION AIRPORT:ALL FACILITIES INCLUDED: ALL EXPECT UPDATE:0400Z REASON: REMARKS: THE FAA COMMAND CENTER NATIONWIDE GROUND STOP HAS BEEN CANCELLED. EFFECTIVE TIME:
140119-140200 SIGNATURE:
26/11/14 01:19
Don’t push updates to prod on Friday afternoon.
Right when I left the airport.
BTW, Uber surge pricing felt absurd in this situation.
I wonder if the drivers actually get an advantage in this situation or if it’s just uber milking things.
FWIW Uber did waive cancellation fees on reservations to/from an airport.
There is a lot of talk of legacy hardware. What hardware are they referring to? I love to nerd out and read up on this stuff.
One time I plugged an Ethernet cable in the wrong port and created a loopback. That's my guess.
is the root cause some no bid contract ?
Found stack of 3.5” floppies and rolled back
Doesn’t it feel wrong to anyone else that there’s a single point of failure for all fights in the US? I can understand if there are regional zones and the East coast gets grounded. But everything at once??
Given that many flights are between regions, it makes sense to me that you would need a nationwide system. Even a flight within a region may depend on a plane or crew coming in from another one.
Having it degrade into regional systems would be a nice property, but risks making a complex system more complex. Put all of your eggs in one basket and then make sure it's a really good basket.
It's a miracle that US air traffic control isn't having incidents like this more often. Admiral Cloudberg has a great write-up [1] that touches on some of the systemic problems in American aviation, oh boy it's a mess.
[1] https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...
I guess if you consider safety the top priority, you might let a local failure pause the whole system. So I’m not sure quite how much to criticize this point
Is it a bad idea? Absolutely. But also it's to be expected. People like to have a single point of control. You can't do that without introducing single points of failure.
I wish more was being done on partition tolerance. Consistency is over rated.
I didn't know we could stop all fights in the US. Tell me more!
I took the opportunity to ask Fable what the 50% over/under on my 7am flight being delayed tomorrow, and it used 33% of my four hour window to come up with an answer of 15 minutes. Somehow, I doubt it.
Per AP: "Mullin said the introduction of some new technology into an old air traffic control system overwhelmed it."
So it was the AI?
If this is indeed the case, this event should be used as an argument for load tests as part of test suites in software.
I mean, it’s not like there’s established processes for doing this that have been around for 30 years
MARKWAYNE MULLIN: We call it SI around here, remember, SI.
SI? No thanks. My car gets 40 rods to the hogshead and that's just the way I likes it.
"we" being him (ie. not the reporter) because he's part of the executive branch; that's the limit of the authority of the executive order, after all...
yes but we need to upgrade to SMART Ultra
Clearly they asked their new AI tool to "make air travel safer" and it found a zero-day to ground all flights.
This felt at best like a "we'll fix it in prod" type of situation.
"we don't need a rollback plan, what could possibly go wrong..." -Probably someone at the FAA
"Just fix forward" - Sean Duffy, the reality TV star from the 90s turned... U.S. Secretary of Transportation.
Who could have seen this coming?
Reason: "A bunch of smart guys made it a long time ago and nobody knows how it works"
We fired the remaining smart people two years ago, choosing loyalty as the only retention criteria.
Ahh the great Kubernetes cluster in the sky finally finished reconciling and got the new nodes up and running. I bet this time it only took 3 faxes and 9 different signatures from heads of the redundant departments of redundancy.
Pet theory: the fancy new solution used an LLM and they realized it couldn't handle production workflow so they had to rollback hard.
What if grounded pilots killed time in Flight Simulator, which overloaded Azure?
They jumped from 3 airports to nationwide in one step, and are now framing it as the existing hardware not being compatible with their new software, rather than the other way around.
If you count meatware as hardware, the failure more could be described as ‘antiquated hardware’.
They found the missing floppy disk
They need $50 billion to give to Accenture according to the latest press release
And they'd still be using the sliding nameplate system from the 1960's
>That's a nice airspace you have there, shame if something were to happen to it. Now sign this $50B check for our buddies and it'll all be ok
https://youtube.com/watch?v=Mh2q-y0wIu8&t=876
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system".
So, it was a careless update, probably the one deploying their new 875 millions AI tool.
And no journalist at the conference picked that up.
Bro was fighting for his life to not say AI:
"They were putting new equipment um and uh new um......technology"
I just happened to have vibecoded an ATS myself, pats.ai.
Unrelated I only fly ocunus because I don't want a blue haired pilot.
Here's the cause: https://th.bing.com/th/id/R.d3a3769838f29ecfe301e8f69fb0499b...