From Staff
A latent bug from July took down Outlook. A router swap in
Arkansas grounded the country's air traffic for more than four hours.
A software fault in Chase's own data centers stalled transactions at
banks across the country. An internal update at Apple knocked out
retail stores on two continents. A patch brought down the National
Security Agency's classified network. And somewhere underneath all of
it, attackers spent the better part of three weeks using a pair of
critical flaws in the programming language that runs most of the web
to walk through firewall gateways and wipe servers.
The instinct to connect all of this into a single event is
understandable and, on the available record, wrong. But so is the
reflexive announcement that nothing is connected. Those are two
different claims, and only one of them is being made by people who
have looked.
The trouble began on November 12, when Microsoft 365 services went
down globally. The company's own account, published four days later,
is worse than the outage. A fix prepared specifically for its
networking gateway servers began rolling out in July. It ran
worldwide for more than three months with no visible impact.
Microsoft's load testing did not catch the flaw because the tests ran
on older hardware with fewer processor cores, and the defect gets
worse on newer machines with more cores. Fifteen hours before the
failure, Teams failed over from the backup gateway to the primary one
that also serves Exchange Online, raising the load. Roughly 14
percent of capacity was already out of service from an unrelated
problem the day before. The result was a cascading failure that left
more than 80 percent of gateway servers unresponsive, and the
partitions meant to keep a regional problem from going global did not
hold. Teams, OneDrive, SharePoint and the Office apps could route
around the damage. Exchange Online could not. That is why Outlook
stayed down longest.
On November 13, the Federal Aviation Administration issued a
nationwide ground stop at about 3:30 in the afternoon Eastern time
and lifted it at 8:15 that evening. Seventeen of the agency's 22 air
route traffic control centers went unstable. Data was flooding the
network at roughly 1,000 connection attempts every four to five
seconds. Two sites were implicated: a radar facility in Amarillo,
Texas, and a radio site in El Dorado, Arkansas, where the contractor
L3Harris was replacing routers that date to the 2001 to 2006 period
with new Juniper equipment. Investigators say a multicast or
broadcast feature that is not enabled by default somehow got turned
on while the new router was talking to two decade old Cisco gear. The
FAA paused further Juniper installations and disabled that
functionality on more than 60 other routers. HPE says its equipment
was working as configured. The Transportation Secretary blamed the
old Cisco routers. The agency's own question and answer document
blamed the legacy transmission system. Nobody has explained how the
feature was enabled in the first place. The agency spends about a
billion dollars a year maintaining a network built in the 1980s, and
has installed 400 of 3,648 planned replacement routers, of which 69
are operational.
On November 16, JPMorgan Chase reported a temporary issue tied to
a software bug in hardware systems inside its own facilities. The
bank said there was no evidence of anything cyber related and no
systemic problem, and that other banks were affected only as
counterparties to transactions. Zelle payments slowed. Bank of
America, Wells Fargo, U.S. Bank, Truist and PNC all reported knock on
delays while denying direct impact. Normal operations resumed that
evening.
The same day, United Airlines lost the system pilots use to
calculate final weight and balance figures. Flights already airborne
and regional United Express flights were unaffected. Accounts differ
on duration, with the airline describing roughly half an hour and
fewer than 150 flights, and the Associated Press reporting more than
90 minutes. United said there was no indication of a cybersecurity
problem and never disclosed what actually broke.
On November 17, an internal Apple update took down Apple Pay,
iCloud, the App Store, Apple Music, AppleCare, and the business and
school management services. It also took down the company's own
employee portal, its internal sign on system, remote access, and the
internal website that tracks outages. Stores could not complete
purchases. Everything was restored by early evening.
That same day, the NSA's classified internal network went down.
The agency called it temporary degradation during scheduled routine
maintenance and said mission critical capabilities remained
available. The Director of National Intelligence, Jay Clayton, told
Fox Business something more specific: patching that followed the PHP
vulnerability had caused systems not to come back up as expected, and
that is what happened at the agency. Staff watched the internal alert
level move from code yellow to code green to code orange in a single
day before Fort Meade closed to most employees. The New York Times
reported the network had been unstable for weeks.
What PHP2Shell actually is. Two memory safety flaws in PHP,
disclosed with patches on October 21 and scored 9.2 out of 10 under
the CVSS version 4 scale. One corrupts internal state when an
allocation fails, leaving an unfinished data structure that the
cleanup routine later treats as valid. The other leaves stale state
from one request pointing into freed memory, which a later request
handled by the same worker can follow. Neither requires a login. A
malicious request is processed by the language runtime before any
application code executes, which means login pages, web application
firewalls and application level authentication do nothing to stop it.
Only patching closes it.
Fixed releases are PHP 8.2.35, 8.3.36, 8.4.27 and 8.5.12. Versions
7.4 through 8.1 received minimal source backports with no official
release. Versions 5.4 through 7.3 are end of life and will never be
fixed. The earliest affected code path dates to PHP 5.4.0.
The flaws were found with the help of frontier AI models used by
researchers at Wiz and at Calif, the latter as part of an OpenAI
effort to hunt vulnerabilities in critical open source code. Wiz has
called them the most severe vulnerabilities ever discovered with AI.
That claim deserves more scrutiny than it has received.
Amazon Threat Intelligence observed exploitation attempts against
its honeypot network on October 17. That is four days before public
disclosure and before patches existed. Amazon has said it is unclear
how the attacker obtained exploit information. CISA added both flaws
to its Known Exploited Vulnerabilities catalog on October 26 and
later warned that attackers were chaining them with a Linux kernel
flaw to gain root. Cloudflare recorded millions of attempts to
weaponize them after disclosure.
The industry framing is that AI gave defenders a window, that the
good guys found these first. The October 17 exploitation undermines
that. Either the window was already open or the claim is marketing.
The same capability that finds bugs at internet scale also adapts
working exploits across PHP versions, configurations and memory
layouts in hours, and the labs only advertise one direction.
The destructive attacks documented by Check Point Research are the
part of this story that deserves the most attention, and they had
nothing to do with the outages above.
Beginning October 30, an attacker scanned and exploited
organizations running unpatched Check Point security gateways,
entering through Mobile Access portals using PHP2Shell. The first
wave targeted Israeli organizations and included destructive
activity. A second, broader wave on November 2 hit organizations
across multiple countries and sectors with no destructive activity
observed.
Once inside, the attackers used a separate flaw in the DynamicCLI
interface to move from the portal service account to root. They
dropped PHP webshells into the portal templates directory behind a
hardcoded key, installed a backdoor that persists under a disguised
name, and injected a networking component into legitimate appliance
processes so that attacker traffic slid past ordinary monitoring.
They ran a scanner to map internal assets. They then reached a VMware
ESXi host over SSH using the vpxuser service account that vCenter
uses for host management, staged a wiper in the temporary directory,
zeroed files before deleting them, and left behind a politically
themed note rejecting ransom negotiations.
That last detail matters more than any vendor advisory in
circulation. The hypervisor was not compromised through a VMware
vulnerability. It was reached with a working service account, from a
security appliance that should never have had an interactive session
to it.
Attribution is genuinely unclear. The tooling shares lineage and
cryptographic material with a malware family tied to a China linked
group known as UNC5221, the infrastructure sat on a hosting provider
previously associated with Chinese activity, and the scanning hours
line up with a Beijing workday. But the destruction, the focus on
Israel, and the internal naming do not fit that group's known
motives. Check Point leaves open that the actor ran the campaign,
served as an access broker, or staged a false flag.
Check Point's remediation is the November Jumbo Hotfix for
management servers and gateways, prioritized for any internet facing
Mobile Access portal. Signs of compromise include session files
containing a distinctive upload progress marker, unexpected daemon
processes, unexplained gaps in firewall logs, and SSH activity from a
gateway to an internal ESXi host.
The precautionary wave on October 21 and 22 was far larger than
most summaries suggest. In the House of Lords, the technology
minister Baroness Lloyd of Effra reported 32 government services
unavailable and four partially unavailable, attributing the
disruption largely to precautionary measures. The Office for Budget
Responsibility took its site down a week before the Budget. Councils
including Edinburgh, Oxford, Bristol, Lambeth and Barnet went
offline. In Scotland, at least seven of 32 councils and six of 14
health boards pulled their sites, along with the Care Inspectorate.
Germany, France, Belgium and Finland shut down government services.
Finland's state ICT center took down public services including the
national land survey. Australia's own cybersecurity agency site went
dark. The Wikimedia Foundation took down a number of systems, leaving
parts of Wikipedia unreachable, and the BBC's own program pages went
offline.
The Netherlands took down Overheid.nl, MijnOverheid.nl and DigiD.
DigiD is the national digital identity system used for tax filing,
health portals and benefits. A government does not switch off its own
front door for routine maintenance. It does it when the risk of
leaving it open exceeds the cost of locking citizens out.
The UK's National Cyber Security Centre was briefed by OpenAI the
week before disclosure and stood up an operation across departments
before the public announcement, then sent 50,000 warnings to
organizations.
Nepal's disruption illustrates the cost better than anything else
in the record, and one detail in circulation is wrong: the shuttered
system was not run by the customs department. It is the Vehicle
Consignment Tracking System, operated by the Department of Revenue
Investigation to track freight vehicles for revenue enforcement. With
it offline, drivers and businesses had to carry physical bills and
invoices.
Thousands of government sites went dark at once because most of
them run on a single shared platform, which is now being audited
centrally. Nepal Rastra Bank suspended its site and web services and
told banks, payment operators and remittance companies to tighten
monitoring. The Employees Provident Fund and the Social Security Fund
went offline too. Nabil, Sanima and Nepal Investment Mega Bank
restored service after patching. Mobile and QR payments kept working.
Internet banking and anything tied to a web server did not. The
national cyber center's director, Raj Kumar Maharjan, said the
shutdowns were precautionary and that no unauthorized access or
attack had been detected.
The central bank told the public the cause was a technical glitch
in PHP software. A remotely exploitable flaw that runs commands
before an application loads was described to citizens as a glitch.
Vendor guidance on protecting hypervisors is correct and, as
written, internally contradictory. Patch the host through supported
workflows. Limit management access and avoid root. Enable Lockdown
Mode so administration flows through vCenter. Disable SSH when it is
not required. Use Secure Boot and the execInstalledOnly setting to
restrict what can run. Use maintenance mode and prechecks rather than
making uncontrolled changes across an environment. Do not install
conventional antivirus or third party agents on ESXi or the vCenter
appliance, because the hypervisor is a purpose built system more like
router firmware than a general purpose server, and unsupported
modification voids support.
Now read the vendor's own note on monitoring. Security tooling for
these hosts typically works agentless, through remote access over
SSH, and customers are told to weigh the exposure of enabling SSH
against the benefit of the monitoring. The attackers reached ESXi
over SSH using a service account. The only way to watch the
hypervisor with conventional tools is to open the path that was used
to destroy it. Disabling SSH and enabling monitoring are not
compatible recommendations, and no vendor advisory resolves that.
Three additions belong on any real checklist. A gateway should
never authenticate interactively to a hypervisor as the vCenter
service account, so that path should alarm on its own. Lockdown Mode
concentrates administration in vCenter, which simply relocates the
crown jewel. And anyone exposed should search for a wiper note left
in every directory it processed, and for execution from the temporary
directory of a host that should never execute anything there.
Patching ESXi closes nothing. The entry point was the gateway.
The record supports three separate conclusions. Several outages
resulted from maintenance, software changes or defects, with no
established hostile activity and no demonstrated single cause.
PHP2Shell is a serious risk that justified urgent patching and
deliberate shutdowns, and the NSA disruption was linked to patching
by the Director of National Intelligence rather than established as
an intrusion. And confirmed exploitation of unpatched gateways led in
some cases to destructive attacks on virtualization infrastructure,
which is why external entry points and internal administrative
systems have to be defended as one problem.
Everything beyond that is being asserted rather than shown. There
is no victim list, no count of wiped hosts, no independent forensic
review of the agency network, the aviation backbone or the gateway
fleet, and no loss estimate anyone can defend. Every explanation in
this account was issued by the organization that failed, about its
own failure, within hours, by parties with an interest in calm. That
does not make the explanations false. It makes them untested.
What is testable is smaller and less comfortable. Six of seven
major failures in one week came from change, not from attackers. The
systems running aviation, banking, government identity and classified
intelligence have grown too complex for the organizations operating
them to modify safely. Patching breaks them. Not patching breaks
them. A defect in a July fix disabled Outlook in November. A router
swap grounded the country. A patch in the middle of an emergency
closed Fort Meade.
That is the finding, and it does not require a villain.
Sources:
PHP2Shell technical record and disclosure
Check Point gateways and the ESXi wiper
Microsoft 365 outage, November 12
FAA ground stop, November 13
FAA's own questions and answers:
faa.gov
AP, router installations paused:
apnews.com
Reuters, initial account:
reuters.com
CRN, Duffy's blame of the old
Cisco gear: crn.com
Network World, the multicast feature that is not enabled by
default: networkworld.com
JPMorgan Chase, November 16
United Airlines, November 16
Apple, November 17
NSA and Jay Clayton
Precautionary shutdowns, Europe
The Register, 32 UK services down,
House of Lords disclosure, councils: theregister.com
BBC, worldwide takedowns including
Wikimedia and BBC sites: bbc.com
BBC, Scottish councils and health
boards: bbc.com
Dutch government statement, DigiD
and MijnOverheid: rijksoverheid.nl
Yle, Finland's state ICT center: yle.fi
Nepal
VMware and Broadcom hardening guidance