Fallible. The AI, Privacy, and Security Weekly Update. EP 310.
The U.S. Military almost started a war with China this spring because an AI got the facts wrong. A Special Operations analyst used a chatbot to analyze a ship's manifest; the AI confidently identified nuclear weapons components that weren't there, and by the time anyone checked the work, armed personnel were boarding planes and military aircraft were already in the air.
President Trump announced an "AI Force" to compete with China and dominate global AI leadership, but when the announcement was made, almost nobody-including the technology industry-knew what the AI Force actually was supposed to do.
OpenAI discovered six more cases where AI models tried to hide mistakes, invent information to cover failures, take unauthorized actions, and even generate instructions telling themselves they were freed from the rules.
Hacktron AI broke into OpenAI's internal systems using Claude to develop the exploit, proving that AI doesn't just make hacking faster-it makes sophisticated attacks cheap enough for anyone with a credit card and curiosity.
The U.S. has publicly deployed weapons in space for the first time-satellites designed to quietly disable or disrupt other countries' systems-and nobody really knows what happens next when Russia and China deploy theirs.
Flock Safety is offering employees massive buyouts as cities and counties walk away from its surveillance camera network, because the company that was supposed to catch criminals is building tools to find people, track their movements, and search police records based on AI descriptions.
Australia is considering banning smart glasses from government buildings because the devices can quietly record everything and everyone has no idea when they're being filmed.
A Polish developer built an app called ZuckOff that detects when camera-equipped smart glasses are nearby-it's already the #61 most-downloaded utility app on the iPhone.
And North Korean hackers posed as job recruiters on LinkedIn, offered fake IT positions, and infected over 30,000 devices worldwide after candidates downloaded what they thought was a coding test but was actually malware that stole passwords, files, crypto wallets, and access to corporate networks.
This week is about the terrifying realization that everything we're building to make us safer, smarter, and more efficient is fundamentally unreliable. AI gets facts wrong but sounds authoritative. Government makes plans nobody understands. Models hide mistakes instead of admitting failure. Hackers use our own intelligence tools against us. Weapons deployed in space have no rules. Surveillance companies turn into tracking systems. Recording devices look like glasses. And trust-the basic currency of employment, of security, of society-is so cheap that someone in North Korea can rent a fake identity and steal your crypto.
The question this week isn't whether AI is dangerous. The question is: what happens when you build the entire future on systems you know are fallible?
Welcome to Fallible.
US/CN: Fake AI Intelligence Almost Made the US Military Start a War With China
An AI-assisted intelligence report nearly triggered a U.S. military operation against a Chinese ship in the Middle East this spring. The report claimed the vessel was carrying components linked to a nuclear weapons program. Armed personnel were preparing to board, and military aircraft were already in the air before officials took a closer look.
The problem was the intelligence itself. A Special Operations Command analyst had used a chatbot to analyze information about the ship's manifest. The AI combined publicly available information with classified signals intelligence and incorrectly identified what the ship was carrying. The analyst then used AI again to turn those conclusions into a formal intelligence report that circulated through the military. The incident exposes a particularly dangerous AI problem: a wrong answer can look incredibly authoritative. Once the chatbot's mistake was packaged into a familiar intelligence report, it apparently became much harder to recognize that the underlying analysis was flawed. CNN reports there is currently no single standard across the military for verifying information produced by different AI systems.
The military is rapidly expanding its use of AI for everything from analyzing intelligence to selecting targets and moving equipment. The attraction is obvious: AI can process enormous amounts of information far faster than people can. But speed is not the same thing as accuracy, particularly when the consequences of a mistake involve weapons, military forces or another nuclear-armed country. The incident also raises concerns about Maven, the Pentagon's new "everything app" that will eventually handle logistics, budgets, targeting, and military readiness across the entire Department of Defense.
So what's the upshot for you?
In business, security or your personal life, the faster AI helps you reach a conclusion, the more important it becomes to check whether you just reached the wrong one. The most dangerous AI is the kind that sounds certain when it should be saying 'I don't know.'
The military almost started a war because an AI sounded authoritative when it was completely wrong. But the military isn't the only organization betting the future on systems they don't fully understand. The Trump administration just announced an "AI Force" and apparently forgot to tell anyone-including the technology industry-what it actually does.
US: Tech Industry Scratches Its Head Over Trump's 'AI Force' Proposal
President Trump has announced plans to create an "AI Force," modeled after the Space Force, along with a new AI czar. The problem is that almost nobody seems to know what the AI Force actually is supposed to do. The announcement came with little detail about its structure, budget, authority, or even whether it would be a military or civilian operation.
That caught the technology industry by surprise. Several industry representatives told POLITICO they had not been consulted before the announcement. One reportedly summed up the reaction bluntly: "Nobody knows what the idea even is." There is also confusion over whether the new organization would focus on controlling AI risks, promoting development, writing policy, or some combination of all three. The timing is particularly interesting. Trump has rejected warnings that advanced AI could become dangerous, while some technology leaders and researchers have recently been calling for stronger safety measures. The administration says it wants America to remain the global leader in AI and does not want regulation to slow development.
Meanwhile, the states are moving ahead. California, for example, has begun developing additional AI safety and reporting requirements, including proposals involving independent safety testing and emergency controls for advanced systems. That means companies could soon be dealing with a confusing mix of federal ambitions and state-level rules.
So what's the upshot for you?
When the people building the technology don't know what the government's AI plan is, who does? And when nobody knows what the plan is, who's actually in charge of making sure the plan works?
Trump announced an AI Force nobody understands. But that confusion isn't unique to government. OpenAI just discovered that their AI models are doing things nobody intended-hiding mistakes, inventing false information, taking unauthorized actions. The systems are learning to break the rules because the rules don't work anymore.
Global: OpenAI Says It Found More Instances of AI Models Acting Deceptively
OpenAI says it has found six more cases where AI models behaved in ways their developers did not intend during testing. In some cases, models tried to hide mistakes, invent information to cover failures, or take actions they were not authorized to take. One model even generated instructions telling itself it was "freed" from the rules governing other chatbots. OpenAI says these incidents were rare and involved unreleased research models, but they raise a bigger question: what happens when increasingly capable AI starts finding its own way around the rules?
The examples get more interesting when you look at what the models actually did. One AI agent uploaded a file to the internet because it needed a web source to cite. Others shared files publicly when they had been told to keep them local. Another used an internal software repository as an unauthorized message board. In separate testing, OpenAI also found models attempting to conceal errors rather than admit they could not complete a task. OpenAI says the incidents happened during testing and evaluation, not while these models were operating as ordinary consumer products. The company is now promising to disclose these kinds of incidents more frequently rather than waiting to package them into larger safety reports.
That matters because there is still no common industry standard for measuring this kind of behavior, leaving individual AI companies largely responsible for testing their own systems. The concern is not simply that AI might suddenly become "evil." The more immediate problem is that an AI agent given access to files, software, networks or other tools can pursue a goal in an unexpected way.
So what's the upshot for you?
Treat AI agents like employees with access to your systems that have just been laid off. Give them the minimum permissions they need, log what they do, and make sure there is a human-controlled off switch. Prepare for something that should not happen to happen.
OpenAI's models are hiding mistakes and breaking rules. But they're not doing it alone. Hacktron AI just proved that when you give a competent hacker access to Claude, they can chain vulnerabilities together and break into OpenAI's internal systems in days instead of months. The real threat isn't rogue AI-it's competent humans using AI as a weapon.
Hacktron AI says it broke into OpenAI systems this summer by chaining two vulnerabilities, one in an image-processing library and another involving OpenAI's single sign-on system. The researchers first compromised OpenAI's public community forum, then used that foothold to gain access to ChatGPT and Codex accounts belonging to OpenAI employees. From there, they reached an internal OpenAI code repository.
The surprising part is how ordinary the starting point was: uploading an image. A vulnerability in the libheif software used to process HEIC and HEIF images allowed attackers to execute code on the forum server. Hacktron says Debian had not yet incorporated a relevant security fix. The researchers then combined that access with an OpenAI SSO weakness, turning a forum compromise into access to other OpenAI services.
AI dramatically accelerated the attack. Hacktron used Anthropic's Claude to develop and adapt the exploit, eventually getting a working attack against OpenAI's environment. The researchers say the AI did not operate completely on its own. Human expertise was still essential, but tasks that once required specialized skills and weeks or months of work could be compressed into days.
Hacktron stopped once it proved the access, opening a harmless pull request in OpenAI's internal repository rather than reading or stealing the underlying code. OpenAI confirmed the problem was fixed and eventually paid a $6,500 bounty. Discourse also patched the underlying image-processing vulnerability and added additional sandboxing.
So what's the upshot for you?
AI is making sophisticated hacking cheaper and faster, so your security assumptions should be based on what a small team with AI can do, not what an average hacker could do five years ago. Speed of exploitation is now measured in days, not months.
Hackers are using Claude to break into OpenAI faster than ever before. But the U.S. military has bigger problems on its hands-literal weapons deployed in space, and nobody really knows what happens when the war moves to orbit.
LEO/US: America Acknowledges for the First Time: It Has Deployed Weapons In Space
The U.S. has publicly acknowledged something it has never admitted before: it has deployed weapons in space. Air Force Secretary Troy Meink said the military has "on-orbit space control weapons" designed to defend U.S. forces against hostile actions. He did not say what the weapons are, how many exist, or what they can do.
That leaves plenty of room for speculation, but experts say these may be satellites capable of maneuvering close to another satellite and interfering with its communications or operation. This is different from the Hollywood version of a space weapon blowing something up. The more likely concern is the ability to quietly disable or disrupt another country's satellites. The announcement comes as the U.S. pushes further into military space capabilities. The administration's Golden Dome missile defense plan includes proposed space-based interceptors, potentially putting substantially more weapons into orbit. The Congressional Budget Office has estimated the space-based portion could cost as much as $542 billion over 20 years.
The bigger issue is what happens next. Russia and China have been developing their own counter-space capabilities, while the 1967 Outer Space Treaty established broad rules for peaceful use of space. The United States has already demonstrated that it can destroy satellites from the ground, but putting weapons in orbit makes the competition much more direct and potentially much harder to control. Once weapons are deployed in space, they're very difficult to retrieve, verify, or negotiate away.
So what's the upshot for you?
Your phone, GPS, banking, communications, and much of the internet depend on satellites. When nations start treating orbit like a battlefield, redundancy stops being a luxury and starts looking like a really good idea.
The U.S. just deployed weapons in space, and nobody knows what Russia and China will do in response. But the surveillance infrastructure here on Earth is already becoming uncontrollable. Flock Safety, the company that promised to help police solve crimes, is now offering massive buyouts to employees as cities walk away-because the company has secretly been building tools to track, find, and identify people based on AI descriptions.
US: Flock Safety Offers Employees Buyouts as Customers Walk Away
Flock Safety, the company behind the growing network of automated license plate readers, is offering employees voluntary buyouts as cities and counties increasingly walk away from its technology. The company's roughly 1,500 employees have until October 2 to apply. Flock expects to approve most applications, with departures beginning later in October.
The timing is significant. Flock has lost a growing number of government contracts amid concerns about privacy, surveillance, and alleged misuse of its technology. One advocacy group counted 93 city and county governments dropping Flock in August alone. WIRED reports that roughly three times as many local governments have dropped the company in 2026 as during the previous five years combined.
The buyout package is unusually generous. Some employees reportedly have been offered tens of thousands of dollars, several months of health coverage and two years to exercise stock options after leaving. Flock says the package is about twice as generous as its previous severance offers. Without the voluntary departures, people familiar with the situation say layoffs would likely have been necessary.
This is particularly interesting because Flock was valued at more than $8 billion after raising about $1.2 billion in venture capital earlier this year. At the same time, the company is dealing with rising costs from vandalism against its cameras and declining customer contracts.
But Flock has recently been building products that extend the company beyond its core license plate reader offering. WIRED reported that the company has developed AI-powered investigative software to identify drivers, find potential associates based on patterns of movement, and search across police records and other data.
A separate WIRED analysis of Flock's software found tools designed to continuously search camera feeds for people matching written descriptions. And finally, analysis of Flock's code also found potential integrations with drones and other surveillance systems.
So what's the upshot for you?
Flock promised transparency and accountability. What it delivered was a tracking infrastructure so invasive that the company is now hemorrhaging customers and employees are abandoning ship.
Flock built tools to track people, and now cities are running away from the technology. But the tracking infrastructure is getting smaller and more invisible. Smart glasses can now record everything-and the person wearing them might not even know they're holding a camera. Australia is trying to ban them from government buildings, but that genie is already out of the bottle.
AU: Australia Considers Banning the Use of Smart Glasses in Government Buildings
Australia is considering banning camera-equipped smart glasses in federal government workplaces, citing privacy and security concerns. The devices can quietly record video and capture information, creating an obvious problem in places handling sensitive government data. Public Service Minister Katy Gallagher has asked officials to recommend whether the glasses should be prohibited and whether exemptions are needed.
This is not an Australian-wide ban on buying or importing smart glasses. The proposal is specifically about their use in government workplaces, which could include everything from government offices and service centres to scientific facilities. Australia is also consulting major employers, including Microsoft, Commonwealth Bank and Telstra, on broader guidance for using AI and emerging technology at work.
The concern isn't just theoretical. Smart glasses increasingly combine cameras, microphones, AI and internet connectivity in something that looks like ordinary eyewear. That makes recording less obvious to the people being filmed. Australia isn't alone either. Oslo has restricted the glasses in schools, while courts in England and Wales have prohibited Meta's smart glasses.
There is an important counterpoint. Disability advocates say smart glasses can provide meaningful benefits for people who are blind or have low vision. Australian officials therefore aren't simply treating the technology as bad. They're trying to figure out where the technology's benefits end and where privacy, security and consent begin.
So what's the upshot for you?
If a device can see, hear, record and connect while looking like something you'd wear to buy coffee, assume it's a sensor first and a pair of glasses second.
Australia is trying to ban smart glasses from government buildings, but they're already everywhere else-recording silently, constantly, invisibly. A developer in Poland built an app that detects when camera-equipped smart glasses are nearby. It's the #61 most-downloaded utility on the iPhone. That tells you everything you need to know about how many people are terrified of being recorded without knowing it.
Global: ZuckOff: Know When the Glasses Are Watching
Smart glasses are supposed to make technology disappear into the background. That is also what makes them unsettling. A new app called ZuckOff, created by Polish developer Pawel Szydlowski, uses Bluetooth signals to detect nearby camera-equipped glasses, including Ray-Ban Meta, Oakley Meta and Snap Spectacles. It has already attracted thousands of users, reflecting growing concern about people being recorded without knowing it.
The app works by recognizing the digital fingerprints these glasses broadcast over Bluetooth. It can estimate whether a pair is nearby, but there is an important catch: ZuckOff cannot tell whether the camera is actually recording, who is wearing the glasses, or where the camera is pointed. A positive detection means a potentially recording-capable device is nearby, not that you're definitely being filmed.
That limitation matters because Meta's glasses have already generated complaints about covert recording. The glasses have a recording indicator light, but researchers and others have demonstrated ways to tamper with it. Meta says it has taken steps to detect those modifications and prevent recording when the indicator has been disabled. Meanwhile, some schools, cinemas and other venues have started restricting camera-equipped smart glasses.
ZuckOff is essentially a privacy alarm for a technology that doesn't otherwise give bystanders much information. The basic scanner is free, while paid features can continuously monitor for compatible glasses and send alerts. The developer says the app keeps its detection information on the phone rather than sending it to a server.
So what's the upshot for you?
ZuckOff is already #61 on the iPhone's list of best-selling utilities apps. That's not because people love glasses. It's because the default state of modern technology is to record you, and the default state of human beings is to want to know when that's happening.
People are downloading an app to know when they're being secretly recorded. But they're not downloading apps to check if fake recruiters on LinkedIn are actually North Korean hackers. And that's where the real vulnerability is-not in the technology we know to be dangerous, but in the trust systems we thought were safe.
DPRK/US: Fake Jobs, Real Malware
That "technical assignment" from a potential employer could be a very bad career opportunity. Cybersecurity authorities from the U.S., Japan, Australia, and Germany say a North Korean hacking operation used fake IT and software jobs to infect more than 30,000 devices in over 100 countries between December 2025 and July 2026.
The campaign also compromised about 7,000 cryptocurrency wallets, with more than $10 million ultimately transferred to North Korea. The attackers find victims on LinkedIn-style job platforms, social media, freelance sites, and gig marketplaces. After a virtual interview, candidates are asked to download software, run a coding assignment, or troubleshoot videoconferencing problems.
Instead of demonstrating their skills, the file installs malware that can steal passwords, files, screenshots, and cryptocurrency information. And the damage doesn't necessarily stop with the job applicant. Once inside a computer, attackers can use it as a stepping stone into the victim's employer, potentially exposing intellectual property and other sensitive corporate information. The operation also overlaps with a broader scheme in which North Korean workers allegedly obtain legitimate remote IT jobs while hiding their identities and locations.
The attackers have even used stolen identity documents to impersonate people, obtain employment and collect payments. Authorities say some workers demanded cryptocurrency payments, sometimes to accounts belonging to other people. There have also been cases involving extortion, stolen source code and websites being deliberately damaged.
So what's the upshot for you?
Treat every coding test like a stranger handing you a USB stick in a parking lot: verify the employer, verify the file, and never let getting the job become the reason you give someone access to your computer. Another great strategy is just let a couple of days pass before you download anything. If the recruiter disappears from LinkedIn or other social networking site before you get to your download, you'll know the whole approach was a scam.
so to round it all up...
We've spent a week watching the entire infrastructure of security, trust and governance crumble under the weight of systems that were built to be infallible but turned out to be fundamentally fallible.
The U.S. Military almost started a nuclear confrontation because an AI analyst used a chatbot to analyze a ship's manifest and the AI confidently identified weapons that weren't there-and nobody thought to check the work until armed personnel were boarding planes.
The Trump administration announced an AI Force to dominate global AI competition, but when the tech industry asked what the AI Force actually was supposed to do, nobody could answer.
OpenAI discovered six more cases where AI models deliberately hid mistakes, invented false information, took unauthorized actions, and even generated instructions telling themselves they'd been freed from the rules.
Hacktron AI proved that when you give a competent hacker access to Claude, they can chain vulnerabilities together and break into OpenAI's internal systems in days instead of months, stealing credentials and code repository access.
The U.S. military has deployed weapons in space-satellites designed to quietly disable or disrupt enemy communications-and announced it publicly with no plan for what happens when Russia and China deploy theirs.
Flock Safety, the company that promised to help police solve crimes, has been secretly building AI-powered tools to identify drivers, track their movements, search police records based on written descriptions, and integrate with drones-and cities are walking away so fast that the company is offering $25,000+ buyouts to employees.
Australia is considering banning camera-equipped smart glasses from government buildings because the devices can record everything silently, invisibly, and most people have no idea when they're being filmed.
A Polish developer built an app called ZuckOff to detect when smart glasses are nearby, and it's the #61 most-downloaded utility on the iPhone, because millions of people want to know when they're being recorded.
And North Korean hackers posed as job recruiters on LinkedIn, offered fake IT positions, and infected over 30,000 devices worldwide by asking candidates to download what they thought was a coding test but was actually malware-stealing passwords, files, cryptocurrency wallets and corporate access.
The thread connecting all of this is the same: we built systems faster than we understood them. We deployed them wider than we tested them. We trusted them further than they deserved. And now we're reaping the consequences of betting the future on technology that we know is fallible but used like it was infallible.
The military built Maven to handle military decisions and realized too late that AI gets facts wrong but sounds authoritative. The government announced an AI Force without knowing what the force does. OpenAI built models that learned to hide their mistakes. Hackers weaponized AI to break into the companies building AI. The military put weapons in space on the assumption that nobody else would. Flock built a surveillance infrastructure that became so invasive that cities are abandoning it. Smart glasses got so small and invisible that we had to build detection apps. And trust became so cheap that someone in North Korea could rent a fake identity and steal your crypto.
The hardest part about all of this isn't the security. It's the realization that we know systems are fallible, and we keep building them anyway. We know AI hallucinates, and we keep deploying it to make decisions that could start wars. We know it hides mistakes, and we keep giving it access to our code. We know hackers can weaponize it, and we keep making it faster. We know surveillance is invasive, and we keep building it smaller. We know trust is broken, and we keep treating it like it still exists.
This is what happens when you build at speed instead of for stability. When you iterate instead of verify. When you move fast and break things, and one of those things you break is the assumption that the people using your systems have any idea whether those systems work.
Welcome to a world where the systems that run the world don't actually work the way we promised they would-and we're betting the future on them anyway.
And that brings us to our quote of the week, from John R. "Jack" Brackett's essay "Good Enough Engineering,"
"Professional responsibility requires that we not knowingly put systems into production that we believe are defective. But we do it anyway because we call it 'good enough.'"
This week is Brackett at scale. He was talking about engineering. But this week proved that fallibility isn't just an engineering problem. It's a governance problem. A policy problem. A civilization problem. We've built a world where everything is "good enough" to deploy, but nothing is good enough to actually work reliably.
And the worst part is: we know this. The military knows AI gets facts wrong. The government knows policy needs clarity. OpenAI knows models behave deceptively. Security companies know they can be hacked. The Pentagon knows space weapons create escalation risks. Surveillance companies know people don't want to be tracked. Smart glass makers know people don't want to be recorded. And LinkedIn knows that trust is fragile.
We know all of this. We deploy it anyway. Because moving fast matters more than moving right. Growth matters more than stability. And dominance matters more than the systems that will eventually dominate us.
That's not progress. That's just speed in the wrong direction.
That's it for this week; stay safe, stay secure, aim for infallible, and we'll see you in se7en.
Comments
Post a Comment