Artificial Insecurity: how AI tools compromise confidentiality
In the first of our three-part blog series on the dodgy_ digital security practices underlying advanced artificial intelligence (AI) tools, we unpack how large-language models (LLMs) can _jeopardize the confidentiality of people’s data.
Whatever you think about the promises or perils of AI, it’s becoming increasingly impossible to ignore that these tools are beset by glaring security vulnerabilities. From exposing user data to facilitating hacks, from undermining information integrity to creating supply chain vulnerabilities, AI tools are underpinned, and undermined, by dodgy security practices. As we’ll explore in this series, this has grave consequences for the confidentiality of our data, for information integrity, and for access to and availability of systems — all problems that a human rights-respecting approach can help solve.
// Why do we need to talk about AI and digital security?
It’s important to note that these days when we talk about ‘advanced AI’ tools such as chatbots, image generators, and ‘AI agents,’ what we are really talking about are systems built on ‘large language models’ (LLMs). LLMs are a type of machine learning model trained on enormous amounts of data including text, images, and video, which can generate content in response to prompts and perform complex tasks with varying degrees of reliability.
To analyze the specific security risks associated with LLMs, it’s helpful to use the confidentiality-integrity-availability (CIA) triad, a widely used model that guides how organizations handle data security. When applied to LLMs, this framework helps us understand their security risks, as well as showing why human rights safeguards are essential to mitigate those risks. For instance, as we’ll discuss below, the CIA triad is a way to understand how people’s individual security and their digital rights can be jeopardized by LLMs — because what happens to the confidentiality of the data you input into a chatbot when there’s a data breach? But it also allows us to examine how LLMs jeopardize digital security more widely and systemically; as we’ll discuss in parts two and three of this series, this includes compromising information integrity through ‘AI slop’ or concentrating market power in such a way that the availability of such systems is undermined.
// How can AI compromise confidentiality?
According to the U.S. National Institute of Standards and Technology (NIST), confidentiality means “[p]reserving authorized restrictions on information access and disclosure, including means for protecting personal privacy and proprietary information.” In other words, no unauthorized people should be able to access your information. This is especially important given how people are using LLM-based tools for everything from therapy and medical advice, to companionship, while businesses, governments, and nonprofits are integrating these tools into workflows that deal with sensitive data.
The security flaws plaguing LLM-based tools are often quite basic. For example, while established players like Microsoft and Google offer multifactor authentication (MFA), many popular AI apps have limited native account protection, meaning accounts can be easily hijacked by attackers. Data breaches, whether accidental or intentional, are also a regular occurrence. In January 2025, Wiz Research uncovered a publicly accessible database belonging to Chinese company DeepSeek, which contained people’s chat history, secret keys, backend details, and other highly sensitive information. And barely three months ago, an attacker used a third-party analytics provider to hack OpenAI, leaking private information including names, email addresses, location data, operating system, and browser information.
Even tools that promise to boost security may in fact undermine it. Security researchers at Koi recently discovered how a number of Urban Cyber Security Inc.’s virtual private network (VPN) browser extensions, many of which promised “AI protection” for sensitive data, were actually harvesting data on all prompts entered into LLMs, the responses received, and timestamps, metadata, and information about the AI tools used by eight million people. According to Koi, Urban Cyber Security Inc. was then sharing this information with data brokers.
// What are the wider privacy risks associated with AI?
The New York Times is currently suing Open AI for copyright infringement, alleging that people are using ChatGPT to circumvent its online paywall. To prove this, the news outlet wants access to over 20 million private ChatGPT conversations. Whatever the merits of the case, granting access to these conversations would be a huge violation of privacy — yet it is not even an isolated case, with requests for AI chat logs to be made available in legal proceedings becoming increasingly frequent.
One of the major risk factors here is that end-to-end encryption (E2EE) is not a standard, or even necessarily an available, feature for chatbots such as ChatGPT or Gemini, meaning that people’s chat histories face exposure. E2EE is fundamental for protecting human rights, including privacy. In addition to the problem of chatbot platforms not encrypting chat histories, LLM-based AI ‘agents’ or assistants, at the operating system or application level, are undermining the security promise of E2EE.
In April 2024, Meta rolled out its AI chatbot, Meta AI, for WhatsApp, which uses E2EE, and there is no option to remove it. This means that, upon request from another user and without your consent, Meta AI can access and summarize messages between you and that user, with the summaries passing through Meta’s servers. As the Electronic Frontier Foundation explains, when the person you are chatting with asks Meta AI a question, “that part of the conversation, which you can both see, is not end-to-end encrypted, and is usable for AI training.” This is a major step in the wrong direction. While some slight risk to your WhatsApp correspondence remaining private has always existed (e.g. the person you are chatting with might screenshot or copy the encrypted content, or report your messages to WhatsApp for alleged guideline violations) Meta AI removes the expectation of privacy as a default.
WhatsApp claims that, if you don’t want Meta AI to summarize your shared conversations, all you need do is activate its advanced chat privacy feature. But shifting the onus onto individuals isn’t good enough, especially because this feature must be manually activated per individual chat; there is no easy, one-step way to automatically apply it across all your conversations. Integrating Meta AI by default, diluting WhatApp’s privacy and security promises in the process, is part of a wider trend of LLM-based tools being forced on people with or without their consent.
// What are the dangers of AI (double) agents?
Alongside the integration of new AI features into existing apps, ‘agentic AI’ software that functions at an operating system level is also creating new risks. AI enthusiasts sing the praises of integrating LLMs into every facet of our lives, championing the widespread adoption of AI agents capable of executing commands on our behalf, whether by booking flights, messaging potential romantic partners, or playing the stock market.
This is what Meredith Whittaker, President of the Signal Foundation, calls the “root permission problem,” whereby giving AI agents total access to our systems and data, including ‘memories’ of our interactions, opens up a wealth of attack opportunities and can even undermine E2EE platforms. Simon Willison has framed this problem as a lethal trifecta in which AI agents haveaccess to private data, gain exposure to untrusted content, and can communicate externally to pass your data onward. For example, AI agents are susceptible to prompt injection attacks, where attackers trick the AI agent into doing something you didn’t intend for it to do, such as exposing your credit card details. These risks are currently playing out in real-time thanks to OpenClaw, an open source, self-hosted AI assistant software that, as it turns out, allowed thousands of people to set up AI agents with abysmal security settings.
// What security solutions have been suggested?
To date, the security measures implemented for LLM-based tools have not kept pace with the growing risks. In its response to The New York Times’ request for chat histories, Open AI indicated that it is working on “client-side encryption for your messages with ChatGPT” — yet even here the company hints at deploying “fully automated systems to detect safety issues in our products,” which sounds very much like client-side scanning (CSS). CSS, which involves scanning the content on an individual’s device for some class of objectionable material, before it is sent onwards via an encrypted messaging platform, is a lose-lose proposition that undermines encryption, increases the risk of attack, and opens the door to mission creep.
By contrast, the open source community has made positive strides in prioritizing confidentiality. OpenSecret’s MapleAI supports a multidevice end-to-end encrypted AI chatbot, while Moxie Marlinspike, co-author of Signal’s E2EE protocol, has launched ‘Confer,’ an open source AI assistant that protects all user prompts, responses, and related data. But for now at least, such rights-respecting solutions remain the exception rather than the norm.
Unbridled AI adoption combined with depressingly lax security practices demands urgent action. The security issues associated with advanced AI tools are the consequences of deliberately prioritizing profit and competitiveness over the security and safety of at-risk communities, and they will not resolve on their own. While we would love to see companies self-correct, governments should not shy away from demanding that these companies prioritize security and human rights, especially when public money is being spent to procure and build ‘public interest’ AI tools. In the meantime, we can all also choose to support open, accountable rights-respecting alternatives to the big name models and tools where possible.
PART 2 – ARTIFICIAL INSECURITY: THREATS TO INFORMATION INTEGRITY
In part two of this series on digital security and AI, we will explore how and why even the most privacy-preserving AI tools can’t guarantee the integrity of the information they share.
coming soon