Questions For
Meredith Whittaker
Meredith Whittaker has been warning us about what she calls the surveillance-driven business models of Big Tech since long before lawmakers were sounding the alarm. Now, with the demands of advanced and agentic AI, she thinks we could be heading into a world in which privacy is essentially over.
After joining Google back in 2006, she rose to fame in 2018 when she helped lead the employee effort to prevent the company’s involvement with Project Maven, the Pentagon’s flagship artificial intelligence initiative to integrate machine learning and computer vision into military workflows. The company walked away from the project. Around the same time, she founded the AI Now Institute at New York University with the researcher Kate Crawford, to produce interdisciplinary research on the social implications of AI and the concentration of power in the tech industry. In 2019, she left Google, citing retaliation for her internal organizing, to focus on AI Now.
In September 2022, she became president of the Signal Foundation, the nonprofit behind the encrypted messaging app Signal. She has positioned the organization as a deliberate counter to the data-extractive economics of Big Tech, arguing that Signal’s nonprofit structure removes any financial incentive to compromise user privacy. Whittaker is generally seen as a privacy absolutist, viewing it as a fundamental human right that should not be betrayed for any reason. She has repeatedly said Signal would withdraw its services from a country rather than weaken its encryption, taking that stance publicly against proposed legislation in the United Kingdom, Sweden and Australia, as well as against the EU’s “chat control” scanning proposals. Along the way, she has become a sharp critic of the current business models around AI, which she argues are built on the back of surveillance.
Almost a decade since she fought Google’s surveillance work with the Pentagon, the world looks very different. In China, AI has been layered onto existing surveillance systems to build a more granular picture of citizens. In the EU, flagship regulations such as the AI Act have constrained uses of biometric surveillance, while national security exemptions have left room for the bloc to build biometric databases covering migrants and travelers. In the US, Google and OpenAI have both signed wide-ranging deals with the US government allowing their AI to be used for military applications.
We spoke to Whittaker about how advanced AI puts privacy under more strain than ever, why the internal revolt that killed Maven hasn’t recurred and whether there is any commercially sustainable model for AI that protects consumer privacy.
What follows has been edited for clarity and length.
People use AI in very different ways from other technologies, sometimes sharing intimate details of their lives by asking advice about personal problems or sharing tasks for work. Do you think people realize how much they’re giving away?
Let’s let’s zoom out a little bit, because the personal data you might dump into the window of a chatbot is not really the extent of our concerns.
The large-scale language models, which have kind of come to occupy almost the definition of what we think of as AI, are built on the mass collection of data. The paradigm is scale at all costs. There’s a data hunger, and so you’re seeing an erosion of privacy. You have Meta dropping end-to-end encryption from Instagram DMs. You have Apple seemingly reconsidering its strategy as it moves to Google Gemini and away from some of the private cloud compute assurances they had. So there are structural imperatives for increasing invasions of privacy.
And then of course there’s what you dump into the [chat] window [of an AI]. What are you telling it? What is it asking you? What type of profile is it creating about you? It’s certainly more information than you would type into a search window, maybe more information than you would type into an intimate email. Under the hood, that data is going to a corporation which is going to do whatever it wants with it. They will use it to model your preferences. They will use it to train their AI systems. They will turn it over to the government in response to a subpoena.
Would anything convince you that people are making informed choices when sharing their data?
Informed by what? The documentation that is shipping with new operating system updates — Android and iOS being two key examples — is not even clear in many cases whether the data is being processed on the device or off the device. What data are they collecting? Is it being paired with credit data that they buy from data brokers? Is it being paired with geolocation data that they’re sucking off my device? I don’t have the answer to that. Informed consent is not really what we’re working with.
It seems like people never fully came to terms with the true privacy implications of social media. Maybe the closest we got was the Cambridge Analytica scandal, which raised the issues in the public consciousness and brought them more fully to the attention of policymakers. Are we going to have to wait for AI’s version of that scandal before we see any change?
My sense is that the public consciousness exists. It’s just the agency to do anything about it that doesn’t. And so without the power to shift this, people resort to narratives around, “It’s too late anyway. Nothing I can do about it.”
After the Tumbler Ridge school shooting, there were almost immediate calls for mandatory flagging of problematic AI conversations. Prior to the shooting, OpenAI had flagged and banned the ChatGPT account of the perpetrator due to conversations involving gun violence, but did not report this to law enforcement. You’ve said in the past that a lack of data is rarely the problem in these cases — but here things were different. Wasn’t that a pretty strong case for a duty to report?
[Pause] I don’t want to answer that question. [Pause] The reason I’m sitting with this is because I think we need to reframe that question to get at the roots of it. There was a tragic shooting. [But] I want to zoom out. [Flagging a conversation to law enforcement] means scanning every single message that is sent, against let’s say, a database or a model that identifies keywords, to be like: Is this something we need to report to law enforcement? That is a tool that can be expanded or contracted to include dissident activity [or] criticizing ICE. A junior developer could [build] that, could vibe code that easily once [the approach was] greenlit by the powers that be.
And then we look at this [specific] problem, this extremely gnarly pathology [of high school shootings] that is metastasizing in our world. What are we solving with that sort of surveillance system that basically flags bad speech to whoever is in power at that time? Who do we think would get caught up in that dragnet? What other issues in a world moving toward the authoritarian would be added to that roster? This is sensitive, and it’s important that when I give an answer, I do it with a duty of care, so that it’s very clear what I’m saying.
It’s almost a decade since you helped oppose Project Maven so that Google backed out of the contract. Now we have Google and OpenAI signing contracts with the Pentagon that reportedly give wide-ranging access to AI tools for use in defense applications. People within those companies have tried to oppose those contracts internally without success. Is something different this time?
There’s so many things that are different. In some sense, there is an incentive alignment between governments, who want access to these tools and the surveillance infrastructure that underlies them, in service of geopolitical positioning, and publicly traded companies, whose ultimate objective function is profit and growth. [We’re still in] this early phase [of AI development], assuming we continue on this trajectory, in which workflows and practices and norms get instantiated at the level of bedrock around which other things are built, making them very, very difficult to rip out. Think about, like, ripping out Microsoft Excel? And so there is a massive incentive that is aligned with the incentives of the shareholder-driven corporation to ink these contracts.
You’ve said agentic AI is structurally incompatible with end-to-end encryption.
Many of the integrations we’re seeing at the operating system level are incompatible with enabling application developers to use end-to-end encryption to provide privacy with integrity.
Is there any architecture in which AI agents can maintain user privacy?
Certainly there are guardrails and harnesses and access controls you can put on agentic systems. I think there is a paradigmatic tension insofar as a fully realized agent that is doing everything for you on your behalf seamlessly, with full knowledge of your preferences, et cetera, et cetera, needs huge amounts of access to your data, and needs a kind of pervasive ability to act without permission, both of which are oppositional to privacy. Most agents are not built that way, but that’s the goal that the quote-unquote “most capable” or “most enabled” agents are leaning toward.
That puts Signal in a tough spot. You have a well-defined philosophy about how data should be handled, and third parties are taking different tacks. Where does that leave your organization?
The ecosystem is against us right now. People very, very much want privacy. We see Signal’s user numbers trending up. We see people really recognizing the value of Signal. We see concern about the invasions of privacy. But the ecosystem is building and trending in a way that does damage Signal’s ability to continue to provide privacy at the application layer with the integrity that we are committed to. The [most recent Microsoft Windows] operating system has made choices around data access that create what is effectively a backdoor that bypasses our strong encryption. That is a huge concern.
If AI becomes the dominant interface of computing, does that mean we’re going to lose the fight to keep data private?
It could very well mean that. It could mean that Signal effectively can’t ensure privacy.
Is there any commercially sustainable model for AI that maintains privacy?
I’m thinking about this. I don’t have a pat answer right now. There’s a demand for it, certainly. I think the definition “that maintains privacy,” given the different axes on which AI is in tension with or undermines privacy, is something we would need to spend some time with.
We would also need to look at what would have to change about the core economic incentives of the tech industry in general, and how that would displace monoliths and the winners of the current AI paradigm. We have the S&P 500 wrapped around Big Tech and AI companies. So there’s a lot more at stake there than simply “Is there a demand for it, and could we, if we wanted to, change things to meet that demand?” What we’re talking about is a complex political and economic configuration, not simply a series of technical choices.
You were making critiques of AI’s impact on privacy a decade before it was fashionable. Yet we’re still sitting here discussing very similar problems 10 years later. Is there anything that could have been done a decade ago that would have gotten us to a different place now?
I don’t have an elegant answer to this question because, in part, being right is not a strategy. Was it a question of narrative, of incentives and power asymmetries, a business model that had baked-in path dependencies on some of this stuff? I don’t know.
I might leave it with this … A lot of policymaking over the last couple decades that I’ve witnessed, in general and in tech in particular, has assumed that the good guys would always be in power, and that these capabilities would be always used [for good]. “Of course, we would never weaponize surveillance. Of course, we would never use the power of the administrative subpoena to demand access to information about people who criticize the government anonymously online.” Don’t be paranoid, right? [But I think a ] healthy dose of paranoia and the humility of recognizing that power shifts hands [would have gone a long way] to steering us in a direction of curbing these business models, these infrastructures and these surveillance capabilities.