Minding Mindful Machines: AI Agents and Data Protection Considerations
Thank you for the contributions of Rob van Eijk, Marlene Smith, and Katy Wills
We are now in 2025, the year of AI agents. In the last few weeks, leading large language model (LLM) developers (including OpenAI, Google, Anthropic) have released early versions of technologies described as “AI agents.” Unlike earlier automated systems and even LLMs, these systems go beyond previous technology by having autonomy over how to achieve complex, multi-step tasks, such as navigating on a user’s web browser to take actions on their behalf. This could enable a wide range of useful or time-saving tasks, from making restaurant reservations and resolving customer service issues to coding complex systems. However, AI agents also raise greater and novel data protection risks related to the collection and processing of personal data. Their technical characteristics could also present challenges, such as those around safety testing and human oversight, for organizations seeking to develop or deploy AI agents.
This analysis unpacks the defining characteristics of the newest AI agents and identifies some of the data protection considerations that practitioners should be mindful of when designing and deploying these systems. Specifically:
- While agents are not new, emerging definitions across industries describe them as AI systems that are capable of completing more complex, multi-step tasks, and exhibit greater autonomy over how to achieve these goals, such as shopping online and making hotel reservations.
- Advanced AI agents raise many of the same data protection questions raised by LLMs, such as challenges related to the collection and processing of personal data for model training, operationalizing data subject rights, and ensuring adequate explainability.
- In addition, the unique design elements and characteristics of the latest agents may exacerbate or raise novel data protection compliance challenges around the collection and disclosure of personal data, security vulnerabilities, the accuracy of outputs, barriers to alignment, and explainability and human oversight.
What are AI Agents?
The concept of “AI Agents” or “Agentic AI” arose as early as the 1950s and has many meanings in technical and policy literature. In the broadest sense, for example, it can include systems that rely on fixed rules and logic to produce consistent and predictable outcomes on a person’s behalf, such as email auto-replies or privacy preferences.
More recently, however, the technologies that several companies have unveiled are AI systems, typically enabled by advances in LLMs and machine and deep learning techniques, that are capable of completing complex, multi-step tasks, and exhibit greater autonomy over how to achieve these goals.
Advances in AI research, particularly around machine and deep learning techniques and the advent of LLMs, have enabled organizations to develop agents that can tackle novel use cases, such as purchasing retail goods and recommending and executing transactions. From finance to hospitality, these technologies could help individuals, businesses, and governments save time they would otherwise dedicate to completing tedious or monotonous tasks.
Companies, civil society, and academia have defined the latest iteration of AI agents, examples of which are provided in the table below:
Source | Definition |
“Building effective agents,” Dec. 19, 2024, Erik Schluntz and Barry Zhang, Anthropic | “[S]ystems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.” |
“Navigating the AI Frontier: A Primer on the Evolution and Impact of AI Agents,” Dec. 2024, Larsen et al., World Economic Forum and Capgemini | “[A]n entity that senses percepts (sound, text, image, pressure etc.) using sensors and responds (using effectors) to its environment. AI agents generally have the autonomy (defined as the ability to operate independently and make decisions without constant human intervention) and authority (defined as the granted permissions and access rights to perform specific actions within defined boundaries) to take actions to achieve a set of specified goals, thereby modifying their environment.” |
“Visibility into AI Agents,” Chan et al., ACM FAccT ’24, June 3–6, 2024, Rio de Janeiro, Brazil | “AI agents [are] systems capable of pursuing complex goals with limited supervision,” having “greater autonomy, access to external tools or services, and an increased ability to reliably adapt, plan, and act open-endedly over long time-horizons to achieve goals.” |
“Agents,” Sept. 2024, Julia Wiesinger, Patrick Marlow, and Vladimir Vuskovic, Google | “[A] Generative AI agent can be defined as an application that attempts to achieve a goal by observing the world and acting upon it using the tools that it has at its disposal. Agents are autonomous and can act independently of human intervention, especially when provided with proper goals or objectives they are meant to achieve. Agentscan also be proactive in their approach to reaching their goals. Even in the absence ofexplicit instruction sets from a human, an agent can reason about what it should do next to achieve its ultimate goal.” |
“Regulating advanced artificial agents,” Apr. 5, 2024, Cohen et al., Science | Defining long-term planning agents as “an algorithm designed to produce plans, and to prefer plan A to plan B, when it expects that plan A is more conducive to a given goal over a long time horizon.” |
“What are AI agents?,” July 3, 2024, Anna Gutowska, IBM | “An artificial intelligence (AI) agent refers to a system or program that is capable of autonomously performing tasks on behalf of a user or another system by designing its workflow and utilizing available tools. |
Table 1. Definitions of “AI agents”
These definitions highlight common characteristics of new AI agents, including:
- Autonomy and adaptability: Users generally provide an agent with the task they want it to achieve, but neither they nor the agent’s designers specify how to accomplish the task, leaving those decisions to the agent. For example, upon being instructed by a business to project the sales revenue of its flagship product for the next six months, the agent may decide that it needs sales figures from the last two years and use certain tools (e.g., a text retriever) to obtain these details. If it cannot find these figures or if they contain errors, it may determine that the next step is to seek information from other documentation. Agentic systems may incorporate human review and approval over some or all decisions.
- Planning, task assignment, and orchestration to solve complex, multi-step problems: An AI agent may make use of additional components, such as sensing, decision-making, planning, learning, and memory. In time, these systems may become more complicated with the development of multi-agent systems that feature a group of agents collaborating with one another to solve challenging tasks, such as diagnosing and devising treatments for rare medical conditions.
These characteristics enable advanced agents to achieve goals that are beyond the capabilities of other AI models and systems. However, they also raise questions for practitioners about the data protection issues organizations may encounter when developing or deploying these technologies.
Emerging Privacy and Data Protection Issues with Agentic AI
While the latest AI agents may raise similar risks to consequential decision-making and LLMs, they can also exacerbate or pose novel privacy and data protection considerations. The economic and social impact of AI agents is a topic of heated debate and significant financial investment, but there has been less attention on the potential impact of agents on privacy and data protection. In order to effectuate tasks and decision making with autonomy, especially for consumer-facing tools and services, AI agents will need access to data and systems. In fact, much like human assistants, AI agents may be at their most valuable when they are able to assist with tasks that involve highly sensitive data (e.g., managing a person’s email, calendar, or financial portfolio, or assisting with healthcare decision-making).
As a result, many of the same risks relating to consequential decision-making and LLMs (or to machine learning generally) are likely to be present in the context of agents with greater autonomy and access to data. For example, like some LLMs, some AI agents transmit data to the cloud due to the computing requirements of the most powerful models, which may expose the data to unauthorized third parties (e.g., the recent Data protection impact assessment on the processing of personal data with Microsoft 365 Copilot for Education). As with chatbots that use LLMs, AI agents with anthropomorphic qualities may be able to steer individuals towards or away from conducting certain actions against the user’s best interest. Other examples of cross-cutting data protection issues include challenges related to having a lawful basis for model training, operationalizing data subject rights, and ensuring adequate explainability. These legal and policy issues for LLMs, which are the subject of ongoing debate and legal guidance, are only heightened in the context of agentic systems with enhanced capabilities.
In addition, more recent AI agents may present some novel privacy implications or exacerbate data protection issues that go beyond those associated with LLMs.
Data collection and disclosure considerations: The latest AI agents may need to capture data about a person and their environment, including sensitive information, in order to power different use cases. As with LLMs, the collection of personal data by agents will often trigger the need for having a lawful ground in place for such processing. When the personal data collected is sensitive, additional requirements for lawfully processing them often apply too. While current LLM-based systems may train and operate using personal data, they lack the tools (e.g., application programming interfaces, data stores, and extensions) to access external systems and data. The latest AI agents may be equipped with these tools, which could enable them to obtain real-time information about individuals. For example, some agents may take screenshots of a user’s browser window in order to populate a virtual shopping cart, from which intimate details about a person’s life could be inferred. As the number of individuals using AI agents and its use cases grow, so too could AI agents’ access to personal data. For example, AI agents may collect many types of granular telemetry data as part of their operations (e.g., user interaction data, action logs, and performance metrics). Increasingly complex agents may collect large quantities of telemetry information, which may qualify as personal data under data privacy legal regimes.
Security vulnerabilities: Advanced AI agents’ design features and characteristics may make them susceptible to new kinds of security threats. Adversarial attacks on LLMs, such as the use of prompt injection attacks to get these models to reveal sensitive information (e.g., credit card information), can impact AI agents too. Besides causing an agent to reveal sensitive information without permission, prompt injection attacks can also override the system developer’s safety instructions. While prompt injection is not a threat unique to the latest AI agents, new kinds of injection attacks could take advantage of the way agents work to perpetuate harm, such as installing malware or redirecting them to deceptive websites.
Accuracy of outputs: Hallucinations, compounding errors, and unpredictable behavior may impact the accuracy of an agents’ outputs. LLM hallucinations—the making up of factually untrue information that looks correct—may affect the accuracy of an agent’s outputs. These hallucinations are closely tied to the “temperature” parameter that controls randomness in the model’s attention mechanism: higher temperatures increase creativity and the risk of hallucinations, while lower temperatures reduce hallucinations but may limit the agent’s adaptability. However, errors that affect agent outputs may have different implications for individuals, such as misrepresenting a user’s characteristics and preferences when it fills out a consequential form. In addition to hallucinations, the latest AI agents may experience compounding errors, which could occur while the systems perform a sequence of actions to complete a task (e.g., managing a customer’s account). Compounding errors is the phenomenon where the agent’s accuracy decreases the more steps a task takes. For example, an AI agent creating a travel experience may experience an error while making a one-day hotel booking, which cascades into misaligned restaurant reservations and museum tickets. This holds true even when the model’s accuracy is high. Some AI agents may act in unpredictable ways due to dynamic operational environments and agents’ non-deterministic nature—producing probabilistic outcomes, adapting to new situations, learning from data, and exhibiting complex decision-making—leading to malfunctions that affect output accuracy. These accuracy issues may be challenging to redress through risk management testing and assessments and exacerbated when different AI agents interact with each other.
Barriers to “alignment”: Some AI agents may pursue tasks in ways that conflict with human interests and values, including data protection considerations. AI alignment refers to designing AI models and systems to pursue a designer’s goals, such as prioritizing human well-being and conforming to ethical values. Misalignment problems are not new to AI, but continued technological advances with agents may make it challenging for organizations to achieve alignment through safeguards and safety testing. LLMs can fake alignment by strategically mimicking training objectives to avoid undergoing behavioral modifications. These challenges have data protection implications for the latest AI agents. For example, an agent may decide that it needs to access or share sensitive personal data in order to complete a task. Such behavior could implicate an individual’s data protection interest in having control over their data when personal data is processed during deployment. Practitioners must be mindful of the need for safeguards to constrain this behavior, although research into model alignment has focused more on safety issues rather than privacy.
Explainability and human oversight challenges: Explainability barriers arise when users cannot understand an agent’s decisions, even if these decisions are correct. Users and developers may encounter difficulties in understanding how some AI agents reach decisions due to their complex processes. The black box problem, or the challenge of understanding how an AI model or system makes decisions, is not unique to agents. However, the speed and complexity of AI agents’ decision-making processes may create heightened roadblocks to realizing meaningful explainability and human oversight. AI agents utilizing language models can provide some of their reasoning in natural language, but these “chain-of-thought” insights are becoming more complicated and are not always indicative of the agent’s actual reasoning. These challenges may make it more difficult to reliably interrogate agents’ decision-making processes and manage risks.
Looking Ahead
Recent advances in AI agents could expand the utility of these technologies across the private and public sectors, but they also raise many data protection considerations. While practitioners may be aware of some of these considerations due to the relationship between LLMs and the latest AI agents, the unique design elements and characteristics of these agents may exacerbate or raise new compliance challenges. For example, an agent may manage privacy settings (e.g., accepting cookies so that it can continue working on a task) as part of its operations, although companies can establish safeguards to address this risk. In closing, practitioners should remain abreast of technological advances that expand AI agents’ capabilities, use cases, and contexts where they can operate, as these may raise novel data protection issues.