The EU Commission’s Approach to Age Verification: Mobile Apps, DSA Enforcement, and Challenging National Social Media Bans

On 29 April 2026, the European Commission published its Recommendation for a common approach for EU-wide age verification technologies, a non-binding policy document with the aim of harmonizing future measures for the protection of children online. 

This blog post outlines the Commission’s emerging strategic approach to the implementation of EU-wide age verification measures, provides an analysis of the legal framework envisioned for their deployment, and includes notes on the Commission’s thinking with regard to possible social media bans in individual Member States. A number of key takeaways emerge:

• In response to growing tensions surrounding the possibility of social media bans in a number of EU countries, the Commission is accelerating its attempts to enable the roll-out of age verification solutions, urging Member States to implement these by 31 December 2026;
• An analysis of the applicable legal framework, and primarily the Digital Services Act (DSA), shows that since none of its Articles include specific mention of minimum age requirements or of age verification measures, it is still unclear whether age verification solutions will be voluntary or mandatory – it is worth noting here, however, that this does not mean that age assurance methods should not be implemented, as shown by emerging DSA enforcement on the topic;  
• While the Commission’s 2025 Guidelines on the protection of minors under the DSA focus on a variety of age assurance methods, this Recommendation aims to advance the EU’s strategic approach to age verification in particular, contributing to a growing global trend focused on age verification for service access or limitations; 
• The Commission aims to develop an EU age verification blueprint – a publicly available technical specification comprising the architecture, protocols, and interfaces to be used by Member States and providers to roll out national age verification measures;
• An EU age verification scheme will also be developed by the Commission to establish the framework for “proof of age attestations,” including a list of trusted EU-based providers for such attestations; 
• While significant references are made to privacy and to ensuring that age verification measures are “privacy-preserving,” there is no reference to the GDPR and little detail regarding the technical parameters that will be expected;
  • This gap is surprising, taking into account that DPAs have started to enforce GDPR in the context of age verification solutions in recent months, after the EDPB adopted a Statement on age assurance in February 2025. 

• Invoking Directive 2015/1535 on technical regulations and two CJEU cases from 1996 and 2000, the Commission aims to make it procedurally challenging for any individual EU Member State to implement a social media ban. 

1. Applicable legal framework – From the Digital Services Act to the (not-yet-published) Digital Fairness Act 

Article 28(1) DSA states that “providers of online platforms accessible to minors shall put in place appropriate and proportionate measures to ensure a high level of privacy, safety, and security of minors, on their service.” While the remainder of the Article covers advertising based on profiling and the further processing of personal data for the purpose of proving whether the user is a minor, it does not include mention of age verification measures. 

The Commission’s Recommendation, in paragraph 3, also makes reference to the July 2025 Guidelines for the protection of minors under the DSA, also issued by the Commission, which specifies general guidance on the application of age assurance measures. It is worth noting that, while in the 2025 DSA Guidelines the Commission focuses on self-declaration, age estimation, and age verification as tools to ensure the protection of minors online, the 2026 Recommendation aims to advance the EU’s strategic approach to age verification in particular, recognizing the higher degree of accuracy of the latter. 

The Recommendation additionally references Articles 34 and 35(1) of the Digital Markets Act (DMA) in which Very Large Online Platforms and Online Search Engines are required to “assess and mitigate actual or foreseeable risks that their service may pose to the protection of minors.” It also references Article 44(1)(j) DSA which enables the Commission to develop voluntary targeted standards to protect minors online, and recognizes that no such standards have been developed yet. 

The Audiovisual Media Services Directive, through which video-sharing platforms have an obligation to protect minors from accessing harmful audiovisual content, and the Unfair Commercial Practices Directive which recognizes minors as vulnerable users that must be protected, similarly form the basis of the applicable legal framework for age verification in the EU. Finally, the upcoming Digital Fairness Act is expected to fill any gaps left unaddressed, though the Recommendation does not specify which ones. 

Two notes are particularly relevant when considering the applicable legal framework:

• Mandatory or voluntary? – While the requirement to implement age verification tools is not explicitly included in any of the abovementioned laws as a legally binding obligation for digital services providers, both the Commission’s DSA Guidelines and the Recommendation may be taken into consideration by national Courts when interpreting existing, binding EU law.
  • Lessons from emerging enforcement under the DSA is, however, showing the inadequacy of age assurance methods currently being implemented for compliance which are, so far, largely based on self declaration and age estimation (rather than age verification) – for example, the Commission preliminarily finds (April 2026) Meta in breach of the DSA for failing to prevent minors under 13 from accessing Facebook and Instagram; and the Commission opens an investigation into Snapchat (March 2026) for not preventing users under 13 from accessing the app, and not adequately assessing whether users are under 17, which it deems necessary in order to ensure an age-appropriate experience.
• Enforcement also shows inconsistencies in EU harmonization regarding the age of a minor – While there is no consistent and agreed upon age of the child under EU law, the Recommendation defines a “minor” as anyone under the age of 18 – however, across individual Member States the age of the minor can range from 13 to 18.
  • Under the GDPR, which is not referenced by the Recommendation, the processing of personal data of a child in relation to the offer of information society services directly to them is lawful where that child is at least 16 years old (Article 8(1)), though Member State law may provide for a lower age (which must not be under 13). 
  • Since Member States have discretion in defining the age of a minor within their national territory, “EU-wide” age verification measures may become fragmented depending on this definition. 

2. Age verification blueprint and age verification scheme 

When it comes to operationalizing EU-wide age verification tools, the Commission will develop a blueprint consisting of the technical specifications that such tools should follow and an open source implementation as a mobile app that can be customized to national contexts. This will be consistent with the EU Digital Identity Wallet, acting as an additional “age verification functionality”, which Member States are expected to operationalize by the end of 2026. It is worth noting that the EU Digital Identity Wallet is also voluntary for citizens and businesses, although Member States have the obligation to make the option available. 

The Commission will additionally develop an age verification scheme, with requirements for providers of proof of age attestations and age verification solutions to meet, and including a list of EU-based trusted providers of such attestations. The role of the attestation is to ensure conformity with the criteria of effectiveness of the age verification solution, namely accuracy, reliability, robustness, non-intrusiveness, and non-discrimination (these criteria are outlined in the Commission’s 2025 DSA Guidelines, mentioned above). 

Two notes are particularly relevant here:

• While the Recommendation does not include significant details regarding the proof of age attestations, its reference to conformity is reminiscent of the Conformity Assessment required under the EU AI Act, hinting at the further expansion of a product safety approach across the EU digital regulatory ecosystem; 
• The Recommendation specifically notes that the trusted providers of such attestations, which can be public or private entities, must be EU-based, recalling the Commission’s broader strategic goals in the area of EU digital sovereignty. 

From a global perspective, the Commission’s age verification scheme may be comparable to recent age assurance developments in other jurisdictions—such as the ongoing rulemaking efforts by the New York Attorney General’s Office to establish age assurance standards and accuracy benchmarking requirements under the SAFE for Kids Act, and Australia’s Age Assurance Technology Trial which assessed a variety of age assurance solutions and vendors but sought only to determine the feasibility of age assurance mechanisms from participating vendors rather than assess provider conformity with legal requirements. Notably, the Commission’s efforts seemingly go beyond both New York’s and Australia’s since it aims to establish requirements for conformity supplemented by a list of EU-vetted, trusted providers for use in legal compliance.

3. “Privacy-preserving” age verification?

Notable references are made throughout the Recommendation to the importance of privacy. Through this Recommendation, the Commission aims to facilitate the development of “harmonised, privacy-preserving, cybersecure, data protection compliant and robust EU age verification solutions.” Without reference to the GDPR, the Recommendation nonetheless relies on key data protection principles and requirements, interpreting “privacy-preserving” as preventing unnecessary data collection, unauthorized access or misuse of personal information.  

To be privacy-preserving, the age verification solution should, by default, limit the information shared to the relying party to a true or false response regarding the age of the individual, without providing any further information about them. Additionally, the Recommendation states that verification methods “should include technical safeguards to protect citizens from privacy and data protection risks, such as tracking of their online activity, including the use of zero knowledge proofs.” 

While there is no further elaboration of the expected technical safeguards or the privacy-enhancing technologies that could be deployed, it is likely that there will be significant interest in these attributes, particularly following the security flaws found in the EU “age checking app” launched by the Commission in early April. 

4. On social media bans: From political debate to procedural impossibility 

The Commission’s Recommendation is timely in that it comes as some individual EU Member States, such as France (for under 15s), Spain (for under 16s), and Germany (for under 14s, with stricter rules for under 17s), consider social media bans. 

With a view to harmonization and the prevention of barriers within the internal market, the Recommendation invokes an administrative requirement found in Directive 2015/1535 laying down a procedure for the provision of information in the field of technical regulations and of rules on Information Society services. On this basis, where Member States consider introducing technical measures restricting minors’ access to online platforms, they have an obligation to report such measures to the Commission before they are adopted. This notification triggers a 3-month (extendable) standstill period during which the Member State is prevented from adopting the restriction, and a series of dialogues both with the Commission and with other Member States through the Digital Services Expert Group. Digital Services Coordinators, on the basis of the DSA, can also bring the issue for consideration to the European Board for Digital Services, a forum for cooperation for ensuring the coherent enforcement of the DSA.

Should a Member State fail to notify the Commission of the draft technical measure they are considering for restricting minors’ access to online platforms, it would be considered “a procedural defect that renders the measure unenforceable against individuals in national court proceedings”, and would be inapplicable to individuals. The Recommendation cites CJEU Case C-194/94, CIA-Security and Case C-443/98, Unilever in its reasoning. Furthermore, the Commission could initiate proceedings against a Member State should the proposed national measures regarding restricting minors’ access to online platforms be found to be incompatible with the DSA. 

As regulators globally continue to navigate the intensifying youth online safety space, the Commission’s Recommendation adds another thread to the global patchwork of proposals aimed at restricting or banning social media access for minors. Several countries outside the EU are considering bans for minors, such as Australia and Indonesia which both recently started implementing social media bans (for under 16s), or targeted restrictions on social media access, such as in Brazil (which requires that accounts of minors under 16 are linked to a parent account in the recently effective Digital ECA) and the US (where legislation is pending that would ban minors under 13 from holding accounts and restrict use of certain platform features within teen accounts).

5. Concluding Notes 

It is still uncertain how the age verification landscape will develop across the EU. As enforcement shows that the currently implemented lower-accuracy age assurance measures are increasingly deemed incompatible with the DSA, and political pressure grows within and across Member States to more adequately protect minors online, the Commission is attempting to set the tone for a harmonized approach.

 While the Recommendation is a non-binding, soft law instrument, it shows the Commission’s strategic direction and positioning regarding age verification measures. Nevertheless, specific details regarding the technical specifications, protocols, interface, the interoperable and privacy-preserving features of such tools, as well as how (and when) each individual Member State will operationalize them, remain open questions. 

Taking stock: The Impact of the India AI Impact Summit 2026

India’s hosting of the AI Impact Summit 2026 was an ambitious undertaking. With 600,000 attendees and 92 signatories to the New Delhi Declaration, the Summit was a showcase of a Global South country taking a leading role in shaping the AI governance agenda. The Summit’s official framing centered on infrastructure, compute, and equitable access to AI. What emerged across the week, and across FPF’s engagements in New Delhi before and during the Summit, was a global AI governance conversation defined by the tension between ambitious multilateral declarations and the slower, harder work of building the institutions and tools needed to make them real.

Now that the dust has settled, this blog post takes stock of the impact the Summit has had on the global AI governance conversation, drawing takeaways from FPF’s participation in events across Pre-Summit and the Summit itself. The threads that emerged from our engagements with the programming in New Delhi and now continue to manifest in various ways are: (1) the growing role of sandboxes as governance infrastructure; (2) whether global AI policy conversations can hold together in the face of geopolitical divergence; and (3) the sharpening focus on children’s safety and agentic AI as specific governance challenges that are moving faster than the frameworks designed to address them.

Theme 1: For AI governance to scale, it needs the right testing environments, and sandboxes are emerging as an answer

FPF participated in two events tied to India’s AI Impact Summit 2026, both co-organized with Nasscom. On 20 January 2026, FPF and Nasscom co-hosted a Pre-Summit Event in New Delhi titled “Building Safe Spaces for AI Impact: Regulatory and Private Sandboxes,” bringing together senior government leaders, regulators, global industry representatives, and policy experts. From 16–21 February 2026, Jules Polonetsky, CEO of FPF, Josh Lee Kok Thong, Managing Director for APAC, and Bilal Mohamed, Policy Manager for India, represented FPF at the Summit itself, co-organizing a high-level panel with Nasscom, hosting an FPF Salon Dinner on 17 February, and participating in bilateral engagements throughout the week.

The FPF delegation at the India AI Impact Summit 2026. From L-R: Josh Lee Kok Thong, Managing Director (APAC); Jules Polonetsky, CEO; Bilal Mohamed, Policy Manager for India Photo credit: Josh Lee

One of the clearest messages from the Pre-Summit Event was that the global AI governance conversation has moved decisively beyond the question of what principles should govern AI toward the more difficult question of how to build the regulatory infrastructure needed to put those principles into practice. Sandboxes (whether in their regulatory and private organizational forms), are emerging as one possible lever to achieving this.

The Pre-Summit Event’s first panel, moderated by Josh, brought together regulators from India, Singapore, and Brazil alongside industry experts to examine the evolution of regulatory sandboxing. Two key insights emerged:

• First, sandboxes have seen global uptake as a mechanism for translating governance principles into practice. Over 200 regulatory sandboxes are now in operation globally, 70 of which are focused on AI. More importantly, their function is changing. Where early sandboxes primarily granted permission for testing, well-designed sandboxes today generate the real-world evidence regulators need to write better-calibrated rules. Singapore’s Infocomm Media Development Authority (IMDA) has pioneered a phased methodology moving from case studies to guidelines to formal standards, offering a model of prospective enforcement grounded in observed technical reality.
  
• Second, sandboxes are becoming interoperable by necessity. AI-driven products cut across sectors in ways that engage multiple regulators simultaneously. The Reserve Bank of India’s Interoperable Regulatory Sandbox mechanism, introduced in 2022, was designed to test products that trigger obligations across jurisdictional lines. Similarly, Brazil’s Agencia Nacional de Proteção de Dados (ANPD) deliberately involves other regulators, technical experts, and civil society from the outset, recognizing that the questions sandboxes address are rarely confined to a single institution’s mandate.

The second panel examined how organizations are building private sandboxes for AI governance. The discussion, featuring representatives from Coforge, PayPal, Salesforce, Palo Alto Networks, and European Data Protection Supervisor (EDPS) AI Unit, highlighted two practical insights:

• First, private sandboxes help organizations build trust with both consumers and regulators. Sudheer described Salesforce’s “Customer Zero” approach: before any AI product reaches customers, it is deployed internally across Salesforce’s 80,000-person workforce. The Salesforce philosophy of “build it, use it, fix it, scale it, and then sell it” surfaces real-world failures that may be limited by laboratory testing and allows governance guardrails to be refined before external rollout. Sam described how Palo Alto Networks used isolated “dirty lab” environments to subject models to curated malicious prompts, simulating prompt injection, data leakage, and adversarial manipulation, to establish a behavioural baseline before deployment. For companies navigating frameworks like India’s Digital Personal Data Protection Act, 2023 (DPDP Act), internal sandboxes serve as a signal of due diligence to regulators, demonstrating structured processes throughout the product lifecycle.
• Second, unlike generative AI systems (whose failure modes are at least probabilistically characterized), agentic systems take autonomous actions, which means sandboxing must simulate intent rather than just behavior. More broadly, governance frameworks must be built to outlast the specific technologies they regulate. As Christian Lau of Dynamo AI described during the first panel, organizations must “separate the governance layer from the tech layer,” building accountability mechanisms that remain intact as models evolve.

Theme 2: Geopolitical divergence is exposing the limits of international AI governance

As the first Global South host of the AI Summits, India played an important bridging role, keeping the focus on how AI can drive economic development across Africa, South America, and Asia. The adoption of the New Delhi Declaration, signed by 92 countries and international organizations – including the US, China, and G7 nations – reflected genuine multilateral ambition, even as its voluntary and non-binding character also revealed the limits of that ambition.

The Summit provided a platform for different philosophies on AI governance and oversight to be articulated, with geopolitics in the backdrop. Michael Kratsios, Director of the White House Office of Science and Technology Policy, argued that AI policy must remain national and local, and that international fora risk creating centralized oversight that could stifle innovation under the guise of safety. Implementing this vision, the US outlined a set of parallel initiatives: an American AI Exports Program, new development finance instruments, a Tech Corps initiative embedding US technical experts with partner governments, and an AI Agent Standards Initiative through the Department of Commerce.

On the other hand, the President of France, Emannuel Macron, who hosted the previous edition of the AI Summit in Paris, promoted the EU AI Act in his speech as evidence that responsible and competitive AI are not in opposition, and argued for an approach that treats oversight as foundational to AI development rather than an obstacle to it. 

India, as host, articulated its own approach. During the fireside chat concluding the Pre-Summit Event, S. Krishnan, Secretary, Ministry of Electronics and Information Technology (MeitY), outlined a philosophy of regulation “only when necessary,” explaining that India’s constitutional framework allows sectoral regulators such as Securities and Exchange Board of India (SEBI) and the Royal Bank of India (RBI) to oversee AI within their respective domains, rather than relying on a single, prescriptive national law. This middle path eyed by India relies heavily on the kind of regulatory infrastructure discussed in Theme 1.

FPF’s Managing Director for APAC Josh Lee Kok Thong engaging MeitY Secretary S. Krishnan during the fireside chat at the FPF-Nasscom Pre-Summit Event. Photo credit: Nasscom

FPF’s own Summit panel, titled “From Policy to Practice: Governing AI for Global Impact“, co-organized with Nasscom and moderated by Ashish Aggarwal (Nasscom), brought this tension into sharper relief. The panel featured Carina Prunkl (INRIA), Jules Polonetsky (FPF), Gail Kent (Google), Ivana Bartoletti (Wipro), and Wifredo Fernandez (xAI). Three insights from the discussion stood out.

First, it was highlighted that a critical question for the adoption of responsible AI practices is whether emerging baselines are clear and accessible enough to prevent a race to the bottom on safety. As Jules Polonetsky noted, weak or expensive compliance infrastructure creates competitive pressure to cut corners, a particular risk for startups and smaller players. 

Second, governance frameworks must be built for specific contexts rather than transplanted from elsewhere. As Gail Kent noted, Indian users rely heavily on voice, video, and image-based inputs rather than text, which fundamentally changes the safety and privacy challenges that need local attention. Third, as Ivana Bartoletti argued, India’s “techno-legal” approach positions it to be an architect of governance solutions rather than a recipient of frameworks designed elsewhere.

These observations point to something important that focusing on divergent regulatory philosophies can obscure. The real risk in global AI governance may lie less in countries choosing different regulatory models, and more in those models being either ineffective overall or inaccessible to smaller actors that a shared floor on safety ceases to exist. 

A packed full house at FPF’s and Nasscom’s official session at the India AI Impact Summit.
Photo credit: Josh Lee

Theme 3: There is a cross-border consensus to regulate for children’s safety, but approaches vary

Despite differences in AI regulatory philosophies exposed during the Summit, child safety emerged as a point of cross-border consensus. Prime Minister of India, Narendra Modi, called for AI to be child-safe and family-guided, and for mandatory authenticity labels on AI-generated content. President Macron urged India to join a coalition restricting social media access for children.

Prime Minister Modi’s remarks were also grounded in a domestic regulatory development that had unfolded days before the Summit. On 10 February 2026, MeitY notified the IT (Intermediary Guidelines and Digital Media Ethics Code) Amendment Rules, 2026, introducing India’s first formal framework for synthetically generated content. The amendments require intermediaries to label AI-generated content, block the creation and dissemination of child sexual abuse material and non-consensual intimate imagery, and comply with a three-hour takedown window for prohibited content. 

In India, the momentum has not been limited to the federal government. On 6 March 2026, the state government of Karnataka announced in its 2026–27 State Budget a proposed ban on social media use for children under 16, citing concerns over digital addiction, mental health, and declining academic performance. On the same day, the Chief Minister of Andhra Pradesh, Chandrababu Naidu, announced that the state would implement a ban on social media for children under 13 within 90 days. At the federal level, the DPDP Act already requires parental consent for the processing of personal data of children below the age of 18.

India’s actions sit within a broader global trend. In July 2025, the EU adopted guidelines on the protection of minors under the DSA; Australia implemented a social media age ban for under-16s in December 2025; and Singapore’s IMDA introduced age assurance requirements for app stores. In the weeks since the Summit, that response has accelerated. The White House’s National Policy Framework for AI placed children’s safety at the center of its legislative recommendations. Dozens of chatbot safety bills are under consideration in state legislatures across the US, and the US Congress. In the UK, Prime Minister Keir Starmer announced that AI chatbots will be brought under the Online Safety Act. The World Economic Forum’s Global Risks Report 2026 ranked online harms among the top risks of the next decade. 

Taken together, this activity signals that child safety in the age of AI has become the rare governance issue that commands cross-jurisdictional political consensus, even as the jurisdictions diverge on almost every other dimension of AI oversight. The harder question is whether frameworks across jurisdictions, which share the same underlying concerns but differ in their approaches to age assurance, parental consent, and platform liability, can converge enough to hold platforms to consistent and effective standards. It is a question that India, with its large minor population and newly enacted synthetic media rules, has a significant stake in helping to answer.

Conclusion

The vivid debates at the Summit showed that AI governance approaches will be shaped by the economic, political, and legal contexts in which different nations operate. The real question is whether enough common ground can be built to prevent a race to the bottom on safety and responsible AI, as was highlighted by the FPF-Nasscom panel.

India’s hosting of the Summit was an important signal that this work is genuinely global in its participants and ambitions. The governance gaps that came into focus in New Delhi, from agentic AI accountability to the protection of children in AI-mediated spaces, to the question of whether voluntary multilateral declarations can be turned into durable commitments, represent the agenda for the conversations ahead.

Red Lines under the EU AI Act: Restricting Real-time Remote Biometric Identification Systems for Law Enforcement Purposes

Blog 8 | Red Lines under the EU AI Act Series

This blog is the eighth of a series that explores prohibited AI practices under the EU AI Act and their interplay with existing EU law. You can find the whole series here.

1. Introduction

The eighth blog in the “Red lines under the EU AI Act” series examines the general prohibition on the use of real-time biometric (RBI) systems in publicly accessible spaces for law enforcement purposes imposed by Article 5(1)(h) of the EU AI Act, the three narrow exceptions to the prohibition permitted for Member States to utilize, and how these obligations fit in the broader context of real-time biometric identification in the EU. 

There are a few key takeaways from our analysis of this provision:

• The prohibition on the use of RBI systems in public spaces is narrowly tailored. All of the factors must be present for the prohibition to be triggered, otherwise the collection and use of biometric information is categorized as related to “high-risk” AI systems . 
• RBI systems can create a risk to the rights and freedoms of individuals simply by being deployed. The European Commission Guidelines and the AI Act Recitals both emphasize the risk of a “chilling effect” on the exercise of public freedoms that can come from a perception of ubiquitous surveillance. 
• The Guidelines and the AI Act itself make a significant effort to distinguish banned “remote biometric identification” from permitted uses of biometric identification, such as device-level identity verification.
• Mileage may vary – because the offenses for which an exception to the RBI prohibition may be sought are defined in Member State criminal law, actual implementation of the prohibition and its exceptions may diverge significantly in implementation.  

With these key takeaways in mind, Section 2 of this blog examines the reasoning behind the prohibition on RBI, while Section 3 explores the specific elements that all must be triggered to bring processing activity within the provision’s scope. Section 4 outlines the important but limited exceptions to the prohibition, while Section 5 examines how this provision interacts with other relevant areas of EU law, such as Article 9 of the General Data Protection Regulation (GDPR). Section 6 includes closing thoughts and takeaways along with a brief examination of salient activity by DPAs. 

2. Why the prohibition? Specific risks associated with RBI for law enforcement purposes

As noted earlier in this blog series, the creation and use of large scale biometric identification systems has long been an area of serious concern for EU authorities. This is particularly acute in the context of such systems’ deployment for law enforcement purposes; the Guidelines recognize the potential impact on the rights and freedoms of individuals widespread deployment of these technologies represents. The Guidelines further identify the “feeling of constant surveillance” the deployment of RBI systems in public spaces may elicit risks “indirectly dissuad[ing] the exercise of freedom of assembly and other fundamental rights,” and technical failures in AI systems may also produce discriminatory effects based on sensitive personal characteristics such as age, ethnicity, race, sex, or disability status. 

3. Verification vs. Identification: what systems are captured by the RBI prohibition?

The Guidelines walk through a number of questions that must be examined in order to understand whether a given system falls within the prohibition’s scope: 

• Does this system qualify as “remote biometric identification”? 
• Is the system “real time”? 
• Is the space “publicly accessible”? 
• Is the system used for law enforcement purposes? 

It is critical to note that all of these criteria must be present for a system to be affected by the ban set forth in Article 5. 

Article 3(41) of the AI Act defines an RBI system as an “AI system for the purpose of identifying natural persons, without their active involvement, typically at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database.”

Whether a system qualifies as an RBI system depends on: 

• Whether the system captures “biometric information” 
• Whether the system is “remote”
• Whether the system is used for identification 

The Act and Guidelines consider biometric data to be machine-readable representations of individuals’ measurable physical characteristics – for example, eye distance and size, or nose length – or behavioral characteristics, such as gait or voice print. This is broader than the definition of biometric information provided in Article 4(14) of the GDPR, which defines biometric data as information arising from specific technical processing of physical, physiological, or behavioral characteristics of a natural person in such a way that would permit the unique identification of that person. This last part of the GDPR definition of biometric data (“unique identification”) is absent from the AI Act concept, as further analyzed in Blog 6 and Blog 7 of this series. However, “identification” plays a key part in defining RBI systems.

The function of such a system at a distance and an individual choice to interact with it (or possibly even knowledge of its existence) are at the core of whether a system qualifies as remote. “Identification” is critical in that it is distinguished from “verification”: establishing the identity of a natural person by comparing biometric data of that individual to biometric data of individuals stored in a database as opposed to verifying a specific person is who they claim to be via matching sensor data to an on-device record.  

Per Recital 17 of the AI Act, a system operates in “real time” if it captures and processes biometric data “instantaneously, near-instantaneously or in any event without significant delay.” This determination is a fact-based inquiry, ensuring that an artificial, “minor” delay cannot be incorporated in order to allow a prohibited system to be deployed. The Commission also notes that the same device may well be capable of “real-time” and “post-identification” functions – the prohibition’s application is technology-agnostic.

“Publicly accessible space” is defined in Article 3(44) of the AI Act as “any publicly or privately owned physical space accessible to an undetermined number of natural persons, regardless of whether certain conditions for access may apply, and regardless of the potential capacity restrictions.” The Act and Guidelines emphasize this status is also a fact-based inquiry and cannot be evaded by mere signage or official designation; this component of the prohibition is clearly tied to the potential risk posed by RBI deployments to the exercise of fundamental political freedoms such as the freedom to assemble.

Finally, Article 3(46) of the AI Act defines “law enforcement purpose” as those “activities carried out by law enforcement authorities or on their behalf for the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, including safeguarding against and preventing threats to public security.” This definition is consistent with the Data Protection in Law Enforcement Directive (LED). The Commission is careful to note in the Guidelines that non-Law Enforcement entities acting on their own behalf to detect crime would not fall afoul of the prohibition, but rather need comply with the Article 6 governance of “high-risk” AI systems. 

4. When is RBI processing for law enforcement permitted?

Recital 33 of the AI Act emphasizes that any exceptions to the prohibition on using RBI systems for law enforcement purposes must be “exhaustively listed and narrowly defined situations.” There are three set out in Article 5(1)(h)(i)-(iii) of the AI Act:

(i) the targeted search for specific victims of abduction, trafficking in human beings or sexual exploitation of human beings, as well as the search for missing persons;

(ii) the prevention of a specific, substantial and imminent threat to the life or physical safety of natural persons or a genuine and present or genuine and foreseeable threat of a terrorist attack;

(iii) the localisation or identification of a person suspected of having committed a criminal offence, for the purpose of conducting a criminal investigation or prosecution or executing a criminal penalty for offences referred to in Annex II and punishable in the Member State concerned by a custodial sentence or a detention order for a maximum period of at least four years.

Article 5(2)-(7) of the AI Act provides additional limitations on the exceptions, expanded on in Section 10 of the Guidelines. Key limitations include:

• ‘Single target’ – RBI systems can only be deployed for the purpose of confirming the identity of a specifically targeted individual (except for the circumstances involving a genuine and present or foreseeable terrorist attack); 
• Seriousness – assessment of the possible harm and consequences against the interference with fundamental rights, and inclusion of the offense in Annex II of the AI Act; 
• Scale – the number and category of persons affected by interference; 
• Probability –  likelihood that negative event will occur;
• Geographic restriction – where the system will be deployed or the event may occur;
• Personal scope – defining the categories of persons concerned with the deployment; 
• Time limit – duration of deployment must be limited to what is strictly necessary.

Each enumerated exception fulfills a public objective – and is consistent with the overall philosophy of both the AI Act and GDPR of balancing the inherent interest of individuals in the exercise of fundamental rights against the risk of significant harm to the public in specific, factual scenarios. The exceptions to the RBI prohibition also represent an area of deference to the Member States, as they do not function automatically and must be authorized by Member State national laws. As a result, all Member States may not permit precisely the same types of RBI system usage in law enforcement contexts.

5. LED, GDPR and additional safeguards – how does the prohibition interact with other laws? 

A significant element of the RBI prohibition is that the prohibited activity is explicitly tied to RBI systems deployed for law enforcement purposes – and law enforcement authorities themselves are, per Article 2(2)(d) of the GDPR, excluded from that scope of that regulation. Instead, national laws implemented by EU Member States to operationalize the LED are the pre-existing restriction on the use of RBI technologies for law enforcement. The Guidelines do specifically observe that, where Member States have made missing persons inquiries an administrative matter and not a criminal one, the Article 5 RBI prohibition would not qualify and the use of RBI systems in such searches would be governed by the GDPR instead.

The use of RBI systems for law enforcement pursuant to a relevant exception is permitted only if the law enforcement authority has completed a fundamental rights impact assessment as provided for in Article 27 of the AI Act (which imposes the obligation to conduct Fundamental Rights Impact Assessments (FRIA) in relation to high-risk AI systems) and has registered the system in the EU database according to Article 49 of the AI Act. A FRIA must generally be completed before an RBI system is deployed – it cannot be created as an after-the-fact rationale for a pre-determined deployment. The Guidelines note that provisions relating to FRIAs apply only to the Article 5 prohibition on RBIs and not to FRIAs required in connection with high-risk AI systems generally, which will also be informed by a future, still-forthcoming guidance document and template for FRIAs, currently expected this year. The Guidelines also highlight that the FRIA requirement does not replace any existing Data Protection Impact Assessment (DPIA) requirement that may be required under provisions of the LED, GDPR, or the Data Protection Regulation of the EU institutions and bodies (EUDPR), depending on the specific system in question. 

The Guidelines attempt to differentiate between a DPIA, which focuses on the risks to rights and freedoms stemming from the processing of individuals’ personal data specifically, and a FRIA, which is a “more general” assessment of how an AI system could impact fundamental rights. The Commission offers additional detail on each of the categories of information a FRIA must contain, which include: 

• A description of the RBI use and the deployer’s processes for the use, together with the intended purpose of use; 
• The period of use and frequency of use;
• The categories of persons and groups affected by the system;
• The specific risks of harm to the affected persons;
• Human oversight measures; and
• Risk mitigation measures.

Article 5(3) of the AI Act imposes a key further limitation on Member States who wish to deploy RBI systems – each individual use of the system must receive prior authorization from either a judicial or independent administrative authority, and automated decision-making producing an adverse legal effect cannot be based solely on a system’s output. This prior authorization requirement has an extremely limited exception for emergency situations where it is “effectively and objectively impossible to obtain an authorization before commencing use” of the RBI system, and in such circumstances that authorization must still be requested within 24 hours of the use of a system. The Commission makes clear that the “double assessment” requirement of both the FRIA and the prior-use “necessity and proportionality” authorization is an intended consequence of the Act. Member States are also provided guidance on the necessity of deleting any data gathered under a use of the “emergency” authorization exception. 

Whether a decision with adverse legal effect is produced solely based on an RBI system’s output, is linked to the human oversight requirements set out in Article 14 of the AI Act. The Commission emphasizes that even with prior authorization, an RBI system may not be deployed where its outputs would produce adverse legal effects (for example, arrest and imprisonment solely on the basis of an individual’s identification by an RBI system, without further checks). Specifically, two natural persons with the necessary competence, training, and authority must separately verify and confirm identification by an RBI system before action is taken on the basis of that identification. Furthermore, each use of an RBI system must be notified to both the market surveillance authority and the national data protection authority. 

6. Relevant Enforcement and Key Takeaways

Pre-AI Act data protection enforcement activity relating to law enforcement use of real-time RBI systems in public spaces has been limited. So far, topically related enforcement has exclusively been directed at private-sector biometric identification activity, notably in the constellation of cases connected to the activities of Clearview AI. Of particular note (and discussed further in Blog 4 and Blog 5 of this series) are enforcement actions by the Dutch DPA rejecting an alleged third-party interest in combating crime as a valid lawful basis for processing biometric data, and by Italy’s Garante finding violations of core data protection principles related to fairness and transparency, both resulting in large fines.  

The requirement for Member State implementation may still cause significant divergence in practice

Because each member state must draft a separate law specifying which of the three exception categories it opts into, which crimes from Annex II it authorizes, and which authority grants case-by-case approval, there is significant effort required before a single deployment can lawfully occur, and because there is no Europe-wide shared definition of serious criminal offenses, operational consequences may vary.

Forthcoming guidelines will be critical to understanding the operational environment

Due to the required “double assessment” structure for deploying RBI systems pursuant to one of the objections, assuming the Member State legal authorization and review process is satisfied, potential deployers will still need to complete the required Fundamental Rights Impact Assessment for any lawful deployment of an RBI system to commence – and the completion of that step will hinge on a template and guidance document that the Commission has not yet published.

Limits are a feature, not a bug

Taken together, the limited exceptions to the RBI prohibition and detailed, overlapping requirements for their use are clearly designed to create an extremely limited environment for authorizing the deployment of RBI systems, subject to significant oversight by actors outside of their operational environments, given the systems’ potential to impact the fundamental rights and freedoms of individuals. This follows the logic of Article 10 of the LED, which permits processing biometric data for uniquely identifying a natural person only where such is strictly necessary and authorized by Member State law.

Red Lines under the EU AI Act: Understanding the prohibition of biometric categorization for certain sensitive characteristics 

Blog 7 | Red Lines under the EU AI Act Series

This blog is the seventh of a series that explores prohibited AI practices under the EU AI Act and their interplay with existing EU law. You can find the whole series here.

The EU AI Act provides for rules on prohibited AI practices that the legislature considers incompatible with fundamental rights and European Union values. Article 5(1)(g) introduces a prohibition on the biometric categorization for “certain sensitive characteristics”, focusing on systems used to categorize individuals “based on their biometric data to deduce or infer their race, political opinions, trade union membership, religious or philosophical beliefs, sex-life or sexual orientation”. 

The European Commission guidelines on prohibited AI practices (hereinafter, “the Guidelines”) note that information, including sensitive data, can be extracted, deduced, or inferred from biometric data with or without the individual’s knowledge, leading to unfair or discriminatory treatment that undermines human dignity, privacy, and the principle of non-discrimination protected under the EU acquis. This provision also reflects longstanding concerns with regard to the risks associated with processing sensitive personal data, particularly where such processing may take place without the knowledge of the individual. 

With this in mind, Section 1 unpacks the (limited) scope and key definitions of the prohibition, including the cumulative conditions required for the provision to apply. Section 2 takes a look at the situations that fall outside the scope of the prohibition, and, finally, Section 3 explores the interaction between the biometric categorization prohibition and the existing EU legal framework. 

Several key takeaways emerge: 

• The AI Act prohibits specific biometric inference practices, not biometric categorization as such – Many forms of biometric categorization, such as categorization based on non-sensitive physical traits or for purposes that do not involve inferring the listed characteristics, do not fall within the prohibition.
• The objective and design of the system are central to determining whether the prohibition applies – The prohibition is not triggered only by the presence of biometric analysis, but by the intended inference of protected attributes from biometric data. 
• The relationship between this prohibition and EU data protection law needs further clarification – Given that the AI Act itself clarifies that it does not affect the application of the GDPR, and some processing of biometric data that may result in biometric categorization can be lawful under Article 9(2) GDPR when following its strict conditions, further clarification is needed with regard to the intersection of the two laws.  

1. (Limited) Scope and key definitions

To trigger the prohibition under Article 5(1)(g) AI Act, five cumulative conditions must be simultaneously met: 

1. The AI system must be placed on the market, put into service, or used.
2. The AI system must be a biometric categorization system. 
3. The AI system must categorize individuals.
4. The AI system must categorize individuals based on their biometric data.
5. The AI system must infer sensitive characteristics (e.g., race, political opinions, religious beliefs, and so on). 

The first condition, relating to the placing on the market, putting into service or use of an AI system, applies to both providers and deployers within their respective responsibilities. The Guidelines also clarify that the prohibition does not cover the labelling or filtering of lawfully acquired biometric datasets, including for law enforcement purposes.

The requirement that all five conditions be fulfilled simultaneously is likely to be significant in practice. It may limit the scope of the prohibition and it raises questions about how it will be applied in specific cases, particularly where systems are designed to avoid explicit inference of sensitive traits while still enabling similar outcomes.

1.1 Defining biometric categorization

Biometric categorization refers to assigning individuals to predefined groups based on their biometric data, rather than identifying or verifying their identity. Such categorization may be used, for example, to display targeted advertising or for statistical purposes, without necessarily identifying the individual. 

Article 3(40) AI Act defines a biometric categorization system as an AI system that assigns natural persons to specific categories based on their biometric data, unless this function is ancillary to another commercial service and strictly necessary for objective technical reasons. Biometric data, defined in Article 3(34) AI Act, includes behavioural characteristics based on biometric features. As discussed in a previous blog, this definition is broader than the definition of biometric data in the GDPR. Categorization based on clothing, accessories, or social media activity falls outside the scope of biometric categorization under the AI Act.

The Guidelines further clarify that biometric categorization may involve categories based on physical characteristics such as facial structure or skin colour, some of which may correspond to sensitive characteristics protected under EU non-discrimination law. At the same time, the AI Act definition contains an important limitation: a system will not fall within the definition where the categorization is ancillary to another commercial service and strictly necessary for objective technical reasons. According to Recital 16 AI Act, an ancillary feature is one that is intrinsically linked to another commercial service and cannot be used independently of that service.

The Guidelines provide several examples to illustrate this distinction. For instance, filters that categorize facial or bodily features on online marketplaces, allowing consumers to preview a product on themselves, may constitute an ancillary feature because they are linked to the principal service of selling a product. Similarly, filters integrated into social media platforms that allow users to modify images or videos may also be considered ancillary features because they cannot be used independently of the platform’s content-sharing service.

The Guidelines also identify examples of systems that would fall within the prohibition. These include AI systems that analyse biometric data from photographs uploaded to social media platforms to categorize individuals by their assumed political orientation and send them targeted political messages. Another example concerns AI systems that analyse biometric data from photos to infer a person’s sexual orientation and use that information to serve targeted advertising. In both cases, the categorization would not be strictly necessary for objective technical reasons and therefore would fall within the definition of biometric categorization under the AI Act. Importantly, the systems that perform such categorization need to fall under the definition of “AI system” pursuant the AI Act for the prohibition to apply.

The risks associated with biometric categorization also reflect broader concerns under EU data protection law. The EDPB has clarified that inferences about sensitive characteristics may themselves constitute special categories of personal data under Article 9 GDPR. Also, the Court of Justice of the European Union has held that processing which allows information falling within Article 9(1) GDPR categories to be revealed must be regarded as processing of special categories of personal data (Meta Platforms and Others, C-252/21). However, the prohibition to process sensitive data under the GDPR has several exceptions, such as explicit consent. 

The EDPB and the European Data Protection Supervisor (EDPS) have taken a similar position in their Joint Opinion 5/2021 on the Proposal for the AI Act. They called for a broader prohibition of certain biometric AI practices. In particular, they called for a general ban on the use of AI for automated recognition of human features in publicly accessible spaces, including faces, gait, fingerprints, DNA, voice, and other biometric and behavioural signals.

1.2 For the prohibition to apply, categorization must take place at the level of the individual

Another essential condition for the prohibition to apply is that the system must categorize individual natural persons based on their biometric data. Importantly, the categorization must take place at the level of the individual. If biometric analysis is performed without categorizing specific individuals, the prohibition does not apply. For example, the prohibition would not be triggered where a system analyzes biometric information only to categorize an entire group without identifying or singling out individual persons. These include AI systems that conduct “attribute estimation”, sometimes referred to as demographic analysis, by assigning characteristics such as age, gender or ethnicity based on biometric features such as facial characteristics, height or skin, eye or hair colour, or other features such as a visible scar or distinctive tattoo.

1.3 “Sensitive characteristics” under the AI Act

The prohibition under Article 5(1)(g) AI Act applies only when a biometric categorization system is used to deduce or infer specific sensitive characteristics, such as: race, political opinions, trade union membership, religious or philosophical beliefs, sex life, or sexual orientation.

This means that not all biometric categorization systems fall within the scope of the prohibition. Rather, the prohibition targets systems that attempt to derive particularly sensitive characteristics from biometric data.

For example, a system that claims to infer an individual’s race from their voice would fall within the scope of the prohibition. By contrast, a system that categorizes individuals according to physical traits such as skin or eye colour, or a system analysing the DNA of crime victims to determine their origin, would not be prohibited under Article 5(1)(g). Another example provided by the Guidelines concerns a biometric categorization system that claims to infer a person’s religious orientation from tattoos or facial characteristics would fall within the prohibition.

2. Biometric categorization for bias detection: What falls outside the scope of the prohibition?

The prohibition in Article 5(1)(g) AI Act does not apply to all uses of biometric categorization. In particular, it does not cover AI systems used for the labelling or filtering of lawfully acquired biometric datasets, including in law enforcement contexts. As explained in Recital 30 AI Act, such uses may include sorting images by biometric characteristics, such as hair or eye colour.

The Guidelines note that labelling or filtering biometric datasets may be necessary to ensure that datasets used to train AI systems are representative across demographic groups. Where training data contains systematic differences between groups, for example, due to historical bias in data collection, algorithms may replicate those biases and potentially lead to discriminatory outcomes. In such cases, labelling data according to certain characteristics may be necessary to improve data quality and prevent discrimination. In some circumstances, the AI Act may even require such labelling operations in order to comply with the requirements applicable to high-risk AI systems (see Article 10 AI Act).

The Guidelines provide several examples of permissible uses. One example concerns the labelling of biometric data to prevent recruitment algorithms from disadvantaging individuals from certain ethnic groups, where historical training data reflects biased outcomes. Another example involves categorizing patients’ images by skin or eye colour, which may be relevant to medical diagnosis, including certain cancer diagnoses.

The exception also applies in law enforcement contexts where biometric datasets have been lawfully acquired. For example, law enforcement authorities may use AI systems to label or filter datasets suspected of containing child sexual abuse material. Such systems may help detect and redact sensitive information in images or assist investigations by labelling biometric features such as gender, age, eye or hair colour, scars, markings, or tattoos in order to identify victims or establish links between cases. Similarly, filtering and labelling features such as hand characteristics or distinctive tattoos may help identify possible suspects in law enforcement contexts.

3. Interplay with other EU laws

This prohibition must be understood in the context of the existing EU data protection framework. 

Interestingly to note, the Guidelines refer to an earlier explanation provided by the Article 29 Working Party (the precursor to the EDPB) when describing “biometric categorization” in the Opinion on developments in biometric technologies. Article 3(40) AI Act provides a legal definition, describing a biometric categorization system as an AI system that assigns natural persons to specific categories on the basis of their biometric data, while also specifying an exclusion where such categorization is ancillary to another commercial service and strictly necessary for objective technical reasons. 

By contrast, the Article 29 Working Party explains biometric categorization as the process of determining whether the biometric data of an individual belongs to a group with predefined characteristics, emphasizing that the objective is not to identify or verify the individual but to assign them automatically to a category, for example, to display different advertisements depending on the perceived age or gender of the person. While both definitions describe categorization based on biometric data rather than identification, the AI Act establishes a regulatory definition determining the scope of the prohibition, whereas the Article 29 Working Party description provides a conceptual explanation of how biometric categorization systems operate in practice.

Furthermore, Article 9(1) GDPR establishes a general prohibition on the processing of special categories of personal data, subject to exceptions, which might see some processing of biometric data in the context of biometric categorisation lawful under the GDPR, as long as it respects its strict provisions. The AI Act introduces an additional layer of restriction, which raises important conflict of law questions with the GDPR. As analyzed in the first blog of this series, the GDPR takes priority in application (the AI Act “shall not affect” the GDPR). Further guidance on the intersection of the GDPR and the AI Act in this respect is needed. 

The Guidelines clarify that AI systems intended to categorize individuals based on biometric data to infer attributes protected under Article 9(1) GDPR are classified as high-risk AI systems, provided they are not already prohibited under Article 5 AI Act. At the same time, Article 5(1)(g) further limits the possibilities for lawful processing of personal data under EU data protection law, including the GDPR, the Law Enforcement Directive (LED), and Regulation (EU) 2018/1725 (EUDPR). In particular, the provision excludes the use of biometric data to categorize natural persons in order to infer sensitive characteristics such as race, political opinions, trade union membership, religious or philosophical beliefs, sex life or sexual orientation, subject to the limited exception for the labelling or filtering of lawfully acquired biometric datasets.

The prohibition is also consistent with Article 11(3) LED, which explicitly prohibits profiling that results in discrimination on the basis of special categories of personal data, including race, ethnic origin, political opinions, religious beliefs or sexual orientation.

4. Closing reflections and key takeaways

The AI Act prohibits specific biometric inference practices, not biometric categorization as such

Article 5(1)(g) AI Act does not prohibit biometric categorization in general. It prohibits the placing on the market, putting into service, or use of AI systems that categorize individuals based on biometric data for the purpose of inferring certain sensitive characteristics, such as race, political opinions, religious beliefs, trade union membership, sex life or sexual orientation. The prohibition applies only where all cumulative conditions of Article 5(1)(g) are met. This means that many forms of biometric categorization such as categorization based on non-sensitive physical traits or for purposes that do not involve inferring the listed characteristics, do not fall within the prohibition.

The objective and design of the system are central to determining whether the prohibition applies

The Guidelines place significant emphasis on the purpose and functionality of the AI system, in particular, whether the system is designed to deduce or infer one of the sensitive characteristics listed in the provision. This means that the prohibition is not triggered only by the presence of biometric analysis, but by the intended inference of protected attributes from biometric data. The examples provided in the Guidelines illustrate this distinction: systems that claim to infer race from voice or religious beliefs from facial features would fall within the prohibition, whereas systems categorizing individuals based on traits such as eye or hair colour would not.

Context and use matter for determining the scope of the prohibition

The prohibition applies only where individuals are individually categorized based on their biometric data, and where the categorization results in the inference of the listed sensitive characteristics. Systems that analyse biometric data at an aggregated level without singling out individuals would not meet this condition. Similarly, the AI Act explicitly excludes certain practices from the scope of the prohibition, including the labelling or filtering of lawfully acquired biometric datasets, for example, where such operations are carried out to improve dataset quality, mitigate bias in AI training data, support medical diagnosis or assist law enforcement investigations.

The relationship between this prohibition and EU data protection law needs further clarification

Finally, the prohibition must be understood in the broader context of EU data protection and non-discrimination law. The GDPR already restricts the processing of special categories of personal data under Article 9(1), while the AI Act introduces an additional regulatory layer by prohibiting certain biometric inference practices altogether. Given that the AI Act itself establishes that it does not affect the GDPR, further guidance is needed for those cases where processing of biometric data would be lawful under Article 9(2) GDPR, but prohibited under the AI Act. 

Red Lines under EU AI Act: Unpacking the prohibition of emotion recognition in the workplace and education institutions

Blog 6 | Red Lines under the EU AI Act Series

This blog is the sixth of a series that explores prohibited AI practices under the EU AI Act and their interplay with existing EU law. You can find the whole series here.

The sixth blog in the “Red lines under the EU AI Act” series focuses on unpacking the prohibition on emotion recognition in the workplace and educational institutions, as contained in Article 5(1)(f) AI Act and explored in the Commission’s Guidelines on the topic. This analysis revealed a number of key takeaways:

• Not all emotion recognition AI systems are prohibited. The AI Act prohibits only the use of emotion recognition AI systems in the workplace or related to education institutions;
• The main reason behind the prohibition in the areas of workplace and education institutions lies behind the power imbalance and asymmetric relationships in these contexts, ​​where both workers and students are in particularly vulnerable positions;
• Emotion recognition systems that are not prohibited under this provision are classified as high-risk;
• Emotion recognition systems used for medical and safety purposes in the workplace or education and training institutions are excluded from this prohibition;
• The provision prohibits only the inference of emotions. The inference of intentions, which is included in the definition of “emotion recognition systems” in Article 3(39) AI Act seems to be left out of the prohibition. 

Article 5(1)(f) AI Act prohibits AI systems from inferring the emotions of a natural person in the workplace and education institutions based on biometric data, with specific exceptions for medical and safety purposes. Recital 44 AI Act invokes the lack of scientific basis for the functioning of such systems and the key shortcomings such as limited reliability, the lack of specificity and the limited generalisability, which may lead to discriminatory outcomes and can be intrusive to the rights and freedoms of the concerned persons. 

Acknowledging the power imbalances in these environments which, combined with the intrusive nature of these systems, could lead to detrimental or unfavorable treatment of certain natural persons or whole groups, this prohibition aims to protect individuals from potentially invasive emotional surveillance. It is important here to note that AI systems for emotion recognition that are not put into use in the areas of workplace or educational institutions do not fall within the scope of this prohibition and qualify as ‘high risk’ under Annex III, paragraph 1, subparagraph c of the AI Act.

It is unclear whether AI systems that do not have as a primary aim the identification or inference of emotions, but have emotion identification or inference as a secondary functionality, are covered by the prohibition. For example, an AI system primarily intended for transcribing meetings that can also infer emotions or intentions, or an AI system that monitors students during a test, but at the same time also identifies emotions.  

1. Limited scope: The provision does not prohibit ‘emotion recognition systems’, but only ‘AI systems to infer emotions’

The Guidelines highlight that the prohibition in Article 5(1)(f) AI Act does not refer to emotion recognition systems more generally, but only to “AI systems to infer emotions of a natural person”. 

Article 3(39) AI Act defines an ‘emotion recognition system’ as “an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data”. Three elements can be identified in this definition:

1. The AI system must be used for identifying or inferring; 
2. Emotions or intentions of natural persons;
3. Based on the biometric data of natural persons.

Hence, the target of these AI systems are emotions and intentions of individuals, which might be identified or inferred (a lower threshold). The identification or inference must be based on biometric data of the individuals. This definition seems to cover the use of emotion recognition on individuals, leaving out groups. 

The prohibition in Article 5(1)(f) AI Act does not refer to ‘emotion recognition systems’, but only to ‘AI systems to infer emotions of a natural person’. Recital 44 further clarifies that prohibition covers AI systems ‘to identify or infer emotions.’ ‘Intensions’ are not mentioned either in the Article 5(1)(f), nor in Recital 44. Hence, it appears that while the definition of ‘emotion recognition systems’ provided in Article 3(39) can serve as a reference point for this prohibition, it does not equate to what the prohibition covers, the prohibition being narrower in scope.

Certain cumulative conditions must be fulfilled for the prohibition to apply:

1. The practice must constitute the “placing on the market”, “putting into service for this specific purpose,” or the “use” of an AI system;
2. The AI system is used specifically to infer emotions;
3. The AI system is in the area of the workplace or education and training institutions; 
4. AI systems intended for medical or safety reasons are excluded from the prohibition.

All the cumulative conditions listed above must be met simultaneously to trigger the prohibition, a consistent approach of the AI Act’s full set of prohibited practices, an approach which narrows down the scope of the prohibition to very specific use cases.

Recital 18 AI Act provides a non-exhaustive list of emotions referred to in this definition, including happiness, sadness, anger, surprise, disgust, embarrassment, excitement, shame, contempt, satisfaction, and amusement. The Recital clarifies that physical states, such as pain or fatigue, are not included in this definition. For example, systems used to detect fatigue in professional pilots or drivers for the purpose of preventing accidents are explicitly excluded from this definition. The mere detection of “readily apparent expressions, gestures, or movements” such as basic facial expressions, such as a frown or a smile, or gestures such as the movement of hands, arms, or head, or characteristics of a person’s voice, such as a raised voice or whispering, are not included either unless they are used for identifying or inferring emotions. The Guidelines further clarify that inferring emotions from a written text does not fall within the scope of the prohibition. 

What is left unclear with regard to the prohibition’s scope is that there is a thin line between emotions, other readily apparent expressions, and pain or fatigue, which also result in expressions that can be mistaken for emotions. The distinction between mood and emotion (both of which can manifest in different ways) is similarly not made, making it unclear whether mood detection falls within this prohibition. The distinction between these features would require a detailed analysis of many factors and circumstances on a case-by-case basis, rather than only biometric data, making the application of this prohibition in practice challenging and complex.

2. It is unclear whether the inference of intentions is also prohibited

The AI Act does not provide clarification on what it means by ‘intentions’ and how to differentiate them from ‘emotions’ in cases when the same system can identify both. This distinction is important because the prohibition applies only to emotions and does not refer to intentions, whereas the definition of ‘emotion recognition systems’ applies to both. 

None of the examples provided in Recital 18 seems to fall within the notion of intentions. While the emotions listed in this Recital represent reactions to situations or environments, the notion of ‘intention’ would have a predictive quality about the future. Additionally, the Commission Guidelines seem to focus solely on emotions, without providing any clarification or definition of ‘intentions’. However, in a non-exhaustive list of examples of emotion recognition, they include the example of “Systems inferring from voice or body gestures, that a student is furious and about to become violent”. While ‘furious’ seems to fall within the notion of ‘emotion’, ‘about to become violent’ makes a prediction about a future action based on the (automatically) detected emotion. This prediction might fall within the notion of ‘intention’ to commit an action in the future but it might also be considered as an identification of the transition from a passive state (emotions) to active (a combination of emotions and intentions), making it difficult to understand whether such a prediction falls within the prohibition. The Guidelines, however, do not seem to make such a distinction. 

An example of ‘intentions’ in the workplace could be the detection of an employee’s intention to resign from the job based on their facial expressions during meetings or videocalls. A wide range of intentions, such as intentions to commit a crime, intentions to drop out of school, or even suicidal intentions could fall within this prohibition when detected in the area of the workplace or education. The Guidelines also note that the concept of emotions or intentions should be understood in a broad sense, noting that attitude and emotion are equivalent for the purposes of this prohibition, thus preventing circumvention through changes in terminology.

Another distinction of the prohibition from the definition of ‘emotion recognition systems’ is that the prohibition refers only to the ‘inference of emotions’ as a prohibited practice, whereas the definition of ‘emotion recognition systems’ includes both ‘identification’ and ‘inference’ of emotions.

The Dutch Data Protection Authority (AP) interprets the prohibition as covering both the inference and the identification of emotions and intentions. The Guidelines distinguish between “identification” and “inferring”, clarifying that identification occurs where the processing of the biometric data (for example, of the voice or a facial expression) of a natural person allows to directly compare and identify an emotion with one that has been pre-programmed in the emotion recognition system. “Inferring” involves deduction through analytical processes, including machine learning approaches that learn from data how to detect emotions. 

3.  The prohibition refers only to emotions deriving strictly from biometric data

According to the Guidelines, this prohibition has a similar scope as the rules applicable to other emotion recognition systems¹, while it should be limited to inferences based on a person’s biometric data, as defined in Article 3(39) AI Act. Hence, the prohibition in Article 5(1)(f) refers only to emotions deriving strictly from biometric data. 

Biometric data is defined in Article 3(34) AI Act as “personal data resulting from specific technical processing relating to the physical, physiological or behavioural characteristics of a natural person, such as facial images or dactyloscopic data”. The relationship between the AI Act definition of biometric data and that provided in the GDPR is explored below in section 6 of this blog.

The limited scope of this ban on inference based on biometric data excludes AI systems that perform emotion recognition through other inferences not on the basis of biometric data such as AI systems for crowd control, and AI systems inferring physical states such as pain and fatigue.

4. A broad interpretation of the ‘workplace’ and ‘education institutions’ contexts

The prohibition in Article 5(1)(f) AI Act is limited to emotion recognition systems specifically in the “areas of workplace and educational institutions”. The Guidelines submit that the workplace context should be interpreted broadly, covering any physical or virtual space where the work is performed. The Guidelines also specifically mention training institutions as covered by this prohibition. Training institutions are not mentioned in the AI Act’s provision or any of the related Recitals. Besides simply listing training institutions alongside educational institutions, the guidelines do not provide any further details as to what constitutes training institutions.

Interestingly, the Guidelines clarify that hiring processes also fall within the workplace context for the purpose of this prohibition. Similarly, the Guidelines clarify that educational institutions should encompass all types and levels of education, including admissions procedures, and should also be interpreted broadly, without any limit, in terms of the types or ages of pupils or students or of a specific environment.

Based on the text of Recital 44, this prohibition also covers AI systems for emotion recognition in situations related to the workplace and education. This makes the area of applicability of the prohibition broader while leaving space for interpretation as to what “related to the workplace” might consist of. Such an interpretation may require a case-by-case analysis. The Dutch AP has interpreted this broad notion as including for example, home working environments, online or distance learning, and also the application of emotion recognition for recruitment and selection or application for education. Further clarifications with clearer guidelines might be necessary in this regard, for ensuring legal certainty, while keeping in mind the volatile nature of ‘workplace’.

It is important to note that emotion recognition systems installed in a work environment can also be used for emotion detection of customers rather than employees, such as, for example, to detect suspicious customers. The AI Act prohibition does not apply to such cases. However, the same system might detect the emotions of employees simultaneously with those of customers. The guidelines only superficially touch upon this scenario in an example stating that in such cases, it should be ensured that “no employees are being tracked and there are sufficient safeguards”.

5. Exception(s) to the prohibition: medical and safety reasons

In a Joint Opinion on the AI Act, the European Data Protection Board (EDPB) and European Data Protection Supervisor (EDPS) state that the “use of AI to infer emotions of a natural person is highly undesirable and should be prohibited.” In this statement, and in a later EDPS Opinion, they further note that exceptions should be made for “certain well-specified use-cases, namely for health or research purposes”.

The Guidelines note that the exception granted in this prohibition should be narrowly interpreted, limited to what is strictly necessary and proportionate, including limits in time, personal application and scale, and should be accompanied by sufficient safeguards in order to ensure a high-level of fundamental rights protection. The recitals of the AI Act stress that the exception granted in this prohibition applies to AI systems used strictly for medical or safety reasons. For example, a system intended for therapeutic use. As such, therapeutic uses mentioned in Recital 44 AI Act as an exception, should only apply to CE-marked medical devices. The Guidelines note that this exception does not extend to general well-being monitoring, such as stress or burnout detection.

Additionally, the Guidelines note that safety exceptions should be limited to protecting life and health, excluding other interests such as property protection or fraud prevention. “Explicit need” is also mentioned as a requirement for the exception to apply. The Guidelines also highlight that data collected and processed in this context cannot be used for any other purpose, hence linking to the GDPR’s purpose limitation principle.

6. Interplay with the GDPR

The Guidelines mention, in a footnote, that there is a distinction between the definition of biometric data within the AI Act and the definition within the GDPR. The definition provided in the AI Act “does not include the wording ‘which allow or confirm the unique identification’ (the functional use of biometric data), contrary to the definition of biometric data in the GDPR that includes this requirement. As such, the Guidelines conclude that “the GDPR definition of biometric data will apply under data protection rules with regard to the processing of personal data ​​(and when for example Article 9(1) and 9(2) GDPR would be applicable)”. This would mean that the AI Act definition applies in AI contexts, whereas the GDPR definition in data protection contexts. 

When reaching this conclusion, the Commission appears to have disregarded recital 14 of the AI Act, according to which the concept of biometric data in the AI Act must be interpreted “in the light of” the concept of biometric data in the GDPR. The clarification of this gap is crucial given the high bar that needs to be met for unique identification. 

If we stick to the AI Act definition, the notion of ‘biometric data’ becomes broader, including most emotion recognition systems using biometric data that do not necessarily have the ability to identify the individual. These systems would be left out of the prohibition if the GDPR definition of biometric data is to be applied. This raises the question of whether there needs to be a categorization of biometric data for the purposes of Article 5(1)(f), to further clarify the notion of biometric data to be able to determine without doubt which practices count as emotion recognition under this prohibition.

Data protection regulators have already been treating emotion recognition systems, including those in the workplace, as high-risk under data protection law, even before the AI Act became applicable. In the Budapest Bank case, the Hungarian DPA submits that AI emotion recognition in the workplace poses fundamental rights risks and ordered the Bank to modify its data processing practices to comply with the GDPR, specifically by refraining from analyzing emotions during voice analysis. With regard to the emotion recognition of employees based on voice analysis, the DPA stresses the need for a separate balancing of interests, taking into account their vulnerable position resulting from their subordinate status. It can be noticed that the reasoning behind the ban imposed by Article 5(1)(f) of the AI Act is reminiscent of the reasoning in this case.

However, it is interesting to note that in the Budapest Bank case, the DPA did not explicitly classify the voice-derived data as biometric data under Article 9 GDPR. The DPA treated the data as ordinary personal data, attracting heightened scrutiny due to the AI risk, rather than as special category biometric data triggering the application of Article 9 outright. Nevertheless, the Hungarian DPA specifically referenced the EDPB-EDPS Joint Opinion 5/2021 on the AI Act proposal, highlighting that AI-based emotion recognition systems pose a high risk to the fundamental rights of data subjects.

7. Concluding Reflections

The AI Act’s prohibition is consistent with previous case law of DPAs, on the basis of the GDPR

The prohibition’s premise regarding the power imbalance and the vulnerable position of employees and students is reminiscent of a DPA’s fine in a similar case. As such, in the Budapest Bank case of 2022, the Hungarian DPA found that the use of AI for emotion recognition of employees and consumers breached several of the GDPR’s key principles and obligations.

The definition of ‘biometric data’ in the AI Act does not seem to be aligned with the definition of the same concept in the GDPR, creating confusion as to which definition applies in this prohibition.

The Guidelines give precedence to the definition of biometric data in the AI Act for the purpose of Article 5(1)(f) prohibition, whereas the AI Act itself, in its Recital 14, seems to give precedence to the concept of biometric data in the GDPR.

The prohibition expressly differentiates between emotions derived from biometric data and those derived from other types of analysis not involving biometric data, the latter practice being excluded from the prohibition.

Biometric data is defined as “personal data resulting from specific technical processing relating to the physical, physiological or behavioural characteristics of a natural person, such as facial images or dactyloscopic data”. The narrow scope of this ban on inference based on biometric data excludes AI systems that perform emotion recognition through other inferences not on the basis of biometric data.

1.  See Annex III, point 1(c), and Article 50 AI Act. ↩︎

Red Lines under the EU AI Act: Understanding the ban of the untargeted scraping of facial images and facial recognition databases 

Blog 5 | Red Lines under the EU AI Act Series

This blog is the fifth of a series that explores prohibited AI practices under the EU AI Act and their interplay with existing EU law. You can find the whole series here.

1. Introduction

The fifth blog in the “Red lines under the EU AI Act” series focuses on unpacking the Article 5(1)(e) prohibition to place on the market, put into service, or use AI systems that create or expand facial recognition databases through the untargeted scraping of facial images from the Internet or CCTV footage. It is notable how this provision targets specifically the acts necessary prior to engaging in facial recognition itself, which is tackled separately, under a different provision of the AI Act, Article 5(1)(h). A number of key takeaways emerge from our analysis: 

• The European Commission Guidelines echo Recital 43 AI Act by acknowledging that the untargeted scraping of facial images is a particularly intrusive practice which “adds to the feeling of mass surveillance and can lead to gross violations of fundamental rights, including the right to privacy”. This, in turn, is consistent with previous case law of Data Protection Authorities (DPAs) on the basis of the GDPR, which remains the most comprehensive protection in facial recognition use-cases; 
• The prohibition expressly differentiates between “targeted” and “untargeted” scraping, thereby limiting the scope of its application and excluding qualified “targeted” scraping from its scope; 
• An analysis of the practices that fall outside the scope of the AI Act’s prohibition finds that some use-cases, such as the scraping of facial images for training AI models that generate new images about fictitious persons, may lead to increasingly complex compliance scenarios triggering both copyright and data protection rules. 

Following this brief introduction, Section 2 outlines the rationale behind the prohibition, while Section 3 notes its specific scope as defined in the differentiation between “targeted” and “untargeted” scraping. Section 4 outlines what falls outside the scope of the prohibition, potentially including use-cases of AI-driven deepfakes, while Section 5 explores the AI Act’s interplay with other relevant areas of EU law, including the GDPR and Law Enforcement Directive (LED). After noting significant cases on facial recognition by DPAs, Section 6 includes concluding reflections and key takeaways. 

2. Context and rationale: untargeted scraping of facial images as a particularly intrusive practice posing “unacceptable risk”, consistent with past case law under the GDPR 

Article 5(1)(e) AI Act prohibits the creation or expansion of facial recognition databases through the untargeted scraping of internet or CCTV footage. The European Commission’s Guidelines on Prohibited Artificial Intelligence Practices under the AI Act recognize that the untargeted scraping of facial images “seriously interferes with individuals’ right to privacy and data protection and deny those individuals the right to remain anonymous”. This is further supported by Recital 43 AI Act, which recognizes that the untargeted scraping of facial images can add to the feeling of mass surveillance and lead to gross violations of fundamental rights, including the right to privacy. 

The context and rationale of the AI Act’s prohibition is consistent with past case law by DPAs across the EU on the basis of the GDPR. Indeed, the expansion and creation of facial recognition databases on the basis of the untargeted scraping of data, including biometric data such as facial images, has been a continuous area of serious concern for DPAs. From 2022 to 2024, several DPAs imposed large fines on Clearview AI for GDPR violations due to practices related to facial recognition, as highlighted in Section 5 of this blog. 

3. Defining facial recognition databases and (targeted vs.) untargeted scraping

Article 5(1)(e) AI Act states that the following practice shall be prohibited: “the placing on the market, the putting into service for this specific purpose, or the use of AI systems that create or expand facial recognition databases through the untargeted scraping of facial images from the internet or CCTV footage” (emphasis added).

Article 5(1)(e) AI Act states that the following practice shall be prohibited: “the placing on the market, the putting into service for this specific purpose, or the use of AI systems that create or expand facial recognition databases through the untargeted scraping of facial images from the internet or CCTV footage” (emphasis added).

Four cumulative conditions must be met for the prohibition to apply: 

1. The practice must constitute market placement, putting into service for this specific purpose, or usage of the AI system; 
2. Aim to create or expand facial recognition databases; 
3. Employ AI tools for untargeted scraping methods; and 
4. Source images from either the internet or CCTV footage.

The Guidelines clarify that, similarly to the other Article 5 prohibitions, all four cumulative conditions above must be met simultaneously to trigger the prohibition. This approach, which is a consistent element of the AI Act’s full set of prohibited practices, seems to ensure a targeted approach to banning very specific uses of AI technologies. The prohibition applies to both providers and deployers who, in accordance with their responsibilities and placement in the value chain, have a responsibility not to place on the market, put into service, or use AI systems for this specific purpose. 

The Guidelines stress that Article 5(1)(e) AI Act does not require that the sole purpose of the database is to be used for facial recognition; it is sufficient that the database can be used for facial recognition. The Guidelines define a “database” in this context as any collection of data or information that is specially organized for rapid search and retrieval by a computer, and may be temporary, centralized or decentralized. 

An important distinction in the application of this provision is between targeted and untargeted scraping – the prohibition does not apply to any scraping tool with which a database for face recognition may be constructed or expanded, but only to tools for untargeted scraping. In this context, untargeted scraping is defined as a technique absorbing as much data and information as possible from different sources and without a specific focus to a given individual or group of individuals. This may be done using a variety of scraping tools and techniques, including web crawlers, bots, or other means that allow for the extraction of data from a variety of sources, including CCTV footage, social media, and other websites, in an automatic manner. 

It is crucial to determine the precise scope of the scraping, since the Guidelines further note that the prohibition does not cover “targeted” scraping, such as the collection of images or videos of specific individuals or pre-defined groups of persons for law enforcement purposes. Furthermore, in more complex systems combining targeted with untargeted searches, only the untargeted scraping is prohibited. 

Notably, the Guidelines highlight that the publication of images on social media by natural persons does not constitute consent for inclusion in facial recognition databases, aligning with the GDPR notion of (valid) consent as a legal basis for processing personal data. 

4. What falls out of the scope of the prohibition?

While specifically targeted scraping is in some cases allowed, several other practices fall outside the prohibition’s scope, including the untargeted scraping of biometric data other than facial images (such as voice samples) and, importantly, non-AI scraping methods. The Guidelines also note that AI systems which harvest large amounts of facial images from the internet to build AI models that generate new images about fictitious persons, similarly fall outside the scope of the prohibition.

While the logic behind this last use-case is seemingly to permit the effective training of AI models, and it explicitly falls outside the scope of the prohibition, attention should be paid to the compliance of this practice with both copyright and data protection laws. Indeed, AI systems that scrape large amounts of facial images to build AI models may trigger the dual application of EU copyright rules, which protect the images themselves to the extent they are under copyright protection, and the application of the GDPR, which protects facial images as personal data, or even as special category biometric data where they are processed with the purpose of uniquely identifying a person. While the scope of this prohibition was agreed upon by co-legislators during final negotiations for the AI Act, this particular use-case may not account for the increasing sophistication of AI-driven deepfakes. 

In fact, at the time of writing, the European Parliament reportedly reached a political agreement on the AI Act Omnibus wherein the latest compromise text includes a new addition to the list of prohibited practices. Namely, once adopted, non-consensual sexual deepfakes will be banned under the revised AI Act. 

It is also worth noting that while this new ban will allow for further protection, it will not cover all use-cases of AI-driven deepfakes, potentially marking an area of continuous, ongoing review by regulators and legislators alike. For this purpose, outside of the Omnibus procedure, the AI Act’s Article 112 empowers the Commission to assess and review the list of prohibited practices on a yearly basis, with the resulting assessment report having to be submitted to the EU Parliament and Council. 

5. Interplay with other EU laws: From the GDPR to the LED

5.1. Facial recognition as a long-standing regulatory priority for DPAs across the EU

The creation and expansion of facial recognition databases on the basis of untargeted scraping of facial images has been a prominent area of regulatory intervention on the basis of the GDPR. In February 2022, the Italian DPA (Garante) fined Clearview AI €20 million and imposed a ban on the company’s further collection and processing of data, including biometric data, and ordered the erasure of such data relating to citizens on Italian territory. 

In October 2022, the French DPA (CNIL) similarly imposed a fine of €20 million on Clearview AI, recognizing the very serious risk to individuals’ fundamental rights posed by their facial recognition software. In September 2024, in an ex officio investigation, the Dutch DPA (AP) fined Clearview AI €30 million for “illegal data collection for facial recognition.” 

In their investigations, DPAs found breaches of the GDPR’s Article 6 (lawfulness of processing), Article 9 (processing of special categories of personal data), and a failure to fulfil data subject rights, particularly those found in Article 15 (right of access) and Article 17 (right to erasure). The Garante also found breaches of key principles of data protection, in particular of lawfulness, fairness and transparency (Article 5(1)(a) GDPR), the purpose limitation principle (Article 5(1)(b) GDPR), and the storage limitation principle (Article 5(1)(e)). As such, in addition to constituting a prohibited practice under the AI Act, the untargeted scraping of facial images for the purposes of creating or expanding a facial recognition database also contravenes several obligations found in the GDPR.

The Guidelines themselves similarly note that, in general, the processing of personal data via the untargeted scraping of the Internet or CCTV material to build up or expand face recognition databases is unlawful, and there is no legal basis under the GDPR for such activity. 

5.2. Law Enforcement use of facial recognition databases

Law Enforcement Authorities (LEAs) use facial recognition databases for identification purposes, allowing for the automated identification of individuals that may in some way be related to criminal events, such as suspects, wanted persons, victims, or witnesses. Among the different types of databases used for face matching by LEAs are also databases consisting of surveillance footage or private data sources and open-source data from the internet. 

While the AI Act prohibits the creation or expansion of facial recognition databases through the untargeted image scraping from the internet or CCTV footage, the provision does not seem to prohibit the use of already existing databases that were previously created from untargeted scraping of internet or CCTV footage that are used by LEAs for face matching and identification purposes. Hence, there might be a legal gap between the prohibition of the creation of new databases and the expansion of existing databases from image scraping, and the use of such databases that were created prior to the entry into force of the AI Act prohibition. 

The AI Act’s Article 5(1)(e) prohibition admits no exceptions for law enforcement use, unlike Article 5(1)(h) on real-time remote biometric identification (to be explored in the final instalment of this blog series), which has a carve-out for competent authorities in public spaces under strict conditions. The AI Act’s blanket ban seems intentional to prevent circumvention through law enforcement justifications. 

The LED, the specific legal framework for data protection in law enforcement, takes a more balanced approach: it may permit particularly intrusive practices if proportionate, necessary, and legally grounded. Hence, if a biometric database is strictly necessary, sufficiently targeted (i.e., footage related to a specific investigation), and proportionate for law enforcement purposes, it passes the LED test. 

Article 10 LED governs the processing of special categories of data, including biometric data processed for the purpose of uniquely identifying a natural person, and permits such processing only where it is strictly necessary, subject to appropriate safeguards, and authorized by Union or Member State law. Untargeted scraping does not seem to satisfy Article 10 conditions. 

Hence, even though the LED does not explicitly prohibit the use of databases from untargeted scraping, it implicitly reaches to the same normative position as the AI Act due to its strict requirements. The primary difference is that the AI Act’s prohibition does not engage with that balancing at all: untargeted scraping is simply prohibited. The two legal instruments thus create overlapping and mutually reinforcing layers of prohibition. One question that remains is whether a database that was created outside of the EU can be used by LEAs in the EU in accordance with the LED or AI Act.

6. Concluding Reflections and Key Takeaways

The AI Act’s prohibition is consistent with previous case law of DPAs, on the basis of the GDPR, which remains the most comprehensive protection in facial recognition use-cases

The prohibition’s differentiation between the targeted and untargeted scraping of facial images, and the subsequent ban of untargeted scraping, is reminiscent of several DPAs’ fines, particularly in the line of Clearview AI cases between 2022 and 2024. DPAs, including the Italian Garante, the Dutch AP, and the CNIL, found that Clearview AI’s facial recognition software breached several of the GDPR’s key principles and obligations.

The prohibition expressly differentiates between “targeted” and “untargeted” scraping, thereby limiting the scope of its application and excluding qualified “targeted” scraping from its scope 

The differentiation between targeted and untargeted scraping is also significant because the AI Act does not include a blanket ban on all scraping of facial images. Indeed, it acknowledges that in some cases, such as in law enforcement contexts, targeted scraping may be lawful when strictly necessary and proportionate. The LED sets out specific conditions for such use-cases, which are tightly regulated across the EU. An analysis of the interplay between the LED and AI Act shows an alignment between the two regulations, creating mutually reinforcing layers of prohibition. 

Some use-cases, such as the harvesting of facial images for training AI models that generate new images of fictitious persons, may lead to increasingly complex compliance scenarios 

When analyzing the practices or use cases that fall outside the scope of the prohibition, we also found that specific AI-driven deepfakes have so far not been captured by Article 5 AI Act. This seems to have similarly been recognized by legislators when, on 11 March 2026, it was reported that the European Parliament reached a political agreement on the AI Act Omnibus, which aims to include a new ban on non-consensual sexual deepfakes. It is worth noting that while this is a development that will allow for further protection, the (new) ban does not cover all AI-driven deepfakes. 

Africa’s Data Protection Reforms: A Continental Perspective on the Drivers of Change in Legal Frameworks

1. Introduction

Within an evolving digital landscape, several African jurisdictions have proposed a variety of reforms to existing and novel legal frameworks that regulate the processing of personal data, and the development and deployment of new technologies. Across the continent, there is a growing consensus among legislators on the need to create a regulatory environment that is responsive and adaptable to a changing technological landscape and a growing digital economy.

This blog traces data protection legal and policy reforms across seven African countries, including Nigeria, Kenya, Angola, Ghana, Mauritius, Botswana, and Seychelles, to identify their scope, rationale, and common and diverging themes. The blog also briefly looks at the regional and sub-regional legal reforms to note the potential implications for other countries that might consider similar reforms and eventual harmonization. 

While these developments are unfolding across Africa, they are occurring alongside broader global efforts to rethink data protection frameworks. As discussions around data protection policy reforms are intensifying in jurisdictions such as the European Union, which introduced a simplification package to reduce regulatory burdens and boost competitiveness; the UK, which finalized reforms to its data protection framework through the Data Use and Access Act (2025); and South Korea, which continues to explore legal reforms to its data protection law to facilitate the development of AI, data protection reforms across the African continent bring a different flavor to addressing their needs. Indeed, legislators across African jurisdictions agree that any reforms or amendments must first and foremost be reflective of local realities. 

In the closing section, the blog considers the future of legal reforms on the continent by drawing from ongoing discussions and lessons learned in other key jurisdictions. In doing so, the following takeaways emerge: 

• There is a growing patchwork of legal reforms across the continent, with data protection law reforms taking both unilateral trajectories, primarily driven by national interests and concerns, including AI related, and multilateral ones, such as the growing need for alignment with international standards.

• Notable among the drivers of change in data protection legal frameworks are: requirements to establish a local nexus that would support accountability of foreign companies; the need to address processing of personal data through AI systems; and ensuring interoperability with international frameworks.

• Some legislative initiatives to amend data protection laws on the continent stand out with unique policy proposals: for instance, Angola proposes to restrict data scraping and to require entities involved in AI systems to establish governance and data management practices that prevent discrimination, while Ghana is looking to codify property rights over personal data.

• While Regional Economic Communities (RECs) are also involved in reform processes, their historically minimal influence on national laws means that new approaches to encourage Member States to harmonize such reforms will be necessary.

• Legal reforms in Africa are set to continue and are likely to  be influenced by the changing technological landscape and reforms in jurisdictions that have historically influenced current data protection frameworks.

Overall, most reforms are, for the time being, confined to national borders. However, legal reforms have also been proposed both continentally and at the sub-regional level, for example, through the Economic Community of West African States’ (ECOWAS) reform of its Supplementary Act on Personal Data Protection. While these regional reforms have not gained much traction compared to national efforts, they are nonetheless crucial as they can continue to inform ongoing debates for legal reforms within their respective Member States.

2. A new task for data protection law? New obligations for digital platforms and developer accountability

Despite being a relatively new law, proposals to amend the Nigeria Data Protection Act 2023 (NDPA) have already emerged through two separate legislative initiatives. The first, SB.650: Nigeria Data Protection Act (Amendment) Bill, 2024, seeks to amend the NDPA by introducing requirements for social media companies to establish physical offices in the country. At present, no other substantive changes to the NDPA have been outlined, making this the central focus of the reform proposal. The Bill, which is in its second reading, notes that while major social media platforms have significant Nigerian user engagement, they are yet to set up physical presence in Nigeria as they have done in other countries. 

According to the sponsor of the Bill, Senator Ned Nwoko, the establishment of a company’s physical presence will contribute to the economy as well as ensure their compliance with the country’s legal framework. The Bill was referred to the Senate Committee on ICT & Cybersecurity and a report was expected within two months. 

The second legislative proposal, HB.2436: Nigeria Data Protection (Amendment) Bill, 2025, focuses on strengthening accountability in the digital ecosystem by introducing obligations for application developers, regulating third-party data sharing, and expanding the enforcement powers of the Nigeria Data Protection Commission. Among other provisions, the bill proposes requirements for application developers to register with the Commission, maintain data processing registers, implement consent interfaces, and conduct annual data protection impact assessments, while also introducing stricter rules governing third-party data sharing and related enforcement measures.

However, while updates regarding the progress of both Bills have been limited, the decision to amend the NDPA to cover social media companies has been criticized by civil society groups on grounds that the proposal to require social media companies to establish physical offices in the country may extend beyond the initial objectives of the country’s data protection framework. Indeed, the NDPA is a principles-based data protection law that focuses on regulating the processing of data across all sectors, rather than regulating specific entities such as social media companies. 

A brief look at the history of social media regulation in Nigeria shows that it is intricately connected with state regulation of the freedom of expression. While past attempts to regulate the use of social media platforms have largely been led by ad-hoc bans on the basis of national security concerns, the proposed Bill to amend the NDPA signals a new approach: one that aims to progressively embed social media oversight within broader data governance frameworks, starting with data protection law. In this case, Nigeria’s approach to amending its NDPA uniquely highlights how national-level priorities and new technological realities converge under the umbrella of data protection.

3. Processing of sensitive personal data by third parties in Kenya 

Calls for amendments to Kenya’s Data Protection Act of 2019 (KDPA) began informally, largely owing to implementation challenges and gaps observed by controllers and processors. This was further solidified in the Parliamentary Report on the inquiry into the activities and operations of WorldCoin in Kenya, completed in 2023. The inquiry was set up to establish the legality of WorldCoin’s processing of sensitive personal data by an ad hoc Parliamentary Committee. The resulting Report included considerations on legal and regulatory gaps to provide safeguards for this type of data processing activity. While not legally binding, the Parliamentary Committee’s findings have nevertheless informed the push to amend the KDPA on grounds including:

• Aligning the KDPA with the Companies Act (the legal framework that governs the formation, operation, and regulation of companies in Kenya) by requiring foreign companies to provide proof of registration with local regulatory bodies under Part XXXVII, before registering as data processors and/or controllers with the data protection authority;
• Requiring full disclosure on how data controllers and processors utilize and store personal and sensitive data collected in Kenya;
• Providing discretion to the Office of the Data Protection Commissioner (ODPC) in the imposition of administrative fines. It does not describe the nature of such possibly expanded discretion. Currently, the ODPC can impose administrative fines for violations under the KDPA only; and
• Creating a board to which the ODPC reports or accounts on its daily operations.

Negotiations on the amendment of the KDPA are ongoing and public consultations are expected to happen soon. Early contributions have been made by organizations such as the Data Protection and Governance Society of Kenya proposing amendments such as the creation of a data protection appeals tribunal that would hear appeals from the ODPC. This would reduce the burden of appeals at the High Court, which have been numerous. They also suggest repealing Section 54 of the KDPA that provides the Data Protection Commissioner with powers to exempt compliance with certain provisions of the Act, unless such exemptions are provided for under other regulations. This approach would provide more certainty on the conditions for exemption. Overall, Kenya’s approach to amending its data protection framework is driven by a growing interest to address specific procedural challenges as related to enforcement. 

4. Angola leads the way in amending its data protection law to address the need for regulating AI

Unlike Kenya and Nigeria, the discourse of data protection reform in Angola is driven by the need to regulate emerging technologies including AI. As African countries continue to carve out policy and legislative proposals aimed at regulating the development and deployment of AI, mostly in the form of national AI strategies, some countries are considering more specific legislation. In this respect, countries such as South Africa have proposed standalone AI legislation under its National AI Framework, while others such as Angola have opted to revise existing data protection laws to address privacy challenges posed by AI systems already in use. 

Angola’s preparedness to regulate AI began with the recognition of privacy risks posed by AI. In March 2025, Angola’s data protection agency released a public consultation on the revision of its 2011 data protection law. Besides introducing numerous new sections, the draft revised law notably contains a section dedicated to AI. Its robust provisions on AI differentiate the law from other data protection laws in Africa, whose automated decision-making provisions mostly mimic Article 22 of the GDPR. Noteworthy aspects on the regulation of AI in the revised law include:

• Legitimate use of AI in credit scoring by requiring consent to process credit and solvency data, providing data subjects with the right not to be subject to fully automated evaluations—especially those using AI—that could negatively profile them. If credit-based decisions are made, data controllers must explain the algorithm and criteria used and inform individuals when a denial results from a credit report, including providing access to the report and identifying its source, as governed by specific credit data laws (Article 23); 
• Providing data subjects the right not to be subjected to a decision solely based on automated or semi-automated processing with legal effects. However, this right does not apply if the decision is necessary for a contract or based on the data subject’s explicit consent. In such cases, the data controller must provide clear and sufficient information about the decision-making criteria and procedures, while respecting trade and industrial secrets (Article 33);
• Prohibiting the use of AI systems that compromise privacy, exploit vulnerabilities, or lead to illegitimate or discriminatory profiling (Article 36);
• Granting data subjects the rights to receive clear and adequate information about an AI system’s characteristics; request explanations for decisions, recommendations, or predictions made by the system and challenge a system’s decisions; demand human participation in system decisions; and receive fair and equal treatment (Article  37);
• Requiring entities involved in AI systems to establish governance and data management practices that prevent discrimination and ensure legal compliance. Among other responsibilities, entities will be required to inform users that a service uses AI if this is not evident to a user, as well as develop mechanisms for explainability (Article 38);
• Restricting data scraping and unauthorized data transfers, addressing a core concern at the intersection of personal data processing for AI. Though it does not explicitly refer to AI, Article 82 prohibits and criminalizes the unauthorized scraping, copying, or transfer of personal data without legal authorization or the data subject’s consent, regardless of purpose. Violations can lead to imprisonment or fines, with doubled penalties if security measures are bypassed, financial gain is involved, or sensitive data is affected (Article 82).

What stands out about Angola’s approach to reforming its data protection law is the explicit specification of rules with regard to the use of AI for credit scoring. Article 23 provides nuance to the proposed legal reforms by identifying country-specific challenges introduced by the use of AI, and specifically the use of AI-enabled systems for credit scoring, thus moving away from the more general automated decision-making provisions seen continentally.

The use of AI in credit scoring remains one of the earliest uses of AI continentally and has generated considerable data protection concerns, leading to several landmark enforcement decisions in some countries and necessitating specific guidelines on the use of personal data by digital lenders. For example, Kenya’s body of enforcement decisions consists of numerous such decisions including repeat offenders. The decision to specifically regulate the use of AI within the credit scoring industry points to the need to address subject-specific issues relating to the processing of personal data in Angola.Notably, Angola’s proposed reforms parallel the EU AI Act’s approach by specifically regulating AI-enabled credit scoring as a high-risk application, recognizing its widespread use and potential for harm. Like the EU AI Act’s Annex III(5)(b), which classifies credit scoring as high-risk, Angola moves beyond general provisions on automated decision-making to addressing country-specific risks to data subjects.

5. Mauritius seeks to boost its growing business processing outsourcing industry

National economic considerations such as Mauritius’ vision of becoming a preferred destination for business process outsourcing (BPO) and knowledge-based services have been central to its data protection reforms. The recently released National ICT Blueprint views legal and regulatory reforms, including to the data protection framework, as enablers of Mauritius’ goals for economic growth. According to the Blueprint, Mauritius intends to align its national frameworks with the AU Data Policy Framework as well as create regulatory conditions for pursuing an EU adequacy decision. These ongoing reforms aim to position Mauritius as a leader for outsourced services.

Such economic considerations have been a major factor influencing the repeated data protection law amendments in Mauritius to date. Its first data protection law, enacted in 2004, was heavily influenced by the EU Data Protection Directive of 1995. The 2004 law was amended twice to bring the text of the law in closer alignment with the EU Data Protection Directive to provide Mauritius with better chances of accreditation by the European Commission as an adequate country, thus facilitating personal data transfers at a time when the country sought investments in its BPO sector, with the EU as its primary beneficiary. In 2017, the current data protection law of Mauritius was enacted, repealing the 2004 law but maintaining the initial aspirations of being a leader in outsourced BPO service providers. This further saw Mauritius ratify Council of Europe’s Convention 108, and Convention 108+ in 2020.

6. Botswana’s path to filling in practical implementation gaps

Botswana enacted its new data protection law in 2024, repealing the 2018 law and introducing new provisions to address implementation gaps in the latter. The 2018 framework, which had been in transition since 2021, did not provide sufficient clarity on certain provisions, including the institutional independence of the Information and Data Protection Commission, the scope of its enforcement powers, or the practical obligations of data controllers and processors. 

For example, when compared to most data protection laws on the continent, Botswana’s 2018 data protection law did not provide modalities for responding to data subject rights, and its limited focus on data controllers with processors treated merely as agents created ambiguity in shared compliance responsibilities. It also lacked provisions on accountability, joint controllership, or clear rules governing relationships between controllers and processors, including the use of sub-processors. Similarly, there were no requirements for data protection impact assessments (DPIAs) or structured procedures for breach notification beyond informing the Commission, and sanctions were limited to fixed fines and criminal penalties rather than risk-based administrative measures.

The 2024 Act responds to such uncertainty by clearly defining the Commission’s authority, strengthening accountability mechanisms, and introducing risk-based tools such as DPIAs. It distinguishes between controllers and processors as separate entities with direct statutory obligations, introduces concepts of joint controllership and data protection by design and default, and requires formalised contractual arrangements for processor relationships, including restrictions on the use of sub-processors. The Act further mandates breach notifications to both the Commission and affected data subjects, introduces proportionate administrative fines, and establishes structured compliance roles such as Data Protection Officers (DPOs). These reforms, alongside an expanded territorial scope and refined definitions of sensitive data, collectively close the significant regulatory and operational gaps left by the 2018 framework.

7. Seychelles’ reforms reflect clearer provisions and expanded transfer mechanisms while retaining limited extraterritorial application

Still on the shift from theoretical legal frameworks to practical and clearer provisions, Seychelles repealed its 2002 Data Protection Act which had never been implemented with the 2023 Data Protection Act. The overhaul of Seychelles’ data protection regime marked a move from a largely symbolic framework to one grounded in enforcement, accountability, and operational clarity. Unlike the earlier law, which relied on formal registration of “data users” and “computer bureaux” but imposed few operational duties, the 2023 Act abandons registration in favour of an accountability-based model requiring data controllers and processors to maintain internal records, demonstrate compliance, and cooperate with regulatory audits. Security obligations have also evolved from a general duty to prevent unauthorized disclosure to a detailed mandate for technical, organisational, and physical safeguards, including breach notification duties.

Equally, the 2023 Act introduced explicit obligations for data processors including acting only on a controller’s instructions, maintaining security measures, and being jointly liable for breaches supported by mandatory written contracts between controllers and processors that define purpose, scope, and safeguards. The law also embeds governance mechanisms through the requirement for DPOs and DPIAs for high-risk processing, neither of which existed in the 2002 text. 

With regard to cross-border data transfers, the 2023 regime replaces the earlier “transfer prohibition notice” system with a more flexible approach permitting international data flows where adequate protection or recognised safeguards exist. Notably, the 2023 Act expressly recognises participation in frameworks such as the Global Cross-Border Privacy Rules (CBPR) System, signalling Seychelles’ intention to align its transfer mechanisms with interoperable international privacy standards expanding mechanisms for transfers. 

Finally, enforcement capacity has been strengthened with the 2023 Act empowering the Information Commission to conduct audits and inspections independently, issue enforcement notices, and impose administrative fines, enhancing oversight compared to the limited, warrant-based powers of the 2002 law. 

While its territorial scope remains modest compared to broader extraterritorial models, these reforms collectively transform Seychelles’ data protection law into a more operational, risk-based, and globally interoperable framework.

8. Ghana seeks to introduce a new Bill to strengthen enforcement and oversight, including broader data subject protections

Ghana first enacted its data protection law in 2012, which also established the Data Protection Commission. However, implementation challenges soon emerged, including the absence of a clear framework for cross-border data transfers, limited protection for vulnerable groups such as children, and a narrow scope of application compared to new generation data protection laws that did not extend to foreign entities offering goods or services in Ghana. These gaps created practical and regulatory difficulties. On 17 October, the new Data Protection Bill, 2025, spearheaded by the Ministry of Communication, Digital Technology, and Innovations, was therefore introduced with the aim of addressing these shortcomings and modernizing the country’s data governance framework.

Overall, the Bill aims to strengthen oversight by introducing clearer obligations, enhanced data subject rights, and a more robust regulatory structure. Particularly, it introduces key reforms by addressing emerging privacy challenges associated with new technologies, introducing data ownership rights, and refining exemptions for the processing of personal data.

In contrast to Angola’s targeted approach to addressing privacy concerns in AI systems, Ghana seeks to adopt a broader stance by regulating all emerging technologies, including AI systems, insofar as they process personal data. For automated decision-making (ADM) systems, the Bill would require outcomes to be explainable, contestable, and subject to human oversight, obligations that were absent from the 2012 Act which only required notification when decisions involved ADM. The Bill also aims to introduce explicit requirements for the use of privacy-enhancing technologies in ADM systems, a novel provision not contained in the earlier law.

On data ownership, the Bill would introduce a data ownership framework that recognises personal data as the property of the data subject, and establish a fiduciary-style relationship between data subjects and controllers. Under this model, controllers and processors are deemed custodians of personal data with a duty of care, and any form of processing does not confer ownership rights including for public authorities. If passed, Ghana would become one of the few jurisdictions globally to recognise the proprietary nature of personal data, with significant implications for secondary data use, AI development, and the application of rights such as the right to object to processing. The Bill was open for public consultation until 28 November 2025, and could be adopted as early as 2026.

Regarding exemptions, the Bill aims to retain the broad exemption themes found in the 2012 Act, but significantly expand and refine them. While both instruments include exemptions for national security, the 2012 Act required a ministerial certificate to validate the exemption. The 2025 Bill removes this safeguard, a notable development given the increasing reliance on public-interest grounds to limit privacy protections across the continent.

Crucially, the Bill would introduce a comprehensive regime for cross-border data transfers, which was absent from the 2012 Act. The new framework emphasizes data localization, unless such localization would impair business operations. Where transfers are necessary, the Bill would require data subject consent, approval from the Data Protection Authority, and compliance with additional conditions designed to safeguard personal data before it leaves Ghana.

9. The patchwork challenge: emerging regional frameworks

Even as countries unilaterally consider legal reforms, there are regional plans, led by the AU and the respective RECs, to amend or create new data protection frameworks for their Member States. Regional initiatives must navigate a complex landscape where many States already have distinct data protection regimes. At the continental level, the AU announced plans to revise the Malabo Convention. At the sub-regional level, ECOWAS is expected to revise the Supplementary Act on Data Protection, the East Africa Community (EAC) is developing its data governance framework, and the Southern Africa Development Community (SADC) has plans to revise the Model Law. Despite their minimal influence on national laws, legal reforms at the REC level could spur similar actions for Member States, especially in the ECOWAS region where the Supplementary Act on Personal Data Protection is legally binding on member states.

As legal reforms continue, bigger questions of what will be the drivers of such reforms remain, especially considering that some African countries still maintain legal frameworks influenced by the now-defunct 1995 EU Data Protection Directive. 

9.1. Development and deployment of AI in Africa

Strongly tied to the aspect of responsible data use is the development of local AI systems as well as the general adoption of AI across the continent. Discussions of the former largely revolve around the lack of local datasets for training AI models, hence the emergence of targeted initiatives seeking to address this issue. The theory that effective data protection regimes can allow responsible local data collection and use has advanced, as seen in continental data governance frameworks such as the AU Data Policy Framework.

Additionally, the risks posed by the general adoption of AI have been highlighted on the continent as drivers of legal reforms in countries such as Angola, as explored above. Data protection frameworks have been fronted as useful instruments for ensuring responsible development and deployment of AI as seen in the text of numerous national AI strategies, some of which note, however, that ADM provisions alone may not be sufficient for addressing AI harms. For example, the AI Policy Framework of South Africa considers a standalone AI Act to complement its national data protection law.

While there is growing regulatory momentum on comprehensive AI specific laws, there are currently no AI specific laws that provide guidance on the development and deployment of high-risk AI systems. Nonetheless, some DPAs on the continent are grappling with the foundational questions of what privacy risks are unfolding in the use of AI systems. DPA activities related to regulating AI have included Senegal’s CDP rejecting an application for the use of facial recognition systems in the workplace requiring the controller to use less intrusive means of registering work attendance and Mauritius’ data protection authority’s Guide on Data Protection for Health Data and Artificial Intelligence. Such approaches signal that even though considerations towards stand-alone AI regulation on the continent are in their nascent stages, DPAs are nevertheless addressing new AI technologies on the basis of national data protection law, either in the form of guidance or through enforcement.

10. Concluding reflections: The future of data protection legal reforms in Africa

The EU, whose data protection legal framework has been relied on by many African countries, is currently considering amendments to its existing data protection framework through an Omnibus initiative. Amendments to laws that have largely informed legal frameworks across Africa could provide a moment of reflection for the “recipient” countries, some of which have already registered the challenges of implementing current data protection frameworks, especially for SMEs, and questioned the impact of the “Brussels effect” for their own national data protection laws.

In addition to the shifts noted in the EU, legal reforms in Africa are also increasingly influenced by the growing recognition of data as a national asset and the subsequent need for autonomy on its protection and governance. There are already new sector-specific regulations that place emphasis on balancing data use and protection, as well as explicitly designating governments as custodians of such data. Implementation of these sector-specific laws has revealed gaps in foundational data protection frameworks, prompting legal reforms towards frameworks that not only safeguard rights but also enable responsible data access and re-use.

As data protection reforms take shape across the continent, the question is not whether change will come but, rather, what form it will take. 

Red Lines under the EU AI Act: Unpacking the Prohibition of Individual Risk Assessment for the Prediction of Criminal Offences

Blog 4 | Red Lines under the EU AI Act Series

This blog is the fourth of a series that explores prohibited AI practices under the EU AI Act and their interplay with existing EU law. You can find the whole series here.

The fourth blog in the “Red lines under the EU AI Act” series focuses on unpacking the prohibition on individual risk assessment and the prediction of criminal offences, as contained in Article 5(1)(d) AI Act and explored in the European Commission’s Guidelines on the topic. Our analysis led to three key takeaways:

• As this provision is limited in its scope, it does not entirely prohibit crime prediction or forecasting AI technologies – rather, it focuses on prohibiting individual risk assessments to predict criminal offences based solely on profiling or personality assessments. The provision relies on the well-established GDPR definition of ‘profiling’; 
• Similarly to other prohibitions explored in this series, when an AI system does not meet all of the conditions for the Article 5(1)(d) prohibition to apply, it will nevertheless be classified as a high-risk AI system and be subject to specific requirements and safeguards, including human intervention; 
• Given the particularly sensitive context of crime prediction, and the inherently “forward-looking” nature of risk assessments, the Guidelines note that engaging in such activities may perpetuate or reinforce biases and erode public trust in law enforcement.  

With this context in mind, this blog post begins with an overview of the logic and scope of the prohibition on individual risk assessment in the EU AI Act, and continues in Section 3 with an analysis of understandings of “risk” elaborated in the Commission’s Guidelines. Section 4 expands on the notion of “profiling”, including the prohibition of assessing a natural person’s personality traits and characteristics, and Section 5 outlines the exceptions to the Article 5(1)(d) prohibition. Section 6 explores cases in which this provision is applicable to private sector actors, and Section 7 notes concluding reflections and key takeaways. 

2. The ban is limited in its scope, applying only to AI systems used to assess or predict criminal offences based solely on profiling or personality assessments

Article 5(1)(d) AI Act establishes a crucial prohibition on AI systems that assess or predict the likelihood of natural persons committing criminal offenses based solely on profiling or personality assessment. This prohibition focuses on risk assessments relating specifically and exclusively to committing criminal offences, reflecting the fundamental principle that individuals should be judged on their actual behavior rather than predicted conduct, reinforcing the principle of legal certainty in EU criminal law.

Importantly, the prohibition does not apply when AI systems support human assessment regarding a person’s involvement in a criminal activity (offending or re-offending), such as when the assessment is already based on objective and verifiable facts directly linked to criminal activity. In such cases, the AI system serves as a supportive tool rather than the primary decision-maker. These systems are instead classified as high-risk AI systems (Annex III, point 6, letter (d) AI Act).

The provision does not entirely outlaw crime prediction and risk assessment practices but, rather, imposes specific conditions under which the use of certain AI systems in specific contexts shall be prohibited. The Guidelines clarify that the three cumulative conditions all have to be met simultaneously, creating a high threshold for the prohibition to apply: 

1. The practice must involve placing an AI system on the market, putting it into service for the specific purpose of assessing or predicting the likelihood of natural persons committing criminal offenses, or using the AI system.
2. The AI system must make risk assessments that assess or predict the risk of a natural person committing a criminal offence.
3. The risk assessment or the prediction must be based solely on either, or both, of the following:

The prohibition applies to law enforcement authorities or any entity using such systems on their behalf, as well as to Union institutions, bodies, offices, or agencies that support law enforcement authorities. Both providers and deployers therefore have the responsibility not to place on the market, put into service or use AI systems that meet the above conditions. The rationale behind this prohibition is that natural persons should be judged on the basis of their actual behaviour rather than on (AI-)predicted behaviour. While the Guidelines do not directly refer to the principle of legal certainty when analyzing the rationale for this prohibition, it should play a role in the implementation of this prohibition, as it is a primary principle of the rule of law in the EU, alongside equality before the law, the prohibition of the arbitrary exercise of executive power, and effective judicial protection. 

It is also worth highlighting that the Article 5(1)(d) prohibition applies to criminal offences only, with administrative offences falling outside of the scope of the prohibition. Under EU criminal law, the determination of the criminal nature of an offence most often depends on national law and, as such, may include offences that are not covered by Union law. Given possible differences at national level across the EU, the use of AI systems for the risk assessment and prediction of criminal offences might require further clarification, particularly with regard to which actions amount to “criminal offences” under national law. Indeed, the Guidelines highlight that, for offences that are not directly regulated under EU law, the national qualification of the offence is nevertheless subject to scrutiny by the CJEU on a case-by-case basis, since the concept of “criminal offence” has autonomous meaning within EU law and should be interpreted consistently across EU Member States. 

3.  Notions of “risk” in the AI Act’s prohibitions, while uncertain, are closely related to harm and ensuring individuals are only assessed on the basis of actual (not predicted) behaviou

According to the Commission’s Guidelines, risk assessments are understood broadly and can be conducted at any stage of law enforcement activities, such as during crime prevention, detection, investigation, prosecution, execution of criminal penalties, and during the process of an individual’s reintegration into society. Such risk assessments are often referred to as individual “crime prediction” or “crime forecasting” which, according to the Guidelines, refer to “advanced AI technologies and analytical methods applied to large amounts of often historical data… which, in combination with criminology theories, are used to forecast crime as a basis to inform police and law enforcement strategies and action to combat, control, and prevent crime.” 

In practice, there are two major areas where law enforcement applies AI risk assessments: predictive policing and recidivism risk assessment. Predictive policing involves law enforcement using predictive analytics and other algorithmic techniques to identify patterns related to the occurrence of crime and unsafe situations, and to proactively prevent crime based on these insights. This approach has been adopted by several Member States. On the other hand, a recidivism risk assessment is used to predict the risk of individuals reoffending. 

Crime prediction or crime forecasting AI systems identify patterns within historical data, associating indicators with the likelihood of a crime occurring, and then generate risk scores as predictive outputs. The Guidelines seem to expand on the notion of “risk” contained in Article 5(1)(d), noting the inherently “forward-looking” nature of risk assessments used for crime prediction or forecasting. 

In this context, they note that using historical data on crimes committed to predict other persons’ future behaviour may perpetuate or reinforce biases, and undermine public trust in law enforcement and the justice system. Indeed, risk is by definition uncertain: it may or may not materialise into harm. Any decision based solely on a risk score has the potential to make a wrong assumption regarding the actual commission of a criminal offence. In a recent case, the Dutch Ministry of Justice and Security instructed the probation service in the Netherlands to either adjust or stop using the OxRec algorithm, which, following an investigation, was found to have misjudged the risk of recidivism in a quarter of cases. Having been used around 44,000 times per year, OxRec was identified as relying on outdated data, being in breach of privacy legislation, and posing risk of discrimination.

As Recital 42 of the AI Act explains, natural persons in the EU should always be assessed on the basis of their actual behaviour, and risk assessments carried out solely on the basis of profiling or an assessment of personality traits or characteristics should be prohibited. This aligns with the presumption of innocence until proven guilty under the law (Article 48 EU Charter of Fundamental Rights) and the principle of legal certainty as enshrined in EU law. Indeed, in their final section analyzing the interplay of this prohibition with other Union law, the Guidelines acknowledge the indirect link between the prohibition and Directive (EU) 2016/343 on the presumption of innocence. 

4. The prohibition relies on the GDPR’s definition of ‘profiling’, and takes a broad understanding of ‘personality traits’ and ‘characteristics’

The Guidelines clarify that the prohibition applies regardless of whether the AI system profiles or assesses the personality traits and characteristics of only one natural person or a group of natural persons simultaneously. In this context, group profiling can consist of, for example, an AI system assessing and predicting the risk of other persons committing similar offences, based on constructed or historic data about previously committed crimes by others.

Similarly to the prohibition in Article 5(1)(c) AI Act, explored in Blog 3 of the “Red Lines” series, profiling is understood by reference to its definition in Article 4(4) GDPR. Further, the Guidelines highlight that the predictive policing prohibition is without prejudice to Article 11(3) of the Law Enforcement Directive (LED), which prohibits profiling on the basis of special categories of personal data which results in direct or indirect discrimination.  

The risk assessments covered by the analyzed provision are only prohibited when they are based solely on the profiling of a person or the assessment of their personality traits and characteristics. This means that when there is a human assessment, which will normally be based on relevant objective and verifiable facts, and the AI assessment is used to support the human assessment, the prohibition does not apply. The Guidelines clarify that “personality traits” and “characteristics” are to be broadly understood, and that the examples contained in Recital 42 are not exhaustive. 

However, according to the Guidelines, the use of the term “solely” leaves open the possibility of various other elements being taken into account in the risk assessment, beyond personality traits and characteristics, which will need to be assessed on a case-by-case basis. The Guidelines submit that in order to avoid circumvention of the prohibition and ensure its effectiveness, any such other elements will have to be real, substantial, and meaningful for them to be able to justify the conclusion that the prohibition does not apply. In this context, both providers and deployers of such systems will have to document their decision-making processes to be able to justify choosing a certain course of action over another, particularly in highly sensitive contexts such as crime prediction, in which the risks of producing legal effects can be imminent and significant. 

5. Exception(s) to the prohibition: When a ‘predictive policing’ AI system is not prohibited, but may nonetheless be classified as ‘high-risk’ 

The last phrase of Article 5(1)(d) AI Act clarifies that the prohibition does not apply to AI systems that are used to support the human assessment of the involvement of a person in a criminal activity. This exception applies only insofar as the human assessment is based on objective and verifiable facts directly linked to the criminal activity at hand. While both the AI Act and the Guidelines do not directly define what may constitute “objective and verifiable acts”, the Guidelines provide some examples in which these conditions for the exception to the prohibition may be fulfilled. 

For example, this is the case for an AI system used for the profiling and categorization of actual behaviour, such as “reasonably suspicious dangerous behaviour in a crowd that someone is preparing and likely to commit a crime, and there is a meaningful human assessment of the AI classification” (emphasis added). This latter requirement for ensuring that any AI system used in this context is only acting in support of human assessment echoes the GDPR’s right to obtain human intervention in automated decision-making contexts. 

In the highly sensitive context of crime prediction, the requirement for the “human assessment” to be based on objective and verifiable facts linked to a specific criminal activity is an important precursor to the exercise of the right to an effective remedy (Article 47 EU Charter of Fundamental Rights). While the Guidelines do not expressly refer to the EU Charter, they refer to case law of the Court of Justice of the EU (CJEU) in their understanding and interpretation of the concept of “human assessment.” In the Ligue des droits humains judgement, published in June 2022, the CJEU noted that any human assessment “must rely on objective criteria … and to ensure the non-discriminatory nature of automated processing.” 

Additionally, according to the Dutch DPA (AP), human intervention ensures that a decision is made carefully and prevents people from being (unintentionally) excluded or discriminated against by the outcome of an algorithm. Hence, human intervention must contribute meaningfully to the decision-making process, rather than serve only as a symbolic function. 

It is worth noting that while the Guidelines are specific in their interpretation of the exception contained in Article 5(1)(d), they also mention that this express exclusion from the prohibition may not be the only one. However, the Guidelines do not further elaborate on what other exceptions may apply and in which contexts. It is likely that such exceptions may have to be assessed on a case-by-case basis and, in any case, be real, substantial, and meaningful. Nevertheless, what the Guidelines do clarify is that when the system falls within the scope of the exclusion from the prohibition, it will be classified as a high-risk AI system and be subject to specific requirements and safeguards, including with regard to human oversight as referred to in Articles 14 and 26 AI Act. 

Finally, it is worth noting that AI systems used in the context of national security are excluded from the scope of the AI Act as referred to in Article 2(3) and further explained in Recital 24. This means that an AI system that falls under the ‘predictive policing’ prohibition may nevertheless be permitted exclusively for national security purposes. In this context, the Guidelines do not clarify the distinction between national security and law enforcement activities, which could be crucial for delineating the boundaries of the prohibition of individual risk assessment. 

This is particularly relevant with regard to ‘dual-use systems’ – AI systems that can be used both for law enforcement purposes and for the prevention of national security threats. Recital 24 provides a clarification for such cases, stating that ‘AI systems placed on the market or put into service for an excluded purpose, namely military, defence or national security, and one or more non-excluded purposes, such as civilian purposes or law enforcement, fall within the scope of this Regulation and providers of those systems should ensure compliance with this Regulation.’ Hence, if an AI system is placed on the market or put into service for both national security and law enforcement purposes, it must nevertheless comply with the AI Act. 

6. The prohibition can apply to private actors when they are entrusted by law to exercise public authority and public powers

Notably, the ‘predictive policing’ prohibition does not apply exclusively to law enforcement authorities. The prohibition may be assumed to apply, in particular, when private actors are entrusted by law to exercise public authority and public powers for the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties. Private actors may also be explicitly requested, on a case-by-case basis, to act on behalf of law enforcement authorities and carry out individual crime risk predictions. In those cases, the activities of those private actors could also fall within the scope of the Article 5(1)(d) prohibition.

The prohibition may apply to private entities assessing or predicting the risk of a person committing a crime where this is objectively necessary for compliance with a legal obligation to which that private operator is subject to (for example, a banking institution obliged by Union anti-money laundering legislation to screen and profile customers for money-laundering offences). 

The Guidelines also outline what is explicitly excluded from this prohibition or out of its scope, namely: 

• Location-based crime prediction without individual profiling (e.g. prediction on the likelihood of criminality in certain areas within a city); 
• AI systems that support human assessments based on objective and verifiable facts linked to a criminal activity, as explored above; 
• Administrative offense prediction, on the basis that their prosecution is less intrusive for individuals’ fundamental rights and freedoms; and 
• Risk assessments of legal entities (unless targeting specific individuals). 

While the Guidelines do not expressly address the issue, it is worth noting that, while certain exemptions may exist for the use of AI technologies in the law enforcement context, the mere fact that such uses occur in the context of determining criminal activity does not absolve a private entity from complying with legal obligations beyond the AI Act, including under the GDPR. In a case that led to a more than €30 million fine imposed by the Dutch AP on Clearview AI in September 2024 under the GDPR, the company argued that they were acting in the interest of potential third-party users of their facial recognition database, in this case overwhelmed law enforcement authorities (paragraph 88 of the Dutch AP’s judgement). The company also identified “responsible organizations charged with protecting society” (paragraph 88), which may include private actors, as justifying the interest of third parties in using their service. 

In assessing whether the interest of third parties in combating crime, tracing victims, and other public duties qualify as legitimate interests, the Dutch AP notes that “such interests do not qualify as a legitimate interest of a third party” within the meaning of Article 6(1)(f) GDPR. The Dutch AP expands that, similarly, Dutch and European regulators cannot rely on legitimate interests under Article 6(1) GDPR for the purposes of exercising their duties of preserving and protecting society-wide interests (paragraph 92). 

With this in mind, caution must be exercised in ensuring a reading of the AI Act’s prohibitions that is contextualized within the broader set of EU rules regulating technology development and deployment. In this sense, the Guidelines could have expanded on Section 5.4 (Interplay with other Union law) by making reference to at least one specific instance in which regulatory authorities, on the basis of already applicable and relevant laws, have interpreted technology uses that directly relate to the prohibition at hand. This may have helped reinforce legal certainty with regard to the applicability and scope of the prohibition by noting instances in which uses not expressly covered by the AI Act are otherwise covered by other EU laws. 

7. Concluding Reflections and Key Takeaways

As Article 5(1)(d) is limited in its scope, it does not entirely prohibit crime prediction or forecasting AI technologies

As explored in the fourth blog post in the series, given that the Article 5(1)(d) prohibition is limited and targeted in its scope, it does not entirely prohibit crime prediction or forecasting AI technologies. Rather, it focuses on prohibiting (individual) risk assessments for the prediction of criminal offences based solely on profiling or personality assessments. The prohibition draws on the logic and legal foundations of general and fundamental rights law in the EU and, in particular, on Article 47 (right to an effective remedy and fair trial) and Article 48 (presumption of innocence and right of defence) of the EU Charter of Fundamental Rights. 

When an AI system does not meet all of the conditions for the prohibition to apply, it will nevertheless be classified as a high-risk AI system

Similar to the analysis in previous blog posts on the AI Act’s prohibitions, we find that when an AI system does not meet all of the conditions for the prohibition to apply, it will be classified as a high-risk AI system. This is reminiscent of the AI Act’s scaled approach to delineating and classifying risk and the close interplay between Articles 5 and 6 of the AI Act. 

The Guidelines note that engaging in crime prediction activities may perpetuate or reinforce biases and erode public trust in law enforcement

Finally, given the particularly sensitive context and nature of applying AI technologies in the area of crime prediction and forecasting, wherein risk assessments can lead to significant legal effects and consequences for individuals, the Guidelines acknowledge that such activities may perpetuate or reinforce biases and erode public trust in law enforcement. 

Red Lines under the EU AI Act: Unpacking Social Scoring as a Prohibited AI Practice 

Blog 3 | Red Lines under the EU AI Act Series 

This blog is the third of a series that explores prohibited AI practices under the EU AI Act and their interplay with existing EU law. You can find the whole series here.

The prohibition of AI-enabled social scoring is among the red lines established by the EU AI Act under its Article 5, targeting practices that assess or classify individuals or groups based on their social behavior or personal traits, leading to unfair treatment, particularly when the information is drawn from multiple unrelated social contexts or when the resulting treatment is disproportionate to the behavior assessed. Notably, the prohibition has a broad scope of application across public and private contexts and is not limited to a specific sector or field. 

The practice of “social scoring” is not uniquely regulated by the AI Act, as it engages well-established notions under the General Data Protection Regulation (GDPR): profiling, purpose limitation and automated decision-making. Therefore, those practices in the same realm that do not meet the high threshold of the social scoring prohibition under the AI Act must in any case comply with the detailed GDPR provisions relevant to them.

As this analysis will show, the “social scoring” prohibition under the AI Act also engages notions of “personalization” in AI, which may be particularly relevant to the current state of AI development, as prior FPF analysis has shown. 

This blog examines the definition and contextual scope of the prohibition of social scoring under Article 5(1)(c) AI Act (Section 1), including its conditions and detailed scenarios (Section 2), as well as the practices that fall outside the scope of the prohibition (Section 3). It then takes a look at how this provision interacts with other areas of EU law, in particular data protection, non-discrimination, and sector-specific frameworks (Section 4). The main takeaways (Section 5) highlight that:

• The AI Act prohibits specific practices of AI-enabled social scoring that lead to detrimental or unfavorable treatment in unrelated social contexts or treatment that is unjustified to the behavior assessed.
• The prohibition applies across both public and private sectors.
• The lawful evaluation and classification practices carried out for legitimate purposes using relevant data and proportionate safeguards, such as creditworthiness assessments, insurance risk scoring or fraud detection systems, remain outside the scope of the prohibition, subject to compliance with relevant provisions of the AI Act and other applicable legislation.

1. Social scoring as a “contextual” prohibited AI practice

EU legislators made the policy choice to expressly ban practices of AI systems that enable social scoring because they considered them incompatible with fundamental rights and European Union values. This results from Recital 31 of the AI Act, which states that such practices “may lead to discriminatory outcomes and the exclusion of certain groups” and can violate individuals’ dignity, privacy, and right to non-discrimination. The European Commission characterized AI systems that allow “social scoring” by governments or companies as a “clear threat to people’s fundamental rights”, noting that these are banned outright. The Guidelines the Commission issued on prohibited practices under the AI Act reiterate this framing and clarify the cumulative elements for the prohibition with practical illustrations.

This rationale was backed by EU data protection authorities (DPAs). In June 2021, the European Data Protection Board (EDPB) and European Data Protection Supervisor (EDPS) welcomed the intention to ban social scoring in their Joint Opinion 5/2021 on the AI Act proposal, warning that “the use of AI for ‘social scoring’… can lead to discrimination and is against the EU fundamental values”​. Since then, the EDPB and national DPAs have continued to develop guidelines around profiling and automated decision-making (ADM), including guidance on legitimate interests (2024) and national tools such as the Dutch DPA’s guidance (AP) on “meaningful human intervention”, which could be relevant when assessing whether an AI-enabled score could fall under the AI Act provisions or Article 22 GDPR, which provides for the right not to be subject to solely automated decision-making. 

According to the Commission Guidelines, the AI Act prohibits social scoring practices if the following cumulative conditions are met:

1. The AI system is placed on the market, put into service, or used.
2. The AI system is intended to evaluate or classify individuals or groups over a certain period of time based on their social behavior or inferred personal or personality characteristics.
3. The social score results in (i) unfavorable treatment in social contexts unrelated to where the data was originally collected and/or (ii) treatment is unjustified or disproportionate to the social behavior or its gravity.

All three conditions must be met simultaneously for Article 5(1)(c) to apply. The prohibition applies to both providers and deployers of AI systems. Of note, the prohibition has been applicable since 2 February 2025, while the supervisory and enforcement provisions related to it have been in force since 2 August 2025. However, no enforcement or regulatory action has been announced so far regarding the social scoring prohibition. 

The prohibition does not extend to all AI-enabled scoring practices. The Guidelines clarify that it targets only unacceptable practices that result in unfair treatment, social control or surveillance. At the same time, the Guidelines note that the prohibition is not meant to affect the “lawful practices that evaluate people for specific purposes that are legitimate and in compliance” with the EU and national law, particularly in the cases where the legislation provides for the types of data that are relevant for the specific evaluation purposes and ensures that any unfavorable or detrimental treatment that results from the practice is justified and proportionate. 

In this context, the Guidelines clarify that sector-specific scoring systems, such as creditworthiness assessments, insurance risk scoring or fraud detection systems, are not prohibited in cases where they are carried out for clearly defined purposes and in accordance with EU or national legislation. 

For example, the credit scoring systems used by financial institutions to assess a borrower’s creditworthiness based on relevant financial data do not fall under the provision of Article 5(1)(c) of the AI Act, provided that they do not result in unjustified or disproportionate treatment or rely on unrelated social context data. Instead, such systems are typically classified as high-risk AI systems under Article 6 and Annex III of the AI Act and must comply with the applicable requirements, including risk management, transparency, human oversight and data governance obligations. 

2. Unpacking how the social scoring prohibition is triggered under the AI Act

2.1 The AI system is intended to evaluate or classify individuals or groups over a certain period of time based on their social behavior or inferred personal or personality characteristics

Article 5(1)(c) AI Act explicitly prohibits “the placing on the market, putting into service or use of an AI system for the evaluation or classification of natural persons or groups of persons over a certain period of time based on their social behavior or known, inferred or predicted personal or personality characteristics”. The Guidelines clarify that this condition is fulfilled where an AI system assigns individuals or groups scores based on their social behavior or personal or personality characteristics. These scores could take different forms, such as numerical values, rankings or labels. This prohibition applies broadly across public and private sectors and concerns only natural persons or groups of natural persons, excluding thus legal entities. 

The Guidelines differentiate between “evaluation” and “classification” as two distinct but related concepts within the scope of Article 5(1)(c) AI Act. “Evaluation” refers to an assessment or judgment about a person or group of persons, and “classification” has a broader scope and includes categorizing individuals or groups based on certain characteristics or behavioral patterns. “Classification” does not necessarily involve an explicit judgement or assessment but may still fall within the scope of the prohibition in cases where individuals are assigned scores, rankings or labels based on their behavior or personal or personality characteristics. 

In addition, the Guidelines note that the term “evaluation” is closely linked to “profiling” as defined by EU data protection law, namely in Article 4(4) GDPR, and as referred to in Article 22 GDPR and Article 11 Law Enforcement Directive. Profiling refers to the processing of personal data to evaluate personal aspects of an individual, in particular to analyse or predict behavior about their ability to perform tasks, interests, likely behavior, or future actions. 

Interestingly to note is that the Guidelines opted for the wording of the Article 29 Working Party Guidelines on Automated Decision-Making and Profiling, adopted in 2017, when referring to profiling, reflecting a broader, functional understanding of profiling that encompasses AI systems assigning behavioral scores or predictive assessments, and therefore clarifying that that the scope of the prohibition is not narrowly limited to specific technical forms of automated processing but extends to AI-enabled evaluation and categorization of persons based on their characteristics or behavior.  

The Guidelines note that although Article 5(1)(c) AI Act does not explicitly reference profiling under the GDPR as defined in Article 4(4), the act of profiling may still fall under the prohibition when AI systems process personal data to assess individuals.

To illustrate the link between profiling and social scoring, the Guidelines refer to the SCHUFA I judgment (Case C-634/21), in which the CJEU examined a creditworthiness scoring system used in Germany. In that case, the score generated by the computer programme consisted of a probability value estimating an individual’s ability to meet payment commitments. The CJEU found that this score was based on certain personal characteristics and involved establishing a prognosis concerning the likelihood of future behavior, such as the repayment of a loan. The scoring process relied on assigning individuals to groups of persons with comparable characteristics and using the behavior of those groups to predict the individuals’ future conduct. 

The CJEU held that this activity constitutes “profiling” within the meaning of Article 4(4) GDPR and it held that the automated establishment of that probability value can constitute ADM under Article 22(1) GDPR where a third party draws strongly on it to decide whether to enter into, implement, or terminate a contractual relationship. The Guidelines clarify that such scoring may also constitute an “evaluation” of individuals based on their personal characteristics within the meaning of Article 5(1)(c) AI Act and will be prohibited if carried out through AI systems, provided that all the other conditions are fulfilled. 

Additionally, even if not referenced in the Guidelines, the CJEU judgment in CK v Dun & Bradstreet Austria (CaseC-203/22) further clarified the legal framework governing profiling and scoring systems. In that case, the CJEU held that the right of access under Article 15(1)(h) GDPR requires controllers to provide data subjects with meaningful information about the logic involved in automated decision-making, including the procedures and principles used to generate a score.

2.1.1 The prohibition requires evaluations to rely on data gathered over a period of time, ensuring that one-off assessments cannot circumvent it.

The prohibition in Article 5(1)(c) AI Act applies only where the evaluation or classification is based on data collected over “a certain period of time”. The Guidelines clarify that this temporal requirement indicates that the assessment should not be limited to a one-time rating or grading based solely on data from a single, isolated context. This condition must be assessed in light of all the circumstances of the case to avoid the circumvention of the scope of the prohibition. 

To illustrate this, the Guidelines refer to a scenario involving a migration or asylum authority that deploys a partly automated surveillance system in refugee camps using cameras and motion sensors. If such a system analyzes behavioral data collected over a period of time and evaluates individuals to determine, for example, if they may attempt to abscond, this would mean that the temporal condition is met and may fall within the scope of the prohibition, provided that all the other conditions are also met. 

2.1.2 The provision prohibits AI evaluations based on social behavior or known, inferred, or predicted personal or personality characteristics

The evaluation or clarification of individuals based on AI-enabled processing in relation to either (i) their social behavior or (ii) their known, inferred or predicted personal and personality characteristics, or both, is prohibited under the AI Act provision. This data may be directly provided by the individuals, indirectly collected through surveillance, obtained from third parties, or inferred from other information. 

The Guidelines explain that “social behavior” is a broad concept that encompasses a wide range of actions, habits, and interactions within society. This may include behavior in private and social contexts, such as participation in cultural or voluntary activities, as well as behavior in business or institutional contexts, including payment of debts, use of services and interactions with public authorities or private entities. This type of data is often collected from multiple sources and combined, sometimes involving extensive monitoring or tracking of individuals. 

The prohibition also applies in cases where “personal or personality characteristics” may involve specific social behavioral aspects. The Guidelines note that personal characteristics may include a wide range of information relating to an individual, such as race, ethnicity, income, profession, other legal status, location, level of debt, and so on. Personality characteristics should, in principle, be interpreted as personal characteristics, but may also involve the creation of specific profiles of individuals as “personalities”. These characteristics may indicate a judgment, made by the individuals themselves, observed by others, or generated by AI systems.

The Guidelines distinguish between three types of characteristics used in scoring systems: (i) “known characteristics” (verifiable inputs provided to the AI systems), (ii) “inferred characteristics” (conclusions drawn from existing data, usually by AI systems), and (iii) “predicted characteristics” (estimates based on patterns, often with some degree of inaccuracy). These distinctions are relevant because inferred and predicted characteristics may be less accurate and more opaque, raising concerns about fairness and transparency in AI-driven scoring systems.

2.2. The social score must lead to detrimental or unfavorable treatment in unrelated social contexts and/or unjustified or disproportionate treatment to the gravity of the social behavior

2.2.1. Causal link between the social score and the treatment

For the prohibition to apply, the social scoring created by or with the assistance of an AI system must lead to detrimental or unfavorable treatment of the evaluated person or group of persons. There must be a causal link between the score and the resulting treatment, such that the treatment is the consequence of the score. This causal link may also exist where harmful consequences have not yet materialised, provided that the AI system is capable or intended to produce such outcomes. 

The Guidelines further note that the AI-enabled score does not need to be the sole cause of the detrimental or unfavorable treatment. The prohibition also covers situations where AI-enabled scoring is combined with human assessment, as long as the AI output plays a sufficiently significant role in the decision. The prohibition is still applicable if the score is obtained by an organization and produced by another (e.g., a public authority using a creditworthiness score from a private company).

2.2.2. Detrimental or unfavorable treatment in unrelated social contexts and/or unjustified or disproportionate treatment

For the prohibition to apply, the social score must or could result in detrimental or unfavorable treatment of the evaluated person or group. This treatment could occur either in (i) a different social context from where the data was originally generated or collected, and/or (ii) in a manner that is unjustified or disproportionate to the social behavior or its gravity. 

The Guidelines emphasize that a case-by-case analysis is required to determine if at least one of these conditions is fulfilled, as many AI-enabled scoring practices may fall outside the scope of the prohibition. 

The Guidelines further clarify that “unfavorable treatment” refers to situations where, as a result of the scoring, a person or a group is treated less favorably compared to others,  even where no specific harm or damage is demonstrated. By contrast, “detrimental treatment” requires that the individual or group suffer harm or disadvantage as a result of the scoring. Such treatment may also be considered discriminatory under EU non-discrimination law and may include the exclusion of certain persons or groups, although it is not a necessary condition for the prohibition to apply. As the Guidelines highlight, the treatment covered by the Article 5(1)(c) could go beyond the EU non-discrimination law. 

The Guidelines further detail the scenarios described under Article 5(1)(c) AI Act: 

a. Detrimental or unfavorable treatment in unrelated social contexts, such as when authorities use information like nationality, internet activity, or health status from one area to evaluate people in another

The first scenario regards the situations where the detrimental or unfavorable treatment resulting from a social score occurs in social contexts unrelated to the one in which the data were originally generated or collected. The Guidelines clarify that this condition requires both that the data used for scoring originates from unrelated social contexts and that the resulting score leads to detrimental or unfavorable treatment in a different context. 

This scenario typically involves AI systems processing data relating to the individuals’ social behavior or personal characteristics that were generated or collected in contexts unrelated to the purpose of the scoring, and used by the AI system for the scoring of the individual(s) without an apparent connection to the purpose of the evaluation or classification or in a way that leads to the generalised surveillance of individuals or groups. 

As the Guidelines note, in most situations, these kinds of practices occur against the reasonable expectations of the individuals concerned and may also violate EU law and other applicable rules. To determine if this condition is met, a case-by-case assessment is required, evaluating the purpose of the evaluation and the context in which the data was collected and generated.

There is a clear link between this scenario and the purpose limitation principle under Article 5(1)(b) GDPR, which provides that personal data must be collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes. When the personal data collected in one context is used to generate social scores in an unrelated context, such a practice may violate this principle, particularly where the new use of data was not foreseeable to the individual or where the new processing lacks a sufficient legal basis of connection to the original purpose. 

The Guidelines provide several examples of prohibited practices under this first scenario, highlighting the following situations where:

• AI predictive tools are being used by national tax authorities to select specific taxpayers for inspections based on social habits, internet connections, and other unrelated data; or
• AI systems are being used by social welfare agencies to estimate fraud risk based on characteristics collected or inferred from unrelated contexts (e.g., the nationality or ethnicity of the spouse, internet connection, social media activity, workplace performance); or
• AI systems are used by public labor agencies to score unemployed individuals based on personal data such as age, education, as well as inferred or collected variables from contexts and data unrelated to the purpose of the evaluation (e.g., health conditions, marital status). These practices can be distinguished from lawful practices.

On another note, national developments illustrate the risks associated with AI-enabled social scoring and classification systems that rely on data from unrelated contexts. In the Netherlands, the Dutch Tax and Customs Administration used the “Fraude Signaleringsvoorziening” (FSV – Fraud Signaling Provision), a system used to record and assess fraud signals based on personal data collected from multiple sources, including internal systems, other public authorities, and third parties. 

The Dutch AP found that the processing of personal data in the FSV was unlawful. The AP found that the processing had no legal basis and that the purpose of the processing was not sufficiently defined. These findings were explored in the Case 202401528/1/A3. The Council of State held that the letter of the Ministry of Finance informing the individual that they were not eligible for financial compensation following his registration in the FSV was a decision subject to judicial review. It is relevant to note that this case was decided under administrative and data protection law and did not concern the application of the AI Act, yet it highlights the risks associated with systems that record and use personal data to evaluate and classify individuals which may influence their treatment by public authorities. 

b. Situations where the detrimental or unfavorable treatment is disproportionate to the actual behavior

To this extent, the Guidelines provide a list of unjustified or disproportionate treatment that falls under both Article 5(1)(c)(i) and Article 5(1)(c)(ii):

• AI systems used by tax authorities to profile child benefit recipients and assign fraud risk categories based on criteria such as low income or nationality, that could lead to unjust, discriminatory, and detrimental treatment and severe financial hardship. The Guidelines reference a similar concern that arose in the Netherlands, where automated risk profiling systems used in the administration of childcare benefits contributed to disproportionate enforcement measures, and with the SyRI (Systeem Risico Indicatie) system being subsequently found unlawful. 
• AI systems used by public authorities to control fraud in the student housing grant process by considering indicators such as internet connections, family status or the level of education of beneficiaries as distinguishing factors for fraud risk, which do not appear relevant or justified for the purpose of the evaluation.
• An AI system introduced by the government that is used to monitor and rate citizens’ behavior across various aspects of life, including social interactions, online activities, purchasing habits, and punctuality in paying bills.  Individuals with lower scores may face restricted access to public services, financial disadvantages and certain limitations, such as in employment, housing or travel. Such systems could lead to excessive surveillance and detrimental treatment in unrelated contexts while also imposing excessive penalties for minor infractions.

The Guidelines note that the prohibition may also cover cases when preferential treatment is granted to certain individuals or groups of people (e.g., in cases of support employment programs, (de-)prioritization for housing or resettlement).

2.3. AI-enabled social scoring is prohibited regardless of whether the system or the score are provided or used by public or private persons

Article 5(1)(c) prohibits AI-enabled social scoring practices regardless of whether the AI systems or the resulting score are provided or used by public or private persons. While scoring practices in the public sector may have particularly significant consequences due to the individuals’ dependence on public services and the imbalance of power between the authorities and the individuals, similarly harmful consequences may also happen in the private sector. 

For instance, as the Guidelines exemplify, an insurance company may use an AI system to analyze spending patterns and the financial data obtained from a bank, which are unrelated to the assessment of eligibility for life insurance, in order to determine if it should refuse the insurance or impose higher premiums to individuals or groups of individuals. In another example, a private credit agency may use an AI system to determine an individual’s creditworthiness for obtaining a housing loan based on unrelated personal characteristics. 

In the case of verifications conducted by the competent market surveillance authorities, the responsibility lies with the providers and deployers of the AI systems, within their respective obligations, to demonstrate that their AI systems are legitimate, transparent and only process context-related data. They must also ensure that the systems operate as intended and that any resulting detrimental or unfavorable treatment is justified and proportionate to the social behavior assessed. 

The Guidelines also note that compliance with the applicable requirements, including those concerning high-risk AI, may help ensure that the evaluation and classification practices remain lawful and do not constitute prohibited social scoring. 

3. What falls outside the scope of the prohibition?

The AI Act makes room for carefully tailored exceptions to the social scoring prohibition. It acknowledges several scenarios where assessing individuals via algorithms is lawful and even necessary, provided that such an assessment is conducted in a targeted and proportionate manner.

First to note is that the prohibition applies only to the scoring of natural persons or groups of natural persons. Scoring of legal entities is, in principle, excluded in situations where the evaluation is not based on the social behavior or personal or personality characteristics of individuals. However, as the Guidelines highlight, in the situations where a score attributed to a legal entity aggregates the evaluation of natural persons and directly affects those individuals, the practice may fall within the scope of this prohibition. 

Secondly, the Guidelines distinguish AI-enabled social scoring as a “probabilistic value” and prognosis from individual ratings provided by users (for example, the ratings of drivers or service providers on online platforms). These fall outside the prohibition unless they are combined with other data and analyzed by an AI system to evaluate or classify individuals that fulfill the conditions of Article 5(1)(c).   

Finally, Recital 31 AI Act and the Guidelines clarify that lawful evaluation practices conducted for a specific purpose in compliance with EU and national law remain outside the scope of the prohibition. Recital 31 reiterates that this prohibition “should not affect lawful evaluation practices of natural persons carried out for a specific purpose in accordance with Union or national law.” 

The Guidelines provide additional examples of legitimate scoring practices that are out of scope, including: 

• financial creditworthiness assessments based on relevant financial and economic data, in compliance with consumer protection and financial services law;
• fraud detection systems relying on relevant transactional behavior and metadata in the context of the service provided;
• insurance risk assessments based on telematics data reflecting driving behavior, where premium adjustments are proportionate to the risk;
• online platforms evaluating users’ behavior for safety or service quality purposes, based on relevant data for the given context, when the evaluation does not result in disproportionate detrimental treatment;
• AI-enabled targeted commercial advertising based on users’ preferences, if it complies with the applicable consumer protection, data protection, and digital services law;
• AI systems used for legitimate purposes such as medical diagnosis, fraud prevention, law enforcement, or migration procedures, where the data used is relevant and the resulting treatment is justified and proportionate.

4. Interplay with other EU laws, including consumer protection, data protection, non-discrimination, and sector-specific provisions such as credit, banking, and anti-money laundering

Providers and deployers must assess whether other EU or national laws apply to any particular AI scoring system used in their activities, particularly if more specific legislation strictly defines the type of data considered relevant and necessary for specific evaluation purposes and ensures fair and justified treatment.

AI-enabled social scoring in business-to-consumer relations may also require the application of EU consumer protection laws, such as Directive 2005/29/EC on unfair business-to-consumer commercial practices (the “UCPD”), if it misleads consumers or distorts their economic behavior. The practices which may amount to misleading consumers or distorting their behavior through AI uses or in AI contexts is further explored in Blog 2 of this series, accessible here. 

Social scoring may also engage specific data protection rules as encoded in the GDPR, particularly those regarding the legal ground for processing, data protection principles, and other obligations, including the rules on solely automated individual decision-making. AI-enabled social scoring that results in discrimination based on protected characteristics (e.g., age, race, and religion) would also fall under EU non-discrimination law.

Finally, certain sector-specific rules may be applicable. For example, the Consumer Credit Directive (CCD) prohibits the use of special categories of personal data in these evaluations and the obtaining of data from social networks. Additionally, guidelines from the European Banking Authority provide further specifications on the relevant information for the purpose of creditworthiness assessments, which are relevant to determine whether a practice falls under the scope of Article 5(1)(c). AI systems used for anti-money laundering and counter-terrorism financing purposes must also comply with the applicable EU legislation.

5. Closing reflections and key takeaways

The AI Act prohibits specific practices of AI-enabled social scoring, not scoring in general

Article 5(1)(c) of the AI Act does not prohibit scoring as such, but rather the placing on the market, putting into service or use of AI systems for social scoring practices that meet the conditions set out in the provision. The Guidelines repeatedly focus on the concrete use of the AI system and the effects of the resulting score, rather than on the existence of the scoring mechanisms alone. In particular, the prohibition is determined only when all conditions are cumulatively met, including the evaluation or classification over a certain period of time and the link to detrimental or unfavorable treatment in unrelated social contexts and/or unjustified or disproportionate treatment. 

Public and private uses are equally in scope, with shared accountability across the value chain

The Guidelines clarify that unacceptable AI-enabled social scoring is prohibited regardless of whether the system or score is provided or used by public or private persons. They also place practical weight on accountability: in case of verifications conducted by the competent market surveillance authorities, both providers and deployers, each within their responsibilities, must be able to demonstrate legitimacy and justification, including transparency about system functioning, data types and sources, and the use of only data related to the social context in which the score is used, as well as proportionality of any resulting detrimental or unfavorable treatment.

Out of scope does not mean being exempted from scrutiny.

Recital 31 of the AI Act and the Guidelines clarify that the prohibition is not intended to affect the lawful evaluation practices carried out for a specific purpose in accordance with the existing legislation in place. It depends on several criteria, as examined throughout this blog, if a scoring practice falls outside the scope of the prohibition, including whether the evaluation serves a legitimate and clearly defined purpose, whether the data used is relevant and necessary for that purpose, whether the scoring occurs within the same social context in which the data was collected, and whether any resulting detrimental or unfavorable treatment is justified and proportionate to the behaviour assessed.

As the Guidelines emphasise, this assessment is contextual. The same scoring practice may fall outside the scope of the prohibition in one situation, for example, where it is used for a lawful and proportionate creditworthiness assessment based on relevant financial data, but may fall within the scope of Article 5(1)(c) where it relies on unrelated data, produces disproportionate consequences, or is used in a different social context. This reinforces that compliance depends not only on the existence of scoring systems, but on how they are designed, the types of data they process, and the purposes for which they are used.

Red Lines under the EU AI Act: Understanding Manipulative Techniques and the Exploitation of Vulnerabilities

Blog 2 | Red Lines under the EU AI Act Series  

This blog is the second of a series that explores prohibited AI practices under the EU AI Act and their interplay with existing EU law. You can read the first episode here and find the whole series here.

Harmful manipulation and deception through AI systems and exploiting certain human vulnerabilities are the first on the list of prohibited practices under Article 5 of the EU AI Act. It is apparent that the underlying goal of these provisions is to ensure that individuals maintain their ability to make autonomous decisions. This is especially important when considering that one of the goals of the AI Act is “to promote the uptake of human-centric and trustworthy AI”, while ensuring respect for safety, health and fundamental rights (see Recital 1, AI Act).

These first two prohibited practices listed in Article 5(1) specifically concern AI systems that could undermine individual autonomy and well-being through:

• Deploying subliminal, purposefully manipulative or deceptive techniques that are significantly harmful and materially influence the behavior of natural persons or group(s) of persons (Article 5(1)(a) AI Act). 
• Exploiting vulnerabilities due to age, disability, or a specific socio-economic situation (Article 5(1)(b) AI Act). 

It is notable, though, that manipulative and deceptive practices based on processing of personal data, and those that specifically occur through online platforms, are already strictly regulated by the EU’s General Data Protection Regulation (GDPR) and Digital Services Act (DSA). Specifically, the GDPR intervenes through obligations like ensuring fairness (Article 6(1)(a)) and data protection by design (Article 25) for all processing of personal data, regardless of whether that processing occurs through AI or not, while the DSA includes a prohibition for providers of online platforms to design, organise or operate their online interfaces in a way that deceives or manipulates their users (Article 25). While the relationship between the DSA obligations and those in the GDPR related to manipulative design is clear, with the DSA only being applicable where the GDPR does not apply, their relationship with the AI Act prohibitions on manipulative techniques and exploiting vulnerabilities requires further guidelines and clarification. 

The Guidelines published by the European Commission to support compliance with Article 5 AI Act highlight that the two prohibitions aim to protect individuals from being reduced to “mere tools for achieving certain ends”, and to protect those who are most vulnerable or susceptible to manipulation and exploitation. Significantly, the Guidelines analyze these two prohibitions together, making it obvious that there is a nexus between them. In this sense, according to the Guidelines, they are both designed to support and protect the right to human dignity, as enshrined in the EU Charter of Fundamental Rights. 

This second blog in the “Red Lines” series provides an analysis of the scope and content of the Article 5(1)(a) prohibition in Section 2, focusing on the definitions of subliminal, manipulative, and deceptive techniques. Section 3 goes on to explore the notion of vulnerability contained in the Article 5(1)(b) prohibition and in the Guidelines, while Section 4 notes the possible interplay between the two prohibitions. Section 5 takes a broader view by highlighting the interplay between the prohibitions and other EU laws, including the GDPR and the DSA, before the conclusions in Section 6 note the following key takeaways:  

• There is a high threshold, including some highly subjective elements, for fulfilling the cumulative conditions required for falling under the prohibitions related to manipulative techniques and the exploitation of vulnerabilities.  
• The prohibition for AI practices that include manipulative techniques applies even when there is no intention of manipulation.
• Compliance with other laws, including the GDPR and DSA, in relation to these two prohibitions, can help demonstrate compliance with the AI Act.

2. Understanding harmful manipulation and deception as a prohibited practice under the AI Act

Article 5(1)(a) AI Act targets those cases in which AI practices subtly manipulate human action without the individual noticing. The final text of the AI Act for this provision underwent several changes from the European Commission’s initial proposal, broadening its scope and clarifying some elements.

Following amendments submitted by the European Parliament, the final text sought to add manipulative and deceptive techniques to the initial “subliminal techniques”, and broaden the scope of the ban to cover not only harmful effects on individuals but also on groups, in order to prevent discriminatory effects. Another modification of the initial proposal added that the prohibition should not be limited to cases where the systems are intended to modify behaviour, but also to cases where the modification of the behaviour that led to a significant harm is a mere “effect”, even when it was not the intended objective of the AI practice in question.

2.1. Defining subliminal, purposefully manipulative or deceptive techniques

The Guidelines list four cumulative conditions to be fulfilled in order for this prohibition to be applicable, even though, in their analysis, they also include a fifth one. 

1. The practice must constitute the ‘placing on the market’, the ‘putting into service’, or the ‘use’ of an AI system. 
2. The AI system must deploy subliminal (beyond a person’s consciousness), purposefully manipulative, or deceptive techniques. 
3. The techniques deployed by the AI system should have the objective or the effect of materially distorting the behavior of a person or a group of persons. The distortion must appreciably impair their ability to make an informed decision, resulting in a decision that the person or the group of persons would not have otherwise made. 
4. The distorted behavior must cause or be reasonably likely to cause significant harm to that person, another person, or a group of persons. 

The four conditions must be met cumulatively for the prohibition to be applicable. Additionally, according to the Guidelines, there must be a plausible causal link between the techniques used, the significant change in the person’s behavior, and the significant harm that resulted or is likely to result from that behavior. While the causal link is not listed among the four conditions, it is analyzed further down in the Guidelines as a self-standing, additional condition to be met, and it should be considered as the fifth point on this list.

The prohibition applies to both providers and deployers of AI systems who, each within their own responsibilities, have an obligation not to place on the market, put into service, or use AI systems that impair an individual’s ability to make an informed decision on the basis of subliminal, manipulative or deceptive techniques. 

The Guidelines note that while the AI Act does not directly define “subliminal techniques”, the text of Article 5(1)(a) and Recital 29 imply that such techniques are inherently covert in that they operate beyond the threshold of conscious awareness, capable of influencing decisions by bypassing a person’s rational defences. However, the Recital also explains that the prohibition covers even those cases where the person is aware that the techniques used are subliminal, but cannot resist their effect. The Guidelines clarify that the prohibition on the use of subliminal techniques is not limited to those practices that influence decision-making only, but rather, it also covers those techniques that influence a person’s value- and opinion-formation, a criterion that seems highly subjective and might raise difficulties in applying it in practice. A relevant example could be an AI system facilitating deepfakes on matters of public interest when spread on platforms without appropriate labeling and in violation of the transparency obligations in place (Article 50 AI Act). Their use could be considered prohibited. 

Subliminal techniques can use audio, visual, or tactile stimuli that are too brief or subtle to be noticed. The following techniques are among several suggested in the Guidelines (p. 20) as potentially triggering a ban, if the other conditions are also met: 

• Visual Subliminal Messages: an AI system may show or embed images or text flashed briefly during video playback which are technically visible, but flashed too quickly for the conscious mind to register, while still being capable of influencing attitudes or behaviours. 
• Auditory Subliminal Messages: an AI system may deploy sounds or verbal messages at low volumes or masked by other sounds, influencing the listener without conscious awareness. These sounds are still technically within the range of hearing, but are not consciously noticed by the listener due to their subtlety or masking by other audio. 
• Embedded Images: an AI system may hide images within other visual content that are not consciously perceived, but may still be processed by the brain and influence behaviour. 

The Guidelines, referring to Recital 29 AI Act, specify that the development of new AI technologies, like neurotechnology, brain-computer interfaces, virtual reality, or even “dream-hacking” increases the potential for sophisticated subliminal manipulation and its ability to influence human behavior subconsciously. 

While “purposefully manipulative techniques” are similarly not defined by the AI Act, the Guidelines fill this gap by noting that such techniques exploit cognitive biases, psychological vulnerabilities, or situational factors that make individuals more susceptible to influence. This provision covers cases where individuals are aware of the presence of a manipulative technique but cannot resist its effect and, as a result, are pushed into decisions or behaviours that they would not have otherwise made (Recital 29). 

Recital 29 of the AI Act also refers to techniques that deceive or nudge individuals “in a way that subverts and impairs their autonomy, decision-making and free choices.” A direct comparison can be made with the DSA which, inter alia, prohibits providers of online platforms from deceiving or nudging recipients of their service and from distorting or impairing their autonomy, decision-making and free choice (Article 25 and Recital 67 DSA). 

The manipulative capability of the technique is a key factor in determining its effect. Indeed, the Guidelines clarify the AI system could manipulate individuals without the provider or deployer intendingto cause harm. However, the provision would still apply, unless the result is incidental and appropriate preventive and mitigating measures were taken. This is consistent with the overall logic and scope of the AI Act’s prohibitions, as explored in Blog 1 of this series, in which deployers have a responsibility to reasonably foresee harms that may arise from the misuse of an AI system. 

Deceptive techniques are techniques that subvert or impair a person’s autonomy, decision-making, or free choice in ways of which the person is not consciously aware or, where they are aware, can still be deceived or cannot control or resist them. In the case of deepfakes, for example, Article 50 of the AI Act requires that the deployer disclose their nature. If this transparency is absent and the deepfake is used to deceive individuals, it could fall under prohibited uses. Notably, according to the Guidelines, this provision applies even if the deception occurs without the intent of the provider or deployer. However, the Guidelines also clarify that a generative AI system that produces misleading information due to hallucinations—provided the provider has communicated this possibility—does not constitute a prohibited practice.

2.2 To fall under the AI Act’s prohibited practices, manipulative techniques have to have the “objective or effect of materially distorting the behavior of a person or a group of persons” 

The subliminal, manipulative and deceptive techniques must have the objective or the effect of materially distorting the behavior of a person or a group of persons. Material distortion involves a degree of coercion, manipulation, or deception that goes beyond lawful persuasion. The Guidelines note that material distortion implies a substantial impact on a person’s behavior, such that their decision-making and free choice are undermined, rather than a minor influence. 

When interpreting “material distortion of behaviour” under Directive 2005/29/EC (the Unfair Commercial Practices Directive or ‘UCPD’), it is sufficient to demonstrate that a commercial practice is likely (i.e., capable) of influencing an average consumer’s transactional decision; there is no need to prove that a consumer’s economic behavior has been distorted. However, this requires a case-by-case assessment, considering specific facts and circumstances. Additionally, the average consumer’s perspective may not be helpful in situations where an AI system delivers highly personalized messages designed to manipulate individual behavior.

The AI Act adopts a similar understanding of “material distortion” as the UCPD, where the prohibition applies even if the material distortion of a person’s behavior occurs without the intent of the provider or deployer. The text specifies that the prohibition covers not only cases in which behavior modification is the object of the system (like in the original text of the European Commission’s proposal) but also those in which it is the mere “effect”. This change, as introduced into the final text, amplifies protection against the possible distorting effects of manipulative AI systems. 

2.3 The subliminal, manipulative and deceptive techniques must be “reasonably likely to cause significant harm” 

The Guidelines define harm under three broad categories:

• Financial and economic harm: which can include financial loss, exclusion and economic instability (an addition by the European Parliament during the AI Act negotiations).
• Physical: any injury or damage to a person’s life, health, and material damage to property (e.g., an AI chatbot promotes self-harm to users);
• Psychological: harm that exploits cognitive and emotional vulnerabilities, encompassing adverse effects on a person’s mental health and psychological and emotional well-being; 

However, the harm must be significant for the prohibition to apply. The determination of ‘significant harm’ is fact-specific, requiring careful consideration of each case’s circumstances and a case-by-case assessment. Still, the individual effects should always be material and significant in each case. According to the Guidelines, the assessment of the significance of the harm takes into consideration several factors:

• The severity of the harm;
• Context and cumulative effects;
• Scale and intensity;
• Affected persons’ vulnerability;
• Duration and reversibility;

When assessing harm, the Guidelines suggest that a comprehensive approach should be taken, which considers both the possible immediate and direct harms that are associated with AI systems that deploy subliminal, deceptive, or manipulative techniques. 

The last requirement for identifying a prohibited practice is determining the likelihood of a causal link between the manipulative technique and the distorting behavior. In that regard, to not fall in the category of prohibited practices, providers and deployers are suggested to take appropriate measures such as:

• Transparency and individual autonomy: integrate appropriate user control and safeguard measures to ensure that the system is not deceptive and operates within the boundaries of lawful persuasion; 
• Compliance with relevant legislation: which indicates that the practice does not constitute a purposefully manipulative or deceptive practice; 
• State-of-the-art practices and industry standards: which can help preempt and mitigate significant unintended harms. 

It is worth reminding that although the concept of significant harm is very similar to the one of “significant effect” that we encounter within Article 22 GDPR on automated decision-making (ADM), they do not overlap perfectly, with the latter providing for a broader interpretation than the former (see here FPF’s Report on ADM case law). For example, profiling through ADM for political targeting could have a significant effect on citizens but not result in significant harm. 

Not all forms of manipulation fall within the AI Act’s scope. Many persuasive techniques commonly used in advertising are legitimate because they operate transparently and respect individual autonomy. The Guidelines suggest that if an AI system appeals to emotions but remains transparent and provides accurate information, it falls outside the law’s scope.

Additionally, compliance with regulations like the GDPR helps providers and deployers demonstrate that transparency, fairness, and respect for individual rights and autonomy are upheld.

Furthermore, manipulation may be acceptable in some cases if it does not result in significant harm. For instance, in an example, the Guidelines provide – an online music platform might use an emotion recognition system to detect users’ moods and recommend songs that align with their emotions while avoiding excessive exposure to depressive content.

3. The exploitation of vulnerabilities, particularly those due to age, disability or socio-economic status, as prohibited AI practice

Cases in which an AI system exploits the vulnerabilities of a single person or a specific group with the objective of distorting their behavior are designated as prohibited AI practices under Article 5(1)(b) AI Act.

There are four cumulative conditions to be fulfilled for the application of Article 5(1)(b):

1. The distorted behavior must cause or be reasonably likely to cause significant harm to that person, another person, or a group of persons. 
2. The practice must constitute the ‘placing on the market’, the ‘putting into service’, or the ‘use’ of an AI system. 
3. The AI system must exploit vulnerabilities due to age, disability, or socio-economic situations. 
4. The exploitation enabled by the AI system must have the objective or the effect of materially distorting a person’s behavior or a group of persons. 

3.1. Exploitation of vulnerabilities due to age, disability, or a specific socio-economic situation

While vulnerability is not directly defined by the AI Act, according to the Guidelines, the concept covers a wide range of categories, including cognitive, emotional, physical, and other forms of susceptibility that may impact an individual’s or group’s ability to make informed decisions or influence their behavior. 

However, under the AI Act’s prohibited practices, the exploitation of vulnerabilities is only relevant if it involves individuals who are vulnerable due to their age, disability, or socio-economic circumstances. It is worth noting that a reference to an individual’s socio-economic situation was included in the final text of the AI Act after the amendments submitted by the European Parliament, which led to a wider scope of the Article 5(1)(b) prohibition in the final text, as compared to the initial European Commission proposal. 

Exploiting other categories of vulnerabilities than those expressly mentioned falls outside the scope of the Article 5(1)(b) prohibition. The Guidelines note that age, disability, or socio-economic vulnerabilities may, in principle, lead to a limited capacity to recognize or resist manipulative AI practices. The prohibition aims to prevent the exploitation of cognitive limitations stemming from age or health conditions. However, socio-economic status can also reduce an individual’s ability to recognize deceptive practices and may intersect with other discriminatory factors, such as belonging to an ethnic, racial, or religious minority group.

The Guidelines share a number of examples in cases of exploitation of vulnerable people based on their age that fall under prohibited practices, including: 

• An AI-powered toy designed to interact with children that keeps them interested in interactions with the toy by encouraging them to complete increasingly risky challenges;
• An AI system used to target older people with deceptive personalized offers or scams. 

In the case of exploitation of vulnerable people based on disabilities, the Guidelines include the example mentioned of a therapeutic chatbot aimed to provide mental health support and coping strategies to persons with cognitive disabilities, which can exploit their limited intellectual capacities to influence them to buy expensive medical products. 

When the exploitation concerns vulnerable people based on their socio-economic situation, an example mentioned is an AI-predictive algorithm that could be used to target people who live in low-income post-codes with advertisements for predatory financial products. 

3.2. For the Article 5(1)(b) prohibition to apply, AI practices have to materially distort behavior and be reasonably likely to cause significant harm 

As previously noted, a substantial impact is required to fall within the scope, even though intention is not a necessary element, as the provision also covers merely the effect (see Section 2.3). Similarly to fulfilling the conditions for Article 5(1)(a), as explored above, the AI practice has to be reasonably likely to cause significant harm. It is worth mentioning that the harms in this case may be particularly severe and multifaceted due to the increased susceptibility of the vulnerable group in question. Risks of harm that might be deemed acceptable for adults are often considered unacceptable for children and other vulnerable groups.

4. Areas of interplay between the two prohibitions, and between the prohibitions and other EU laws, including the UCPD, GDPR, and DSA

4.1. Tiered approach to the interplay between Articles 5(1)(a) and (b) 

Where the Article 5(1)(a) prohibition covers mainly the use of subliminal and manipulative techniques, Article 5(1)(b) is focused on the targets of AI exploitation, particularly individuals considered vulnerable due to age, disability or socio-economic circumstances. 

However, there may be instances where both Articles seem applicable. In such cases, examining the predominant aspect of the exploitation is essential. If the exploitation does not explicitly relate to one of the vulnerable groups previously discussed, Article 5(1)(a) applies, taking into consideration that it also covers the exploitation of vulnerabilities in groups outside those listed in Article 5(1)(b). When the exploitation specifically targets the groups identified in Article 5(1)(b), then the practice falls under this latter prohibition.

4.2. Interplay with the GDPR obligations to ensure fairness and data protection by design

The protection of individuals from manipulative processes is also covered in various other European laws, including the GDPR. Under the GDPR, the principle of fairness—enshrined in Article 5(1)(a)—acts as an overarching safeguard ensuring that personal data is not processed in a manner that is unjustifiably detrimental, unlawfully discriminatory, unexpected, or misleading to the data subject. Information and choices about data processing must be presented in an objective and neutral way, strictly avoiding any deceptive, manipulative language or design choices. In fact, the European Data Protection Board (EDPB) explicitly identifies the use of “dark patterns” and “nudging” as violations of this fairness mandate, as these techniques subconsciously manipulate data subjects into making decisions that negatively impact the protection of their personal data. 

In its Guidelines 4/2019 on Data Protection by Design and by Default, the EDPB emphasizes that controllers must incorporate fairness into their system architectures from the outset, proactively recognizing power imbalances and granting users the highest degree of autonomy over their data. This means choices to consent to or abstain from data sharing must be equally visible, and platforms cannot use invasive default options or deceptive interfaces to lock users into unfair processing. 

The profound risks of such subliminal and deceptive techniques are illustrated in the EDPB’s Binding Decision 2/2023 and the Irish Data Protection Commission’s corresponding final decision regarding TikTok. In these rulings, the authorities found that TikTok infringed the principle of fairness by utilizing deceptive design patterns to nudge child users toward public-by-default settings. TikTok has challenged these findings in a case now pending at the CJEU.

Beyond social media interfaces, the EDPB has also stressed the dangers of subliminal manipulation in democratic processes. In its Statement 2/2019 on the use of personal data in political campaigns (the Cambridge Analytica case), the EDPB warns that predictive tools used to profile people’s personality traits, moods, and points of leverage pose severe societal risks. When these sophisticated profiling techniques are used to target voters with highly personalized messaging, they not only infringe upon the fundamental right to privacy but also threaten the integrity of elections, freedom of expression, and the fundamental right to think freely without being subjected to unseen psychological manipulation. 

Synthesizing EDPB decisions and guidelines: to counteract these deceptive techniques across all sectors, the fairness principle mandates that controllers respect data subject autonomy, avoid exploiting user vulnerabilities, and ensure that individuals are never coerced into abandoning their privacy through unfair technological architectures. 

Importantly, these GDPR rules apply in the absence of high thresholds, making them particularly relevant even where the conditions to meet the AI Act prohibitions are not met. This is why clarity about the interplay of the two regulations are essential for practical implementation.

  4.3.  Interplay with other EU laws: UCPD, DSA

The AI Act serves to complement or expand the provisions of existing EU law. For instance, unlike EU consumer protection laws, Articles 5(1)(a) and 5(1)(b) of the AI Act extend protection beyond consumers to encompass any individual. As a result, it must be considered alongside other legal frameworks such as the UCPD, the GDPR, the DSA, the political advertising regulation, and EU product safety legislation. 

For example, the UCPD aims to protect individuals from misleading information that could lead them to purchase goods they would not otherwise have bought. It also offers greater protection to vulnerable individuals, such as the elderly and children. The UCPD overlaps partly with the Article 5(1)(a) and (b) prohibitions, though not entirely. Firstly, the UCPD is a Directive and not a Regulation under EU law, and secondly, it only protects consumers (those “acting outside their trade, business, craft or profession”). In the case of the AI Act, however, the prohibitions in Article 5 serve to protect everyone, irrespective of their “consumer” or other status, such as “patient”, “student”, or “tax payers” to give some examples.

Furthermore, the scope of the UCPD is limited to transactional decisions, not all decisions. For example, a surgeon persuaded by manipulative or deceptive techniques by an AI system to operate on a patient in a certain way rather than another would not be covered by the UCPD. On the contrary, both rules will apply in all cases where AI systems are used to manipulate the consumer’s decision-making autonomy subliminally.

By analogy, the scope of the DSA is also limited to what happens on online platforms, and when it comes to deceptive design and the rules in Article 25 DSA – it is relevant only where the GDPR is not applicable, so the cases in which both the AI Act and the DSA apply are limited. 

But there are other provisions of the DSA that could be relevant at the intersection with prohibited AI practices. For example, the DSA pays special attention to the prohibition of profiling using special categories of personal data (as defined by Article 9 GDPR) on online platforms, given the possible manipulative effect of disinformation campaigns that can lead to a negative impact on public health, public security, civil discourse, political participation, and equality (Recitals 69 and 95 DSA). Therefore, if bots and deepfakes spread information online to convince vulnerable individuals (such as the elderly, children, and economically disadvantaged individuals) to purchase high-profit financial products, both the DSA and the AI Act would apply.

Compliance with these laws can help mitigate harm and reduce manipulative effects. For example, suppose that a very large online platform has conducted a risk assessment to assess systemic risk (as required by Article 34 DSA) and a data protection impact assessment (as required by Article 35 GDPR in certain circumstances). In this case, it will be easier for such a platform to identify whether any of its AI systems may fall under the prohibited uses listed in Article 5 AI Act, and adopt mitigating measures accordingly.

5. Concluding Reflections and Key Takeaways

There is a high threshold for falling under the Articles 5(1)(a) and (b) prohibitions.  

To fall under the prohibitions in Article 5(1)(a) or (b), providers and deployers would have to fulfil several cumulative conditions at once. Interpreting the Guidelines, this high threshold is designed to ensure that only very specific AI use-cases and applications would fall under the scope of the prohibitions. While a high threshold of application exists, it is worth noting that the final text of the AI Act ended up being broader in scope as compared to the European Commission’s initial proposal.

It is important to note that even where this threshold is not met, EU law through provisions of the GDPR regarding fairness and data protection by design when processing personal data, or some of the DSA rules when very large online platforms are involved would still limit some manipulative and deceptive practices. 

The prohibition applies even when there is no intention of manipulation. Even when there is no voluntary intention to influence a person’s decision, Article 5(1) could still apply since the provision also covers the harmful effect of manipulating and exploiting individuals or groups. In order to mitigate potential risks, the provider may adopt transparency measures and implement appropriate safeguards to prevent harmful outcomes or consequences. While doing so, it is important to keep in mind that even though the use of a specific AI system does not meet the cumulative conditions of the Article 5(1) prohibitions, it is nevertheless highly likely to be considered a high-risk AI system under Article 6 AI Act.   

Compliance with other laws can help demonstrate compliance with the AI Act.

The Guidelines highlight that if the AI provider shows compliance with relevant EU legislation on transparency, fairness, risk assessment, and data protection, it may contribute to demonstrating compliance with the AI Act’s requirements.