FPF Releases Comparative Analysis of California and U.K. Age-Appropriate Design Codes

The Future of Privacy Forum (FPF) today released a new policy brief comparing the California Age-Appropriate Design Code Act (AADC), a first-of-its-kind privacy-by-design law in the United States, and the United Kingdom’s Age-Appropriate Design Code. While there are distinctions between the two codes, the California AADC, which is set to become enforceable on July 1, 2024, was modeled after the UK’s version and represents a significant change in the regulation of the technology industry and how children will experience online products and services. 

Download the POLICY BRIEF: Comparing the UK and California Age-Appropriate Design Codes.

“Understanding the requirements of both the UK and California codes, and in particular where they differ, is critical for companies in the US and abroad who may soon be covered under one – or both – codes,” said Chloe Altieri, Youth & Education Privacy policy counsel for FPF and an author of the report. “The explanations and examples in the UK code, many of which are not yet defined in California’s version, may provide helpful compliance insights.”

The report builds on FPF’s in-depth analysis of the California AADC, published in October, and contains a side-by-side comparison of the 15 standards laid out in the UK AADC to the corresponding text of the California AADC, including the “best interests of the child” standard, age assurance, default settings, parental controls, enforcement, and data protection impact assessments.

The report also outlines several broader distinctions between the California and UK codes, including, crucially, how the underlying regulatory frameworks differ. While both codes build on the aims of their respective consumer privacy laws (the UK’s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act), the California AADC is standalone legislation that will be independently enforced, while the UK AADC and GDPR are linked, and enforcement falls to the UK Information Commissioner’s Office (ICO). The UK AADC is also “rooted” in Article 3 of the United Nations Convention on the Rights of the Child (UNCRC), an international treaty ratified by 195 countries, including the UK, but not the US. While the California AADC uses a similar “best interests of children” standard, without the foundation of the UNCRC, there is much less certainty in how businesses should make that determination. 

“As policymakers in other states start to consider similar age-appropriate design code legislation, understanding how and why the California and UK codes differ will be critical,” said Bailey Sanchez, Youth & Education Privacy policy counsel at FPF and an author of the report. “It is not as simple as copying and pasting the UK code in California or anywhere else. California’s version was adapted to fit the legal landscape in the US and the state, which has a unique consumer privacy landscape. Other states will need to make their own adjustments.”

FPF’s youth and education privacy team is closely monitoring the implementation of the California AADC. Catch up on previous blog posts tracking the bill’s progress and our formal analysis once the bill was signed into law. To access the Youth & Ed team’s child and student privacy resources, visit www.StudentPrivacyCompass.org and follow the team on Twitter at @SPrivacyCompass.

New Report Promotes Accountability-Based Approach to Data Protection in the APAC Region

In recent years, there has been an uptick in new (comprehensive) data protection laws in the Asia Pacific (APAC). This trend introduces challenges for cross-border compliance, particularly for the industry, legal practitioners, and the community of data protection regulators. As a result of the difficulties, these stakeholders acknowledge that there is a need for greater consistency in regional data protection frameworks.

Today, the Future of Privacy Forum (FPF) — a global non-profit focused on privacy and data protection — and experts from the Asian Business Law Institute (ABLI) — a Singapore-based think tank non-profit dedicated to providing practical guidance in the field of Asian legal development — released a report providing a detailed comparison of the requirements for processing personal data in 14 jurisdictions in the Asia-Pacific (APAC), including Australia, China, India, Indonesia, Hong Kong SAR, Japan, Macau SAR, Malaysia, New Zealand, the Philippines, Singapore, South Korea, Thailand, and Vietnam. The comparative analysis follows a months-long dissemination of individual reports on each jurisdiction. 

In the movement for enhanced consistency between different jurisdictions’ data protection laws, FPF and ABLI found that many APAC jurisdictions have engaged in conversations around the call to move away from consent-centric privacy practices. Consent practices are often restricted to individual jurisdictions and their legal systems. FPF and ABLI’s new report aims to elevate this discussion by promoting alternatives to consent measures, thereby increasing the accountability of organizations that process personal data.

By using various alternative legal bases other than consent and with an eye on supporting greater accountability, organizations can balance their interest in using personal data with broader societal concerns, such as developing a vibrant digital economy or preventing the harms of crime and fraud to individuals.

“The APAC region is presently undergoing a period of intensive law reform. This APAC comparative report provides an opportunity for lawmakers, governments, and regulators who draft, review or implement data protection laws to have a comprehensive overview and analysis of notice, consent, and related requirements that operate in the data protection frameworks of their respective jurisdictions, regional partners and neighbors,” said Josh Lee Kok Thong, Managing Director of FPF APAC in Singapore. “We hope the report serves as a catalyst for initiating a regional dialogue focused on clarifying existing uncertainties and enhancing the compatibility of regional data protection laws.”

Understanding that compliance-based approaches to data privacy, such as consent, notice, and choice mechanisms, have serious limitations, FPF and ABLI propose an accountability-based approach to data protection. This new system shifts the responsibility of protection to users and controllers of personal data by utilizing data protection impact assessments (DPIAs). Assessing risk and enhancing accountability provides organizations with legal certainty, ensuring better privacy accountability across the region.

Moreover, FPF and ABLI found that this accountability approach is necessary and important as data protection developments continue advancing across the APAC region. The rise of new developments conveys a rare opportunity to clarify existing uncertainties and enhance the compatibility of Asian data protection laws on these crucial privacy issues. The report’s comparative legal analysis and input from industry, privacy professionals, and regional practitioners have demonstrated that despite divergences, commonalities exist and can be leveraged to drive convergence between jurisdictions’ different legal systems.

“ABLI is delighted to see the fruit of its partnership with FPF culminating in the release of this comparative review at a time when digitalization is driving an increase in cross-border data flows,” said Mr. Rama Tiwari, Chief Executive of the Singapore Academy of Law, ABLI’s parent organization. “With the largest economies in APAC estimated to be able to reap economic benefits of approximately US$2 trillion if they fully capture their digital potential, the release of the report provides a timely and enabling reference for policymakers and practitioners alike as they design, reform and refine policies and practices for personal data processing both domestically and across borders.”

The GDPR and the AI Act Interplay: Highlights from FPF and Ada Lovelace Institute’s Joint Event

Authored by Christina Michelakaki, FPF Intern for Global Policy

On November 9, 2022, FPF, along with the Ada Lovelace Institute (Ada), organized a closed roundtable in Brussels where experts met to discuss the lessons that can be drawn from General Data Protection Regulation (GDPR) enforcement precedents when deciding on the scope and obligations of the European Commission (EC)’s Artificial Intelligence (AI) Act Proposal. The event hosted representatives from the European Parliament, civil society organizations, Data Protection Authorities (DPAs), and industry representatives.

The roundtable discussion was based on a comprehensive Report launched by FPF in May 2022 analyzing case law under the GDPR applied to real-life cases involving Automated Decision-Making (ADM). This blog outlines the main conclusions drawn by the speakers on four main topics:

1. the complementarity between the AI Act and the GDPR;
2. the needed clarity on transparency requirements and AI training;
3. the difference between the risk-based approach and an exhaustive high-risk AI list; and
4. the need for effective enforcement and redress avenues for affected persons.    

FPF’s Managing Director for Europe, Dr. Rob van Eijk, gave opening remarks followed by a short introduction by FPF EU Policy Counsel Sebastião Barros Vale and Ada’s European Adviser, Alexandru Circiumaru. 

As outlined in a recent analysis on the matter published by FPF, Barros Vale highlighted that:

• The AI Act and the GDPR share Article 16 of the Treaty on the Functioning of the European Union (TFEU) as a legal basis, so data protection considerations are important when it comes to regulating AI systems under the AI Act Proposal. 
• Regarding the AI value chain, the AI Act’s ‘users’ will generally be the ‘controllers’ under the GDPR in the deployment phase of AI systems; during the development phase, providers will likely be the GDPR controllers where they build AI systems relying on the collection, analysis, or any other processing of personal data.
• Where no processing of personal data is involved, or where no GDPR breaches occur, but AI systems still significantly impact individuals, the AI Act could potentially fill redress gaps. 
• The existing and future GDPR case law on ADM may inspire the future enlargement of the list of high-risk AI systems (HRAIS) provided in Annex III of the AI Act. 

This was followed by short interventions from the participants and a discussion. 

1. The AI Act and the GDPR have different but complementary scopes and obligations.

Speakers seemed to agree on the fact that not following the GDPR’s distinction between data controllers and processors in the AI Act stems from the fact that the EC aims at regulating AI systems entering the EU market from a product safety perspective. 

Industry representatives further explained that trying to understand the requirements of the AI Act from a data protection point of view does not make practical sense. They added that having specific types of AI systems in the Annex III list would not make them lawful per se, notably if they breach GDPR requirements. 

Other panelists noted that the AI Act has a broader scope than the GDPR regarding AI systems, as it covers cases where no personal data processing is involved but that may negatively affect individuals.

Moreover, participants questioned whether the AI Act’s human oversight requirements would render Article 22 GDPR on ADM useless – notably, the rights for individuals to obtain human review of automated decisions. Some participants argued that this does not seem to be the case because users are not required by the AI Act Proposal to implement meaningful human oversight when they deploy AI systems. With regard to human oversight, the current draft of the AI Act only requires providers to embed measures into their AI systems that enable users to include human oversight in their deployment scheme. 

Additionally, some experts claimed that the interplay between the AI Act and the GDPR was neglected in the original Proposal, which may raise further issues. As an example, the initial text seems to overlook the fact that users (likely controllers under the GDPR) are often very dependent on AI providers when it comes to practical GDPR compliance. Moreover, speakers worried about issues arising in the AI system’s deployment phase – where AI users are in control – and pointed to the approach proposed at the Council of Europe’s Convention on AI as more tailored to the responsibilities of each party.

For another speaker, incorporating broader fundamental rights impact assessments into the AI Act could complement Data Protection Impact Assessments (DPIAs) under the GDPR. 

2. Transparency requirements and AI training need further clarity

With regard to certain requirements set forth by the AI Act Proposal, speakers conveyed that:

• Training AI systems by using sensitive data, as allowed under Article 10(5) of the AI Act,  may be challenging for Business-to-Business companies that do not usually process sensitive data since they would need to ask third parties for access to such data. 
• Transparency requirements under the AI Act need to ensure information is intelligible: sharing information that a well-trained expert understands will probably be useless for the average person. On the other hand, sharing too much information about how AI systems work may enable malicious actors to circumvent them (e.g., fraud detection algorithms).

3. The difference between the risk-based approach and an exhaustive high-risk AI list 

As a pushback to having a specific set of strictly-regulated high-risk AI use cases in Annex III and prohibited practices in Article 5 of the AI Act, some voices suggested mimicking the GDPR’s open clauses and risk-based approach. More specifically, a few speakers agreed that not having a closed list but rather a set of overarching principles and risk assessment requirements would increase providers’ accountability and enable enforcers to verify compliance in a more flexible manner. 

Speakers that advocated for such a solution agreed that the concept of ‘risk’ and its underlying assessment criteria should be read in line with the GDPR, which could also provide less prescriptive indications of how to mitigate detected risks.

For other experts in the room, a clear definition of AI along with the list of Annex III is preferable, as it avoids enshrining subjective risk assessment criteria. According to this point of view, relying on providers’ self-assessments to decide what is considered high-risk could benefit larger players with the financial incentives and legal resources to take a less cautious approach to AI development.

Among the participants that defended keeping a high-risk AI list, some called for the ability to easily update the list since novel risky AI use cases are constantly surfacing. One of the experts disagreed that it should fall on the EC to update the list, given the institution’s political driving factors. Instead, the speaker called for a bottom-up approach involving regulators and the public’s participation. Other voices also advocated for incorporating emotion recognition systems and the analysis of biometrics-based data in the list of high-risk AI use cases. 

4. A need for effective enforcement and redress avenues for affected persons 

Lastly, participants touched upon the AI Act’s governance mechanisms and redress avenues. 

• With regard to enforcement, some speakers stressed the need for cooperation between different competent regulators on AI and that DPAs should not be the sole or main enforcers of the AI Act. On the other hand, the idea that DPAs should be the main enforcers was also supported by some participants. On the previous point, some experts claimed that DPAs often lack the resources and expertise to deal with AI systems. Participants also advised against having EU Member-States choose their national authority to avoid fragmentation in regulatory approaches. Some speakers called for the centralization of enforcement of AI rules at the EU level while not excluding a potential role for DPAs.  
• When it comes to redress mechanisms, some speakers called for enshrining rights for individuals affected by AI systems, notably a right to complain to a supervisory authority and to file judicial claims, including class actions. Others highlighted that the recently proposed AI Liability and revised Product Liability Directives could empower individuals to obtain compensation for harms caused by AI systems. 

Further reading:

• FPF’s report: “Automated Decision-Making Under the GDPR:  Practical Cases from Courts and Data Protection Authorities,” May 2022 
• FPF’s blog: “The GDPR and AI Act interplay: lessons from FPF’s ADM case law report,” November 2022
• Ada’s blog: “People, risk and the unique requirements of AI,” March 2022
• Ada’s blog: Regulating AI in Europe: Four problems and four solutions, March 2022

FPF Urges Federal Trade Commission to Craft Practical Privacy Rules

FPF Comments Regarding FTC ANPR Urge the Commission to Provide Individuals with  Strong, Enforceable Rights and Companies with Greater Clarity about their Obligations under Section 5 of the FTC Act.

The Future of Privacy Forum filed comments regarding the Federal Trade Commission’s Advance Notice of Proposed Rulemaking, recommending that the Commission prioritize practical rules that clearly define individuals’ rights and companies’ responsibilities. 

The Commission has spent decades enforcing prohibitions against unfair and deceptive data practices regarding a wide range of established and emerging technologies. Those privacy and security enforcement actions have been based on the FTC’s statutory authority, which provides flexibility to address consumer harms arising from novel technologies and business practices, but which does not articulate granular rights for consumers or requirements for businesses. Clear, practical rules can more specifically define what data practices the Commission considers unfair or deceptive. The current FTC rulemaking is an opportunity to provide individuals with strong, enforceable rights and companies with greater clarity about their obligations under Section 5 of the FTC Act.

FPF’s comments urge the Commission to:

• Codify its “common law” privacy and security norms. FTC enforcement actions are often viewed by practitioners as precedent or guidance. But settlements and consent decrees do not provide explicit, comprehensive rules that companies must follow and upon which consumers can rely. The Commission should codify key aspects of its deception and unfairness settlements while also incorporating lessons from FTC staff reports, workshops, privacy laws, self-regulatory regimes, and commercial best practices. Specifically, the FTC should:
  • require businesses to provide material, clear, and prominently accessible data use policies;
  • require businesses to implement reasonable security measures;
  • require businesses to comply with the representations they make about privacy and security, including self-regulatory commitments; 
  • prohibit companies from circumventing individuals’ clearly expressed privacy preferences without clear, explicit, superseding consent from the individual; and
  • articulate the circumstances in which the FTC considers discriminatory algorithmic decision-making to be an unfair trade practice, the factors the Commission considers when weighing that determination, and the degree to which the Commission’s analysis relates to other anti-discrimination regimes.
• Go beyond its common law privacy and security norms to mitigate important privacy risks and establish increased clarity regarding companies’ responsibilities. When crafting these sorts of rules, the FTC should be guided by three principles:
  • data exists on a spectrum of identifiability, rather than in binary categories of “personal information” or “not personal information,” and privacy enhancing technologies can reduce the identifiability of data and otherwise mitigate risks;
  • standards for evaluating the fairness of “secondary uses” of data are needed to define the boundaries of what secondary uses are compatible, based on a careful evaluation of context, expectations, harms, and benefits of processing, including competition; 
  • It is especially important to consider the harms that sensitive data use can create, the manner in which those harms impact marginalized communities, and the heightened protections that may be appropriate to mitigate those harms. At the same time, sensitive data is essential to a wide range of activities, including detecting and addressing disparate outcomes.

As a practical matter, the FTC acts as the primary U.S. privacy enforcement agency. Although FPF views a new, pragmatic, comprehensive federal privacy law as the ideal mechanism for grappling with complex technologies and data flows, clear and practical FTC rules defining unfair and deceptive practices would benefit individuals and businesses.

Understanding Extended Reality Technology & Data Flows: Privacy and Data Protection Risks and Mitigation Strategies

This post is the second in a two-part series. Click here for FPF’s XR infographic. The first post in this series focuses on the key functions that XR devices may feature, and analyzes the kinds of sensors, data types, data processing, and transfers to other parties that power these functions. 

I. Introduction

Today’s virtual (VR), mixed (MR), and augmented (AR) reality environments, collectively known as extended reality (XR), are powered by the interplay of multiple sensors, large volumes and varieties of data, and various algorithms and automated systems, such as machine learning (ML). These complex relationships enable functions like gesture-based controls and eye tracking, without which XR experiences would be less immersive or unable to function at all. However, these technologies often depend on sensitive personal information, and the collection, processing, and transfer of this data to other parties may pose privacy and data protection risks to both users and bystanders. 

This post examines the XR data flows that are featured in FPF’s infographic, and analyzes some of the data protection, privacy, and equity issues raised by the data that is processed by these devices, as well as strategies for mitigating these risks.

Key risks include:

• Sensitive inferences: XR devices collect, process, and share large quantities of data about users’ bodies and environments. This data could be used to make inferences—whether accurate or not—about sensitive aspects of peoples’ lives, such as their sexual orientation or health conditions.
• Digital fingerprinting: Tracking users’ and bystanders’ bodies could allow for digital fingerprinting and the loss of anonymity in XR environments.
• Bystander data collection: Non-users in proximity to XR technology may be unaware that it is collecting and processing data about them, as well as for what purposes and with whom the technology is sharing this information.

Key mitigation strategies include:

• On-device processing and storage: Processing and storing data on a user’s device, as opposed to remotely on a processor’s server, to ensure that the data remains in the user’s hands and not accessible to others.

• Purpose limitation and data minimization: Limiting data collection, storage, and/or usage, including third-party use, to particular, specified purposes, and requiring data controllers to provide notice to or obtain consent from users if they plan to use this data for a different purpose. 
• Privacy-enhancing technologies (PETs): Certain technological innovations can also be useful tools for managing privacy risks. For example, advances in encryption and differential privacy can allow for privacy-preserving data analysis and sharing, and the use of synthetic data sets can alleviate concerns about data sharing or secondary data use.
• Bystander protections: Designing XR devices so that they ensure bystanders’ data is not unduly collected. This could include automatically blurring bystanders’ faces, or using a system of lights on a head-mounted display to signal to non-users the device is on and potentially collecting data.

II. Processing Large Volumes and Varieties of Sensitive Personal Data

XR technologies raise traditional privacy and data protection risks, but also implicates larger questions around surveillance, social engineering, and freedom of expression. As noted in the first blog post in this series, XR technologies require large volumes and varieties of data about the user’s body and their environment. Certain collection and use limitations may therefore be challenging or impossible to implement, since some of XR’s core functions require extensive data collection and processing. Now and in the future, XR technologies may also transfer data to other users and third parties, such as software companies, hardware manufacturers, and advertisers. While devices generally process raw sensor data on device, they may transmit raw or processed sensor data to an application and other parties for further processing to improve representations of virtual content or enable shared experiences. While these transmissions of data may improve a user’s XR experiences, they can also create new privacy and data protection risks for users and bystanders.

Eye tracking underpins many current and future-facing use cases, such as enhanced graphics, expressive avatars, and personalized content, but it may pose privacy and data protection risks to users. This is due to eye tracking data’s sensitive nature, its potential role in significant decisions affecting users, and the unconscious nature of the behaviors from which some of this data is derived. Organizations could use data related to pupil dilation and gaze to potentially infer information—whether accurate or not—about the user, such as their sexual orientation, age, gender, race, and more. Organizations may also use this data to attempt to diagnose medical conditions, such as ADHD, autism, and schizophrenia. Despite the sensitive nature of this data, users often lack the capacity to meaningfully control its collection or use. Without proper controls, this information may be further shared with third parties. This raises the likelihood of organizations using this data to inform major decisions about a user, which could have real-world impacts on XR users.

Sensors that track a user’s bodily motions may also cause harm due to their potential to undermine anonymity. The first post in this blog series analyzed how tracking a user’s position can enable functions like mapping the user’s space to help place virtual content. But this tracking could also be a means to digitally fingerprint users and individuals, including bystanders, especially given the volume and variety of data that XR devices gather and process. At the same time, this tracking data raises the same de-identification and anonymization concerns that exist regarding similarly granular non-XR data types, such as behavioral biometrics, historical geolocation, and genetic information. Digital fingerprinting may therefore undermine individuals’ ability to maintain anonymity in XR environments. This may discourage users from fully expressing themselves or participating in certain activities due to worries about retaliation.

III. Statutory Obligations 

It is unclear how well current legal protections mitigate the privacy risks posed by certain processing activities in the XR context. Whether or not bodily information like gaze and gait are covered by existing biometric regulations may depend on these laws’ definitions of biometric data. For example, under the EU’s comprehensive privacy law, the General Data Protection Regulation (GDPR), this type of data qualifies as “personal data” if it relates to an identified or identifiable person, such as a user or bystander. Thus, an organization that records, collects, assesses or uses this data in any other manner would be subject to GDPR obligations such as transparency, fairness, data minimization or storage limitation. 

Pursuant to the GDPR, “biometric data” includes personal data resulting from the specific technical processing of a person’s physical, psychological, or behavioral characteristics, and which allows for identification. Organizations are subject to heightened obligations under the Regulation depending on the purpose for which they process biometric data. Specifically, the GDPR prohibits organizations from processing such personal data, unless one of the permissible grounds strictly defined by the law applies. The Regulation defines biometric data to include only that which an organization could use for identification purposes. As described in FPF’s prior blog post, however, an organization may process eye and other bodily information for non-identification purposes, such as to debug applications or improve products . This raises questions as to whether the GDPR’s protections for sensitive data categories would always apply to these XR functions. Notably, even if this eye and other bodily information does not meet the “sensitive data” criteria, the rest of the Regulation would still apply to this data. Furthermore, European ePrivacy rules may apply to a user’s system that connects to or pairs with XR equipment.

Similar lack of certainty exists in U.S. law. For example, the Illinois Biometric Information Privacy Act (BIPA) applies to information based on “scans” of hand or face geometry, retinas or irises, and voiceprints. This definition of “biometric identifiers” does not explicitly cover the collection of behavioral characteristics or eye tracking. Whereas the GDPR may still apply to an organization that processes eye and other bodily information if it is personal data or qualifies as other sensitive data categories, BIPA may not apply at all. This highlights how existing laws’ protections for biometric data may not extend to every situation involving XR technologies. However, protections may apply to other special categories of data, given XR data’s potential to draw sensitive inferences about individuals.

IV. Bystander and Environmental Data

Bystanders’ privacy can also be impacted when XR devices and third parties collect and process sensor data. Some of the privacy and data protection issues affecting bystanders mirror the  privacy risks to XR users. However, unique notice challenges arise with respect to bystanders. Non-users in proximity to an XR user may be unaware that the device is collecting and processing data about them, as well as for what purposes and with whom the device is sharing this information. Like users, bystanders also cannot control the unconscious behaviors that provide the sensor data inputs for XR experiences. Even if a bystander generally understands that a device is collecting data about them, the unconscious nature of some behaviors means that bystanders may neither be aware of the behaviors nor specifically understand that a device is processing data about these behaviors. 

Bystander data could facilitate both use cases that are detrimental to a non-user’s privacy and decisions that negatively affect them. Future XR technologies will likely incorporate facial characterization or analysis technologies that can allegedly sense cognitive states or infer emotions—whether accurate or not—based on sensor data. Insights from these technologies could help organizations construct a portrait of the locations a non-user frequents, their interests, and medical conditions. 

IV. Strategies for Mitigating Risks

Organizations that provide XR technologies can implement a number of strategies to address the risks raised by XR data collection, use, and sharing. While no single intervention by itself mitigates all of these risks, some combination of strategies is likely to decrease risks and help minimize harms that may result. For instance, processing and storing data on a user’s device, as opposed to remotely on a processor’s server, helps ensure that the data remains in the user’s hands and not accessible to others. Organizations can also work to limit data collection, storage, and/or usage, including third-party use, to particular, specified purposes, and provide notice to or obtain consent from users if they plan to use this data for a different purpose. Companies should set policies and guidelines for third-party developers’ data practices, and monitor to ensure compliance with said policies.

Certain privacy-enhancing technologies (PETs) are useful tools for managing privacy risks. For example, advances in encryption and differential privacy can enable privacy-preserving data analysis and sharing, and the use of synthetic data sets can address concerns about data sharing or secondary data use. Another option is to provide greater user controls, allowing users to control the kinds of data collected about them—particularly sensitive data like eye tracking and facial expressions data—and with whom this data is shared. 

Some organizations have chosen to design XR devices so that they ensure bystanders’ data is not unduly collected, for instance by automatically blurring bystanders’ faces, or using a system of lights on a head-mounted display to signal to non-users the device is on and potentially collecting data.

Organizations using XR should be transparent about how they use and plan to use XR data, and publicly commit to guidelines and/or ethical principles. This could also include something akin to an institutional review board (IRB) to ensure compliance with these principles. Finally, organizations can build privacy into an organization’s culture and processes and create bodies like oversight boards to ensure privacy protections endure beyond other changes in mission and values.

V. Conclusion

The complex web of data, sensors, algorithms and automated systems, and parties that enable important and sometimes central XR functions also can raise privacy and data protection concerns. Devices and ML models may collect and process large volumes and varieties of sensitive personal data, over which users and bystanders may lack meaningful controls, and that other parties could use to make important decisions affecting these individuals. The disclosure of this data may also undermine user anonymity, which could discourage users from freely expressing themselves due to fears of retaliation. Providing bystanders with notice that communicates that a device is collecting data about them, let alone for what purpose and to whom the data is transmitted, is challenging and may not be possible. This creates difficulties related to obtaining affirmative express consent to data processing activities in XR, where consent is predicated on the individual being informed. There is also uncertainty about how existing laws interact with XR technologies, such as how body-based data fits within existing legal definitions of biometrics. The risks to users and bystanders outlined in this post underscore the importance—and, sometimes, challenge—of ensuring appropriate safeguards exist at the technical, policy, and legal level to mitigate against harms that may arise in this space.

Brussels Privacy Convening Focuses on Empowering Vulnerable and Marginalized People, Launches New Project

The Future of Privacy Forum (FPF), a global non-profit focused on data protection and privacy, and the Brussels Privacy Hub of Vrije Universiteit Brussel (VUB) will jointly present the sixth edition of the Brussels Privacy Symposium on November 15, 2022. The in-person event will convene in Brussels, bringing together policymakers, academic researchers, civil society, and industry representatives to discuss privacy research and scholarship. 

In line with this year’s topic, “Vulnerable People, Marginalization, and Data Protection,” participants will explore the extent to which data protection and privacy law — including GDPR and other modern data protection laws like Brazil’s LGPD — safeguard and empower vulnerable and marginalized people. They will also debate how to balance the right to privacy with the need to process sensitive personal information to uncover and prevent bias and marginalization. Stakeholders will discuss whether prohibiting the processing of personal data related to vulnerable people serves as a protection mechanism. 

The event marks the launch of “VULNERA,” the International Observatory on Vulnerable People in Data Protection, led by the Brussels Privacy Hub and supported by the Future of Privacy Forum. The observatory aims to promote a mature debate on the multifaceted connotations surrounding the notions of human “vulnerability” and “marginalization” existing in the data protection and privacy domains.

“I’m excited to begin the groundbreaking and much-needed work we have ahead of us,” Gabriela Zanfir-Fortuna, FPF’s Vice President for Global Privacy, said. Zanfir-Fortuna is also a member of VULNERA’s executive team as a Scientific Coordinator, which is joined by more than 30 members of a broad scientific network. “This initiative will focus on understanding how data protection and privacy law puts safeguards in place to protect the rights of vulnerable and marginalized people in societies increasingly underpinned by digital data flows.”

Professor Gianclaudio Malgieri, Co-Director of Brussels Privacy Hub of Vrije Universiteit Brussel added: “The VULNERA International Observatory will explore theories of vulnerability, marginalization, and intersectionality, examining how data protection law and policy apply to people in certain contexts that may be vulnerable or marginalized, such as women, children, people on a low or zero income, racialized communities, and people of color, ethnic and religious groups, migrants, LGBTQIA+ and non-binary people, the elderly, and persons with disabilities,” 

Representatives from the European Network Against Racism, Dutch Human Rights Council, European Commission, Irish DPC, European Digital Rights (EDRi), European Data Protection Supervisor, and other relevant organizations will share their expertise during the Brussels Privacy Symposium. 

“As we think about the next iteration of the digital age, it’s important that we have a more global consensus on how to protect those who have been historically marginalized,” said Rob van Eijk, FPF’s Managing Director for Europe. “The timing for the launch of VULNERA, and this symposium at-large, could not have been at a more critical juncture.” 

For more information about the event, the agenda, and speakers, visit the FPF site. To learn more about the VULNERA, visit the Brussels Privacy Hub site. 

Event Report: FPF Side Event and Workshop on Privacy Enhancing Technologies (PETs) at the 2022 Global Privacy Assembly (GPA)

The 2022 Global Privacy Assembly (GPA) – which brings together most global data protection authorities (DPAs) every year since 1979, to share knowledge and establish common priorities among regulators – took place between October 25 and 28, in Istanbul (Türkiye). The Future of Privacy Forum (FPF) was invited by the organizers of the GPA (the Turkish DPA) to host a two-part side event during the GPA’s Open Session (on October 25 and 26), in addition to a capacity building workshop for regulators during the Closed Session (on October 28).

These sessions covered the topic of Privacy Enhancing Technologies from three different approaches:

• The regulators’ take: PETs are promising, but no silver bullet. The first part of the FPF Side Event offered the regulators’ perspective, and was titled ‘Regulatory Views on the Role and Effectiveness of PETs’. The session was moderated by Limor Schmerling Magazanik, Director of the FPF-affiliated Israel Tech Policy Institute (ITPI), and counted on the contributions of Rebecca Kelly Slaughter (Commissioner at the US Federal Trade Commission, or ‘FTC’), Tobias Judin (Head of the International Department of the Norwegian DPA, the ‘Datatilsynet’), Gilad Semama (Privacy Commissioner of Israel), and Vitelio Ruiz Bernal (Director of Supervision at the Mexican DPA, the ‘INAI’).
• The view of practitioners: a call for regulatory clarity and predictability. The second part of the Side Event was entitled ‘Lessons Learned from Implementing PETs’, and saw various privacy leaders from the industry share their experiences of leveraging PETs in their compliance efforts. The panel was moderated by FPF’s CEO Jules Polonetsky and had as panelists Anna Zeiter (Chief Privacy Officer at eBay), Emerald De Leeuw-Goggin (Global Head of Privacy at Logitech), Barbara Cosgrove (CPO at Workday), and Geff Brown (Associate General Counsel at Microsoft).
• FPF’s capacity building workshop. The FPF workshop during the GPA’s Closed Session was conducted by FPF’s Vice President for Global Privacy, Dr. Gabriela Zanfir-Fortuna, and Managing Director for Europe, Dr. Rob van Eijk. The session covered the legal qualification of PETs under the EU’s data protection framework – as well as how they could be leveraged to attain compliance with it -, as well as a primer on three PETs, i.e., Differential Privacy, Synthetic Data, and Homomorphic Encryption. This workshop was a condensed version of the Masterclass that FPF hosted at the 2022 Computers, Privacy and Data Protection (CPDP) Conference last May (recorded).

Below we summarize the discussions in the two FPF Side Events with regulators and privacy leaders and highlight key takeaways.

The regulators’ take: PETs are promising, but no silver bullet

Moderator Limor Schmerling Magazanik opened the first discussion by observing that regulators have a dual role regarding PETs: issuing guidance to clarify when and how PETs should be deployed in different scenarios to ensure compliance with privacy laws; and providing tailored advice to lawmakers that wish to promote the use of PETs for the pursuit of public interest tasks and the responsible use of data. 

On this note, Gilad Semama noted that PETs seem to present solutions for combining innovation in the tech sector with the protection of privacy as a Constitutional right in Israel. Semama highlighted that companies have expressed their need for certainty on how they can use PETs to achieve compliance with the privacy framework. The speaker added that it is challenging to find a one-size-fits-all solution in this respect, but that the Privacy Commissioner is trying to pass flexible guidance and answer the public’s queries on PETs for the benefit of businesses and DPOs, by referring to accountability and helping them choose the most appropriate PET for specific use cases. According to Semama, PETs should be complemented with other data security solutions to provide meaningful protections. On the other hand, he noted that companies that are developing PETs in Israel need access to funding and that a recent joint project from the regulator and the Innovation Authority of Israel may be of help. 

Next up, Rebecca Kelly Slaughter stressed the potential that PETs might offer in promoting competition and consumer protection, as they can represent innovation and a positive metric of competition. However, some applications of PETs can be misleading and competition-inhibiting. This means that, according to Slaughter, the value of PETs should be assessed against their concrete effects. The Commissioner stated that the FTC should mainly focus on providing guidance to assist businesses developing and implementing PETs through FTC rulemaking, instead of strict enforcement. However, the FTC will not approve broad safe harbor provisions for the use of specific PETs, as their effectiveness is generally context-specific.

Tobias Judin followed up on Slaughter’s comments, by saying that, when it comes to greenlighting PETs, DPAs should explain that companies do not need to choose between data collection and privacy, or between innovation and data protection. Judin used health research as an example, outlining that often researchers need to collect data about rare diseases across jurisdictions to make the dataset more representative, even knowing that the level of data protection is not equivalent in all targeted countries. In that context, PETs such as homomorphic encryption or differential privacy may provide reassurance to research subjects. Judin also stressed that confidential computing can mitigate security vulnerabilities that often exist when research data is stored on premises and not in the cloud.

Lastly, Vitelio Ruiz Bernal stressed the importance of helping businesses achieve security standards that can help them comply with data protection law. In this respect, he mentioned the INAI’s data protection laboratory, which is dedicated to analyzing apps and web-based applications that are subject to a black-box. The INAI has found that processors which assist controllers in those contexts are often under-resourced and reluctant to use PETs due to their perceived high costs. Bernal revealed that the INAI is currently looking for public-private collaborations to develop accessible PETs and to issue guidelines on specific PETs (e.g., encryption), also inspired by the work of the Berlin Group on the matter. Given Mexico’s specific legal requirements in terms of cloud service security, Bernal mentioned that PETs could potentiate the uptake of cloud services by increasing trust among stakeholders.

The view of practitioners: a call for regulatory clarity and predictability

To frame the second panel of the Side Event, Jules Polonetsky reflected on the privacy community’s eagerness to learn about how industry privacy leaders are integrating PETs into their compliance strategies, their successful and less successful stories. On the other hand, Jules queried the panelists about the actions they would like to see from regulators and policymakers in this space to promote the uptake of PETs. 

Anna Zeiter revealed that eBay has had meetings with its lead DPA in Germany about how PETs could help them comply with the Court of Justice of the European Union (CJEU)’s Schrems II ruling on international data transfers, in particular on the implementation of supplemental measures in accordance with the European Data Protection Board (EDPB)’s guidance. In that context, the DPA focused on measures such as tokenization and encryption (in transit and at rest).

Emerald De Leeuw-Goggin mentioned Logitech’s offerings of PETs as a service for its internal teams of software developers. According to the speaker, this involved making PETs more accessible and scalable within the wider decentralised organisation, the development of privacy engineering capabilities, and the buy-in of Chief Technology Officers. De Leeuw-Goggin noted that PETs are still not mainstream enough for an SME owner to feel confident investing and implementing them, also due to the existing skills gap in the field. As PETs become mainstream, they will also become more understandable and usable across sectors and company sizes.

Barbara Cosgrove stated that B2B companies like Workday tend to receive questions from their customers on how to best implement PETs into their software solutions. This includes masking or pseudonymizing data, or limiting employee access to data. Sometimes, more sophisticated measures – like differential privacy – could be adequate, but companies are reluctant in investing resources in the absence of regulatory clarity, particularly on de-identification. Cosgrove agreed that businesses and regulators need to put their brains together in developing use cases and standards that would increase legal certainty around the effective use of PETs. Co-regulatory solutions like Codes of Conduct could facilitate demonstrations that PETs are used in a compliant manner. 

Finally, Geff Brown highlighted how differential privacy has become usable in multiple apps, allowing providers to process aggregated telemetry data at scale for analytics. Microsoft is using the technique to improve its Natural Language Processing models, including text and speech prediction. In that context, differential privacy allows companies to demonstrate the accuracy of the model without compromising individuals’ privacy. Brown argued that tech savvy companies need to better explain PETs to consumers and corporate customers, but that standardization efforts and favorable DPA positions can also help. In this context, Geff wished for an EDPB update to the 2014 guidance on anonymization, and to have regulators carry out PET testing and share the results with the public, thereby increasing knowledge and trust in the technologies.

Further reading: 

• A Visual Guide to Practical Data De-Identification (FPF, 2016)
• Privacy 2020: 10 Privacy Risks and 10 Privacy Enhancing Technologies to Watch in the Next Decade (FPF, 2020)
• FPF Files Comments on White House Office of Science and Technology Policy Actions to Advance Privacy-Enhancing Technologies (FPF, 2022)

Call for Nominations Open: FPF’s Award for Research Data Stewardship

When companies share data with researchers in a way that protects data, the collaboration can unlock new scientific insights and drive progress in medicine, public health, education, social science, and many other fields. 

FPF is thrilled to announce the open nomination period for FPF’s 3rd Annual Award for Research Data Stewardship. The Award recognizes partnerships between companies and research institutions where a company shares data it holds in a privacy-protective manner with a researcher or research team for scholarly publication. 

An example of an extraordinary award-winning partnership between researchers and a company to advance scientific and medical progress to benefit society through privacy-protective research data sharing is Stanford Medicine researchers and medical wearable and digital biomarker company Empatica. This award-winning collaboration studied whether data collected by Empatica’s researcher-friendly E4 device, which measures skin temperature, heart rate, and other biomarkers, could detect COVID-19 infections before the onset of symptoms. 

The award is presented to the company and its academic partner based on several factors, including the adherence to privacy protection in the sharing process, the quality of the data handling process, and the company’s commitment to supporting academic research. 

Learn more about past award winners.

Save the Date: FPF’s Award for Data Stewardship Virtual Award Presentation (May 10, 2023)

The Award is a part of FPF’s “Corporate Data Sharing for Research: Next Steps in a Changing Legal and Policy Landscape” project to accelerate the safe and responsible sharing of administrative data between companies and academic researchers. This project is supported by the Alfred P. Sloan Foundation, a not-for-profit grantmaking institution whose mission is to enhance the welfare of all through the advancement of scientific knowledge.

GDPR and the AI Act interplay: Lessons from FPF’s ADM Case-Law Report

In May 2022, the Future of Privacy Forum (FPF) launched a comprehensive Report analyzing case-law under the General Data Protection Regulation (GDPR) applied to real-life cases involving Automated Decision-Making (ADM). Our research highlighted that the GDPR’s protections for individuals against forms of ADM and profiling go significantly beyond Article 22 – which provides for the right of individuals not to be subject to decisions based solely on automated processing that produces legal effects or significantly impacts them, and are currently being applied by courts and Data Protection Authorities (DPAs) alike. These range from detailed transparency obligations to applying the fairness principle to avoid situations of discrimination and strict conditions for valid consent in ADM cases. 

As EU lawmakers are now discussing the amendments they would like to include in the European Commission (EC)’s Artificial Intelligence (AI) Act Proposal, what lessons can be drawn from GDPR enforcement precedents–as outlined in the Report–when deciding on the scope and obligations of the Act? 

This blog will explore: the link between the GDPR’s provisions as relevant for ADM and the AI Act Proposal (1); how the AI Act’s concepts of providers and users fare compared to the GDPR’s controllers and processors (2); how the AI Act facilitates GDPR compliance for the deployers of AI systems (3); the opportunities to enhance or clarify obligations under the AI Act through the lens of ADM jurisprudence (4); the overlaps between GDPR enforcement precedents and the AI Act’s prohibited practices or high-risk use cases (5); the issue of redress under the GDPR and the AI Act (6); and a compilation of lessons learned from the FPF Report in the context of the debates around the AI Act (7). 

Note: when referring to case numbers in this blog, the author is using the numbering of cases in the FPF Report. 

1. Both the GDPR and the proposed AI Act are grounded on Article 16 TFEU for the protection of personal data

One of the two legal bases used by the EC to justify the AI Act Proposal is Article 16 of the Treaty on the Functioning of the European Union (TFEU), which mandates the EU to lay down the rules relating to the protection of individuals with regard to the processing of personal data. This means that, at least to some extent, the AI Act’s rules would complement the protections afforded to data subjects under the GDPR, which is also based on Article 16 TFEU. In fact, in their 2021 Joint Opinion on the AI Act, the European Data Protection Supervisor (EDPS) and the European Data Protection Board (EDPB) have suggested making GDPR compliance a precondition for allowing an AI system to enter the European market as a CE marked product under the AI Act. 

AI systems that would be regulated under the initial Proposal of the AI Act are the ones that rely on the techniques and approaches mentioned in Annex I of the Proposal (such as machine learning and logic-based approaches). Such techniques and approaches could constitute or enable ADM schemes to be implemented by controllers covered by the GDPR. However, the AI Act is generally agnostic regarding the decision-making scheme vis-à-vis individuals when the deployment of an AI system is at stake. This means that its scope is broader than ADM falling in or outside of the Article 22 GDPR prohibition (i.e., ‘qualifying’ or ‘non-qualifying’ ADM).

• As an illustration, Article 3(1) AI Act mentions that software that can produce “predictions and recommendations” – which courts and DPAs have generally not considered to be fully automated decision-making thus far – may constitute AI systems covered by the AI Act if they use one or more of the techniques and approaches mentioned in Annex I. 
• Moreover, the AI Act’s rules under Title III, Chapter 2 and 3 on high-risk AI systems (i.e., the ones that are intended to be used as a safety component of a product, or are themselves products, covered by the Union harmonization legislation listed in Annex II, or that fall under the Annex III list) apply to the systems’ providers, users, importers or distributors even if the final decision that affects a natural person should be taken by a human for the user on the basis of the suggestions, signals, prompts or recommendations provided by the AI system. 

However, even where the AI Act does not provide for specific protections for the rights of individuals where AI systems are underpinned by, or result in solely ADM having a legal or similarly significant effect on them, the safeguards provided by Article 22 GDPR will nevertheless kick in. On the other hand, and as mentioned in the Report, even in cases where Article 22 GDPR does not apply to a particular AI system, the rest of the GDPR applies to non-qualifying ADM and generally to the processing of personal data via AI systems. This could include the AI system’s training, validation and input data, as well as the AI system’s outputs, if they qualify as ‘personal data’ under the GDPR, regardless of whether the data are processed by the system’s providers or users. It is also noteworthy that there may be instances of ADM covered by the GDPR that do not involve any use of AI systems, but rather other forms of automating decisions. 

2. The AI Act’s users are generally the GDPR’s controllers in the AI system’s deployment phase

Instead of focusing on the role of the parties in the AI value chain with regard to the processing of personal data, the AI Act Proposal focuses on the entities which develop and market or that use AI systems for commercial purposes, i.e. ‘providers’ and ‘users’ respectively. Each is assigned specific obligations, but most of the regulatory burden is placed on providers, in particular when it comes to high-risk AI systems (HRAIS) and their conformity assessments. 

This way of defining the main actors subject to obligations under the future AI Act may create inconsistency with the GDPR’s definitions, roles, and responsibilities for covered entities. It has also led the EDPB and the EDPS to ask EU policymakers to ensure the AI Act’s obligations are consistent with the roles of controller and processor when personal data processing is concerned. Indeed, ‘providers’ under the AI Act will likely not be considered the data controllers under the GDPR during their AI systems’ deployment phase. 

Under the GDPR, the bulk of obligations, liability and accountability for how personal data are processed are assigned to ‘controllers’. It is the AI Act’s ‘users’ who will rather be the ‘controllers’ under the GDPR in the deployment phase of AI systems. Therefore, even if they have a very limited set of duties under the Act (e.g., Article 29), they maintain liability and accountability about how personal data is used by, or resulting from the use of AI systems, under the GDPR. It is more likely that providers would be qualified as “processors” under the GDPR, processing personal data on behalf of users, notably if they provide support or maintenance services for AI systems involving the processing of personal data on users’ behalf and under their instructions. 

The situation is different in the development phase of AI systems, where ‘providers’ will likely be considered “controllers” under the GDPR whenever they build AI systems relying on collection, analysis, or any other processing of personal data. The same will be the case for the testing phase of AI systems (e.g. bias monitoring) and for post-market monitoring purposes, which may be legally mandatory under Articles 10 and 61 of the AI Act. In these cases, the status of the provider as controller would derive from the law that imposes data processing duties (i.e., the AI Act), as mentioned in EDPB guidance on the concept of controller (para. 24).

Complex questions further arise in relation to potential joint controllership situations under the GDPR, between “users” and “providers” under the AI Act, in the deployment phase of AI systems. For instance, does the legal obligation that providers have under the AI Act to determine the AI system’s intended purpose and technical data collection lead to the qualification of providers as joint controllers with their customers (users), even if they do not obtain access to the AI system’s input or output data, especially after the Court of Justice of the European Union (CJEU)’s Jehovan todistajat ruling? 

3. The AI Act facilitates to a certain extent GDPR compliance for ‘users’ of AI systems 

References to GDPR compliance in the AI Act proposal are scarce. An example is the authorization granted to providers to process special categories of data covered by Articles 9(1) and 10 GDPR when conducting bias monitoring, detection, and correction in relation to HRAIS (under Article 10(5) of the Act). Another is the obligation for users to use the information received from providers’ HRAIS instructions of use when carrying out Data Protection Impact Assessments (DPIAs) under the GDPR, as per Article 29(6) AI Act. However, as the crux of GDPR obligations for controllers in the commercial deployment phase of AI systems will arguably rest with the systems’ users, it is worth exploring whether the AI Act’s obligations imposed on providers of HRAIS may put users in a better position to comply with the GDPR. 

Under Article 13 AI Act, providers have extensive transparency obligations towards their customers (i.e., ‘users’ and, most likely, GDPR ‘controllers’ as explained above), with a view to enable the latter ‘to interpret the system’s output and use it appropriately.’ This transparency comes in the form of instructions of use that should specify, inter alia, the HRAIS’s intended purpose, level of accuracy, performance, specifications for input data, and implemented human oversight measures (as detailed in Article 14 AI Act). Additionally, the HRAIS’s technical documentation that the provider is required to draw up under Article 11 AI Act – and whose elements are listed under Annex IV AI Act – will provide insight about the HRAIS’s general logic, key design choices, main classification choices, relevance of the different parameters, training data sets, and potentially discriminatory impacts, among other features. 

Regardless of the wording under Article 29(6) AI Act, users may use the information obtained from providers under Article 13 AI Act and the HRAIS’s technical documentation not only to comply with their duty to carry out a DPIA but to ensure broader alignment with the GDPR and its transparency imperatives. Such information may also prove useful to comply with other GDPR obligations, such as providing notice to data subjects about profiling and ADM and to complete their records of AI-powered data processing activities under Article 30 GDPR. 

However, it should be noted that, under the AI Act Proposal, duties for providers only exist with regards to HRAIS, whereas the users/controllers’ above mentioned obligations under the GDPR may apply even where the underlying AI systems are not qualified as such. For example, under the EDPB’s criteria, controllers could still be obliged to carry out a DPIA on an AI system that is not included in the Annex III AI Act list of HRAIS, such as a social media recommender system or an AI system used in the context of online behavioral advertising. 

Specifically, with regards to controllers’ notice obligations on profiling and ADM, the FPF Report shows that DPAs and courts in Europe agree that Articles 13(2)(f) and 14(2)(g) GDPR provide for an ex-ante obligation to proactively inform data subjects about the system’s underlying logic, significance and envisaged consequences, and not about the concrete automated decisions that affect them. On the other hand, an obligation to provide decision-level explanations exists where the data subject exercises his or her data access right under Article 15(1)(h) GDPR, as is illustrated by two Austrian DPA decisions (cases 14 and 21 in the Report) and an Icelandic DPA decision (case 38 in the Report). In such instances, DPAs ordered controllers to disclose specific elements of information regarding automated credit or marketing scores attributed to data subjects, notably the algorithm’s parameters or input variables, their effect on the score, and an explanation of why the data subject was assigned a particular score. 

Thus, when complying with the GDPR’s transparency obligations, controllers who qualify as users under the AI Act would find immense value in leveraging the sort of information that Articles 11 and 13 of the AI Act mandate providers to make available with regards to their HRAIS. Recent GDPR case law on ADM could make a case for extending providers’ transparency duties beyond HRAIS, and for ensuring the standard of intelligibility for the information AI providers should make available to users is one that enables the latter to comply with their GDPR obligations. 

A possible avenue to be considered through the legislative process  would be to create a general duty under the AI Act for providers to assist users in their GDPR compliance efforts in relation to the AI systems they sell, even in cases where providers would not act as  data processors for the users. Some impetus for this approach may be found in Article 9(4) AI Act, which mandates providers to inform users about the risks that may emerge from the use of the AI system, and to provide them with appropriate training. 

4. The GDPR’s ADM case law calls for further development or clarification of obligations under the AI Act

Some of the decisions analyzed in the FPF Report may provide indications that the AI Act’s obligations for providers and users – at least when HRAIS are at stake – need further development or clarifications. 

Accuracy and transparency: In a landmark ruling, the Slovak Constitutional Court (case 4 in the Report) established that local law should require additional measures to protect individuals when automated assessments are carried out by State agencies. According to the Court, such measures could include: (i) checking the AI system’s quality, including its error rate; (ii) ensuring that the criteria, models, or underlying databases are up-to-date, reliable, and non-discriminatory; and (iii) making individuals aware of the existence, scope and impacts of automated assessments affecting them. 

• Measures (i) and (ii) seem to be very close to the data quality, accuracy, robustness, and cybersecurity requirements proposed under Articles 10 and 15 AI Act. However, these obligations are geared towards HRAIS’s providers, and not users/controllers. In its decisions against Deliveroo and Foodinho (cases 3 and 6 in the Report), the Italian DPA fined the controllers for not verifying the accuracy and correctness of their automated rider-management decisions and underlying datasets, although these are not explicit requirements under the GDPR’s Article 22(3). Therefore, and at least for HRAIS, the EU legislator could eliminate legal uncertainty by incorporating data quality and accuracy requirements into Article 29, which sets out users’ obligations in this  context. Such a requirement could go beyond merely checking whether the HRAIS’s input data is relevant in view of the intended purpose, as Article 29(3) AI Act currently requires.
• With regards to measure (iii), it should be noted that making individuals aware of the scope and impact of an automated assessment that falls outside of Article 22 GDPR goes beyond Articles 13(2)(f) and 14(2)(g) GDPR. As a rule, DPAs and courts have agreed with the EDPB by stating that the detailed transparency requirements under said provisions only apply to ‘qualifying’ ADM (see Chapter 1.6.c of the Report). Additionally, the Slovak Constitutional Court’s requirement goes further than Article 52 AI Act, which contains disclosure duties for users who deploy certain AI systems that the Proposal considers to be ‘low-risk’, such as emotion recognition systems, biometric categorization systems, and ‘deepfakes’. In the initial text of the AI Act, there are no transparency requirements towards affected persons when HRAIS are at stake, and there are no such obligations for non-high-risk systems other than the ones set out in Article 52 AI Act. Incorporating transparency rights for affected persons in the AI Act, even if only in HRAIS use cases, can reduce information asymmetries between individuals and organizations when decisions are not fully automated (e.g., an AI system whose recommendations merely support human decision-making). 

Lawful grounds for data processing: the AI Act sporadically mentions the interplay with the GDPR’s rules on lawful grounds and exemptions from the prohibition on processing special categories of data. Most notably, Article 54 AI Act creates stringent conditions for further processing of personal data in the context of AI regulatory sandboxes, which do not have an obvious connection to the purpose compatibility test in Article 6(4). Additionally, Article 10(5) AI Act authorizes providers of HRAIS to tackle potential biases through the processing of special categories of data, as long as appropriate safeguards are in place. However, it fails to elaborate on the conditions for the collection of personal data from publicly available sources for mandatory training, validation, and testing of HRAIS. The narrow interpretation of ‘manifestly making data public’ assumed by the EDPB and (more recently) by Advocate General Rantos of the CJEU, together with the enforcement actions against Clearview AI (cases 10 to 13 in the Report), may significantly hinder the possibilities for AI providers to scrape data from the web to test their AI models against bias. Obtaining consent from data subjects for those purposes is often unfeasible, and the legitimate interests lawful ground often plays a limited role when sensitive data are at stake.

Article 10(5) of the AI Act could also potentially facilitate compliance with Article 9(2)(g) GDPR, which allows for the processing of special categories of personal data where their processing is necessary for reasons of substantial public interest, as long as it is based on Union or Member State law. Provided that countering bias would be qualified as “substantial public interest”, the Union law specifically providing for an obligation to process sensitive data – which in this case would be the AI Act, needs to provide for suitable measures to safeguard fundamental rights.

This complexity could offer an opportunity for the EU legislator to set boundaries and clear rules on the collection and use of personal data for training, validation, and testing of AI systems, at least for HRAIS. 

AI risk management through the lens of ADM: Article 9 AI Act forces providers to establish and maintain a risk management system in relation to their HRAIS, including the ‘identification and analysis of [their] known and foreseeable risks.’ National court decisions on ‘legal or similarly significant effects’ of ADM under Article 22 GDPR may provide useful criteria that providers should consider when conducting risk analysis of HRAIS that could affect natural persons. In its Uber and Ola rulings (cases 17 to 19 in the Report), the District Court of Amsterdam analyzed the impact on drivers of the algorithms the companies were relying on for the functioning of their respective mobile applications providing ride-hailing services. The Court looked into: (i) the sensitivity of the data sets or inferences at stake; (ii) the temporary or definitive nature of the effects on data subjects (or their immediacy); (iii) the effects they would have on the drivers’ conduct or choices; and (iv) the seriousness of the financial impacts potentially involved for individuals. Factors such as these could be codified into Article 9 AI Act as valuable guidance for HRAIS providers’ risk management exercises. 

Incorporating human oversight does not rule out the Article 22 GDPR prohibition: a question arises about whether the Article 14 AI Act requirements for the provider to set up human oversight tools for HRAIS would bring such systems outside of Article 22 GDPR. The answer is ‘not necessarily.’ While the prohibition in Article 22 GDPR may apply to AI systems that are not considered ‘high-risk’ under the AI Act, when HRAIS are indeed at stake, Article 14 AI Act only requires providers to incorporate features that enable human oversight, but not to ensure human oversight as a default. 

That will be on the user of HRAIS (i.e., most likely the controller under the GDPR) to ensure via organizational arrangements. If the latter does not, then it may be in breach of Article 22 GDPR, if its ADM scheme is covered by the prohibition. Moreover, we have learned from the decision of the Portuguese DPA against a university that used proctoring software to monitor its students during exams (case 26 in the Report), and the court cases involving Dutch gun applicants and Austrian jobseekers (cases 8 and 9 in the Report), that merely having a human in the loop with the power and competence to make final decisions does not necessarily mean that the decision will not be considered ‘solely’ automated, and thus, that Article 22 GDPR does not apply. For that, human decision-makers need to receive clear instructions or training about why and when they should follow the AI system’s recommendations or not. 

In that respect, the EU legislator could envision that users have an obligation to inform their human decision-makers about the elements listed under Article 14(4) AI Act, so that the latter are able to make informed decisions based on the HRAIS’s output, and avoid so-called ‘automation bias’. 

5. Prohibited AI practices and HRAIS overlap with GDPR enforcement precedents on ADM

In general, the AI Act’s Annex III list of HRAIS seems to be based on litigated uses of AI systems by private and public bodies, including some that were analyzed by courts and DPAs under the GDPR and thereunder deemed to be unlawful for a variety of reasons, as the FPF Report shows. Some examples include:

• Cases of biometric identification and categorisation, where DPAs have often found the underlying collection and storage of data, plus the training of the AI system to be in breach of the GDPR’s rules on lawful grounds (e.g., Clearview AI enforcement cases);
• Systems that select students for university admissions (case 25 in the Report), assess or test them (cases 7 and 26);
• Some uses of AI systems in recruitment processes, which may be justified under the exception in Article 22(2)(a) GDPR (case 3 in the Report);
• AI systems used for worker management or for managing ride sharing apps and the service provided by gig workers were the focus of the Italian DPA (cases 3 and 6), and Dutch courts (cases 17 to 19 in the Report);
• Usages of AI to manage public services and benefits (cases 9, 20 27, and 32 in the Report), where DPAs agree that strong data quality and bias monitoring requirements are essential;
• Automated creditworthiness checks through AI (cases 14, 15, 22, 23, and 36 to 39 in the Report), although Annex III excludes from its scope AI systems that are developed by small scale providers for their own use.

Despite that significant overlap, some AI use cases that were investigated by European DPAs and courts have not been included in the Annex III list, including recommender and content moderation systems (case 24 in the Report), online behavioral advertising systems, and systems used by tax authorities to detect potential fraud (cases 4 and 27 in the Report). Likewise, the EC did not include commercial emotion recognition systems in the HRAIS list, in spite of the recent Hungarian DPA decision against a bank that used an AI system to detect the emotions of the customers who contacted its help center, and the fact that such systems are included in paragraph 6(b) of the Annex when used for law enforcement purposes. These enforcement actions could be an early indicator of the future potential enlargement of HRAIS use cases outlined in Annex III.

Some of these use cases may already be prohibited under the AI Act’s Article 5(1), notably where they would rely on ‘subliminal techniques beyond a person’s consciousness in order to materially distort a person’s behavior’ and may lead to harm, or where they would constitute “social scoring”. The latter could be the case of the SyRi algorithm, which the Dutch Government used when trying to detect instances of benefits fraud in neighborhoods hosting poor or minority groups, and that both the District Court of The Hague and the Dutch DPA deemed to be unlawful (case 27 in the Report). In any case, some of the concepts under Article 5 AI Act (like ‘detrimental or unfavorable treatment’) could be clarified to make the prohibition more certain for AI providers and users, lest the latter will struggle as much as with Article 22 GDPR’s definition of ‘legal or similarly significant effects.’ 

6. The issue of ensuring redress where AI systems not underlined by personal data may significantly affect individuals and communities

It is undeniable that the GDPR provides meaningful rights to individuals who are subject to or affected by AI systems underpinned by processing of personal data. For example, they can access a detailed description of how they were profiled through Article 15, as well as obtain human intervention, express their point of view, and file objections when ‘qualifying ADM’ is at stake. Additionally, providers and users of AI systems have a priori obligations when they act as ‘controllers’ under the GDPR, to implement data minimization, storage limitation, purpose limitation, confidentiality and, most relevant, fairness requirements in how they collect and use personal data, just to name a few of their obligations. 

However, when ‘qualifying ADM’ is not at stake, ‘controllers’ do not have an obligation to ensure human intervention or the possibility to contest an AI-powered decision. This is a gap that could potentially be tackled through the AI Act.

 On this note, although the AI Act is largely not rights-oriented, Article 52 AI Act on disclosure duties for certain AI systems creates a precedent for enshrining judiciable rights for individuals who are exposed to or targeted by AI systems. 

On the other hand, the Slovak Constitutional Court ruling (case 4 in the Report) required the local legislature to enshrine redress rights for individuals to effectively defend themselves against errors of the automated system at issue. Such actions could be possible under the broad effective judicial redress provided by Article 82 GDPR, but only where the underlying processing of personal data would be conducted in breach of GDPR rules such as transparency, fairness, and purpose limitation. For those cases where processing of personal data is not involved, but where AI systems may still significantly impact individuals, the AI Act could potentially fill a redress gap.  

7. Lessons from the ADM GDPR case law in the AI Act context

Some of our findings in the ADM GDPR Case-Law Report are useful for the debate around the AI Act, as shown above. Putting them in the context of the European Commission’s Proposal to regulate AI and the ensuing legislative process, we found that:

1. The AI Act and the GDPR are bound to work in tandem – they are both grounded on Article 16 TFEU and they have many areas where they complement each other, as well as areas where they could be better coordinated so that both their goals are achieved. For instance, the obligations for ‘users’ and ‘providers’ under the AI Act could be further attuned to the GDPR, by enhancing and clarifying transparency requirements of ‘providers’ towards users or by including a general reference to best efforts of providers to facilitate GDPR compliance where ‘users’ are deemed as ‘controllers’. 
2. Data accuracy and data quality requirements under the GDPR could be strengthened with cross-references in the AI Act for HRAIS, such as in Article 29.
3. Further safeguards for the processing of sensitive personal data to counter bias in AI systems could be laid out more clearly for the purpose of protecting fundamental rights.
4. Obligations to train ‘humans in the loop’ on what criteria they should take into account to rely on or to overturn a decision resulting from the application of a HRAIS could complement the protections the GDPR affords through Article 22. 
5. The existing and future GDPR case-law on ADM may be a source for the future enhancement of the HRAIS list provided in Annex III of the AI Act, considering that most of the existing HRAIS list overlaps with ADM systems that were already subject to GDPR enforcement.
6. While the GDPR right to an effective judicial redress can potentially cover most situations where AI systems relying on or resulting in personal data infringe principles like fairness, transparency, and purpose limitation, there is room to consider options for further redress, such as in situations where AI systems rely on non-personal data and significantly affect the rights of individuals or communities. 

Understanding Extended Reality Technology & Data Flows: XR Functions

This post is the first in a two-part series on extended reality (XR) technology, providing an overview of the technology and associated privacy and data protection risks.

Click here for FPF’s infographic, “Understanding Extended Reality Technology & Data Flows.” 

I. Introduction

Today’s virtual (VR), mixed (MR), and augmented (AR) reality environments, collectively known as extended reality (XR), are powered by the interplay of multiple sensors, large volumes and varieties of data, and various algorithms and automated systems, such as machine learning (ML). These complex relationships enable functions like gesture-based controls and eye tracking, without which XR experiences would be less immersive or unable to function at all. However, these experiences often depend on sensitive personal information, and the collection, processing, and transfer of this data to other parties may pose privacy and data protection risks to both users and bystanders.

This blog post analyzes the XR data flows that are featured in FPF’s infographic, “Understanding Extended Reality Technology & Data Flows.” This post focuses on some of the functions that XR devices support today and may support in the near future, analyzing the kinds of sensors, data types, data processing, and transfers to other parties that enable these functions. The next blog post will identify some of the privacy and data protection issues that XR technologies raise.

II. XR Functions

XR devices use several sensors to gather personal data about users and their surroundings. Devices may also log other types of data: data about a person’s location when the device connects to GPS, cell towers, or other connected devices around it; data about the device’s hardware and software; and usage and telemetry data. Devices utilize this data and may further transfer it to enable a variety of functions, which are the technologies that power use cases. For example, eye tracking is a function that enables the use case of optimized graphics.  

A. Mapping and Understanding the User’s Environment

Sensors on XR devices may work in tandem to collect various kinds of data—such as surrounding audio, the device’s acceleration, orientation, and environment depth data—to map and find objects in the user’s physical environment. Mapping the space entails constructing three-dimensional representations of the user’s environment in order to accurately situate users and content within a virtual space. Understanding the user’s environment involves identifying physical objects or surfaces in the user’s space to help place virtual content. These functions may enable shared experiences and other use cases.

To map and identify objects in the user’s environment, XR devices collect data across a number of sensors, such as microphones, cameras, depth sensors, and inertial measurement units (IMUs), which measure movement and orientation. When a sensor is experiencing a performance problem or certain sensor data is not available, the device may utilize other sensors, which may implicate a less accurate data proxy or fallback. For example, if photons from a depth sensor fail to indicate a user’s position, the device may use an AI system to fill in the sensory gap using pixels closest to the area where the depth sensor directed the photons. 

Once the device has gathered data through its sensors, the device and XR applications may need to further process this data to map and identify objects in the user’s physical space. The kind of processing activity that occurs depends on the features a developer wants its application to have. A processing activity that often occurs after a device collects sensor data is sensor fusion, in which algorithms combine data from various sensors to improve the accuracy of simultaneous localization and mapping (SLAM) and concurrent odometry and mapping (COM) algorithms. SLAM and COM map users’ surrounding areas, including the placement of landmarks or map points, and help determine where the user should be placed in the virtual environment. Some types of XR, including certain MR applications, leverage computer vision AI systems to identify and place specific objects within an environment. These applications may also use ML models to help determine where to place “dynamic” virtual content—virtual objects that respond to changes in the environment caused by adjustments to the user’s perspective. These mapping and object identification functions may also allow for shared experiences. For example, in a theoretical pet simulation, multiple users could toss a virtual ball against a building wall for a virtual puppy to catch. 

While XR devices generally do not send mapping and environmental sensor data to other parties, including other users, there are a few exceptions. For example, raw sensor data may be transmitted to XR device manufacturers to improve existing device functions, such as the placement and responsiveness of virtual content that users interact with. An XR device may also process and relay users’ location information, such as approximate or precise geolocation data, to enable shared experiences within the same physical space. For instance, two individuals in a public park could interact with each other’s virtual pets in an AR app, with each player using their own devices that recognize the placement of both the virtual pets and the other player in the park. In other situations, certain parties can observe processed sensor and other data, such as an application developer or an entity controlling the external server that enables an application’s multiplayer functionality. Therefore, the nature of the data, device and application features may affect who can access XR data.

B. Controller and Gesture-Based Interactions with the Environment

Some XR technologies gather and process sensor data to enable controller- and gesture-based interactions with physical and virtual content, including other users. Gesture-based controls allow users to interact with and manipulate virtual objects in ways that are more reflective of real-world interactions. Most devices use IMUs and outward-facing cameras combined with infrared (IR) or LED light systems to gather data about the controller’s position, such as the controller’s linear acceleration and rotational velocity, as well as optical data about the user’s environment. Some manufacturers are introducing new data collection systems that overcome other methods’ deficiencies, such as when the controllers are outside of the cameras’ view. When visual information about a controller’s position becomes unavailable, IMU data may act as a fallback or proxy for determining controller location. For gesture-based controls, devices gather data about the user’s hands through outward-facing cameras. 

XR technologies use algorithms and ML models to provide controller- and gesture-based controls. In controller-based systems, algorithms use data about the controller’s position to detect and measure how far away the controllers are from the user’s head-mounted display (HMD). This allows the user’s “hands” to interact with virtual content. For example, MR or VR maintenance training could allow a mechanic to practice repairing a virtual car engine before performing these actions in the real world. Gesture-based controls utilize ML models, specifically deep learning, to construct 3D copies of the user’s hands by processing images of their physical-world hands and determining the location of their joints. The 3D copies may be sent to developers to enable users to manipulate and interact with virtual and physical objects in applications through pointing, pinching, and other gestures. 

C. Eye Tracking and Authentication

Eye tracking and, to a lesser extent, authentication power a variety of XR use cases, such as user authentication, optimized graphics, and expressive avatars. An XR device can use data from the user’s eye to authenticate the person using the device, ensuring that the right user profile, with its unique settings, applies during a session. Devices may use inward-facing IR cameras to gather information about the user’s eyes, such as retina or iris data, to this end. ML models can then use the collected eye data to determine whether the person is who they claim to be. 

Now and in the future, XR devices will increasingly feature eye tracking to optimize graphics. Graphics quality can affect a user’s sense of immersion, presence, and embodiment in XR environments. One technology that can enhance graphics in XR environments is dynamic foveated rendering, or eye-tracked foveated rendering (ETFR), which tracks a user’s eyes to reduce the resolution that appears in the peripherals of the HMD’s display. This allows the device to display the user’s focal point in high resolution, reduce processing burdens on the device, and lower the chance of motion sickness by addressing a cause of latency. XR devices may also facilitate better graphics by measuring the distance between a user’s pupils or interpupillary distance (IPD), which affects the crispness of the images on the headset display. For example, in a virtual car showroom, a device may utilize the above technologies to enhance the detail of the car feature that the user is looking at and ensure that objects appear at the correct scale. 

To determine the parts of the HMD display that should be blurred and help focus the lenses, a device may use inward-facing IR cameras to gather data about the user’s eyes. Some XR devices may use ML models, such as deep learning, to process eye data to predict where a user is looking. These conclusions inform what parts of the display the model blurs. In the future, XR devices may use algorithms to more accurately measure the distance between a user’s pupils, further improving the crispness of the graphics that appear on the HMD’s display. 

Eye tracking is a subset of a broader category of XR data collection—body tracking. Body tracking captures eye movements (described above), facial expressions, and other body movements, which can help create avatars that reflect a user’s reactions to content and expressions in real-time. Avatars are the person’s representative in a virtual or other artificial environment. Avatars are already featured in popular shared VR experiences, such as VRChat and Horizon Worlds, but several factors limit their realism. Today’s avatars typically do not reflect all of a user’s nonverbal responses and may lack certain appendages, like legs. Going forward, an avatar may mirror a user’s reactions and expressions in real-time. This will enable more realistic social, professional, and personal interactions. For example, in a VR comedy club, a realistic avatar may display facial and other body movements to more effectively deliver or react to a punchline. 

To depict a user’s reactions and expressions on their avatar, XR technologies need data about the eyes, face, and other parts of the user’s body. A device may use IMUs and internal- and outward-facing cameras to capture information about the user’s head and body position, gaze, and facial movements. XR devices may also use microphones to capture audio corresponding with certain facial movements, known as visemes, as a proxy for visuals of the user’s mouth when the latter is unavailable. For instance, the sound of laughter may cause an avatar to show behavior associated with the act of laughing. 

As with the other functions, body-based insights on XR devices may need to transmit data to other parties, which may use algorithms to process collected data to create expressive avatars. XR devices may transmit data about a user’s avatar, such as gaze and facial expression information to app developers. Developers may use ML models, including deep learning, to process information about the user’s eyes and face to detect the face and make conclusions about where a user is looking and the expression they are making. For facial movements, a deep learning model may analyze each video frame featuring the user’s face to determine with which expressions their facial movements correspond. These expressions are then displayed on the avatar. Devices may then share the avatar with central servers so that other users can view and interact with the avatar.

In addition to avatar creation and operation, future XR technologies may monitor gaze and pupil dilation, motion data, and other information derived from the user’s body to generate behavioral insights. XR tech may be capable of using sensor data that is generated in response to stimuli and interactions with content to make inferences about user interests, as well as their physical, mental, and emotional conditions. When combined with information processed by other sensors, such as brain-computer interfaces (BCIs), these body-derived data points could contribute to the creation of more granular individual profiles and insights into the user’s health. In a medical XR application, for example, doctors could use gaze tracking to diagnose certain medical conditions. However, other parties may use the functions to learn about other highly sensitive information, such as a user’s sexual orientation, which could harm individuals. 

III. Conclusion

XR technologies often rely on large volumes and varieties of data to power device and application functions. Devices often feature several sensors to gather this data, such as outward-facing cameras and IMUs. To enable different kinds of XR use cases, including shared experiences, devices may utilize ML models that process data about the user and their environment and transmit this data to other parties. While the collection, processing, and transmission of this data may be integral to immersive XR experiences, it can also create privacy and data protection risks for users and bystanders. The next blog post analyzes some of these risks. 

Read the next blog post in the series: Privacy and Data Protection Risks to Users and Bystanders