[{"id":"privacy/whatIsPrivacy#intro","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"intro","title":"Introduction","content":"Here we present some of the answers different thinkers and disciplines give to the question, \"What is privacy about?\" We do not aim to comprehensively survey theories or take sides in the many debates happening around this topic in philosophy, psychology, anthropology, and other disciplines. Rather, we offer several lenses through which to understand privacy and what it delineates. The goal is to get students thinking critically about privacy and its applications to computer science.","sidebarKey":null,"isDrawer":false},{"id":"privacy/whatIsPrivacy#lens-1-privacy-is-about-the-ability-to-restrict-access-to-oneself","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"lens-1-privacy-is-about-the-ability-to-restrict-access-to-oneself","title":"Lens 1: Privacy is about the ability to restrict access to oneself.","content":"Most people colloquially understand privacy as the condition of restricted access. We have privacy when we can close a door, make a friends-only post on social media, or otherwise limit when and where others perceive us. This conception is in line with **access-based approaches**, which cast privacy as about **who has access to an individual and under what conditions**. Having access to an individual could entail having physical proximity to or contact with them, learning information about them, observing them, etc.\n\n### Access as a Gateway\n\nThrough this lens, privacy functions {gate-we-can-open-close-to-others|as a gate we can open or close to others}.\nUsing the example of a close friends list on social media, we open the access gateway by adding accounts to that list and close the gateway by removing them.\n\n### Anti-informational Definitions of Privacy\n\nTaken to its logical extreme, the access-based lens suggests that \"an individual enjoys perfect privacy when he is completely inaccessible to others\" (Gavison 1980, 428). We therefore lose privacy when our information becomes accessible to others. One way to guarantee data's inaccessibility is to never produce it in the first place, leading some to conclude that privacy is best defined as **protection against the creation of information**. While most scholars have moved away from such anti-informational definitions in recent decades, the idea that privacy is fundamentally opposed to information is earning renewed attention following the publication of Lowry Pressly's [_The Right to Oblivion_](https://www.hup.harvard.edu/books/9780674260528) (2024).","sidebarKey":null,"isDrawer":false},{"id":"privacy/whatIsPrivacy#lens-2-privacy-is-about-the-ability-to-control-the-flow-of-information-about-oneself","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"lens-2-privacy-is-about-the-ability-to-control-the-flow-of-information-about-oneself","title":"Lens 2: Privacy is about the ability to control the flow of information about oneself.","content":"### Classic Control-Based Approaches\n\nControl-based approaches **consider privacy in terms of how data is not only accessed but also collected, shared, and used**. Through this lens, privacy is more than just an access gateway that we can open or close. Advocates promote a control-based approach to privacy as empowering people to actively govern their privacy **throughout the data lifecycle**, from creation through destruction. In accordance with this principle, the [European Union's GDPR](https://gdpr.eu/what-is-gdpr/) (General Data Protection Regulation) and similar data privacy regulations (e.g., the California Consumer Privacy Act) require that data subjects maintain a high level of control over their personal information for the duration of its existence.\n\n**Figure:** Diagram of the research data lifecycle from the [Longwood Medical Area Research Data Management Working Group](https://datamanagement.hms.harvard.edu) at Harvard Medical School. Detailed accessible version available online [here](https://datamanagement.hms.harvard.edu/plan-design/biomedical-data-lifecycle#).\n![The Biomedical Data Lifecycle diagram depicts the core stages of the research lifecycle. The center of the wheel has a grey circle labeled Store & Manage. The layer surrounding this is cut into six segments in this order: Plan &  Design, Collect & Create, Analyze & Collaborate, Evaluate & Archive, Share & Disseminate, and Publish & Reuse. Clockwise guiding arrows are at the edge of each segment, showing that one stage leads into the next. The third layer expands on each topic by listing activities or resources that are involved in each of the seven segments in the other two layers.](/assets/primer-photos/privacy/whatIsPrivacy/dataLifecycle.png)\n\nWestin defines privacy as \"the claim of individuals, groups, or institutions to determine for themselves when, how and to what extent information about them is communicated to others\" (1967, 7). Privacy loss is therefore thought of as the loss of control over personal information (Véliz 2024, 75). For example, a {man-in-the-middle-attack} intercepts communications along what was believed to be a private channel, violating the sender's privacy by undermining their ability to control the flow of information.\n\nContrary to an access-based perspective, a control-based approach views the possibility of unauthorized access to information as eroding privacy, even if that information is never actually accessed. We might use the metaphor of a diary to think through the difference. Say someone takes your diary, which is full of all your most private thoughts, from your desk drawer, locks it in a safe without reading it, and throws away the key. From an access-based perspective, you have not lost privacy because no one read your diary. From a control-based perspective, you have lost privacy because the diary is no longer in your possession.\n\n### Contextual Integrity\n\nIn the last decade, Helen Nissenbaum's theory of {contextual-integrity} has gained traction, especially among computer scientists. Contextual integrity is a theory of privacy developed within the landscape of 21st century computing. Nissenbaum posits that privacy is best understood as the \"appropriate flow of personal information\" (2010, 127). Appropriate flow of information is defined circumstantially according to five parameters: the data subject, sender, recipient, information type, and transmission principle (e.g., confidentially, with notice, with consent). This conception of privacy allows people to define for themselves the boundaries of acceptable data collection, storage, use, and dissemination. It is therefore flexible and responsive to evolving ethical norms.","sidebarKey":null,"isDrawer":false},{"id":"privacy/whatIsPrivacy#lens-3-privacy-is-about-the-separation-of-public-and-private-spheres","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"lens-3-privacy-is-about-the-separation-of-public-and-private-spheres","title":"Lens 3: Privacy is about the separation of public and private spheres.","content":"A lens of privacy focused on the separation of public and private spheres is in line with the idea that certain kinds of information are inherently private. Compared to contextual integrity, this view is more rigid in its approach to boundary-setting. Public-private distinctions have long been thought of in spatial terms. The home, for instance, is a private sphere and the town square a public one. However, modern theories under this umbrella are as concerned with the _who_ and _what_ than the _where_.\n\n### Spheres as Relational\n\nSome claim that privacy is about **moderating relationships between different entities by allowing disparate access to one's private sphere**. One recent way of representing this idea is a layered model of privacy (sometimes referred to as \"the onion model of privacy\"). The layered model presents privacy as nested, with personal and bodily intimacy and privacy at the center. The middle layers are personal relationships, and the outer layer is civil society more broadly. Privacy loss, analogous to removing layers and letting others in, makes us vulnerable. Voluntary disclosure can be viewed as \"a gesture of trust that, when received with sensitivity and trustworthiness, strengthens relationships\" (Véliz 2024, p. 83).\n\n**Figure:** The layered model of privacy.\n![An illustration of semi-circles in descending size from left to right. An arrow pointing to the largest semi-circle is labeled \"Civil society (e.g., things shared with the government).\" An arrow pointing to the medium-sized semi-circle is labeled \"Personal relationships (e.g., things shared with friends).\" An arrow pointing to the smallest semi-circle is labeled \"Personal & bodily intimacy (e.g., things shared with life partner).\"](/assets/primer-photos/privacy/whatIsPrivacy/layered_model_of_privacy.png)\n\n### Spheres as Action-Based\n\nOthers think of privacy **being inherent to specific types of actions (i.e., the what)** rather than spatial or relational contexts (i.e., the where and with whom). These thinkers believe privacy is about a **private sphere of action** that should be free from external interference. Examples of activities that are commonly understood to fall within the private sphere of action include voting, making medical decisions, and engaging in consensual sexual activity.\n\nThis conception of privacy aligns with the \"{constitutionally-protected-reasonable-exprectation-of-privacy}\" established in _Katz v. United States_. Privacy rights in certain spheres are inferred from the Bill of Rights and have historically protected autonomy around \"child rearing and education, family relationships, procreation, marriage, contraception, and abortion\" (Inness 1992, 64).\n\n### Delineating Public and Private\n\nPublic-private distinctions have been continually re-examined throughout history with the proliferation of new technologies. For instance, {Warren-and-Brandeis}' efforts in 1890 to define a legal right to privacy were inspired by the evolution of photographic technology, which could broadcast private or semi-private moments in unprecedented ways. Did the public sphere now include anything the camera's lens could capture? Warren and Brandeis said no. This sort of reasoned delineation between public and private is a key aspect of {developing-appropriate-privacy-protections}.\n\nIn the 21st century, technology enables {previously-unimaginable-intrusions} into private spheres. The home has long been considered the locus of privacy, and the integration of computers into home life creates a \"porousness\" wherein information may leak out.","sidebarKey":null,"isDrawer":false},{"id":"privacy/whatIsPrivacy#lens-4-privacy-is-about-plausible-deniability","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"lens-4-privacy-is-about-plausible-deniability","title":"Lens 4: Privacy is about plausible deniability.","content":"A somewhat recent development in our thinking on privacy comes from Cynthia Dwork and colleagues in cryptography who have developed techniques to define privacy mathematically. Through this lens, data is private when it functionally cannot be uncovered or linked back to a specific individual, thus preserving a person's ability to plausibly deny a piece of information.\n\n[Canetti, Dwork, Naor, and Ostrovsky (1997)](https://link.springer.com/chapter/10.1007/BFb0052229) introduced the concept of deniable encryption. A **deniable encryption** scheme is one in which a message can be decrypted with the real key to reveal the true message or a fake key to reveal some other text. This allows someone to give the appearance of transparency without actually sacrificing privacy.\n\n[Dwork, McSherry, Nissim, and Smith (2006)](https://link.springer.com/chapter/10.1007/11681878_14#Bib1) describes **differential privacy (DP)**, a {framework} used for privacy-preserving statistical analysis and machine learning. Differentially private algorithms rely on the careful introduction of statistical noise to obscure identity without meaningfully altering results. The inclusion or non-inclusion of a particular observation has no statistically significant impact on the outcome, so there is no way to prove a certain individual is represented in the data set. On this basis, a person can always plausibly deny contributing data.\n\n#### Figure: Table summarizing four views of privacy.\n\n![A table summarizing four views of privacy. It lists that privacy is about access restriction, information flow control, the separation of public and private spheres, and plausible deniability. It lists that someone has privacy if their physical self and information about them is inaccessible to others, they are empowered to control if and how information about them is collected, shared, and used, their personal spaces, relationships, and actions are free from interference, and there is a mathematical guarantee their information or identity is unrecoverable.\"](/assets/primer-photos/privacy/whatIsPrivacy/table_of_privacy.png)","sidebarKey":null,"isDrawer":false},{"id":"privacy/whatIsPrivacy#privacy-from-whom","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"privacy-from-whom","title":"Privacy from Whom?","content":"Privacy is relational and must be understood with respect to a certain person or entity. The importance of the \"privacy from whom?\" question is well illustrated in this short example:\n\n\"Consider the case of a couple having a quiet conversation in their home. While neither spouse has privacy with respect to the other and with regard to their feelings (assuming they are being honest), the couple have privacy with respect to the passersby who cannot hear or see them.\" (Véliz 2024, 83)\n\nThe things we want to keep private and how we want our privacy to be protected naturally vary depending on from whom we desire privacy. [Value of Privacy](/privacy/valueOfPrivacy) includes a number of case studies on circumstances in which privacy from certain entities promotes various individual and societal goods.","sidebarKey":null,"isDrawer":false},{"id":"privacy/whatIsPrivacy#further-reading","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"further-reading","title":"Further Reading","content":"Alibeigi, Ali, Abu Bakar Munir, and Md Ershadul Karim. \"Right to Privacy, a Complicated Concept to Review.\" _Library Philosophy and Practice (e-Journal)_, January 1, 2019. https://digitalcommons.unl.edu/libphilprac/2841.\n\nAllen, Anita. \"Privacy-as-Data Control: Conceptual, Practical, and Moral Limits of the Paradigm.\" _Connecticut Law Review_ 32 (January 1, 2000): 861–75. https://scholarship.law.upenn.edu/faculty_scholarship/790.\n\nBhave, Devasheesh P., Laurel H. Teo, and Reeshad S. Dalal. \"Privacy at Work: A Review and a Research Agenda for a Contested Terrain.\" _Journal of Management_ 46, no. 1 (2020): 127–64. https://doi.org/10.1177/0149206319878254.\n\nboyd, danah. \"What Is Privacy?\" _The Message_ (blog), August 1, 2014. https://medium.com/message/what-is-privacy-5ed72c66aa86.\n\nBratman, Ben. \"Brandeis & Warren's 'The Right to Privacy' and the Birth of the Right to Privacy.\" _Tennessee Law Review_ 69 (January 1, 2002): 623. https://scholarship.law.pitt.edu/fac_articles/63.\n\nBye, Kent. \"Primer on the Contextual Integrity Theory of Privacy with Philosopher Helen Nissenbaum.\" _Voices of VR_ (blog), June 24, 2021. https://voicesofvr.com/998-primer-on-the-contextual-integrity-theory-of-privacy-with-philosopher-helen-nissenbaum/.\n\nCanetti, Rein, Cynthia Dwork, Moni Naor, and Rafail Ostrovsky. \"Deniable Encryption.\" In _Advances in Cryptology — CRYPTO '97_, edited by Burton S. Kaliski, 1294:90–104. Berlin, Heidelberg: Springer Berlin Heidelberg, 1997. https://doi.org/10.1007/BFb0052229.\n\nClark, Andy, and David Chalmers. \"The Extended Mind.\" _Analysis_ 58, no. 1 (January 1998): 7–19. http://www.jstor.org/stable/3328150.\n\nDesfontaines, Damien. \"A List of Real-World Uses of Differential Privacy.\" _Ted Is Writing Things_ (blog), March 22, 2025. https://desfontain.es/blog/real-world-differential-privacy.html.\n\nDwork, Cynthia, Frank McSherry, Kobbi Nissim, and Adam Smith. \"Calibrating Noise to Sensitivity in Private Data Analysis.\" In _Theory of Cryptography_, edited by Shai Halevi and Tal Rabin, 265–84. Berlin, Heidelberg: Springer, 2006. https://doi.org/10.1007/11681878_14.\n\nElectronic Privacy Information Center. \"Differential Privacy.\" epic.org. Accessed April 12, 2025. https://epic.org/differential-privacy/.\n\nFPC. \"Fair Information Practice Principles (FIPPs).\" Accessed April 11, 2025. https://www.fpc.gov/resources/fipps/.\n\nFreivogel, William H. \"The Right to Be Let Alone.\" In _The SAGE Guide to Key Issues in Mass Media Ethics and Law_, edited by William H. Freivogel and William A. Babcock, 303–18. 2455 Teller Road, Thousand Oaks California 91320: SAGE Publications, Inc., 2015. https://doi.org/10.4135/9781483346540.n27.\n\nGavison, Ruth. \"Privacy and the Limits of Law.\" _The Yale Law Journal_ 89, no. 3 (1980): 421–71. https://doi.org/10.2307/795891.\n\nGDPR. \"General Data Protection Regulation (GDPR) – Legal Text,\" May 25, 2018. https://gdpr-info.eu/.\n\nGellman, Bart, and Ashkan Soltani. \"NSA Infiltrates Links to Yahoo, Google Data Centers Worldwide, Snowden Documents Say.\" _The Wahington Post_, October 30, 2013. https://www.washingtonpost.com/world/national-security/nsa-infiltrates-links-to-yahoo-google-data-centers-worldwide-snowden-documents-say/2013/10/30/e51d661e-4166-11e3-8b74-d89d714ca4dd_story.html.\n\nHarvard University Privacy Tools Project. \"Differential Privacy.\" Accessed April 11, 2025. https://privacytools.seas.harvard.edu/differential-privacy.\n\nInness, Julie C. _Privacy, Intimacy, and Isolation_. New York: Oxford Univ. Press, 1996. https://doi.org/10.1093/0195104609.001.0001.\n\nKoops, Bert-Jaap, Bryce Newell, Tjerk Timan, Ivan Škorvánek, Tomislav Chokrevski, and Maša Galič. \"A Typology of Privacy.\" _University of Pennsylvania Journal of International Law_ 38, no. 2 (January 1, 2017): 483–575. https://scholarship.law.upenn.edu/jil/vol38/iss2/4.\n\nLindemulder, Gregg, and Matthew Kosinski. \"What Is a Man-in-the-Middle (MITM) Attack?\" IBM, June 11, 2024. https://www.ibm.com/think/topics/man-in-the-middle.\n\nLMA Research Data Management Working Group. \"Biomedical Data Lifecycle.\" Harvard Biomedical Data Management, 2024. https://datamanagement.hms.harvard.edu/plan-design/biomedical-data-lifecycle.\n\nLundgren, Björn. \"A Dilemma for Privacy as Control.\" _The Journal of Ethics_ 24, no. 2 (2020): 165–75. https://doi.org/10.1007/s10892-019-09316-z.\n\nMulligan, Deirdre K., Colin Koopman, and Nick Doty. \"Privacy Is an Essentially Contested Concept: A Multi-Dimensional Analytic for Mapping Privacy.\" _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 374, no. 2083 (December 28, 2016). https://doi.org/10.1098/rsta.2016.0118.\n\nNissenbaum, Helen. _Privacy in Context: Technology, Policy, and the Integrity of Social Life_. Stanford, CA: Stanford University Press, 2010.\n\nOyez. \"Carpenter v. United States.\" Accessed April 11, 2025. https://www.oyez.org/cases/2017/16-402.\n\nOyez. \"Griswold v. Connecticut.\" Accessed April 11, 2025. www.oyez.org/cases/1964/496.\n\nOyez. \"Katz v. United States.\" Accessed April 11, 2025. https://www.oyez.org/cases/1967/35.\n\nOyez. \"Kyllo v. United States.\" Accessed April 11, 2025. https://www.oyez.org/cases/2000/99-8508.\n\nOyez. \"United States v. Jones.\" Accessed April 11, 2025. https://www.oyez.org/cases/2011/10-1259.\n\nOyez. \"Van Buren v. United States.\" Accessed April 11, 2025. https://www.oyez.org/cases/2020/19-783.\n\nPressly, Lowry. _The Right to Oblivion: Privacy and the Good Life_. 1st ed. Cambridge: Harvard University Press, 2024. https://doi.org/10.2307/jj.16394399.\n\nRoessler, Beate, and Judith DeCew. \"Privacy.\" In _The Stanford Encyclopedia of Philosophy_, edited by Edward N. Zalta and Uri Nodelman, Winter 2023. Metaphysics Research Lab, Stanford University, 2023. https://plato.stanford.edu/archives/win2023/entries/privacy/.\n\nSolove, Daniel J. \"A Taxonomy of Privacy.\" _University of Pennsylvania Law Review_ 154, no. 3 (January 1, 2006): 477. https://doi.org/10.2307/40041279.\n\nThomson, Judith Jarvis. \"The Right to Privacy.\" _Philosophy & Public Affairs_ 4, no. 4 (1975): 295–314. https://www.jstor.org/stable/2265075.\n\nVéliz, Carissa. \"In the Privacy of Our Streets.\" In _Surveillance, Privacy and Public Space_, edited by Bryce Clayton Newell, Tjerk Timan, and Bert-Jaap Koops, 1st ed., 16–32. Routledge, 2018. https://doi.org/10.4324/9781315200811.\n\n———. _The Ethics of Privacy and Surveillance_. Oxford Philosophical Monographs. Oxford: Oxford University Press, 2024. https://philpapers.org/rec/VLITEO.\n\nWarren, Samuel D., and Louis D. Brandeis. \"The Right to Privacy.\" _Harvard Law Review_ 4, no. 5 (December 15, 1890): 193–220. https://doi.org/10.2307/1321160.\n\nWheeler, Evan. _Security Risk Management: Building an Information Security Risk Management Program from the Ground Up_. Elsevier Science & Technology Books, 2011. https://doi.org/10.1016/C2010-0-64926-1.","sidebarKey":null,"isDrawer":false},{"id":"privacy/whatIsPrivacy#sidebar-gate-we-can-open-close-to-others","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"gate-we-can-open-close-to-others","title":"Case Study:","content":"The notion of access to private information as a binary operator underpins the Supreme Court’s 2021 decision in [_Van Buren v. United States_](https://www.oyez.org/cases/2020/19-783), which clarified the meaning of the Computer Fraud and Abuse Act (CFAA). In the majority opinion, Justice Amy Coney Barrett wrote, “liability under [two clauses of the CFAA] stems from a gates-up-or-down inquiry—one either can or cannot access a computer system, and one either can or cannot access certain areas within the system” (2021, 13).\n\n## Further Reading\n\nWeb Content Accessibility Guidelines, Level A Checklist, 1.2.1\n\n\"Definition and Overview of Universal Design (UD).\" Centre for Excellence in Universal Design. Accessed April 14, 2025. https://universaldesign.ie/about-universal-design/definition-and-overview","sidebarKey":"gate-we-can-open-close-to-others","isDrawer":true},{"id":"privacy/whatIsPrivacy#sidebar-man-in-the-middle-attack","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"man-in-the-middle-attack","title":"man in the middle attack","content":"For more details on man-in-the-middle (MITM) attacks, see [“What is a man-in-the-middle (MITM) attack?”](https://www.ibm.com/think/topics/man-in-the-middle) from IBM.\n\n**Case:** Edward Snowden publicized evidence that the U.S. National Security Agency (NSA) was intercepting traffic between Yahoo and Google data centers through a surveillance program known as MUSCULAR. See 2013 the Washington Post article [\"NSA infiltrates links to Yahoo, Google data centers worldwide, Snowden documents say\"](https://www.washingtonpost.com/world/national-security/nsa-infiltrates-links-to-yahoo-google-data-centers-worldwide-snowden-documents-say/2013/10/30/e51d661e-4166-11e3-8b74-d89d714ca4dd_story.html) by Gellman and Soltani to learn more.","sidebarKey":"man-in-the-middle-attack","isDrawer":true},{"id":"privacy/whatIsPrivacy#sidebar-contextual-integrity","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"contextual-integrity","title":"contextual integrity","content":"For more, watch and/or read Voices of VR’s [\"Primer on the Contextual Integrity Theory of Privacy with Philosopher Helen Nissenbaum\"](https://voicesofvr.com/998-primer-on-the-contextual-integrity-theory-of-privacy-with-philosopher-helen-nissenbaum/).\n\nContextual integrity is applicable in legal and policy discussions of privacy. Its advocates assert that it is more comprehensive than the predominant [Fair Information Practice Principles](https://www.fpc.gov/resources/fipps/) (FIPPs) model.\n\nThe information flow control (IFC) model in cybersecurity can be seen as operationalizing Nissenbaum’s principle of appropriate flow. IFC focuses on “what information is authorized to be transferred between entities” (Wheeler 2011, 269). To learn more, see Wheeler’s book [_Security Risk Management_](https://ebookcentral-proquest-com.revproxy.brown.edu/lib/brown/detail.action?docID=685406).\n\nNot all aspects of Nissenbaum’s theory are straightforward to operationalize (e.g. some transmission principles such as “with consent”). IFC systems that allow for more nuanced control are an active area of research. The 2024 paper [“Sesame: Practical End-to-End Privacy Compliance with Policy Containers and Privacy Regions”](https://doi.org/10.1145/3694715.3695984) from Brown’s Efficient and Trustworthy Operating Systems (ETOS) Group is an example of such work.","sidebarKey":"contextual-integrity","isDrawer":true},{"id":"privacy/whatIsPrivacy#sidebar-constitutionally-protected-reasonable-exprectation-of-privacy","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"constitutionally-protected-reasonable-exprectation-of-privacy","title":"constitutionally protected reasonable exprectation of privacy","content":"**Case:** In his oft-cited concurrence to [_Katz v. United States_](https://www.oyez.org/cases/1967/35) (1967), Justice Harlan writes that a telephone booth is a private setting, more similar to one’s home, where one has a reasonable expectation of privacy, than to a field, where one does not have such an expectation. The delineation of public and private spheres in Katz is in relation to the Fourth Amendment protection against unreasonable searches and seizures by the government.\n\n**Case:** In [_Griswold v. Connecticut_](https://www.oyez.org/cases/1964/496) (1965), the Court found that a right to marital privacy inferred from the guarantees of the First, Third, Fourth, Fifth, and Ninth Amendments prevented the state from making contraceptive use illegal for married couples. This legal conception of privacy has been continually challenged. In a dissent to the original case, Justice Potter Stewart wrote, “I can find no such general right of privacy in the Bill of Rights, in any other part of the Constitution, or in any case ever before decided by this Court” (530). More recently, Justice Clarence Thomas rejected the argument behind Griswold in his concurrence to [_Dobbs v. Jackson Women’s Health Organization_](https://www.oyez.org/cases/2021/19-1392) (2022), which eliminated the constitutional right to abortion.","sidebarKey":"constitutionally-protected-reasonable-exprectation-of-privacy","isDrawer":true},{"id":"privacy/whatIsPrivacy#sidebar-warren-and-brandeis","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"warren-and-brandeis","title":"Warren and Brandeis","content":"For more on Warren and Brandeis, see the original article [\"The Right to Privacy\"](https://groups.csail.mit.edu/mac/classes/6.805/articles/privacy/Privacy_brand_warr2.html) (1890) or explore their ideas further in [\"The Right to Be Let Alone\"](https://sk.sagepub.com/hnbk/edvol/the-sage-guide-to-key-issues-in-mass-media-ethics-and-law/chpt/20-right-be-let-alone?PageNum=306) (2015) by Freivogel.","sidebarKey":"warren-and-brandeis","isDrawer":true},{"id":"privacy/whatIsPrivacy#sidebar-developing-appropriate-privacy-protections","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"developing-appropriate-privacy-protections","title":"developing appropriate privacy protections","content":"Since 2000, the Supreme Court has considered how privacy protections apply to thermal-imaging ([_Kyllo v. United States_](https://www.oyez.org/cases/2000/99-8508)), GPS tracking ([_United States v. Jones_](https://www.oyez.org/cases/2011/10-1259)), and cellphone records ([_Carpenter v. United States_](https://www.oyez.org/cases/2017/16-402)) among other emerging technologies.","sidebarKey":"developing-appropriate-privacy-protections","isDrawer":true},{"id":"privacy/whatIsPrivacy#sidebar-previously-unimaginable-intrusions","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"previously-unimaginable-intrusions","title":"previously unimaginable intrusions","content":"As Véliz puts it, “What used to be the paradigmatic loci of the private—our homes and our minds—have become the paradigmatic loci of data collection” (2024, 45). For more on the mind component, see [“The Extended Mind”](https://www.jstor.org/stable/3328150) (1998) by Clark and Chalmers.","sidebarKey":"previously-unimaginable-intrusions","isDrawer":true},{"id":"privacy/whatIsPrivacy#sidebar-framework","section":"privacy","sectionTitle":"Privacy","subsection":"whatIsPrivacy","subsectionTitle":"What is Privacy?","anchor":"framework","title":"framework","content":"For more, see the [Harvard University Privacy Tools Project](https://privacytools.seas.harvard.edu/differential-privacy). Or explore [\"A list of real-world uses of differential privacy\"](https://desfontain.es/blog/real-world-differential-privacy.html) by Damien Desfontaines.","sidebarKey":"framework","isDrawer":true},{"id":"privacy/valueOfPrivacy#what-is-privacy-good-for","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"what-is-privacy-good-for","title":"What Is Privacy Good For?","content":"As previously seen, privacy can be [**understood**](whatIsPrivacy) in a myriad of ways. While conceptions of privacy vary, there is broad agreement that privacy facilitates certain {goods} for both individuals and society at large. The way people determine and prioritize these goods depends on their understanding of privacy.\n\nThis section provides a broad set of goods {across-definitions} without any particular ordering.\n\nMany people hold the belief that privacy does not have value. This is often framed in questions similar to “if I’ve done nothing wrong, why do I need privacy?” Rather than countering this argument directly, we present the ways privacy could be useful across contexts. We strive to answer the guiding question: “what is privacy good for?”\n\n### Privacy in Society\n\nOn a societal level, privacy may be useful in supporting a functional, {liberal-democratic-system}, particularly as it facilitates agency, equality, and decision-making. Some of the benefits of privacy may include:\n\n- {Protection-from-Violence-and-Bad-Actors}\n- {Development-of-Countercultures}\n- {Political-Participation-Without-Judgment}\n- {Fairness-by-Obscuring-Identity}\n- {Private-Control-of-Property}\n- {Creating-Different-Relationships}\n\n### Privacy for Individuals\n\nPrivacy may also support several goods that we may want as individuals, separate from the broader benefits that they bring to society. These include:\n\n- {Increased-Agency-Over-Self-and-Actions}\n- {Personal-Development-Through-Experimentation}\n- {Ability-to-not-be-Seen-as-a-Particular-Identity-or-Characteristic}\n- {Allows-Different-People-to-Know-Different-Things-About-Oneself}","sidebarKey":null,"isDrawer":false},{"id":"privacy/valueOfPrivacy#who-benefits-from-privacy","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"who-benefits-from-privacy","title":"Who Benefits From Privacy?","content":"The value of privacy may vary significantly depending on who you are and what your corresponding needs are.\n\nIf you exist as part of a large group (which you do!), then it’s possible to use the characteristics of that group in aggregate to {learn-information-about-all-members} of the group that wouldn’t be available otherwise. Group membership may also be exploited to {gain-knowledge-about-a-specific-individual} that could be used to target them. More intentionally formed groups, such as government agencies, businesses, activist organizations, {might-also-seek-out-privacy} for various purposes.\n\nPeople in {historically-marginalized-groups} may also place increased value on privacy. This may be due to harassment on the basis of identity, from both organizations (state actors) and individuals. Marginalized groups are also more likely to disagree with normative status quo that causes them to be marginalized, and may wish to experiment outside of these boundaries to prioritize justice goals. From this, there may also be a want to not be seen based upon the basis of their identity, or to associate with others in a manner that deviates from social norms and regulations.","sidebarKey":null,"isDrawer":false},{"id":"privacy/valueOfPrivacy#complicating-privacy","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"complicating-privacy","title":"Complicating Privacy","content":"Privacy is highly context-specific. As previously discussed, individuals and groups may place different values and priorities on privacy depending on their background, demographics, and needs.  \nSpecifically, the types and amounts of privacy individuals seek from governments or corporations may be quite different from the privacy they want from family members or friends. In both cases, what’s valued may also vary, as the value of privacy from institutions may relate more to social connections and control than identity and development.\n\nPrivacy may privacy trade-offs—some of which were seen in this section–with other values, and may cause harm in some cases. We step away from this discussion in this section to maintain focus on the role privacy fills.\n\nPrivacy may also {conflict-with-itself} in some scenarios, as two different parties may want privacy in a mutually incompatible way. Here, who receives privacy when is called into question. Regulations serve to systematize these choices.","sidebarKey":null,"isDrawer":false},{"id":"privacy/valueOfPrivacy#sidebar-goods","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"goods","title":"Goods Definition","content":"A good, in this context, is any social or individual benefit or value. Goods may conflict with each other and are often non-universally experienced, typically coming with trade-offs.","sidebarKey":"goods","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-across-definitions","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"across-definitions","title":"Disclaimer","content":"This section does not cover differential privacy. For more information, view the section focusing on the goods and trade-offs that come from differential privacy specifically.","sidebarKey":"across-definitions","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-liberal-democratic-system","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"liberal-democratic-system","title":"Disclaimer","content":"This section focuses on this type of system in alignment with the broadstanding emphasis on issues and philosophies in computing within the West (i.e. US and EU). “Liberal democracy” is used here to describe a form of governance that allows collective, equal, representative decision-making that provides individuals the opportunity to act both politically and personally as they see fit. This definition is open to immense debate, which we sidestep for now. Differing conceptions of privacy exist outside of this framework and lead to different values, which is beyond the scope of this section.","sidebarKey":"liberal-democratic-system","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-protection-from-violence-and-bad-actors","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"protection-from-violence-and-bad-actors","title":"Protection from Violence and Bad Actors","content":"Generally, privacy may act as a shield against having action and identity be observed, preventing violence between people on this basis. This works well when using an access-based approach to privacy.\r\n\r\nThis is visible in a counterexample where someone loses privacy. “Swatting” is a technical instance of this, where the data of specific individuals are compromised and misused to place fraudulent calls about a serious, dangerous event where those compromised people are. These calls [often take advantage of modern phone technologies](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10058781&tag=1) to protect the harasser’s own privacy. In doing this, the goal is to provoke a heavily armed, militant police response (typically with specialized hostage rescue and special weapons teams) to harass and attack the individual. This may happen to [public figures](https://web.archive.org/web/20220816030937/https://www.washingtonpost.com/video-games/2022/08/15/keffals-adin-ross-ishowspeed-swatting-twitch-youtube/) and [non-public figures](https://www.reuters.com/world/us/staff-us-voting-machine-firms-prep-doxxing-misinformation-swatting-2024-11-01/).\r\n\r\nAlternatively, a positive example is [Address Confidentiality Programs (ACPs)](https://heinonline.org/HOL/P?h=hein.journals/dcujl6&i=217). ACPs obscure the home addresses of victims of domestic violence, stalking, and assault by providing them with a generic substitute address that can be used for mail forwarding and public records, preventing bad actors from accessing their location through these records. In turn, this prevents ongoing violence between individuals.","sidebarKey":"protection-from-violence-and-bad-actors","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-development-of-countercultures","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"development-of-countercultures","title":"Development of Countercultures","content":"In private or semi-private settings, individuals may be better able to challenge norms and push back against a wider status quo without fear of social or political retribution. In doing so, broader social norms can be identified as restrictive standards that can be rejected and further improved. Grassroots changes in social norms and institutions may occur as a result. This works well when thinking about privacy as defining private spheres of place and action.\r\n\r\nAnonymous forums serve as an example of this. YikYak, now Sidechat, is a version of these forums that targets college students and allows anonymous posting when a user is within the bounds of a college campus. [Users of the app describe the dual issue with anonymity](https://bruknow.library.brown.edu/discovery/openurl?institution=01BU_INST&vid=01BU_INST:BROWN&aulast=Tripodi&id=doi:10.2307%2Fj.ctt1t89cfr.22&auinit=F&atitle=Yakking%20about%20college%20life:%20Examining%20the%20role%20of%20anonymous%20forums%20on%20community%20identity%20formation&sid=google), noting that people on the app are more likely to express more nuanced and positive feelings than they would otherwise be willing to share because of the anonymity, but also that this same anonymity allows users to also be more hateful. Other research on different online spaces repeats this same tension on anonymity, showing these forums as places where [hateful ideologies can be developed](https://www.taylorfrancis.com/chapters/edit/10.4324/9781315728346-21/none-new-media-alice-marwick), but also where [queer connection can flourish](https://heinonline.org/HOL/P?h=hein.journals/hcrcl38&i=168) and [women may find safety from harassment](https://dl.acm.org/doi/pdf/10.1145/2675133.2675175).","sidebarKey":"development-of-countercultures","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-political-participation-without-judgment","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"political-participation-without-judgment","title":"Political Participation Without Judgment","content":"Privacy can hide the actions someone takes, either completely or only to certain people or groups. As a result, the perceived ethical impact of those actions are also made unavailable. This allows individuals to take actions without broader judgment from others, facilitating individualized decision-making that is free from social pressure. This hiding is often seen as a prerequisite to allowing free and equal decision-making by individuals that supports liberal democracy. This works well when thinking about privacy as something that delineates certain private spheres of action.\r\n\r\nAn example in modern practice is a voting booth. Even as party affiliation and the act of voting may be public information, how someone votes remains secret in most cases. Along a practical dimension, secret ballots prevent voter intimidation, coercion, and buying, allowing a vote to be actually reflective of the individual voter’s interests [rather than of someone who may exercise power over them](https://muse.jhu.edu/pub/1/article/729169/pdf). Ballot secrecy also supports the egalitarian view that anyone in a democratic society should be able to elect a [government without particular qualification](https://www.cambridge.org/core/services/aop-cambridge-core/content/view/85A2E2D6D191AADA21D84BDB56E5F903/S0953820807002634a.pdf/mill_and_the_secret_ballot_beyond_coercion_and_corruption.pdf). This allows elections to be both [free (in that they are not coercive, including violently so) and fair](https://muse.jhu.edu/article/16828) (in that treatment of people does not differ based on their vote). This is complicated when people may break privacy in the voting booth, [either via tampering with electronic voting machines](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1301313) or by taking [ballot selfies](https://heinonline.org/HOL/P?h=hein.journals/jlawp26&i=355).","sidebarKey":"political-participation-without-judgment","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-private-control-of-property","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"private-control-of-property","title":"Private Control of Property","content":"Certain definitions of privacy focus on the separation of the [**public and private spheres**](/privacy/whatIsPrivacy), including how much control an individual has over those areas. This has implications both ways. Using public resources is a public action, which allows for enforcement of particular regulations against how that resource might be used, serving some wider public interest. At the same time, private control of a resource lets someone use that resource for their private interest, providing them agency. This works well when thinking about privacy as a conduit for control or that allows a private sphere of action.\r\n\r\nAn example of this in regulation is the third amendment to the U.S. Constitution. While there might be a prevailing public interest to require that soldiers are able to live in people’s homes, the private interest to have control over intimate and vulnerable spaces takes precedence. In essence, there is a [defined private space that’s separate from the public where someone cannot be coerced](https://heinonline.org/HOL/P?h=hein.journals/valur26&i=271). Some readings of the third amendment [extend this idea to modern tech and surveillance](https://heinonline.org/HOL/P?h=hein.journals/wflron4&i=7), especially as the government may enter homes digitally through consumer technology, enforcing a type of “quartering.”","sidebarKey":"private-control-of-property","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-creating-different-relationships","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"creating-different-relationships","title":"Creating Different Relationships","content":"Knowing certain Information about people–particularly the relationships that they have with others–requires also knowing information about others. Without privacy protections, individuals do not have control over how they are seen by others, losing control over how this affects their relationships. People can no longer moderate how they present themselves, preventing them from cultivating different types of relationships in a productive manner. This can be appreciated from the intuitive sense that we want our significant other to know different things about us than our boss might. These differences in the types of relationships a person might have are broadly viewed as a foundational aspect of cultivating trust and subsequent community building, underscoring privacy as a prerequisite for a functional society. This works well when using a control-based approach to privacy.\r\n\r\nAn example of this in practice covers digital sex workers. Digital sex workers (such as those on pornography or subscription platforms) frequently hold dual roles, working other jobs alongside their digital work. When employers find out about people doing digital sex work, often without the worker’s intent, this can alter the employer-worker relationship and lead to [workplace discrimination](https://www.vice.com/en/article/employers-sex-work-cost-of-living/), [firings](https://www.abc.net.au/news/2024-03-27/sex-work-discrimination-laws-victoria-employment-vcat/103632868), and [difficulties in finding future work](https://www.huffpost.com/entry/sex-work-mainstream-job-employment_n_60a3f040e4b09092480941fc). In educational settings, this type of disclosure and discrimination [is a Title IX violation](https://law.justia.com/cases/federal/district-courts/oregon/ordce/6:2019cv00283/143726/165/).","sidebarKey":"creating-different-relationships","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-fairness-by-obscuring-identity","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"fairness-by-obscuring-identity","title":"Fairness by Obscuring Identity","content":"An obscured aspect of one’s identity is no longer available to others to support segregation or discrimination along those lines. Individuals _must_ be treated equally on the basis of that characteristic, as no information is made available – there is a veil of ignorance. This is distinct from other conceptions of fairness which may incorporate justice and equity principles along with equality. This works well when using an access-based approach to privacy.\r\n\r\nSexuality is an instance where this value may be important. Sexual relationships may be viewed as something that should be private to someone and their partner, which calls into question who should have access to knowledge about these relationships and when. As it’s possible to [infer sexuality from digital behavior](https://www.pnas.org/doi/epdf/10.1073/pnas.1218772110) and [sexual minorities report discrimination due to disclosure](https://pmc.ncbi.nlm.nih.gov/articles/PMC5819984/) in private spheres, being able to obfuscate identity becomes a pressing issue.\r\n\r\nAnother instance may be “Ban the Box” movements, pushing employers to not include questions about prior criminal history on job applications. By not asking about this background, those with criminal histories experience [less discrimination in hiring and are more likely to find employment](https://heinonline.org/HOL/P?h=hein.journals/ilr104&i=1107). At the same time, privacy in this area has spillover trade-offs, with some research suggesting that it [decreases hiring rates overall for other demographics](https://www.journals.uchicago.edu/doi/pdf/10.1086/705880) that already experience discrimination in hiring.","sidebarKey":"fairness-by-obscuring-identity","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-increased-agency-over-self-and-actions","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"increased-agency-over-self-and-actions","title":"Increased Agency Over Self and Actions","content":"People have knowledge about themselves and who they are from facts about themselves, self-understanding, and how others perceive them. These things help create an identity. Privacy allows someone to manage the extent that this information and their identity is exposed to others, exercising agency over how they are publicly perceived. By controlling how others perceive them, this self-creation is an active process that requires agency over one’s own actions. This is connected to control-based conceptions of privacy.\r\n\r\nOne example of a violation of this might be location-based targeted advertising. What we do is often dependent on where we are, often in intimate ways. On a high level, [large online advertising services](https://support.google.com/google-ads/answer/1722043?hl=en) may offer location targeting services, often operating on the level of zip or postal codes. In combination with demographic (identity) data, this can be used to target ads and influence a small group of individuals at any time. In the most extreme cases, this might be used to harass, intimidate, and influence through specialized advertising. Other [services may provide real-time GPS data](https://www.vice.com/en/article/hundreds-bounty-hunters-att-tmobile-sprint-customer-location-data-years/), allowing for, in one instance, women seeking an abortion to be [targeted directly by ads for anti-abortion clinics](https://rewirenewsgroup.com/2016/05/25/anti-choice-groups-deploy-smartphone-surveillance-target-abortion-minded-women-clinic-visits/).","sidebarKey":"increased-agency-over-self-and-actions","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-personal-development-through-experimentation","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"personal-development-through-experimentation","title":"Personal Development Through Experimentation","content":"People may change themselves and their identities over time. To facilitate this change, some safe experimentation with facets of one’s identity is needed. Privacy allows for this, providing a buffer between a public realm, where identity and perception may generally be fixed and have unwanted connotations, and a private realm, where identity may be more easily altered and exist without external connotations. In practice, the facets of one’s identity that are unknown and private to others are changeable without others being aware of the change, providing this fluidity. This is related to the separation that may exist between public and private spheres.\r\n\r\n_Tearoom Trade_ provides both an instance and a counter-example of this creation. _Tearoom Trade: Impersonal Sex in Public Places_ is a book by Laud Humphreys published in 1970, exploring the methods, practices, and demographics of men who have anonymous sex with men in secluded public areas, referred to as “tearooms.” The work highlights the fact that many men in the research presented themselves as heterosexual in other facets of their life, including to their wives. By engaging with tearooms, these men are able to cultivate two contrasting identities and rework popular understandings of what it means to be homosexual.\r\n\r\nAt the same time, Humphreys’s work relied heavily on deceit and tracking to gather data for his work. _Tearoom Trade_ has been [openly critiqued](https://www.jstor.org/stable/10.3998/mpub.11519906.11) on this basis, pointing out that Humphreys openly disregarded the privacy of his research subjects by failing to inform them that they were participating in his research or that he had collected any information about their sexual behaviors. By violating these participants' privacy in this way, his work also challenges the environment of “safe experimentation” that tearooms may provide.","sidebarKey":"personal-development-through-experimentation","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-ability-to-not-be-seen-as-a-particular-identity-or-characteristic","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"ability-to-not-be-seen-as-a-particular-identity-or-characteristic","title":"Ability to not be Seen as a Particular Identity or Characteristic","content":"Privacy allows people to obscure specific aspects of themselves, preempting potential judgement on the basis of those characteristics. Without these characteristics to rely on, others are forced to rely only on the information that is actively shared with them. Privacy, in this sense, acts as a humanizing force, rather than as a reducing one.\r\n\r\nOne instance of this process in action is through students attempting to hide their socioeconomic status at “elite” high schools and colleges. Anthony Abraham Jack discusses this in detail in The Privileged Poor, noting how many poorer students at these institutions will work to control knowledge of their background, both to integrate better with their wealthier peers and reveal hardships when convenient for social mobility. By selectively choosing when to reveal their background, these students prevent themselves from being othered and viewed as poor, managing impressions and forcing their peers to rely on other information–and assumptions–about who they are.","sidebarKey":"ability-to-not-be-seen-as-a-particular-identity-or-characteristic","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-allows-different-people-to-know-different-things-about-oneself","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"allows-different-people-to-know-different-things-about-oneself","title":"Allows Different People to Know Different Things About Oneself","content":"People may want different categories of individuals to know them in different ways. As an example, we can appreciate the fact that someone may want their boss to know them in a different light than their parents or significant other. Privacy allows people to control the flow of information to all of these people in their lives, forcing them to rely on different sets of information about that person in the context relevant to them. This supports the creation of varied, equally important relationships on an individual level, supporting broader flourishing and meaningful connections.\r\n\r\nOne instance of this is people maintaining multiple social media accounts. “Throwaway” accounts on Reddit are an extreme example, as users will make an account to only make one post, separating it from the rest of their user history out of caution and preventing [other users from knowing _anything_ else about themselves](https://dl.acm.org/doi/pdf/10.1145/3359237). Users on Instagram may use “finsta” (fake Instagram) and “rinsta” (real instagram) accounts, [providing different outlooks on their lives](https://dl.acm.org/doi/pdf/10.1145/3512916), often to a cultivated, gatekept audience. The finsta accounts are often more informal, emotional, and off-beat, typically revealing more intimate and uncurated moments on a more frequent basis. The audience on a finsta is more likely to be attuned to the broader context of a user’s life and, in turn, have a different relationship with the user than someone only following that user’s rinsta.","sidebarKey":"allows-different-people-to-know-different-things-about-oneself","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-learn-information-about-all-members","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"learn-information-about-all-members","title":"learn information about all members","content":"This is often articulated in terms of “group privacy,” which might also be thought about as “[the right to huddle](https://heinonline.org/HOL/P?h=hein.journals/rutlj8&i=235).”\r\n\r\n[One version of this is the algorithmic creation of groups from individual data](https://link.springer.com/content/pdf/10.1007/s13347-017-0253-7.pdf), revealing group associations and connections that would otherwise not be visible. By creating this group, the data of one person can inform knowledge from the data of someone else, involuntarily revealing information about them. The creation of these groups might also place unwanted identity labels on individuals, challenging control over their identity.\r\n\r\nCritically, group privacy extends to groups that are intentionally and unintentionally formed by its members. Even as the members themselves might be anonymous, [it’s frequently possible that the information revealed by the group itself is enough](https://library.oapen.org/bitstream/handle/20.500.12657/52825/1/978-3-030-82786-1.pdf#page=97) to identify and draw conclusions about its members. This has technical implications for privacy-conscious systems and political implications for some rights, such as freedom of association.","sidebarKey":"learn-information-about-all-members","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-gain-knowledge-about-a-specific-individual","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"gain-knowledge-about-a-specific-individual","title":"gain knowledge about a specific individual","content":"It’s often possible to use metadata to re-identify and find connections between hidden individuals. For re-identification (or deanonymization), it’s possible to use the data of an anonymous user to successfully re-identify them, extending to [location data](https://www.tandfonline.com/doi/full/10.1080/17489725.2024.2385312#abstract), [open datasets shared across platforms](https://www.sciencedirect.com/science/article/pii/S0268401215301262), and [user metadata](https://ojs.aaai.org/index.php/ICWSM/article/view/15010). On the other hand, it’s possible to learn and predict information about an individual from their group affiliations. This is a form of social network analysis, which has successfully extracted notable individuals, even from [low-information](https://culturalanalytics.org/article/68188-how-network-analysis-uncovers-international-networks-of-smuggling-history-criminals-in-nagasaki-japan-circa-1667), [historical](https://kieranhealy.org/blog/archives/2013/06/09/using-metadata-to-find-paul-revere/) datasets.","sidebarKey":"gain-knowledge-about-a-specific-individual","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-might-also-seek-out-privacy","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"might-also-seek-out-privacy","title":"might also seek out privacy","content":"The primary example of this is governments holding state secrets in the interest of national security. This active withholding of information–rather than just the lack of accessibility–is a form of privacy that governments hold from other countries and, often, their own citizens. Whether this privacy ought to exist [is a debated issue](https://www.tandfonline.com/doi/pdf/10.1080/13698230.2018.1482097), particularly as this may come into conflict with ideals of accountability and transparency. Alternatively, a company may seek out privacy to protect their trade secrets, competitive advantage, and details of their customers, although this profit-motivated right to privacy [is often seen as weaker](https://heinonline.org/HOL/P?h=hein.journals/mnlr99&i=39) than individual rights to privacy. Advocacy groups have a stronger claim to this type of privacy, as they often operate along legally protected characteristics and seek to enact [collective digital privacy by design](https://www.sciencedirect.com/science/article/pii/S016740482200013X).","sidebarKey":"might-also-seek-out-privacy","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-historically-marginalized-groups","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"historically-marginalized-groups","title":"historically marginalized groups","content":"Many formal and informal groups may seek this kind of privacy. This includes, but is not limited to:\r\n\r\n- [Racial and ethnic communities](https://journals.sagepub.com/doi/pdf/10.1177/00936502241273157)\r\n- [LGBT+ communities](https://arxiv.org/pdf/2112.00107)\r\n- [Advocacy groups and political dissidents](https://www.nytimes.com/2024/05/02/nyregion/college-campus-protests-anonymity.html)\r\n- [Incarcerated populations](https://compass.onlinelibrary.wiley.com/doi/pdf/10.1111/soc4.12847)\r\n- [Immigrant communities](https://dl.acm.org/doi/pdf/10.1145/3173574.3173688)\r\n- [Unions and professional organizations](https://dl.acm.org/doi/pdf/10.1145/3555574)\r\n- [Illness and patient groups](https://www.proquest.com/scholarly-journals/what-electronic-health-records-don-t-know-just/docview/2919545076/se-2?accountid=9758)\r\n- [Religious communities](https://warwick.ac.uk/fac/soc/law/elj/jilt/2009_1/cannataci)\r\n- [Specialized interest and hobby groups](https://www.bbc.com/news/articles/c05m5m5v327o)\r\n- [Linguistic communities](https://link.springer.com/content/pdf/10.1007/s13278-022-01017-0.pdf)\r\n- [Low-income groups](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2182773)","sidebarKey":"historically-marginalized-groups","isDrawer":true},{"id":"privacy/valueOfPrivacy#sidebar-conflict-with-itself","section":"privacy","sectionTitle":"Privacy","subsection":"valueOfPrivacy","subsectionTitle":"Value of Privacy","anchor":"conflict-with-itself","title":"conflict with itself","content":"One example of this is [forensic investigation](https://www.jstor.org/stable/43741598?seq=3). By collecting some data of many people to begin with, when an investigation is needed, more people can be excluded based on the beginning data and will not have their privacy needlessly violated. If this doesn’t occur, then more people will need to be investigated deeply, requiring a trade-off in who gets privacy, even if the average amount of privacy between individuals is the same.","sidebarKey":"conflict-with-itself","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#how-should-we-think-about-data-in-the-context-of-surveillance-capitalism","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"how-should-we-think-about-data-in-the-context-of-surveillance-capitalism","title":"How Should We Think About Data in the Context of Surveillance Capitalism?","content":"Human beings constantly generate data. Some data we might commonly think about generating include our search histories, location tracking, social media interactions, and purchases. Perhaps less obviously, we also generate data on our heart rates, streaming preferences, and political opinions, to name a few examples. These data points hold value for companies and governments. Being able to predict which consumers are more likely to purchase a product or service can reduce advertising costs, allowing companies to narrowly target those who are most likely to generate returns. Similarly, having insight into which voters are most likely to be swayed by a particular political message can allow candidates to more effectively tailor their campaigning. Knowing who was in the vicinity when a crime occurred can assist law enforcement officers in identifying suspects.\n\nWhat is it that people are doing when they generate these data points? Is generating data a kind of **labor**, creating something of value for companies, governments, and other individuals? Should data be thought of as a kind of **property** that can be claimed and traded? Or is data something more intrinsic to a person, like an **extension of their body or self**? Using the examples given in the previous paragraph, we can appreciate that it may depend on the type of data we are considering, as there is an intuitive difference between one’s social media interactions compared to personal writings or medical records. The latter are far more difficult to divorce from the individual than the former and may feel more personal or private.\n\nThis primer explores the concept of **data ownership** within the context of {surveillance-capitalism}, a system wherein corporations, largely based in the United States and China, generate significant profit by collecting, processing, and commodifying personal data. Based on how we conceptualize data, we may arrive at different conclusions regarding how data is generated, how data ownership should be decided, and how, if at all, data should be monetized.","sidebarKey":null,"isDrawer":false},{"id":"privacy/dataOwnershipSurveillanceCapitalism#theory-of-surveillance-capitalism","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"theory-of-surveillance-capitalism","title":"Theory of Surveillance Capitalism","content":"It is often remarked that “if a product is free, you are the product.” This idea traces back to a 1973 art piece titled “[Television Delivers People](https://medium.com/@elali.ahmad/if-youre-not-paying-for-the-product-you-are-the-product-a9d3aa9bbd85)”[^16] by Richard Serra and Carlota Fay Schoolman in which they made the point that television audiences were a product sold to advertisers. In the era before highly targeted advertising, simply having many active users was valuable to advertisers, even without the extensive data collection that characterizes today’s digital landscape. **Surveillance capitalism** represents an evolution of this model—the product is not the user themself but rather their data. For example, on free social networking platforms, companies reap value from their users by building detailed profiles that enable prediction about characteristics and behaviors that may be of interest to those in the market for customers (e.g., advertisers), supporters (e.g., politicians, influencers, or celebrities), {insurance-policy-beneficiaries}, or even targets (e.g., private investigators or law enforcement agencies). Such profiles are typically constructed based on the browsing and purchasing histories and demographic attributes (provided or inferred) of the individual user and of other users they know or resemble.\n\nAs surveillance capitalism currently operates, it relies on the premise that corporations rightfully and fully own the **behavioral data** they collect from users. In order to evaluate whether this is a reasonable premise, we must analyze the broader concept of data ownership. Our conclusions might vary depending on our framing and the type of data considered. How we assess corporate ownership of behavioral data shapes larger questions: Should surveillance capitalism continue in its present form? Would stricter limits on data extraction, alternative business models, or entirely new paradigms better safeguard privacy, autonomy, and individual rights?","sidebarKey":null,"isDrawer":false},{"id":"privacy/dataOwnershipSurveillanceCapitalism#different-ways-we-can-think-about-data-ownership","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"different-ways-we-can-think-about-data-ownership","title":"Different Ways We Can Think About Data Ownership","content":"We often think of ownership in terms of property. Ownership of physical property, such as a house, is fairly intuitive because it is tangible. Yet, ownership of intangible property, such as shares of a company, is also widely accepted. Like shares of a company, data is **intangible**, but unlike shares of a company, data can generally be replicated infinitely without reducing its usefulness to others. Data is also **ephemeral**: if data is not collected in the moment it is generated, it may be gone forever. For example, take the number of seconds spent observing an advertisement on a social media platform. If that information is not collected in the moment, it cannot be recovered. Because data is different from more traditional forms of property, data ownership may be different as well. As a result, we may need to think creatively about how ownership should be determined, which may vary across different contexts.\n\nIn legal and economic contexts, {data-ownership-is-necessary} in order to assign rights and responsibilities to those who generate, collect, and process data. In fact, it may be essential for protecting individuals in a data-driven economy.\n\nWe will discuss three notions of data ownership and what they each imply for how we should interact with data: data as labor, data as property, and data as an extension of the body or self.\n\n![An illustration of the three notions of data ownership. Data as labor is depicted as a factory worker standing next to an industrial factory that is emitting 'data partiles.' Data as property is depicted as various objects (boxes and bags) with 'data particles' emenating from them. Data as an extension of the body or self is depicted as a person holding a data particle in their hand.](/assets/primer-photos/privacy/dataOwnership/data_ownership.png)\n\n**Figure:** The three notions of data ownership.\n\n### Data as Labor\n\nFirst, **data as labor** views the generation of data as a form of work that is currently unpaid. After all, in order for behavioral data to be created, users must engage in some observable activity that creates value for others. This notion would suggest that platforms should compensate users for the value they produce in the form of some kind of “wage for data” and even that “data workers” might have some right to collective bargaining with platforms.\n\nIdeally, under the notion of data as labor, users would be able to profit through some type of fair compensation. This could be micro-payments, where users are paid some small amount whenever their data is used; {universal-data-dividends}, where users whose data is collected by a platform are all paid a given amount on a regular interval, similar to stock dividends; collective shares in platform profits, where users receive shares and become part-owners of the platforms their data contributes to; or something else entirely. Corporations would also be required to access data only through established terms, like labor contracts, that may be negotiated with unions or cooperatives that would oversee fair use and distribution. This restriction of access could be regarded as preventing exploitation of “data laborers” and might, by market forces, incorporate data minimization (where companies only collect data that they actually need) by forcing companies to weigh the value of the data against what they would need to pay for it.\n\nIn order to align with this notion of data, we might need to legislate data labor rights and rights to collective bargaining. Under this notion of data ownership, the exchange of data for compensation occurs under an ongoing contract. The next notion we will discuss, data as property, aligns more with a model of one-time exchanges of data ownership.\n\n### Data as Property\n\nNext, **data as property** treats data as the product of the application of labor to some natural resource, which can then be claimed as private property. This notion is rooted in Lockean philosophy. In his _Second Treatise of Government_ (1689),[^12] John Locke argues that private property is obtained legitimately when a person mixes their labor with something that was previously unowned in the state of nature. He justifies this with three reasons: first, he claims that every person owns themself, as well as their own labor, and that when a person expends labor on an unowned object, they extend their self-ownership onto the external world; second, he argues that labor increases the value of things significantly and thus, that the product of that value-creating activity should not be left in common or belong to someone else; and third, he qualifies ownership by requiring that one leaves enough for others and does not waste what they collect. Locke’s labor-mixing theory remains an influential philosophical foundation for modern notions of legitimate ownership.\n\nTo think about how Locke’s principles can be applied to data ownership in the modern world, we will use the example of a user viewing an advertisement on a social media platform. The primary argument we will consider is that the social media platform is doing labor by observing how long the user spends viewing the advertisement—after all, they must write the code to do so, store that information somewhere, and track it—and thus the social media platform would have legitimate ownership over the data instead. Here, the ephemerality of data becomes especially relevant, as without the platform doing the labor of capturing the data, it could not be monetized. In order to effectively make use of the notion of data as property, a consensus would need to be reached on what qualifies as sufficient labor to establish ownership. {Another-argument} is that users are mixing some of their own labor during the process of data generation. This argument is currently debated in the literature, but it might seem that at best in this case, the result would be mixed ownership between the user and the social media platform once both parties have mixed their labor.\n\nIf a consensus was reached in which data subjects received some or all ownership over the data relating to them, then market-based solutions like **'data marketplaces'** could be formed, where corporations could purchase permission to use data from their owners. These data marketplaces [already exist](https://aws.amazon.com/data-exchange/?trk=6fc00089-b729-4209-a46c-8e96bb51c476&sc_channel=ps&ef_id=Cj0KCQiA_8TJBhDNARIsAPX5qxS5d6rncj2KEqqydZS2mR5Dwi5BqauLFVBjNYoHPmRZsYzifICrpEUaAgRHEALw_wcB:G:s&s_kwcid=AL!4422!3!658520966831!!!g!!!19852662473!149878723820&gad_campaignid=19852662473&gbraid=0AAAAADjHtp9SPixwsiyT3jRRK9UHo13Qy&gclid=Cj0KCQiA_8TJBhDNARIsAPX5qxS5d6rncj2KEqqydZS2mR5Dwi5BqauLFVBjNYoHPmRZsYzifICrpEUaAgRHEALw_wcB)[^4] between corporations and would only need to be extended to individuals. This would likely increase user agency as it would give individuals more control over whom their data is sold to or whether it is sold at all. It would also shift some profits from data toward users, compared to the status quo under surveillance capitalism, where profits go entirely to corporations.\n\nIn order to implement this, one approach could be to enact comprehensive data property laws establishing the legal recognition of personal data as ownable assets, reform consent models to require granular permissions, and educate users on data valuation to mitigate the impacts of {the-data-divide}.\n\n### Data as an Extension of the Body or Self\n\nLast, we can think of **data as an extension of the body or self**. Personal data necessarily describes the person it is associated with in some way, so it may be thought of as an intimate part of one’s personal identity. Take, for example, a person’s medical records. Logs of when health visits occurred, for what reason, and what actions were taken are data that certain companies, such as health care providers, may find incredibly valuable. However, that data is also afforded strict privacy and security protections under current laws like the Health Insurance Portability and Accountability Act of 1996 (HIPAA). Here, monetization of one’s personal data might not make sense as it may {breach-ethical-boundaries}.\n\nUnder the notion of data as an extension of the body or self, we may decide that no one should profit from the commodification of certain data. Then, corporations might only be able to access that data for essential, non-commercial purposes, such as service delivery, with strict prohibitions on sales or predictions based on the data.","sidebarKey":null,"isDrawer":false},{"id":"privacy/dataOwnershipSurveillanceCapitalism#applying-notions-of-data-ownership","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"applying-notions-of-data-ownership","title":"Applying Notions of Data Ownership","content":"These three lenses—data as labor, data as property, and data as an extension of the body or self—offer distinct ways to reframe data ownership and guide how society should handle personal data. These notions of data ownership can coexist. It may be the case that some data ownership is best determined by thinking of the data as property, while other data ownership is best determined by thinking of the data as an extension of the body or self. Essentially, we may decide that it is most appropriate to apply different lenses to different kinds of behavioral data.\n\nFor example, it might make sense to think of the data collected on social media platforms through the data as labor lens. This is an ongoing interaction where users’ actions generate valuable data for corporations. It would be cumbersome to require a new agreement to be made in order for corporations to be able to use each new data point as it is generated. This way, a fair agreement, much like a labor contract, can be reached between users and corporations that can persist indefinitely, unless either group decides to renegotiate the terms.\n\nHowever, we might categorize Google’s ranking model and personalized results set under the lens of data as property. Here, users provide observable behavior by entering searches, clicking on certain results, scrolling, and spending time on pages, and Google invests substantial labor in writing sophisticated tracking code, building and maintaining vast server infrastructure, developing algorithms to capture and store these signals in real time, cleaning and structuring the data, and applying machine learning to transform these fleeting behavioral traces into persistent, valuable assets. Without this active, value-adding labor by Google to produce knowledge from the data, the raw observations would remain ephemeral and unmonetizable. Google's efforts \"mix\" with the users' observable actions to create something new and proprietary.\n\nFinally, we might consider most medical data to fall under the lens of data as an extension of the body or self. Here, it is difficult to divorce most data from individuals, and it pertains to something intimate and personal about ourselves. We might be very uncomfortable with the notion of monetizing this kind of data, particularly without recompense. In one case, breathing machines for individuals with sleep apnea secretly sent reports on users’ sleep quality to insurers in order to allow insurers to deny payment. People may have found this so disturbing because of the notion that the data the machines were collecting and reporting was too personal.\n\nWe do not currently have a solution as to how to best allocate data ownership. The current presumption is that corporations have complete ownership of the data they collect, but this is contentious. As we’ve seen through the examples above, the current state of affairs may not always align with the type of data ownership we want, or who we believe should own this data. Because there is no clear consensus, it is worth considering potential alternatives to the status quo.","sidebarKey":null,"isDrawer":false},{"id":"privacy/dataOwnershipSurveillanceCapitalism#alternatives-to-surveillance-capitalism","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"alternatives-to-surveillance-capitalism","title":"Alternatives to Surveillance Capitalism","content":"Fundamental to surveillance capitalism is the model of data extraction and behavioral model generation. This is far from the only way companies can generate profit. If we were to reject surveillance capitalism, what other options remain for profit generation? Would corporations continue to be able to exist as we know them? Or is surveillance capitalism necessary for the world we have built? Currently, we can observe that some companies are already practicing—and profiting from—alternative models to surveillance capitalism.\n\n### Profitable Business Models Beyond Surveillance Capitalism\n\n#### Subscription-Based Services\n\nSome services, like academic journals, are able to avoid extensive data collection on their users by drawing in revenue from a **subscription-based model**. These services charge users directly for premium features or to enjoy advertisement-free experiences, which allows them to rely on recurring revenue directly from their users rather than data sales.\n\n#### Freemium Models\n\nOther services offer a similar model but with tiers. They offer free basic services funded by optional upgrades or partnerships. This allows the service to profit without behavioral prediction sales through providing additional valuable services beyond the base product, like the removal of advertisements or the addition of Artificial Intelligence tools, for example.\n\n#### Hardware-Centric Revenue\n\nSome businesses monetize through product sales and accessories rather than data extraction. They sell devices or ecosystems that promote privacy by prioritizing on-device processing to minimize cloud data, making privacy values central to their business model.\n\n#### Open-Source and Crowdfunding\n\nAnother option adopted by some providers is to release core software as open-source and fund development with donations, grants, or enterprise support contracts. They can build profitability through allowing their development to be guided by community desires rather than the creation of behavioral data.\n\n#### Contextual Advertising Without Tracking\n\nSome services opt to use **non-personalized advertisements** based on content or user-declared preferences and supplement their revenue with affiliate partnerships or merchandise, avoiding the need for extensive surveillance to generate revenue.\n\n### Legal Reforms and Alternatives to Surveillance Capitalism\n\nIn order to reduce surveillance {within-the-framework-of-a-capitalist-model}, regulatory reforms could be implemented. This would entail modifications to the current system aimed at preserving users’ privacy and autonomy over data relating to them, such as strict privacy laws akin to the EU’s General Data Protection Regulation ([GDPR](https://gdpr-info.eu/)).[^10]\n\nTaken maximally, we might decide that some data should not exist at all. As we have seen through our discussion of the ephemerality of data, data is something that must be captured in order for it to be used. If we decide that certain data should not be used, we can prohibit the creation of that kind of data.\n\nAnother legal framework for data handling is the **commons-based approach**. Under this framework, data is treated as a **common good**. Citizens have more direct say in what data is collected and how it is used. Barcelona has adopted such a model, and according to [Barcelona City Council](https://ajuntament.barcelona.cat/digital/en/technology-accessible-everyone/accessible-and-participatory/decode), “Barcelona considers data to be part of the public infrastructure,” and the “common data infrastructure \\[is\\] open to local businesses, cooperatives and social organisations so that they can provide data-focused services and create long-term value for the public.”[^5] Financial incentives drive much of the current collection and storage of data. Under a data-commons model, some of the data collection carried out in a more purely capitalist system might be curtailed by the privacy demands of citizens while still allowing for the data collection that the community determines to be useful to the common good. In other words, it might reduce profit-driven surveillance in service of other values, such as privacy.\n\n[^1]: Ada Lovelace Institute. 2021\\. “The data divide.” adalovelaceinstitute. https://www.adalovelaceinstitute.org/report/the-data-divide/.\n\n[^2]: Allen, Marshall. 2018\\. “You Snooze, You Lose: Insurers Make The Old Adage Literally True.” ProPublica. https://www.propublica.org/article/you-snooze-you-lose-insurers-make-the-old-adage-literally-true.\n\n[^3]: Allen, Marshall. 2018\\. “You Snooze, You Lose: Insurers Make The Old Adage Literally True.” ProPublica. https://www.propublica.org/article/you-snooze-you-lose-insurers-make-the-old-adage-literally-true.\n\n[^4]: Amazon Web Services, Inc. 2026\\. “AWS Data Exchange.” aws. https://aws.amazon.com/data-exchange/?trk=6fc00089-b729-4209-a46c-8e96bb51c476\\&sc\\_channel=ps\\&ef\\_id=Cj0KCQiA\\_8TJBhDNARIsAPX5qxS5d6rncj2KEqqydZS2mR5Dwi5BqauLFVBjNYoHPmRZsYzifICrpEUaAgRHEALw\\_wcB:G:s\\&s\\_kwcid=AL\\!4422\\!3\\!658520966831\\!\\!\\!g\\!\\!\\!19852662473\\!14987872382.\n\n[^5]: Barcelona. n.d. “DECODE.” Barcelona Digital City. https://ajuntament.barcelona.cat/digital/en/technology-accessible-everyone/accessible-and-participatory/decode.\n\n[^6]: Cather, David A. \"Addressing Insurance Price Discrimination in an Era of Diversity, Equity, and Inclusion.\" _Risk Management and Insurance Review_ 26 (2023): 407–429. https://doi.org/10.1111/rmir.12249.\n\n[^7]: Data Dividend Project. 2021\\. “Who We Are.” datadividendproject. https://www.datadividendproject.com/aboutus.\n\n[^8]: Determann, Lothar. \"No One Owns Data.\" UC Hastings Research Paper No. 265\\. February 14, 2018\\. https://ssrn.com/abstract=3123957.\n\n[^9]: Governor Gavin Newsom. 2019\\. “Governor Newsom Delivers State of the State Address.” gov.ca. https://www.gov.ca.gov/2019/02/12/state-of-the-state-address/.\n\n[^10]: Intersoft Consulting. n.d. “General Data Protection Regulation.” General Data Protection Regulation (GDPR) – Legal Text. Accessed April 19, 2026\\. https://gdpr-info.eu/.\n\n[^11]: Kelly, Robert. 2025\\. “Adverse Selection Explained: Definition, Effects, and the Lemons Problem.” Investopedia. https://www.investopedia.com/terms/a/adverseselection.asp.\n\n[^12]: Locke, John. 2021\\. _Second Treatise of Government_. Edited by A. J. Simmons. N.p.: W. W. Norton, Incorporated.\n\n[^13]: Muldoon, James. 2025\\. “Data-Owning Democracy or Digital Socialism?” _Critical Review of International Social and Political Philosophy_ 28 (4): 570–91. doi:10.1080/13698230.2022.2120737.\n\n[^14]: Resnik, David B. 2013\\. “Charging Smokers Higher Health Insurance Rates: Is it Ethical?” The Hastings Center for Bioethics. https://www.thehastingscenter.org/charging-smokers-higher-health-insurance-rates-is-it-ethical/.\n\n[^15]: “What You Need to Know about Surveillance Capitalism.” Wellesley. https://www.wellesley.edu/news/what-you-need-to-know-about-surveillance-capitalism.\n\n[^16]: Whitney Museum of American Art. 2026\\. “Television Delivers People Dec 12, 2007–Feb 17, 2008.” Whitney.org. https://whitney.org/exhibitions/television-delivers-people.Wellesley College. 2020\\.","sidebarKey":null,"isDrawer":false},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-surveillance-capitalism","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"surveillance-capitalism","title":"Surveillance Capitalism","content":"The term “surveillance capitalism” was coined by Shoshana Zuboff. Zuboff uses the term to refer to a new economic model in which corporations claim ownership “of private human experience as free raw material for translation into behavioral data.”[^15] We can see how the social media example fits into this framework: User engagement is the “raw material” used by the platform to create behavioral models that predict users’ future purchases.","sidebarKey":"surveillance-capitalism","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-insurance-policy-beneficiaries","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"insurance-policy-beneficiaries","title":"Insurance Policy Beneficiaries","content":"In 2018, users of breathing machines with sleep apnea discovered the machines were secretly sending reports detailing their sleep quality and usage of the machine to their insurance providers, and even that their insurers could use this information to [deny payment](https://www.propublica.org/article/you-snooze-you-lose-insurers-make-the-old-adage-literally-true).[^2]","sidebarKey":"insurance-policy-beneficiaries","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-data-ownership-is-necessary","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"data-ownership-is-necessary","title":"Data Ownership Is Necessary","content":"In this primer, we assume data ownership exists. An argument can be made that data ownership does not make sense philosophically. This may be because data can depend on multiple people or entities and the contexts in which it is generated, or because some data is deeply personal to data subjects in ways such that we may not feel comfortable assigning ownership rights. One such argument is made by legal scholar Lothar Determann in the article [“No One Owns Data.”](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3123957)[^8]","sidebarKey":"data-ownership-is-necessary","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-universal-data-dividends","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"universal-data-dividends","title":"Universal Data Dividends","content":"In 2019, California’s Governor Gavin Newsom delivered a [State of the State Address](https://www.gov.ca.gov/2019/02/12/state-of-the-state-address/),[^9] where he expressed the view that “consumers should also be able to share in the wealth that is created from their data” and announced that he asked his team “to develop a proposal for a new Data Dividend for Californians.” As of 2025, California has no data dividends program. In 2020, Democratic presidential primary candidate Andrew Yang launched a [Data Dividend Project](https://www.datadividendproject.com/aboutus),[^7] which is geared towards protecting individuals from the sharing and sale of their personal data and promises to assist individuals in being paid for the use of their data, should they choose to allow its use. In practice, it functions more like a platform for organizing class action lawsuits than a true data dividend system. It does not prevent companies from misusing data, but rather provides a potential avenue to compensation after the misuse has occurred. The concept of data dividends has existed in public policy spheres for some time now. In California, official discussions of data dividends have centered around proactive policy, while Andrew Yang’s project focuses on legal remedies. Neither of these examples perfectly conforms to the theory of data dividends, which may be influenced by the fact that they exist within the United States legal framework.","sidebarKey":"universal-data-dividends","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-another-argument","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"another-argument","title":"User Labor","content":"If the information about the time a user spends viewing the advertisement is taken to be the “unowned resource” in this case, then the user could be said to be doing labor by choosing to watch the advertisement. Under this view, the user would then have some degree of legitimate ownership over the resultant data.","sidebarKey":"another-argument","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-the-data-divide","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"the-data-divide","title":"The Data Divide","content":"The [data divide](https://www.adalovelaceinstitute.org/report/the-data-divide/)[^1] refers to the idea that there is an unequal relationship between those who are represented by and able to shape data-driven technologies and those who are not. This divide falls roughly along the lines of those who are and are not capable of collecting and managing large quantities of data, and it exists because some lack access to the necessary technology, some lack the knowledge and skills necessary to use digital technologies, and some do not find using digital technologies to be acceptable or comfortable. The existence of the data divide makes it more difficult for some individuals to understand the value of the data they have and bargain accordingly.","sidebarKey":"the-data-divide","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-breach-ethical-boundaries","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"breach-ethical-boundaries","title":"The Health Insurance Market","content":"One such example is present in the market for health insurance. Insurance providers find significant value in consumers’ health data, as it enables them to charge consumers based on their expected health costs. For very healthy individuals, this might be attractive as it would promise lower premiums. However, vulnerable individuals (like those who are elderly or chronically ill) might be priced out of the market entirely if the premiums offered to them by health insurance providers are too high for them to afford. This might intuitively feel unfair. Bioethicist David B. Resnik writes about some arguments for and against charging smokers higher health insurance rates in “[Charging Smokers Higher Health Insurance Rates: Is it Ethical?](https://www.thehastingscenter.org/charging-smokers-higher-health-insurance-rates-is-it-ethical/)”,[^14] ultimately concluding that the practice is unethical. However, [some studies](https://onlinelibrary.wiley.com/doi/full/10.1111/rmir.12249)[^6] suggest that banning the use of relevant information on a consumer’s identity may lead to [adverse selection](https://www.investopedia.com/terms/a/adverseselection.asp)[^11] in insurance markets, a process in which prices may rise intolerably high for all consumers.","sidebarKey":"breach-ethical-boundaries","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-secretly-sent-reports","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"secretly-sent-reports","title":"secretly sent reports","content":"In 2018, users of breathing machines with sleep apnea discovered the machines were secretly sending reports detailing their sleep quality and usage of the machine to their insurance providers, and even that their insurers could use this information to [deny payment](https://www.propublica.org/article/you-snooze-you-lose-insurers-make-the-old-adage-literally-true).[^3]","sidebarKey":"secretly-sent-reports","isDrawer":true},{"id":"privacy/dataOwnershipSurveillanceCapitalism#sidebar-within-the-framework-of-a-capitalist-model","section":"privacy","sectionTitle":"Privacy","subsection":"dataOwnershipSurveillanceCapitalism","subsectionTitle":"Data Ownership and Surveillance Capitalism","anchor":"within-the-framework-of-a-capitalist-model","title":"Beyond the capitalist model","content":"Some alternatives outside of the capitalist model include:\n\n**Digital Socialism:** James Muldoon discusses the possibility of treating data as a public resource managed by the state or collectives, with profits reinvested in society (e.g., national data funds) in his article [\"Data-Owning Democracy or Digital Socialism?\"](https://www.tandfonline.com/doi/full/10.1080/13698230.2022.2120737#abstract)[^13] This approach emphasizes equitable access and the elimination of private extraction.\n\n**Data Anarchism:** This model rejects centralized authority altogether in favor of decentralized, peer-to-peer data systems built on technologies like blockchain or federated networks. The appeal of this model lies in its resistance to both corporate surveillance and state oversight, returning control of behavioral data to users themselves. That said, this model raises real feasibility concerns. It requires a level of technological literacy that much of the general public does not yet have, and questions about the long-term security of decentralized storage and data transfers remain open.","sidebarKey":"within-the-framework-of-a-capitalist-model","isDrawer":true},{"id":"privacy/privacyTradeoffs#intro","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"intro","title":"Introduction","content":"# What Are The Trade-Offs of Privacy?\n\nAs discussed in [Value of Privacy](/privacy/valueOfPrivacy), although conceptions of privacy vary, it is widely agreed that it can be useful across a variety of contexts. At the same time, privacy comes with trade-offs: more personal data can be traded in return for greater safety, more personalized services, better research outcomes, or more efficient resource allocation. This primer does not seek to indicate whether a particular privacy trade-off “is worth it.” Instead, it focuses on situations where privacy trade-offs exist and asks a guiding question: “What goods might we trade privacy for?”","sidebarKey":null,"isDrawer":false},{"id":"privacy/privacyTradeoffs#privacy-vs-security","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"privacy-vs-security","title":"Privacy vs. Security","content":"Particularly in national security and policing contexts, privacy may be traded in favor of promoting security. Governments and institutions often argue that additional data collection, including internet history, biometrics, travel history, and communications metadata, is necessary to prevent harm. Prominent examples of this are the large-scale surveillance programs expanded after the September 11 attacks that allow the NSA to collect and analyze records of millions of Americans’ phone calls.[^6] Proponents of this view, like philosopher [Adam D. Moore](https://faculty.washington.edu/moore2/PSA.pdf), offer four common arguments in support of security-driven intrusions into privacy: “just trust us,” “nothing to hide,” “security trumps,” and “consent.”\n\n### “Just Trust Us”\n\nThis first argument urges the public to simply trust those in power not to override their privacy without good reason. On this account, those in power are presumed to be generally well-intentioned, to possess more information about threats than the general public, and to be therefore best positioned to balance security and privacy. Allowing them broad discretion is also most efficient: fewer constraints, with access to vast amounts of information, lead to faster decisions in high-stakes scenarios.\n\nOne response to this view is that it overlooks the history of misuse of expanded security powers in the United States. Examples such as the {internment-of-Japanese-Americans|interment of Japanese-Americans}, {mcepi|McCarthy-era political investigations}, and the {cointelpro|FBI's COINTELPRO (Counter Intelligence Program)} show how these institutions have repeatedly committed abuses when granted wide surveillance authority without meaningful oversight and transparency. On this view, the numerous abuses associated with the {patriot|USA PATRIOT Act} point to why “just trust us” risks significant harm.\n\n### “Nothing to Hide”\n\nThe second argument is: “If you are not doing anything wrong, you have nothing to hide.” Under this view, only those who are violating the law should worry about surveillance, and objecting to surveillance is taken to be a _de facto_ admission of guilt.\n\nA common response to this view is that it misrepresents the value of privacy. There are many kinds of information, such as medical history, sexual orientation, or political affiliation, that are not indicative of criminality. Nevertheless, they are sensitive because they reveal intimate details about an individual's identity and their life. People may have an interest in keeping such sensitive information private from employers, the government, their peers, or even members of their own family.\n\nAdditionally, what counts as “wrong” in society changes over time. Laws change, social norms evolve, and data that seems harmless today may later become dangerous under different political climates. The {post-nine-sur|post-9/11 surveillance of Muslim Americans} illustrates how religious identity can be treated as an indicator of suspicion when political climates shift. Therefore, individuals may have an interest in controlling access to sensitive information that is unrelated to any objectionable activity.\n\n### “Security Trumps”\n\nA third view argues that when privacy infringement substantially advances security, security should be prioritized. This view asserts that privacy primarily safeguards secondary interests that are valuable (dignity, autonomy, freedom, etc.) from intrusion, but that are ultimately less important than life and protection from violence. If a surveillance program reduces the risk of catastrophic harm, then the loss of privacy may be justified. National security decisions are also collective decisions: The costs and benefits of surveillance are distributed across society, and individuals may choose to accept some loss of privacy if doing so reduces risks to the public as a whole.\n\nOne such example of this trade-off is the programs authorized under the Foreign Intelligence Surveillance Act (FISA). Enacted by Congress in 1978 and expanded following the September 11 attacks, FISA was designed to allow intelligence agencies to monitor foreign threats that operate through international communication networks. Since many national security threats originate abroad and involve foreign actors communicating with individuals within the United States, intelligence agencies argue that broad surveillance powers are necessary to identify patterns of communication, detect emerging threats, and prevent attacks before they occur. Large-scale collection programs make it possible to detect suspicious connections between individuals or networks that would be difficult to identify through traditional, individualized warrants alone. Thus, the justification for such programs is preventative: by enabling intelligence agencies to identify threats earlier, they may reduce the likelihood of terrorist attacks or other forms of national security harm. However, whether—and to what extent—this program truly improves security remains {controversial} and requires further close analysis.\n\nA response to this view that “security trumps” is that privacy, particularly when conceived as the right to control access to one's body, seems as weighty as security. In fact, security may derive its value from the goods it protects, including privacy itself. If so, appeals to security cannot automatically override privacy without undermining the very interests that make security worth pursuing in the first place.\n\nThe central challenge, therefore, is determining when security benefits justify these privacy costs and how surveillance powers can be constrained to prevent abuse while still allowing the government to respond effectively to genuine threats.\n\n### “Consent”\n\nA fourth view argues that by voluntarily offering information, even private information, on social media, smartphones, or web pages, individuals have effectively consented to its collection. We each shed vast amounts of data (e.g., actions, emotions, choices) by moving through the world, and others may observe and collect this data. In essence, we are seen and can see others. Given that all of this information is shared, we agree that others may watch. In other words, when we use digital services, we agree to others potentially observing and analyzing our activities, and thus cannot then complain about surveillance. This view closely resembles the legal reasoning behind the {tpd|Third-Party Doctrine}, which holds that individuals lose certain constitutional privacy protections when they voluntarily share information with third parties such as banks, phone companies, or internet service providers.\n\nA common response to this view is that it mistakenly assumes there is a meaningful option to opt out of existing information-gathering systems. When users are given robust and meaningful opportunities to opt out of data collection, [many choose to do so](https://itif.org/publications/2017/10/06/economics-opt-out-versus-opt-in-privacy-rules/). However, many individuals do not, in fact, have a substantive option to opt out of modern-day surveillance. Technical tools to evade facial recognition or video surveillance are typically expensive or [legally prohibited](https://code.dccouncil.gov/us/dc/council/code/sections/22-3312.03). Additionally, existing covert surveillance systems may be unknown to the general public, and therefore, individuals cannot consent to such programs. Thus, individuals may not have a robust option to opt out of surveillance.\n\n### Privacy and Security Are Not Always Mutually Exclusive\n\nIn public debate, privacy and security are often presented as opposing goods, where the trade-off is between two mutually exclusive outcomes: to be safer, we must accept more surveillance; to preserve privacy, we must be willing to accept greater risk. This framing, however, often obscures the reality that in a variety of contexts, privacy and security can reinforce one another. In some cases, {security-necessitates-robust-privacy-protections| security necessitates robust privacy protections} to guard against industrial espionage or unwarranted intrusions into private domains.\n\nIn the context of **information warfare**, national security depends on robust encryption and greater privacy for both private and public entities. In these cases, privacy functions not as a counterweight, but as a foundational component of security itself.","sidebarKey":null,"isDrawer":false},{"id":"privacy/privacyTradeoffs#privacy-vs-convenience","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"privacy-vs-convenience","title":"Privacy vs. Convenience","content":"Privacy is often exchanged for convenience in everyday technologies. Many tools that feel seamless or efficient in practice rely on collecting, linking, and inferring from personal data. In these settings, users receive **faster service** and **greater personalization**. At the same time, however, users give up elements of informational control, since, in these cases, the data that enables convenience must be continuously gathered and analyzed. For example, biometric authentication allows people to unlock devices and board flights, personalized recommendation systems reduce the time spent searching for relevant content, and smart home devices reduce friction in maintaining the home. In each of these cases, convenience is achieved by relinquishing control over data.\n\n### What Does Convenience Look Like in More Detail?\n\nOne aspect of convenience is {speed}, i.e., reducing the time it takes to access a product/service. Users choose to disregard, either consciously or unconsciously, the extensive data policies of the product/service to reduce the time or effort it takes to use it. In such cases, people value the product/service's speed and convenience over the loss of privacy.\n\nAnother aspect of convenience is better service and personalization. Greater access to data allows firms to tailor products and services and communicate with individual customers. Personalization can reduce information overload, helping customers make more efficient decisions. Furthermore, personalization can {minimize-customer-search-costs|minimize customer search costs} by presenting the relevant products and advertisements at the right time. However, personalization also enables companies to shape user engagement by curating content that maximizes attention and reduces the likelihood that users exit the page. Thus, while personalization is convenient, it may also allow companies to influence behavior and prolong engagement in ways that prioritize platform objectives over the interests of the user.\n\nAnother aspect of convenience is access to core product features or simply the product itself. Many digital products and services condition full access to features on the acceptance of extensive data-collection practices. By default and through consent mechanisms, many platforms make full functionality contingent on data collection. As a result, opting out of tracking can lead to restricted access, degraded service, or additional costs, effectively making data collection a prerequisite for access. For example, the **“consent-or-pay” model** has been increasingly adopted by news organizations. Under this approach, readers must either agree to the use of tracking cookies and personalized advertising or pay a subscription fee to access content without data collection. One such example of this model is the influential German weekly news magazine, [Der Spiegel](https://www.spiegel.de). When opening the page, users must either “pay with data” by consenting to tracking or pay with money.\n\n### The Privacy Paradox\n\nThe “**privacy paradox**” is one way of describing this tension between privacy and convenience. Over the past three decades, privacy concerns have grown, with people consistently reporting that they prioritize privacy. For example, people [increasingly refuse](https://www.aeaweb.org/articles?id=10.1257/aer.102.3.349) to disclose details like income or zip code in surveys, with overall “**privacy sensitivity**” on the rise. However, in a study conducted by Alessandro Acquisti and his colleagues, when presented with choices with different degrees of privacy, participants typically implicitly assigned limited value to their privacy.[^2] Furthermore, people typically demand more money to give up their data than they are willing to pay to protect it, suggesting that privacy is valued differently depending on how a decision is framed. In other words, the privacy paradox refers to the gap between people’s stated preference for privacy and their actual behavior, which often reveals a willingness to trade privacy for convenience.\n\nThere are several explanations for the “privacy paradox.”\n\nFirst, it may be that consumers lack sufficient information about how their data might be used. Firms often delay the presentation of, or obscure, privacy policies to increase revenue. Thus, consumers may be unaware of the full extent of the trade-off between privacy and convenience.\n\nSecond, the paradox is measured in specific contexts, while self-reported privacy concerns tend to come from general surveys. Therefore, a methodological explanation may clarify how self-reported privacy concerns may not closely align with the paradox measured in specific contexts.\n\nThird, {consumers-willing|consumers may be willing to give up privacy}in favor of network effects, even if they would rather the privacy-compromising product not exist at all. A **network effect** is when a product or service becomes more valuable to each user as more people use it. One example of this explanation of the privacy paradox is a reluctant user who shares their location with friends. Although this person would rather not have anyone know their location, they participate because all of their friends share their locations. Thus, while they would rather not have anyone share their locations, in the case that their friends do, they would like to participate in the location sharing rather than being left out.\n\nIn this light, these explanations suggest the privacy paradox may arise not from an ambivalence towards privacy, but rather from how privacy choices are designed, presented, and constrained.","sidebarKey":null,"isDrawer":false},{"id":"privacy/privacyTradeoffs#privacy-vs-public-health","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"privacy-vs-public-health","title":"Privacy vs. Public Health","content":"Privacy may also be traded in favor of supporting public health goals. Public health practices often rely on collecting, linking, and analyzing data about disease, treatments, and behavior. Examples include:\n\n- {ehr|Electronic Health Records (EHRs)},\n- {tracing|Contact tracing during pandemics},\n- {alerts|Early alerts for epidemics}, and\n- {mapping|Emergency mapping of mental health}.\n\nWhen used responsibly, these initiatives can identify and control outbreaks earlier, manage chronic disease, and improve the safety and efficacy of treatments. Therefore, the trade-off is whether the improvements in population health justify deeper visibility into individuals’ personal data.","sidebarKey":null,"isDrawer":false},{"id":"privacy/privacyTradeoffs#privacy-vs-research","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"privacy-vs-research","title":"Privacy vs. Research","content":"Scientific research often depends on access to personal or behavioral data, setting up a trade-off in which society gains valuable insights, while participants risk reidentification and lose privacy. Although large, detailed datasets enable robust research findings, the same level of detail also increases the likelihood of revealing participants' identities and other sensitive information. Examples include:\n\n- {Medical-research|Medical research},\n- {Urban-planning-research|urban planning research}, and\n- {Economics-research|economics research}.\n\n### Privacy and Research Are Not Always Mutually Exclusive\n\nAlthough research often requires access to sensitive personal data, privacy and research need not always be in direct conflict. In many cases, researchers can collect detailed information while still implementing {safeguards} that reduce the risk of exposing individuals’ identities. In these cases, privacy risks are mitigated not by eliminating data collection entirely but by controlling how the data are stored, accessed, and shared.","sidebarKey":null,"isDrawer":false},{"id":"privacy/privacyTradeoffs#privacy-vs-economic-efficiency","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"privacy-vs-economic-efficiency","title":"Privacy vs. Economic Efficiency","content":"Privacy may be traded in favor of promoting {economic-efficiency|economic efficiency}, maximizing the good derived from the market for the most people, particularly in markets that rely on data to optimize production and target consumers, or for the state to allocate resources efficiently.\n\nBusinesses and legislators often argue that collecting personal information such as purchase history, browsing behavior, geolocation, and payment patterns enables more accurate decision-making about credit, pricing, marketing, and resource distribution, benefiting consumers, firms, and the government. Examples include:\n\n- {Government-distribution-of-aid|Government distribution of aid},\n- {Automated-and-expanded-credit-scoring|automated and expanded credit scoring},\n- {Targeted-advertising-and-price-discrimination|targeted advertising and price discrimination},\n- {Improvements-in-customer-retention|improvements in customer retention}, and\n- {The-data-broker-industry|the data broker industry}.\n\nUltimately, these examples illustrate the broader tension between privacy and economic efficiency: greater access to personal data can improve market performance, policy design, and access to credit, but it simultaneously reduces our control over how and whether information about us is collected, analyzed, and used.","sidebarKey":null,"isDrawer":false},{"id":"privacy/privacyTradeoffs#privacy-vs-privacy","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"privacy-vs-privacy","title":"Privacy vs. Privacy","content":"Privacy trade-offs are often framed as conflicts between privacy and other goods, such as security, convenience, or economic efficiency. Less commonly acknowledged, however, are situations in which prioritizing one form of privacy necessarily diminishes another. In these cases, competing privacy concerns may be irreconcilable, and policy determines whose privacy is protected, against whom, and under what conditions. David E. Pozen outlines three common forms of privacy-versus-privacy trade-offs: distributional, directional, and dynamic trade-offs.[^5]\n\nFirst, {dist-to|distributional trade-offs} shift privacy burdens _from one group in the population to another_. One example is a city increasing the use of license plate readers and officer deployments to high-crime neighborhoods. Residents in those neighborhoods now experience continuous vehicle-location tracking and increased likelihood of searches by police, while residents in other neighborhoods receive marginal gains in privacy through reduced police presence.\n\nAnother potential reform is having communications metadata be retained by private companies or independent entities rather than the NSA. However, moving where data is housed does not eliminate the privacy risk; it simply changes its location and scope. Furthermore, shifting the storage of such data could expand privacy risks by making information accessible for a wider range of non-security purposes.\n\nSecond, **directional trade-offs** select _which individuals or groups are positioned to violate one’s privacy_. For example, using a ride-hailing app instead of taking the subway means we lose privacy by exposing the destination to the app. On the other hand, taking the subway instead of using a ride-hailing app means we lose privacy to other passengers who may notice where we alight. Privacy _from_ one group of people is secured at the cost of losing privacy _from_ another.\n\nThird, **dynamic trade-offs** shift privacy risks across _time_. For instance, trusted traveler or pre-screening initiatives such as CLEAR and TSA PreCheck involve intensive upfront data collection in exchange for reduced scrutiny in the future.\n\nUltimately, these examples illustrate how privacy trade-offs often involve competing distributions of privacy risks rather than absolute gains or losses. Policies that appear to protect privacy may do so by reallocating exposure across populations, institutions, time periods, or dimensions of personal life. Recognizing the trade-off clarifies the central question of whose privacy is prioritized and at what cost to others.\n\n[^1]: Adam Moore, “Privacy, Security and Accountability,” 2015, 174, [https://faculty.washington.edu/moore2/PSA.pdf](https://faculty.washington.edu/moore2/PSA.pdf).\n\n[^2]: Alessandro Acquisti, Leslie K. John, and George Loewenstein, “What Is Privacy Worth?,” _The Journal of Legal Studies_ 42, no. 2 (June 2013): 249–74, [https://doi.org/10.1086/671754](https://doi.org/10.1086/671754).\n\n[^3]: Cass R Sunstein, “Barbies, Ties, and High Heels: Goods That People Buy but Wish Did Not Exist,” _Social Science Research Network_, January 1, 2023, [https://doi.org/10.2139/ssrn.4614036](https://doi.org/10.2139/ssrn.4614036).\n\n[^4]: Charles Doyle, “CRS Report for Congress USA PATRIOT Act Sunset: Provisions That Expire,” January 27, 2005, 26, [https://www2.law.umaryland.edu/marshall/crsreports/crsdocuments/RL3218601272005.pdf](https://www2.law.umaryland.edu/marshall/crsreports/crsdocuments/RL3218601272005.pdf).\n\n[^5]: David Pozen, “Privacy-Privacy Tradeoffs,” n.d., [https://lawreview.uchicago.edu/sites/default/files/10%20Pozen_SYMP_Final.pdf](https://lawreview.uchicago.edu/sites/default/files/10%20Pozen_SYMP_Final.pdf).\n\n[^6]: G. Alex Sinha, “NSA Surveillance since 9/11 and the Human Right to Privacy,” _Loyola Law Review_ 59 (January 1, 2013): 861, [https://scholarlycommons.law.hofstra.edu/faculty_scholarship/1371/](https://scholarlycommons.law.hofstra.edu/faculty_scholarship/1371/).\n\n[^7]: Leonardo Bursztyn et al., “When Product Markets Become Collective Traps: The Case of Social Media,” _SSRN Electronic Journal_, 2023, [https://doi.org/10.2139/ssrn.4597079](https://doi.org/10.2139/ssrn.4597079).\n\n[^8]: National Archives, “Japanese-American Internment during World War II,” National Archives (The U.S. National Archives and Records Administration, August 15, 2016), [https://www.archives.gov/education/lessons/japanese-relocation\\#background](https://www.archives.gov/education/lessons/japanese-relocation#background).\n\n[^9]: Stewart A Baker, “Stewart A. Baker Oversight Hearing on FISA Surveillance Programs Committee on the Judiciary United States Senate,” The Federalist Society, October 21, 2013, [https://fedsoc.org/commentary/publications/stewart-a-baker-oversight-hearing-on-fisa-surveillance-programs-committee-on-the-judiciary-united-states-senate](https://fedsoc.org/commentary/publications/stewart-a-baker-oversight-hearing-on-fisa-surveillance-programs-committee-on-the-judiciary-united-states-senate).\n\n[^10]: Tat Chan et al., “The Value of Verified Employment Data for Consumer Lending: Evidence from Equifax,” _Marketing Science_, December 28, 2021, [https://doi.org/10.1287/mksc.2021.1335](https://doi.org/10.1287/mksc.2021.1335).\n\n[^11]: United States Senate, “U.S. Senate: Senate Select Committee to Study Governmental Operations with Respect to Intelligence Activities,” www.senate.gov, April 29, 1976, [https://www.senate.gov/about/powers-procedures/investigations/church-committee.htm](https://www.senate.gov/about/powers-procedures/investigations/church-committee.htm).","sidebarKey":null,"isDrawer":false},{"id":"privacy/privacyTradeoffs#sidebar-internment-of-japanese-americans","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"internment-of-japanese-americans","title":"Interment of Japanese-Americans","content":"Following Pearl Harbor in World War 2, approximately 112,000 people of Japanese descent, including 70,000 American citizens, were detained and forced into internment camps by the United States military.[^8] The military used broad ethnic surveillance to enable internment under the assumption that anyone of Japanese descent could be a spy or saboteur, substituting ethnicity for evidence. The Supreme Court upheld the policy in _Korematsu v. United States_ (1944), a decision that was later condemned as a landmark civil liberties failure. For more information, see [Japanese-American Incarceration During World War II](https://www.archives.gov/education/lessons/japanese-relocation#background) by the National Archives.","sidebarKey":"internment-of-japanese-americans","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-mcepi","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"mcepi","title":"McCarthy-Era Political Investigations","content":"During the early Cold War, Senator Joseph McCarthy and the House Un-American Activities Committee (HUAC) conducted government investigations targeting suspected communists through targeted surveillance, interrogations, and blacklists that often violated civil liberties and due process. The FBI compiled secret dossiers on thousands of Americans, while HUAC used public hearings to pressure individuals to name associates, transforming surveillance into an instrument of political control. For more information, see [Senator Joe McCarthy’s Oversight Abuses](https://levin-center.org/joe-mccarthys-oversight-abuses/).","sidebarKey":"mcepi","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-cointelpro","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"cointelpro","title":"COINTELPRO","content":"The Counter-Intelligence Program, more commonly known as COINTELPRO, was a series of covert Federal Bureau of Investigation operations from 1956 to 1971 that targeted any domestic political groups deemed subversive, including civil rights organizations, socialist movements, and anti-war activists. The program was notable for violating the First Amendment rights of Americans through illegal wiretapping and harassment, even resulting in a number of deaths.[^11] For more information, see [Spying on America: The FBI's Domestic Counter-Intelligence Program](https://academic.oup.com/jah/article-abstract/80/1/347/672209).","sidebarKey":"cointelpro","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-patriot","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"patriot","title":"USA PATRIOT Act","content":"The USA PATRIOT (Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism) Act greatly expanded the government’s surveillance powers following the September 11 attacks in 2001\\. These expansions include “roving” wiretaps that allow the government to monitor a suspect’s communications across multiple devices with a single warrant and delayed-notification searches.[^4] For more information, see the [Electronic Privacy Information Center’s overview of the Patriot Act’s surveillance provisions and civil liberty concerns](https://epic.org/issues/surveillance-oversight/patriot-act/).","sidebarKey":"patriot","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-post-nine-sur","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"post-nine-sur","title":"Post-9/11 Surveillance of Muslim Americans","content":"Following the September 11 attacks, the New York City Police Department’s Intelligence Division began systematically monitoring Muslim communities in New York, targeting mosques, student groups, and businesses based on religion. A Pulitzer Prize–winning [Associated Press investigation](https://www.ap.org/media-center/ap-in-the-news/2012/ap-series-about-nypd-surveillance-wins-pulitzer/) reported that the NYPD’s Demographics Unit mapped, photographed, or infiltrated [over 250 mosques and Muslim student organizations](https://www.foxnews.com/us/documents-nypd-eyed-250-plus-mosques-student-groups). In 2012, the chief of the NYPD Intelligence Division [admitted under sworn testimony](https://csrr.rutgers.edu/wp-content/uploads/2020/01/when-the-fbi-knocks.pdf) that both the unit and the [broader NYPD surveillance program](https://ccrjustice.org/home/press-center/press-releases/settlement-reached-nypd-muslim-surveillance-lawsuit) failed to produce a single lead.\n\nFor more information, see the ACLU’s [Factsheet: The NYPD Muslim Surveillance Program](https://www.aclu.org/documents/factsheet-nypd-muslim-surveillance-program).","sidebarKey":"post-nine-sur","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-controversial","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"controversial","title":"FISA Controversy","content":"Given the classified nature of national security investigations, the extent to which these programs improve security has been difficult to determine. In 2013, former President Barack Obama [stated](https://www.nytimes.com/2013/06/20/world/europe/obama-in-germany.html) in reference to the programs under FISA that, “we know of at least 50 threats that have been averted because of this information, not just in the United States but in some cases threats here in Germany.”\n\nIn 2014, the Privacy and Civil Liberties Oversight Board (PCLOB) released two separate reports evaluating two programs under FISA. The [first report](https://documents.pclob.gov/prod/Documents/OversightReport/823399ae-92ea-447a-ab60-0da28b555437/702-Report-2.pdf) addressed Section 702, which authorizes the government to monitor electronic communications of non-US persons located abroad.\n\nThe Board found that the program had enabled the government to gather intelligence more quickly and effectively than would have otherwise been possible, and, in 2023, [stated](https://documents.pclob.gov/prod/Documents/EventsAndPress/d6b5ba9e-4d1d-4081-a909-c1e8f073c979/PCLOB%20FISA%20SECTION%20702%20PRESS%20RELEASE%209-28-23.pdf) that Section 702 “is so valuable that not one Member of this Board believes Congress should \\[allow it to expire.\\]” The Board reached the opposite conclusion on the [second report](https://documents.pclob.gov/prod/Documents/OversightReport/ec542143-1079-424a-84b3-acc354698560/215-Report_on_the_Telephone_Records_Program.pdf) concerning the Section 215 bulk telephone metadata collection program, under which the NSA had been gathering records of millions of Americans’ phone calls. It found that the program never directly contributed to the discovery of a previously unknown terrorist plot or attack.\n\nFurthermore, critics (e.g., Adam D. Moore) argue that prioritizing security over privacy without meaningful limits can undermine the security of the people.[^1] When governments are granted broad authority to collect, analyze, and retain personal data without oversight, they may become sources of risk rather than sources of protection.\n\nFor example, critics (e.g., the ACLU) have noted that under FISA, much surveillance happens without individualized judicial warrants, where courts will often authorize broad protocols enabling warrantless collection and review of communications. Under FISA warrants, the National Security Agency (NSA) may obtain communications, including international messages, emails, and phone calls, of non-US persons. However, when Americans are in contact with any foreigners abroad, their communication data is incidentally collected and stored through this program. As the [ACLU points out](https://www.aclu.org/news/national-security/five-things-to-know-about-nsa-mass-surveillance-and-the-coming-fight-in-congress), although the law allows for surveillance of foreigners, the FBI routinely utilizes this extensive source of information by searching the databases to find and examine the communications of Americans for use in domestic investigations. In this way, surveillance powers justified by national security concerns can extend beyond their original targets, increasing the risk that tools intended to protect the public are used to monitor the very population they were meant to defend.","sidebarKey":"controversial","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-tpd","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"tpd","title":"Third-Party Doctrine","content":"For more information on the Third-Party Doctrine, see “[The Fourth Amendment, the Third-Party Doctrine, and Cloud-Stored Data: Do Terms of Service Undermine Our Privacy Expectations in the Digital Age?](https://lawreview.law.uic.edu/news-stories/the-fourth-amendment-the-third-party-doctrine-and-cloud-stored-data-do-terms-of-service-undermine-our-privacy-expectations-in-the-digital-age/)”","sidebarKey":"tpd","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-security-necessitates-robust-privacy-protections","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"security-necessitates-robust-privacy-protections","title":"Security necessitates robust privacy protections","content":"For more information on how strong encryption strengthens national, economic, and infrastructure security, see the following articles:\n\n- [Ex-NSA Chief Defends End-To-End Encryption, Says 'Backdoors' Will Make Us Less Secure](https://www.bgr.com/tech/ex-nsa-chief-hayden-encryption/)\n- [FBI on Chinese Espionage](https://www.fbi.gov/investigate/counterintelligence/the-china-threat)\n- [Ex-Google engineer convicted of stealing AI secrets](https://www.reuters.com/legal/government/ex-google-engineer-convicted-stealing-ai-secrets-chinese-companies-2026-01-29/)","sidebarKey":"security-necessitates-robust-privacy-protections","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-speed","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"speed","title":"Speed","content":"Common examples include fingerprint scanners and Face ID, which allow users to bypass conventional security codes. From 2020 to 2026, [Amazon](https://www.geekwire.com/2026/amazon-is-ending-its-palm-id-system-for-retail-amazon-one-as-it-closes-physical-stores/) allowed individuals to register their palms for payment of goods at Amazon-owned companies. While this technology reduced friction at checkout, it required users to provide sensitive biometric data to a private company, one that is not legally required to handle the information responsibly in the same ways medical providers are.","sidebarKey":"speed","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-minimize-customer-search-costs","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"minimize-customer-search-costs","title":"Minimizing customer search costs","content":"In a large-scale field experiment on the Alibaba E-commerce platform, [Sun et al.](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3962157) banned the use of personal data in the homepage recommendation algorithm involving a random sample of 555,800 customers. The researchers observed a sharp decrease in both customer engagement (clickthrough rate and product browsing) and market transactions (sales volume and amount), ultimately finding purchases fell by 81%.","sidebarKey":"minimize-customer-search-costs","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-consumers-willing","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"consumers-willing","title":"Consumers may be willing to give up privacy","content":"One potential example of this is examined by Cass R. Sunstein in his paper “Barbies, Ties, and High Heels: Goods That People Buy But Wish Did Not Exist.”[^3] He posits that when people are willing to demand payment to give up a good but would also pay to eliminate it, this suggests that people would prefer that such a good did not exist at all, but if it does exist, they would prefer to have access to it.\n\nFor example, research by Leonardo Bursztyn and collaborators found that users would need to _be paid_ $59 to individually deactivate TikTok for a month.[^7] However, users would simultaneously _pay_ $28 to have _all of the world_, including themselves, deactivate TikTok for a month. Therefore, one such explanation for the privacy paradox is that in the context of social media, people would prefer that it not exist at all, thereby preserving their privacy, but would prefer to be able to access it and give up privacy if others are using it.","sidebarKey":"consumers-willing","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-ehr","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"ehr","title":"Electronic Health Records","content":"An EHR is the electronic version of a patient’s record, including medical history, biographical information, and testing and imaging data. The purported benefits of EHR adoption include [improvements in the delivery of care](https://pmc.ncbi.nlm.nih.gov/articles/PMC9647912/), [access to data for research](https://pmc.ncbi.nlm.nih.gov/articles/PMC10946197/), [reduced medical errors](https://pubmed.ncbi.nlm.nih.gov/26136462/), and [expanded care modalities](https://pubmed.ncbi.nlm.nih.gov/19275987/) such as telehealth and e-messaging. Nevertheless, it is important to note the [mixed results of EHR adoption](https://www.ama-assn.org/practice-management/digital-health/electronic-health-record-ehr-use-research) depending on the specific ways it is implemented.\n\nOn the other hand, the [risk of privacy violations is heightened](https://pmc.ncbi.nlm.nih.gov/articles/PMC7349636/) with the adoption of EHRs, which has been further exacerbated by recent growth in EHR access by mobile devices, third-party sharing (including between organizations, clinicians, and federal agencies), and hacking incidents. At the same time, unauthorized access occurs most often through an abuse of privileges or unauthenticated access/disclosures of information. In the context of research, EHR data integration similarly carries security as well as reidentification risks.\n\nAccording to [HIPAA reports](https://www.hhs.gov/hipaa/for-professionals/breach-notification/reports-congress/index.html), from 2010 to 2019, 255.18 million people were affected by 3,051 healthcare data breach incidents caused by hacking, unauthorized access, theft, and improper disposal. Because EHRs are stored digitally, [cyberattacks involving Ransomware have increased](https://ieeexplore.ieee.org/abstract/document/8726303), with individual incidents exposing medical records and social security numbers of up to a million patients.","sidebarKey":"ehr","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-tracing","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"tracing","title":"Contact tracing during pandemics","content":"Contact tracing during public health outbreaks illustrates this trade-off. Researchers in Taiwan [noted](https://pdfs.semanticscholar.org/5139/4ef323281cb5512fbd9a4e216dff7d952bba.pdf) that quarantine is most effective for preventing further infections by identifying infectious, asymptomatic patients and requiring contact tracing. Before the emergence of contact tracing through mobile phone location data, public health workers investigated all close contacts of each individual case through extensive interviews. In the process of these interviews, patients would divulge both where they had been as well as who they had come into contact with. This information was then used by the government to support quicker identification of exposed contacts, faster isolation and quarantine, and ultimately a more targeted approach where only those in direct or indirect contact with the infected are subject to restrictions. To help contain transmission, individuals are often required to disclose sensitive details about their movements, relationships, and daily routines.\n\nGiven the emergence of contact tracing through mobile phone location data, a more recent example is the COVID-19 pandemic. The pervasiveness of this privacy trade-off is heightened in the age of mobile phone location data, given the breadth and accuracy of such data. Across the world, there was a spectrum of models between “privacy-first” approaches, which store minimal data and limit access by public authorities, and “data-first” models, which centralize large volumes of location and identifying information.\n\nOne such example of the “privacy first” model is the [Apple-Google Exposure Notification](https://covid19-static.cdn-apple.com/applications/covid19/current/static/contact-tracing/pdf/ENPA_White_Paper.pdf) framework used in several US states and European countries. The system relied on decentralized Bluetooth proximity signals, stored locally on individual devices, rather than GPS location data, and it prevented public health authorities from accessing contact lists. While users retained greater control over their data and limited government visibility into their movements, public health agencies also had more limited visibility into transmission chains and could not directly enforce quarantine compliance.\n\nIn contrast, China implemented a [QR-code-based health code](https://pmc.ncbi.nlm.nih.gov/articles/PMC7173240/) system representative of a “data first” approach. The Chinese government integrated the codes into popular applications like WeChat and Alipay. The data from these apps was then processed together with medical data to generate QR-codes that determined individuals’ access to public transportation, workplaces, and public spaces. In this case, centralized data collection enabled rapid enforcement of quarantine orders and real-time monitoring, but also required collection, storage, and governmental control of highly sensitive data at scale.\n\nNevertheless, despite differences in designs across countries, testing and/or contact tracing was associated with [reductions in transmission](https://royalsocietypublishing.org/rsta/article/381/2257/20230131/112506/Effectiveness-of-testing-contact-tracing-and) measured by population-level mortality, number of cases per capita, and growth of cases.","sidebarKey":"tracing","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-alerts","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"alerts","title":"Early Alerts for Epidemics","content":"Search history and other online data have been used to [provide early alerts for epidemics](https://www.heinz.cmu.edu/~acquisti/papers/AcquistiTaylorWagman-JEL-2016.pdf). Projects such as [Google Flu Trends](https://datacollaboratives.org/cases/google-flu-trends.html), [HealthMap](https://www.healthmap.org/en/), and the [Program for Monitoring Emerging Diseases](https://www.promedmail.org) use search history coupled with news reports to produce early warnings about seasonal influenza and emerging outbreaks. These initiatives show how repurposed personal data can support public health goals, but it can also easily be used for a variety of unanticipated inferences, such as [social mapping](https://www.nytimes.com/2006/08/09/technology/a-face-is-exposed-for-aol-searcher-no-4417749.html), [political and religious affiliation identification](https://www.pnas.org/doi/10.1073/pnas.1218772110), or [creditworthiness](https://www.wsj.com/articles/SB10001424052748703294904575385532109190198?gaa_at=eafs&gaa_n=AWEtsqdoHMLyc-wkuHO141C-tWQ7S6TRK0uSe1ls3evXBRPaIJXY0y1mzOZVt6Djb1I%3D&gaa_ts=699e5f2f&gaa_sig=dWchPyT4YTeMDyMDz1w4H5Q8zNeuL84QRSCTcb7mncZLQ0zVuSwcJxw022RpHKW5Mgn1iu1yikMUmEgb70pU4Q%3D%3D).","sidebarKey":"alerts","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-mapping","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"mapping","title":"Emergency Mapping of Mental Health","content":"Another opportunity in public health is [emergency mapping of mental health](https://arxiv.org/pdf/1612.00323) using behavioral data collected via smartphones and social media. Currently, the traditional model of care relies on periodic self-reports of symptoms, which suffer from subjectivity, memory biases, and the patient's current mood. Furthermore, individuals with mental conditions usually seek care once a crisis has already happened. Access to novel behavioral data, such as search history and social media posts, may be used to monitor mental health-related symptoms outside a clinical setting without relying on self-reported data. Several studies have shown that digital behavioral data can be utilized to recognize bipolar disorders, mood, and stress. Additionally, researchers at Harvard University used Twitter data, noting affect, linguistic style, and context, to reliably [detect the onset of depression](https://arxiv.org/pdf/1608.07740) several months before diagnosis. While these uses of behavioral data may one day enable more proactive support and treatment, they also enable inferences about highly sensitive details that individuals may never want disclosed.","sidebarKey":"mapping","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-medical-research","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"medical-research","title":"Medical research","content":"Medical research increasingly depends on access to large-scale health and genomic databases. One prominent example is [The Cancer Genome Atlas](https://www.cancer.gov/ccg/research/genome-sequencing/tcga) (TCGA), which aggregates genetic and molecular data from thousands of cancer patients. By enabling researchers to analyze vast genomic datasets, TCGA has helped identify the genetic drivers of many cancers and opened new avenues for targeted therapies.\n\nPopulation-scale biobanks further extend these capabilities alongside increases in the risk of privacy violations. Initiatives in the US ([NIH All of Us Research Program](https://allofus.nih.gov)) and the UK ([UK Biobank](https://www.ukbiobank.ac.uk)) aim to enroll large swaths of the population, collecting EHRs, genomic data, and survey information to support research on the causes of disease. The resource has generated thousands of publications identifying genetic and environmental risk factors for conditions such as heart disease, cancer, and diabetes. Participants in the US also benefit from direct reports on genetic results about ancestry, traits, and certain health-related DNA findings. Nevertheless, similarly to genomic databases, both programs carry risks of reidentification, potentially exposing participants’ sensitive health information. For more information on privacy risks related to open research data, see “[Open research data poses real world risks that need to be managed](https://blogs.lse.ac.uk/impactofsocialsciences/2024/11/19/open-research-data-poses-real-world-risks-that-need-to-be-managed/).”\n\nThe gains from medical research are offset by the risk of the health and genomic data that is collected for medical research being misused, accessed without authorization, and contributing to reidentification. Given that genetic and health data are often uniquely identifying, even datasets that are anonymized or shared for research purposes may expose participants to privacy risks when accessed improperly or combined with other information. Examples of this include:\n\n- The [University of Hawai'i Cancer Center ransomware data breach](https://therecord.media/university-of-hawaii-ransomware-data-breach) exposed the records of 1.2 million people, including specific health and trial data related to over 80,000 participants.\n- Researchers have demonstrated that supposedly [anonymized genomic data can sometimes be reidentified](https://www.nature.com/articles/nrg3723) by combining genomic data with publicly available genealogy databases and demographic information.\n- [Google partnered with Ascension hospitals](https://www.wsj.com/articles/google-s-secret-project-nightingale-gathers-personal-health-data-on-millions-of-americans-11573496790?gaa_at=eafs&gaa_n=AWEtsqeV0dI5W4TmBCW49ahogjWYjf0mBcaQyHT9hJP5xu2mscnDYmPgTzeAAqTqpGY%3D&gaa_ts=69b34a15&gaa_sig=qL25kcoQoMhY8LPO10_qBHZPyXhwUWF6JsgVZnkTnoe-jJZbS98WhxUlpZfrpjbN4mFOcQK1M3K4skj4Y_a_EA%3D%3D) to collect and analyze millions of detailed patient medical records without notifying the patients or their doctors whose data was included.\n- [A fringe group of international scientists](https://www.nytimes.com/2026/01/24/us/children-genetics-race-science.html) thwarted the National Institute of Health safeguards, accessing the data of thousands of children to argue for the intellectual superiority of white people.","sidebarKey":"medical-research","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-urban-planning-research","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"urban-planning-research","title":"Urban planning research","content":"Research in urban planning draws on behavioral data to improve planning and evaluation of city design. For example, [Lauren Alexander](https://pmc.ncbi.nlm.nih.gov/articles/PMC5774986/), a professor of Civil Engineering at MIT, used smartphone location data from millions of people to estimate the origins and destinations of each trip, inferring the purpose, which supports more accurate models of congestion and infrastructure needs. Additionally, [Jeong et al.](https://pubmed.ncbi.nlm.nih.gov/39755829/) used smartphone accelerometer data from the NIH All of Us research program to study “activity inequality” across cities and countries. They found that walkable environments are associated with greater activity across all groups, with the greatest increases among women.\n\nHowever, the types of data that urban planning largely relies on—location, mobility, and transportation data—are susceptible to data misuse and unauthorized access, violating the privacy of the individuals from whom it was generated. Examples of this include:\n\n- [Uber and Lyft unintentionally sending gig workers’ Social Security numbers to social media companies](https://news.northeastern.edu/2024/11/15/uber-privacy-social-security-leak/),\n- [Strava giving away locations of secret US army bases](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases), and\n- [The New York Times analyzing a large location dataset and showing that many of the twelve million phones could be easily reidentified](https://www.nytimes.com/interactive/2019/12/19/opinion/location-tracking-cell-phone.html).","sidebarKey":"urban-planning-research","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-economics-research","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"economics-research","title":"Economics research","content":"In economics, access to real-time anonymized data from private companies has provided researchers and policymakers with more visibility into the health of the economy. For example, the [Opportunity Insights Economic Tracker](https://opportunityinsights.org/tracker-resources) was first created to monitor the economic impact of COVID-19 by aggregating anonymized data on credit card spending, payroll, and job postings. Even now, the tracker continues to aid studies of how different policy responses affect employment and spending across income groups. Furthermore, Edward Knotek, director of research at the Federal Reserve Bank of Cleveland, [found](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwiw9-PXpZ2TAxU1D1kFHQK8FvoQFnoECCEQAQ&url=https%3A%2F%2Fwww.clevelandfed.org%2F-%2Fmedia%2Fproject%2Fclevelandfedtenant%2Fclevelandfedsite%2Fpublications%2Fworking-papers%2F2024%2Fwp2406.pdf&usg=AOvVaw17wyQyr77wISMq15YivCIw&opi=89978449) that greater access to payment data can improve models and predictions about consumption and inflation more accurately than traditional surveys of forecasters.\n\nHowever, the types of financial and behavioral data used to support this research also create privacy risks when collected, shared, or breached. Examples of this include the [Capital Economics Research leak](https://www.twingate.com/blog/tips/capital-economics-data-breach), which exposed the personal and financial information of 264,000 individuals, and the [Equifax data breach](https://www.ftc.gov/enforcement/refunds/equifax-data-breach-settlement), which exposed sensitive financial information, including Social Security numbers and credit histories of over 140 million Americans.","sidebarKey":"economics-research","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-safeguards","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"safeguards","title":"Safeguards","content":"For more information on safeguards, see “The Problem of Reidentification” primer.  \nFor more reading, see the following articles:\n\n- [NIH Genomic Data Sharing Policy](https://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-124.html) outlines strict requirements for de-identification, controlled access, and researcher accountability.\n- For information on controlled access, restricted data, and encryption, see “[Routes for Breaching and Protecting Genetic Privacy](https://www.nature.com/articles/nrg3723).”\n- See “[Balancing Access to Health Data and Privacy: A Review of the Issues and Approaches for the Future](https://pmc.ncbi.nlm.nih.gov/articles/PMC2965886/),” for information on Data Enclaves, which are secure environments where sensitive data can be accessed under supervision without being removed or widely shared.","sidebarKey":"safeguards","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-economic-efficiency","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"economic-efficiency","title":"Economic efficiency","content":"The idea of economic efficiency centers around people getting as much good (surplus) as possible from markets.\n\nIn an efficient market, there will be a range of consumers, each willing to buy the given product at some price. The greatest number of consumers will be willing to buy the product at a very low price, and the fewest (zero) consumers will be willing to buy the product at a sufficiently high price. From this, economists are able to derive the downward-sloping demand curve. Similarly, there will be producers willing to produce some quantity of the product at each given price. The higher the price, the more they are willing to produce. From this, economists derive the upward-sloping supply curve. The market settles (reaches equilibrium) where these curves meet.\n\nAt this equilibrium quantity, there is some number of consumers to the left of it who would have been willing to pay a higher price. We call the sum of the difference between the higher price they were willing to pay and the lower equilibrium price “consumer surplus.” Correspondingly, there are producers to the left of the equilibrium quantity who would have been willing to produce some amount of the product for less than the equilibrium price they receive. We call the sum of the difference between the higher price they received and the lower price they were willing to pay “producer surplus.” Taken together, consumer and producer surplus are referred to as “total surplus.” The market equilibrium point is where total surplus is maximized.","sidebarKey":"economic-efficiency","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-government-distribution-of-aid","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"government-distribution-of-aid","title":"Government distribution of aid","content":"In some cases, more detailed information about firms or individuals can allow governments to distribute resources more efficiently. When policymakers lack granular economic data, assistance programs may be distributed based on incomplete information rather than actual need. Nevertheless, collecting this information would also require firms to disclose more detailed financial data to the government.\n\nOne example of this trade-off is the government distribution of aid during the COVID-19 pandemic. The [US Paycheck Protection Program](https://www.sba.gov/funding-programs/loans/covid-19-relief-options/paycheck-protection-program) provided large, forgivable loans to small businesses, with limited to no information available to the Treasury about each firm’s financial health. As a result, loans were intermediated through local banks and disproportionately flowed to firms with which the banks had strong connections rather than those in greatest need. Economists cited by the [New York Times](https://www.nytimes.com/2021/02/19/business/privacy-open-data-public.html) estimate that “the program spent between $150,000 and $377,000 per job saved, a high price for a program that was guaranteed for only a few months.” More timely and granular data on the financial health of firms would likely have aided the government in allocating loans more efficiently to those in the greatest need. An example of such a solution is the previously mentioned [Opportunity Insights Economic Tracker](https://opportunityinsights.org/tracker-resources/), which could improve the allocation of goods.\n\nNotably, data-driven government distribution of aid would not create a more “economically-efficient” system in a formal sense, as it does not increase surplus in the market. It simply improves the “efficiency” of government fund use in the colloquial sense.","sidebarKey":"government-distribution-of-aid","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-automated-and-expanded-credit-scoring","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"automated-and-expanded-credit-scoring","title":"Automated and expanded credit scoring:","content":"Access to broader and more novel forms of data on individuals can also improve economic efficiency by enabling lenders to make more accurate assessments of credit risk, thereby allowing them to expand credit access. These companies [bridge the gap](https://www.afi-global.org/wp-content/uploads/2025/02/Alternative-Data-for-Credit-Scoring.pdf) created by traditional credit scoring that relies heavily on past financial histories, which excludes individuals who lack a documented credit record, such as recent graduates, new immigrants, or those who are unbanked. In these cases, however, gains in credit access are balanced against the requirement that individuals share more personal behavioral and financial information.\n\nPrivate firms also use data to build decision-making algorithms, including credit scoring models that could expand credit access. [Chan et al.](https://pubsonline.informs.org/doi/10.1287/mksc.2021.1335) found substantial benefits for both borrowers and lenders from expanded credit access enabled by auto loan inquiries submitted digitally to Equifax, one of the three major consumer credit bureaus in the United States. They found that better-verified data increased the average loan origination rate by 35.5%, without substantially raising the interest rates charged on loans. On the other hand, lenders also benefited from an estimated 19.6% increase in profit from expanded credit access.[^10] Therefore, expanded access to data related to credit benefited borrowers, with a limited increase in cost, while simultaneously increasing lender profits.","sidebarKey":"automated-and-expanded-credit-scoring","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-targeted-advertising-and-price-discrimination","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"targeted-advertising-and-price-discrimination","title":"Targeted advertising and price discrimination:","content":"Firms can also utilize access to detailed behavioral data to predict consumer preferences and deliver advertisements that are more likely to generate purchases, thereby matching products with potential buyers. However, in order to achieve this level of targeting, firms collect and analyze extensive information about individuals’ browsing activity, location, purchases, and social networks.\n\nTargeted advertising typically uses data such as search history, location, purchases, social networks, and demographics, collected from users, to decide which ads to show to which individuals and when. Access to such data allows advertisers to [significantly improve click-through rates](https://pubsonline.informs.org/doi/10.1287/mnsc.1100.1246) and show users products they may desire.\n\nThis form of data-driven advertising can also enable price discrimination, where firms offer different prices or promotions to different consumers based on their willingness to pay. On one hand, this enables museums or theme parks to offer student discounts and charge lower prices to groups with lower purchasing power. On the other hand, companies may use detailed consumer data to identify customers who are less price sensitive and charge them higher prices. For example, [Instacart has been criticized](https://www.pbs.org/newshour/nation/instacart-ends-program-where-users-see-different-prices-for-the-same-item-at-same-store) for using personalized pricing and promotions based on consumers’ purchase histories and shopping behavior. This means that some customers saw higher prices or fewer deals than others for the same products. While in the case of theme parks and museums, the use of price discrimination expands access and output for more people, the use of price discrimination by Instacart simply extracted more money from consumers for the same transaction.","sidebarKey":"targeted-advertising-and-price-discrimination","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-improvements-in-customer-retention","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"improvements-in-customer-retention","title":"Improvements in customer retention:","content":"Another example of this trade-off is companies' improvement of [customer retention strategies](https://www.sciencedirect.com/science/article/pii/S0377221711008599). In subscription services, consumer activity data can reveal which customers are at risk of ending their subscriptions. These data can even reveal the likely impacts of different interventions aimed at retaining the customer. Therefore, with greater access to consumers' granular activity data, firms are better able to retain customers. However, these strategies depend on continuous monitoring and analysis of consumers’ activity patterns, which often include detailed records of how individuals use digital services.","sidebarKey":"improvements-in-customer-retention","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-the-data-broker-industry","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"the-data-broker-industry","title":"The data broker industry","content":"Data brokers are companies that aggregate data (publicly available and available for purchase) and then license it to other organizations. On one hand, this data can be used to promote fairer hiring practices, improve credit access, and personalize ads. On the other hand, these lists may also [segment individuals](https://www.cbsnews.com/news/the-data-brokers-selling-your-personal-information/) by health statuses, financial distress, and addiction. The granularity of such data can then be used to target vulnerable groups with predatory offers like casinos and [sports betting sites](https://prospect.org/2022/04/04/rollups-big-data-machine-driving-online-sports-betting/), offering gambling addicts a [free first bet](https://www.gamblingcommission.gov.uk/about-us/guide/consumer-experiences-and-attitudes-to-free-bets-and-bonuses).","sidebarKey":"the-data-broker-industry","isDrawer":true},{"id":"privacy/privacyTradeoffs#sidebar-dist-to","section":"privacy","sectionTitle":"Privacy","subsection":"privacyTradeoffs","subsectionTitle":"Privacy Trade-Offs","anchor":"dist-to","title":"Distributional trade-offs","content":"In the context of national security, former NSA General Counsel Stewart Baker has proposed that this trade-off serves as the foundation for any law enforcement reform. Baker proposes increasing oversight of government officials by reducing their privacy within their law enforcement functions.[^9] He suggests tracking every database search and monitoring how collected data is used in order to deter abuse by analysts. In other words, privacy losses are imposed on institutional actors to safeguard the privacy of the individuals being surveilled. However, such approaches introduce new concerns about internal monitoring, data retention, who has access to such records, and for what purposes. Any similar approach likely creates new vulnerabilities that must themselves be governed and constrained.","sidebarKey":"dist-to","isDrawer":true},{"id":"privacy/consent#intro","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"intro","title":"Introduction","content":"What is the best way to protect privacy? For many people, protecting privacy in digital contexts begins and ends with consent. While consent can be a powerful tool, it is not a perfect, universally-applicable solution. In this primer, we explore the strengths and weaknesses of a consent-based model along with some alternative approaches to safeguarding privacy.","sidebarKey":null,"isDrawer":false},{"id":"privacy/consent#what-is-consent","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"what-is-consent","title":"What Is Consent?","content":"**Consent** is the process of granting another party special rights and obligations to act in a domain where normally only the **consenter** (i.e., the person giving consent) is allowed to act. In this way, consent transforms the morally unacceptable into the morally acceptable.[^1] For instance, the granting of consent distinguishes a guest from an intruder. As a society, we place importance on consent because it is often “an effective means of respecting individuals as autonomous decision makers with rights of self-determination, including rights to make choices, take or avoid risks, express preferences, and, perhaps most importantly, resist exploitation.”[^2]\n\nWe commonly categorize consent as {explicit-tacit-implicit|explicit, tacit, or implicit} based on the level of autonomy granted to the consenter. **Explicit consent** is expressed through opting in and taking affirmative action (e.g., emphatically saying “yes”) to establish consent. **Tacit consent** is expressed through failing to opt out (e.g., remaining silent when offered the chance to object). **Implicit consent** relies on outside interpretation of whether one’s actions are sufficiently indicative of consent (e.g. going to a haunted house indicates agreement to being scared). The idea is that, “by virtue of entering into a situation,” a person has “agreed to the activities that are broadly known to occur in that context.”[^3]","sidebarKey":null,"isDrawer":false},{"id":"privacy/consent#consent-and-digital-privacy","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"consent-and-digital-privacy","title":"Consent and Digital Privacy","content":"As scholar Meg Leta Jones puts it, “Consent is arguably the central moral and legal tenet to both modern privacy and data protection theory, practice, and culture.”[^4] [Article 4](https://gdpr.eu/article-4-definitions/) of the **General Data Protection Regulation (GDPR)** defines consent in the context of data privacy as a “_freely given, specific, informed and unambiguous_ indication of the data subject's wishes by which he or she, by a statement or by a clear affirmative action, signifies _agreement to the processing of personal data_ relating to him or her” \\[emphasis added\\].\n\nThe interrelationship between consent and privacy was established by {Warren-and-Brandeis|Warren and Brandeis} in their [1890 article](https://doi.org/10.2307/1321160) proposing a legal right to privacy.[^5] To various degrees across time and place, consent has been considered a legal and ethical basis for digital privacy since the advent of personal computing in the mid-twentieth century.\n\nThe US perspective on privacy is particularly deferential to consent, taking a **market-based** **perspective** wherein individuals are treated as **privacy consumers** who are entitled to adequate information in order to make decisions about their privacy. Americans enact consent in large part through the products and services they choose to purchase. This paradigm is referred to as **notice and choice**. A. Michael Froomkin argues that notice and consent “reflects principles of individual autonomy, freedom of choice, and rationality” central to American consumer culture.[^6]\n\nThe US approach stands in contrast to the European Union (EU) system, formalized by the GDPR, which takes **rights-based** **perspective** and treats individuals as {data-subjects|data subjects.} The consent question is separate from the commercial transaction; choosing a certain product or service does not nullify one’s rights to make decisions about their data privacy. Additionally, the EU system is more restrictive in terms of what users can consent to. For instance, under EU law, users cannot consent to a vague statement like “We share your data with our partners” in a privacy policy. On the other hand, under US law, agreeing to such a policy is considered valid consent to any and all third-party data processing.","sidebarKey":null,"isDrawer":false},{"id":"privacy/consent#fries-a-framework-for-thinking-about-meaningful-consent","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"fries-a-framework-for-thinking-about-meaningful-consent","title":"FRIES: A Framework for Thinking About Meaningful Consent","content":"The FRIES acronym was originally created by Planned Parenthood to capture the key elements of consent in sexual interactions. Here, we {adapt-fries|adapt} FRIES into a framework for thinking about what meaningful consent looks like in the context of digital privacy.\n\n**F is for freely given**. Sometimes referred to as voluntariness, the requirement for consent to be freely given concerns the absence of manipulation, coercion, and other forms of external influence on someone’s decisions about their digital privacy. Examples of external influences include misleading interfaces, a lack of options when accessing essential services, and power imbalances between parties. A related concept is {conditionality}, the bundling of privacy permissions with non-negotiable and/or crucial parts of a product or service’s terms.\n\n**R is for reversible**. The reversibility of consent concerns the “right to {limit-access|limit access or entirely remove} your data at any time” without detriment.[^8] Users should be informed of this right _before_ consenting. Regarding exercisability, GDPR Article 7(3) specifies, “It shall be as easy to withdraw as to give consent.” For instance, if agreeing to share certain personal data requires simply checking a box, so should the process of removing access. Reversibility matters because both {users-preferences|users’ preferences} and the {behavior-consent|behavior of those who solicited consent} may change over time (e.g., when terms of service are updated).\n\n**I is for informed**. For consent to be informed, users must be provided _accessible_ information about who can access their data, the purposes for which their data will be processed, and how their data will be handled and stored. This means providing upfront notice in clear language rather than burying details in dense legalese. Human-computer interaction (HCI) researchers Batya Friedman et al. remark that “What is disclosed should address the important values, needs, and interests of the individual, explicitly state the purpose or reason for undertaking the action, and avoid unnecessary technical detail.”[^9]\n\n**E is for engaged**. Engaged digital consent is _active_, not passive, and takes place in a context that allows users to make intentional decisions. As opposed to tacit (opt-out) consent and implicit (inferred) consent, explicit consent requires affirmative user action to communicate choice. For example, {clickwrap-browsewrap-shrinkwrap|clickwrap, browsewrap, and shrinkwrap} are three consent mechanisms commonly used in computing that fall in different places along the implicit-to-explicit spectrum. Appropriate contextual factors for engagement include the user’s cognitive capacity to offer meaningful consent,[^10] minimal distraction during the consent process,[^11] and a manageable number of requests for consent so that each may be considered fully.[^12]\n\n**S is for specific.** Una Lee and Dann Toliver of the [Consentful Tech Project](https://www.consentfultech.io) explain specificity to mean only using “data the user has directly given, not data acquired through other means like scraping or buying” and “only in ways the user has consented to.”[^13] An important concept here is **granularity**, the idea that data subjects should be allowed to consent or not consent to each data processing operation separately. Under the GDPR, the data processor must specify a legitimate purpose for each operation. This acts as safeguard against **function creep**, defined as “the gradual widening or blurring of purposes for which data is processed.”[^14]","sidebarKey":null,"isDrawer":false},{"id":"privacy/consent#consent-in-practice","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"consent-in-practice","title":"Consent in Practice","content":"The design and implementation of digital consent interfaces can either undermine or support meaningful consent. “Design is inherently a persuasive act, where the designer creates intentional change in the world that either directly or indirectly induces behavioral or social change,” write Colin M. Gray and colleagues from Purdue University.[^15] These designs are often discussed through the lens of {behavioral-economics|behavioral economics.}\n\nThe visual presentation of options and default settings selected by the consent requester are proven to change users’ behavior, as {majority-of-users|the majority of users do not change privacy settings} from the default.[^17] It is also possible to influence user choice by making the website’s preferred options stand out and deemphasizing or requiring extra steps to access unpreferred options.[^18] Some scholars and practitioners refer to deceptive design schemes as **dark patterns**. Dark patterns are “instances where designers use their knowledge of human behavior (e.g., psychology) and the desires of end users to implement {deceptive-functionality|deceptive functionality} that is not in the user’s best interest.”[^19]\n\nConversely, design can be employed in ways that promote meaningful consent. Graphic design techniques like salience, framing, and layering and separation combined with logically structured information delivery can greatly increase the accessibility of privacy policies, empowering informed consenters.[^20] A number of {frameworks-practice|frameworks} have been proposed to formalize such ethically-aware design practices.","sidebarKey":null,"isDrawer":false},{"id":"privacy/consent#how-is-consent-upheld","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"how-is-consent-upheld","title":"How Is Consent Upheld?","content":"### By the Free Market\n\nIn the US, consent is upheld largely through **free market competition**, which incentivizes companies to adopt good consent practices in exchange for economic benefit. The theory is that subverting consent exposes companies to potential legal, financial, and reputational damage.[^26] For instance, companies should have an interest in addressing privacy concerns during development rather than risk public backlash to security breaches during a faster rollout.[^27]\n\nAn enormous variety of companies, organizations, and individuals create products and offer services that implicate digital privacy. In the for-profit realm, market participants may use {privacy-selling-point|privacy features as selling points} or offer {privacy-add-on|privacy as an optional, paid add-on.} Other market participants are in the business of privacy itself, for example, providing {personal-data-removal|personal data removal} or {virtual-private-network|virtual private network (VPN)} services. Privacy may be central to the mission of both {for-profit-companies|for-profit companies} and {non-profit-initiatives|non-profit initiatives}, leading these organizations to take a privacy-first approach to development and delivery.\n\n### By Users Themselves\n\nWhen users’ consent is undermined, they may take action by equipping modifications for existing products (e.g., {browser-extensions|browser extensions}) or by educating themselves on {how-to-evade|how to evade insidious tracking techniques}.\n\n### By the Government\n\nWhere market forces and individual action are deemed insufficient to uphold meaningful consent, we rely on **{government-regulation|government regulation}** to do so. Government regulation puts guardrails on consent. It sets limits on what people can consent to, defines legal standards for soliciting and documenting meaningful consent, and {holds-accountable|holds accountable} those who violate consent.\n\nDespite persistent {lobbying} since the 1970s, the US is one of the few democracies in the world that does not have a federal **data protection agency**.[^28] {dpas|Data protection agencies (DPAs)} are generally established as administrative and regulatory authorities that work with the government but have, to varying degrees, some separation from the government. In the absence of such a body, the Federal Trade Commission (FTC) has become the de facto enforcer of US privacy law under its authority to police unfair and deceptive trade practices.\n\nRegulation also has the potential to influence how we think about and practice digital consent beyond its jurisdictional scope. Legal and political scholars use the terms {brussels-effect|‘Brussels effect’} and {california-effect|‘California effect’} to refer to the way EU and California law, respectively, increase regulatory standards by making compliance with more stringent requirements the easiest path forward for companies that operate in multiple jurisdictions.[^29]\n\n### By Watchdogs\n\n**Watchdogs** keep tabs on companies and regulators, holding them to account by exposing wrongdoing and compelling transparency. {ngos|Non-governmental organizations (NGOs)}, {intergovernmental-organizations|intergovernmental organizations}, {media-outlets|media outlets}, and {independent-journalists|independent journalists} can all act as watchdogs. Their work helps people make informed decisions about the companies with which they choose to do business, ensuring free market competition respects individual privacy. Watchdogs also attempt to expose flaws in regulatory systems and may advocate for legislative or judicial solutions.","sidebarKey":null,"isDrawer":false},{"id":"privacy/consent#criticisms-of-the-consent-model-for-digital-privacy","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"criticisms-of-the-consent-model-for-digital-privacy","title":"Criticisms of the Consent Model for Digital Privacy","content":"Criticisms of the consent model for digital privacy take two general forms. Some scholars disagree with the idea that consent should be the main tool we reach for to protect privacy, while others raise practical concerns about implementation.\n\nArguing against consent as the primary mechanism for privacy protection, Daniel J. Solove asserts that consent “attempts to be neutral about substance,” meaning it is agnostic about what the consenter is agreeing to. This has the effect of legitimizing nearly any type of data collection and use.[^31] Additional legal scholars, including Ari Ezra Waldman, argue that framing consent strictly in terms of autonomy and self-determination puts the onus on the individual to manage their privacy and shifts the blame for any harm they suffer from the consent requester to the consent granter.[^32] The key ethical (and legal) question becomes ‘Did they consent to what was done?’ instead of ‘Was there a violation of privacy?’. As long as organizations can maintain the appearance of obtaining meaningful consent, they are legally free to handle data as they wish.\n\nTaking a step back from these broad critiques, we can identify a number of flaws in current implementations of consent.\n\n### Informed Consent vs. Time\n\nMany scholars believe that ‘informed consent’ is _not_ an achievable standard around which to build regulation. The concern, especially in the era of **Big Data**, is that the consent model places an excessive burden on users, giving them “the responsibility to decide on complex issues they lack the time, or capacity, to fully comprehend and assess.”[^33] Thoroughly evaluating every consent request one encounters would require an unrealistic amount of both time and techno-legal expertise. Research suggests we would have to dedicate a month of work to reading all the privacy policies we are presented with each year.[^34] The average privacy policy has not only quadrupled in length but greatly increased in complexity since the mid-1990s.[^35]\n\nRepeated, detailed requests for consent cause many people to experience **consent fatigue** (also called consent desensitization), a condition characterized by decreased “motivation and depth of processing,”[^36] which has the ultimate effect of precluding the user’s full engagement.[^37] These factors underlie what communications researchers Jonathan A. Obar and Anne Oeldorf-Hirsch call “**the biggest lie of the internet**”—checking ‘I agree to these terms and conditions’ when, in fact, {we-have-not-even|we have not even read them}.[^38]\n\n### Informed Consent vs. Aggregation\n\nAnother challenge to informed consent is what scholars refer to as the **aggregation effect**. The consent model for digital privacy asks users to continually make decisions about what usually amounts to relatively small pieces of data. Viewed individually, these data points may not reveal much about a person. But as in a pointillist painting, many small points can, when viewed en masse, form a revealing portrait.[^40] It is particularly challenging to weigh the consequences of the aggregation effect when we do not know who has what data on us, potential {harms-are-distant|harms are distant}, or the circumstances from which negative effects might arise {cannot-be-anticipated|cannot be anticipated}.\n\n![Close-up of a male character in Seurat's painting showing the small, distinct dots of contrasting color characteristic of pointillism.](/assets/primer-photos/privacy/consent/parade_de_cirque.png)\n\n**Figure:** Detail from _Parade de cirque_, an 1889 painting by the inventor of pointillism, Georges Seurat.\n\n### Accountability Infrastructure\n\nA further critique is that we lack sufficient **accountability infrastructure** to make our best approximations of informed consent meaningful. For consent to be meaningful, we must trust that there are systems in place to enforce the terms to which we have agreed. In the biomedical context, professional codes of ethics, laws and regulations, oversight bodies (e.g., Institutional Review Boards), medical licensing boards, and avenues for malpractice litigation serve this purpose.[^43] Similar accountability mechanisms are not typically available, at least to most Americans, in the realm of digital privacy and online tracking. What’s more, the harms of privacy violations are difficult to quantify and practically impossible to rectify. Fines and awards of damages have historically {done-little-harm|done little harm to Big Tech’s bottom line} and may be viewed as ‘the cost of doing business’ rather than serious deterrents to unethical behavior.[^44] In light of these facts, some critics refer to our current system as **{consent-theater|consent theater}**, since we enact the process of consent without truly empowering the consenter. Such disillusionment contributes to **{privacy-nihilism|privacy nihilism}** as people feel increasingly {powerless-to-control|powerless to control their data}.\n\n### Informed Consent vs. The Illusion of Choice\n\nAdditionally, the imbalance in **bargaining power** between users and institutions challenges the requirement that consent be **freely given**. “\\[T\\]o “choose” not to use the Internet is in a very real sense to “choose” not to participate in modern society or the modern economy,” contend legal scholars Neil Richards and Woodrow Hartzog.[^45] This extends to other forms of data processing—for example, {signing-away|signing away privacy rights in exchange for welfare} or {submitting-workplace|submitting to workplace surveillance for fear of losing one’s job}. When it comes to products, terms and policies tend to converge within market segments, becoming “so uniform and standardized that the consumer effectively has no choice.”[^46]\n\n### Who Consents?\n\nEven more concerningly, some individuals affected by the data collection, storage, and processing concerned are not asked for their consent at all. For example, tenants may be living under the gaze of {smart-home-devices|smart home devices} for which their landlord configured the privacy settings.[^48] In our increasingly connected world, **consent by association** with network members is hotly debated. For instance, consent by association is highly relevant in discussions of privacy in {genetic-testing|genetic testing}, {social-media-ecosystems|social media ecosystems}, and {human-subjects|human subjects research}. In some cases, people {may-not-be-aware|may not be aware} data is being collected about them. Finally, individuals (e.g., {children}) may be asked to consent when they do not have the capacity to do so.","sidebarKey":null,"isDrawer":false},{"id":"privacy/consent#strengthening-consent","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"strengthening-consent","title":"Strengthening Consent","content":"Scholars and practitioners have suggested ways to improve digital consent as a privacy-protecting tool. Some of their ideas have been tested and implemented, while others remain theoretical.\n\nOn the consenter’s side, auditing of companies’ policies and behavior helps users make informed consent decisions. Some auditing projects rely on {human-expertise|human expertise}, while others employ {artificial-intelligence|artificial intelligence (AI)} to automate review. Experts are also experimenting with AI to {automate-aspects|automate aspects of the consent process} by predicting user preferences and selecting privacy settings appropriately. The idea is to offload the straight-forward decisions to an AI-based tool, keeping the user in the loop as necessary.[^57] Even if such a tool functioned perfectly, however, automation in this sphere raises ethical and legal questions about the validity and meaningfulness of consent given by a non-human proxy.[^58]\n\nSeveral key proposals for reducing the burden of consent management come from the realm of human subjects research. Tiered consent, meta-consent, and dynamic consent have been proposed specifically to handle the issue of {secondary-data-use|secondary data use}.\n\n**Tiered consent** provides potential participants with more granular options and aims to facilitate informed decision-making by {ordering-those-options|ordering those options by risk and sensitivity}.[^61] Two considerations for tiered consent are:\n\n1. That there is no objective, universally applicable way to do this categorization and\n2. That presenting many options is counterproductive to meaningful consent because it can cause information overload.[^62]\n\n**Meta-consent** attempts to support engagement and autonomy by asking research participants during the consent process for the initial use of their data whether or not they want to receive {consent-requests|consent requests for future uses}. Individuals can pick a format, such as blanket consent (or non-consent), broad consent, tiered consent, or case-by-case consent, that aligns with their values, risk tolerance, and logistic preferences.[^63]\n\n**{dynamic-consent|Dynamic consent}** utilizes web-based platforms to empower research subjects to update their meta-consent preferences, respond to real-time data use requests from studies, and {communicate-investigators|communicate with investigators in real time}. Dynamic consent platforms may include an educational component and {assess-participant|assess participant comprehension} during the initial consent process. The most common criticism of dynamic consent platforms is that they are costly and labor-intensive to maintain.\n\nExperts have also put forth more radical ideas on restructuring the practice of digital consent.\n\nComplex systems researcher Juniper Lovato and colleagues suggest empowering users of social networks to make their consent conditional on the consent of others, in effect forming a community with commonly held privacy standards. This model, which they call **{distributed-consent|distributed consent}**, attempts to address the issue of consent by association through a sort of ‘**herd privacy**’ analogous to herd immunity. The researchers further recommend the creation of **consent passports** with predetermined privacy standards that users can easily compare to the Terms of Service of a platform they are considering joining.\n\nThe idea of **{consent-receipts|consent receipts}** has gained traction in recent years as a way of recording the terms agreed upon and requesting changes to or deletion of one’s data. Like receipts in commercial contexts, each party receives a copy that can be used as proof of the transaction. Consent receipts can improve data minimization by serving such authentication purposes, and, “as a systematic and machine-readable mechanism,” they support automation and make it easier for data processors to fulfill the rights of data contributors.[^65]\n\nAdditionally, **encryption** sets up a system in which authorization is needed to access, or **decrypt**, personal data. That authorization may be managed directly between agreed parties or involve a trusted third party. Examples of encryption protocols that help enforce consent include **sticky policies**, where decryption is bound to specific conditions specified in the privacy policy, and **distributed access control enforcement**, where decryption keys are granted on-demand through a decentralized validation system.[^66]\n\nFinally, **{binary-governance|binary governance}** involves the pairing of data subject rights and data processor obligations within a framework that views privacy as a civic good with collective value. The idea of binary governance is that comprehensive legislation should include not only **negative obligations** (requirements to avoid wrongdoing, e.g., violation of individual rights) but **affirmative obligations** (requirements to take positive action). Examples of affirmative obligations imposed on companies by the GDPR and California Privacy Rights Act (CPRA) include regular {risk-assessments|risk assessments}, data minimization, design specifications for consent interfaces, and storage of consent records.[^67] Another proposal is establishing a **{fiduciary-duty|fiduciary duty}** for data holders that would legally obligate them to act in the best interests of data subjects just as financial fiduciaries are required to act in the best interest of their clients.\n\n### Alternatives to a Consent-Based Approach to Privacy\n\nIn its ideal form, meaningful digital consent empowers people to make decisions about their data with the ultimate goal of safeguarding privacy in a manner compatible with their values. However, consent is not the only way to accomplish this end, and, in certain circumstances, alternatives may work better in light of logistical, legal, or ethical considerations.\n\nOne way of accomplishing the end of privacy is through **anonymization**, which, as privacy scholars Solon Barocas and Helen Nissenbaum write, “seems to take data outside the scope of privacy, as it no longer maps onto identifiable subjects.”[^68] While the word ‘anonymous’ literally means nameless, Nissenbaum defines anonymity here as the stricter condition of **being unreachable**.[^69] Anonymization is a great option in theory, but often fails in practice due to data breaches and ever-advancing re-identification techniques.\n\n**Statistical disclosure limitation (SDL) techniques** help ensure privacy in data analysis and sidestep the issue of soliciting consent for each secondary use case. SDL techniques modify or restrict data in ways that prevent individuals from being identified {while-analysis|while still allowing for useful statistical analysis}.\n\n- **{Coarsening}** involves aggregating data into broader categories to make individual records harder to pick out.\n- **{data-swapping|Data swapping}** entails switching values between selected pairs of records to introduce inaccuracies that make identification more difficult without warping the high-level findings of interest.\n- Approaches using **{synthetic-data|synthetic data}** replace sensitive values with simulated records drawn from probability distributions that preserve important statistical relationships present in the original data. Multiple versions of the dataset, which may be partially or fully synthetic, are made available to analysts so uncertainty can be accounted for appropriately.[^70]\n- Finally, **noise infusion** involves injecting random statistical noise into the underlying data or a specific query to obscure individual data points. **Differential privacy (DP)** is a widely used mathematical framework based on calibrated noise infusion.\n\nA **paternalistic** **approach** to privacy places some or all of the decision-making power granted to individuals under a consent-based model in the hands of a governing body tasked with safeguarding individuals from harm. Paternalism is predicated upon the belief that some data collection, ownership, and use practices are morally and ethically wrong regardless of whether a subject consents. Proponents of paternalism argue that, just as we rely on agencies like the National Highway Traffic Safety Administration (NHTSA) and the Food and Drug Administration (FDA) to ensure reasonable safety standards for the cars we drive and products we consume, it makes sense to rely on an expert group to ensure reasonable safety standards for privacy.[^72] “Failing to limit who can ask for informed consent, when they can ask, and how many times ignores the reality that people need time and space if their choices are to be meaningful,” write Richards and Hartzog.[^73]\n\n**Libertarian paternalism** takes a less restrictive approach by nudging users towards more privacy-preserving options without eliminating less privacy-preserving ones. Such applications of {choice-architecture|choice architecture} are sometimes called **light patterns**. Examples of light patterns include making the default option for log-in creation password generation, color coding settings based on level of privacy protection, reminders about privacy implications when entering sensitive information, and {incorporating-personalized|incorporating personalized scenarios into permissions dialogues}.[^74]\n\nPaternalism hinges on who gets to make the rules and using what standards. This raises obvious issues for implementation. Reasonable, well-intentioned people constantly disagree about how to balance {tradeoffs} between privacy and other goods like security, social cohesion, {sci-and-med|scientific and medical progress}, and so on. And if governing entities look to industry norms to set standards of reasonableness, paternalistic systems are likely to further entrench the prioritization of profit over individual rights.[^75]\n\nThe theory of **{contextual-integrity|contextual integrity}** put forth by privacy scholar Helen Nissenbaum understands consent as one possible aspect of the appropriate flow of information, defined according to the subject, sender, recipient, information type, and **transmission principles** (of which ‘with consent’ and ‘with notice’ are options).[^78] An appealing feature of contextual integrity as a privacy protection framework is its ability to reflect evolving ethical norms and societal interests.\n\n<!-- Note: duplicates have not been removed and these are not in alphabetical order. For this case, that represents\na substantial amount of work, so we're skipping it for right now-->\n\n[^1]: A. John Simmons, “Tacit Consent and Political Obligation,” _Philosophy & Public Affairs_ 5, no. 3 (1976): 274–91.\n\n[^2]: Solon Barocas and Helen Nissenbaum, “Big Data’s End Run around Anonymity and Consent,” in _Privacy, Big Data, and the Public Good: Frameworks for Engagement_, ed. Julia Lane et al. (Cambridge University Press, 2014), 56–57.\n\n[^3]: Batya Friedman et al., “Informed Consent by Design,” in _Security and Usability_ (O’Reilly Media, Inc., 2005), 503, https://old.vsdesign.org/publications/pdf/Security\\_and\\_Usability\\_ch24.pdf.\n\n[^4]: “The Development of Consent to Computing,” _IEEE Annals of the History of Computing_ 41, no. 4 (2019): 37, https://doi.org/10.1109/MAHC.2019.2896282.\n\n[^5]: Samuel D. Warren and Louis D. Brandeis, “The Right to Privacy,” _Harvard Law Review_ 4, no. 5 (1890): 193–220, https://doi.org/10.2307/1321160.\n\n[^6]: “Big Data: Destroyer of Informed Consent,” _Yale Journal of Law & Technology_ 21 (2019): 30\\.\n\n[^7]: Meg Leta Jones, _The Character of Consent: The History of Cookies and the Future of Technology Policy_, Information Policy Series (MIT Press, 2024), 71\\.\n\n[^8]: Una Lee and Dann Toliver, “Building Consentful Tech,” And Also Too, 2017, https://www.andalsotoo.net/wp-content/uploads/2018/10/Building-Consentful-Tech-Zine-SPREADS.pdf.\n\n[^9]: “Informed Consent by Design,” 499\\.\n\n[^10]: Neil Richards and Woodrow Hartzog, “The Pathologies of Digital Consent,” Washington University Law Review, Trust and Privacy in the Digital Age, vol. 96, no. 6 (2019), https://openscholarship.wustl.edu/law\\_lawreview/vol96/iss6/11.\n\n[^11]: Friedman et al., “Informed Consent by Design.”\n\n[^12]: Daniel J. Solove, “Privacy Self-Management and the Consent Dilemma,” _Harvard Law Review_ 126 (2013): 1880–903.\n\n[^13]: “Building Consentful Tech.”\n\n[^14]: Article 29 Working Party (WP29), “Guidelines on Consent under Regulation 2016/679,” April 10, 2018, 12\\.\n\n[^15]: Colin M. Gray et al., “The Dark (Patterns) Side of UX Design,” _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_, April 21, 2018, 2, https://doi.org/10.1145/3173574.3174108.\n\n[^16]: Max Witynski, “Behavioral Economics, Explained,” _UChicago News_, n.d., accessed January 1, 2026, https://news.uchicago.edu/explainer/what-is-behavioral-economics.\n\n[^17]: Friedman et al., “Informed Consent by Design.”\n\n[^18]: Midas Nouwens et al., “Dark Patterns after the GDPR: Scraping Consent Pop-Ups and Demonstrating Their Influence,” _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_, April 21, 2020, 1–13, https://doi.org/10.1145/3313831.3376321.\n\n[^19]: Gray et al., “The Dark (Patterns) Side of UX Design.”\n\n[^20]: Ramona Schmidt et al., “Challenges and Solutions in Implementing Informed Consent in Digital Environments: A Scoping Review,” _IEEE Access_ 13 (2025): 71974, https://doi.org/10.1109/ACCESS.2025.3562773.\n\n[^21]: Batya Friedman and David G. Hendry, _Value Sensitive Design: Shaping Technology with Moral Imagination_ (The MIT Press, 2019), https://doi.org/10.7551/mitpress/7585.001.0001.\n\n[^22]: Mary Flanagan and Helen Nissenbaum, _Values at Play in Digital Games_ (The MIT Press, 2014), https://doi.org/10.7551/mitpress/9016.001.0001.\n\n[^23]: Katie Shilton, “Values Levers: Building Ethics into Design,” _Science, Technology, & Human Values_ 38, no. 3 (2013): 374–97, https://doi.org/10.1177/0162243912436985.\n\n[^24]: Jeffrey Bardzell and Shaowen Bardzell, “What Is ‘Critical’ about Critical Design?,” _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_, April 27, 2013, 3297–306, https://doi.org/10.1145/2470654.2466451.\n\n[^25]: Phoebe Sengers et al., “Reflective Design,” _Proceedings of the 4th Decennial Conference on Critical Computing: Between Sense and Sensibility_, August 20, 2005, 49–58, https://doi.org/10.1145/1094562.1094569.\n\n[^26]: Friedman et al., “Informed Consent by Design.”\n\n[^27]: Data & Society Research Institute, “Data & Society Research Institute Comments to NTIA on ‘Stakeholder Engagement on Cybersecurity in the Digital Ecosystem,’” Data & Society, May 27, 2015, https://datasociety.net/pubs/dcr/Data\\&Society\\_NTIA-comments\\_May2015.pdf.\n\n[^28]: Electronic Privacy Information Center, “Enforcement of Privacy Laws,” Epic.Org, 2025, https://epic.org/issues/data-protection/enforcement-of-privacy-laws/.\n\n[^29]: Wikipedia contributors, “Brussels Effect,” in _Wikipedia_ (Wikipedia, The Free Encyclopedia, November 24, 2025), https://en.wikipedia.org/wiki/Brussels\\_effect.\n\n[^30]: Jones, _The Character of Consent_.\n\n[^31]: “Privacy Self-Management and the Consent Dilemma,” 1880\\.\n\n[^32]: Ari Ezra Waldman, “Privacy, Practice, and Performance,” _California Law Review_ 110 (2022), https://doi.org/10.15779/Z38JD4PQ3D.\n\n[^33]: Fida K. Dankar et al., “Dynamic-Informed Consent: A Potential Solution for Ethical Dilemmas in Population Sequencing Initiatives,” Computational and Structural Biotechnology Journal 18 (2020): 919, https://doi.org/10.1016/j.csbj.2020.03.027.\n\n[^34]: Jones, The Character of Consent, 3\\.\n\n[^35]: Isabel Wagner, “Privacy Policies across the Ages: Content of Privacy Policies 1996–2021,” _ACM Transactions on Privacy and Security_ 26, no. 3 (2023): 1–32, https://doi.org/10.1145/3590152.\n\n[^36]: Caspar Barnes et al., “Enabling Demonstrated Consent for Biobanking with Blockchain and Generative AI,” _The American Journal of Bioethics_ 25, no. 4 (2025): 96–111, https://doi.org/10.1080/15265161.2024.2416117.\n\n[^37]: Woodrow Hartzog, “The Case Against Idealising Control,” _European Data Protection Law Review_ 4 (2018), https://scholarship.law.bu.edu/cgi/viewcontent.cgi?article=4050\\&context=faculty\\_scholarship.\n\n[^38]: Jonathan A. Obar and Anne Oeldorf-Hirsch, “The Biggest Lie on the Internet: Ignoring the Privacy Policies and Terms of Service Policies of Social Networking Services,” _Information, Communication & Society_ 23, no. 1 (2018): 128–47, https://doi.org/10.1080/1369118X.2018.1486870.\n\n[^39]: Paul Bernal, “Collaborative Consent: Harnessing the Strengths of the Internet for Consent in the Online Environment,” _International Review of Law, Computers & Technology_ 24, no. 3 (2010): 287–97, https://doi.org/10.1080/13600869.2010.522335.\n\n[^40]: Daniel J. Solove, “Digital Dossiers and the Aggregation Effect,” _TeachPrivacy_, November 25, 2024, https://teachprivacy.com/digital-dossiers-and-the-aggregation-effect/.\n\n[^41]: Froomkin, “Big Data: Destroyer of Informed Consent.”\n\n[^42]: Froomkin, “Big Data: Destroyer of Informed Consent.”\n\n[^43]: Solon Barocas and Helen Nissenbaum, “On Notice: The Trouble with Notice and Consent,” _Proceedings of the Engaging Data Forum: The First International Forum on the Application and Management of Personal Electronic Information_, October 2009, 6, https://ssrn.com/abstract=2567409.\n\n[^44]: Rene Millman, “Big Tech Needs Less than a Month to Pay off over $7 Billion in 2025 Fines, Proton Warns,” _TechRadar_, January 28, 2026, https://www.techradar.com/vpn/vpn-privacy-security/big-tech-could-need-only-one-month-to-pay-off-over-usd7-billion-in-2025-fines-proton-warns.\n\n[^45]: Richards and Hartzog, “The Pathologies of Digital Consent,” 1487\\.\n\n[^46]: Nancy Kim, “Clicking and Cringing,” Oregon Law Review 86 (2007): 821, https://scholarlycommons.law.cwsl.edu/cgi/viewcontent.cgi?article=1054\\&context=fs.\n\n[^47]: Aiha Nguyen, _The Constant Boss: Work Under Digital Surveillance_ (Data & Society, 2021), https://datasociety.net/wp-content/uploads/2021/05/The\\_Constant\\_Boss.pdf.\n\n[^48]: Data & Society Research Institute, “Data & Society Research Institute Comments to NTIA on ‘Stakeholder Engagement on Cybersecurity in the Digital Ecosystem.’”\n\n[^49]: Juniper L. Lovato et al., “Limits of Individual Consent and Models of Distributed Consent in Online Social Networks,” _2022 ACM Conference on Fairness Accountability and Transparency_, June 21, 2022, 2252, https://doi.org/10.1145/3531146.3534640.\n\n[^50]: Alan Mislove et al., “You Are Who You Know: Inferring User Profiles in Online Social Networks,” Proceedings of the Third ACM International Conference on Web Search and Data Mining, ACM, February 4, 2010, 251–60, https://doi.org/10.1145/1718487.1718519.\n\n[^51]: James P. Bagrow et al., “Information Flow Reveals Prediction Limits in Online Social Activity,” Nature Human Behaviour 3, no. 2 (2019): 122–28, https://doi.org/10.1038/s41562-018-0510-5.\n\n[^52]: Jones, The Character of Consent, 180\\.\n\n[^53]: Froomkin, “Big Data: Destroyer of Informed Consent,” 33\\.\n\n[^54]: Barocas and Nissenbaum, “Big Data’s End Run around Anonymity and Consent,” 61\\.\n\n[^55]: Luger and Rodden, “Terms of Agreement,” 230\\.\n\n[^56]: Richards and Hartzog, “The Pathologies of Digital Consent,” 1490\\.\n\n[^57]: Meg Leta Jones et al., “AI and the Ethics of Automating Consent,” _IEEE Security & Privacy_ 16, no. 3 (2018): 64–72, https://doi.org/10.1109/MSP.2018.2701155.\n\n[^58]: Jones et al., “AI and the Ethics of Automating Consent.”\n\n[^59]: Jones et al., “AI and the Ethics of Automating Consent,” 69\\.\n\n[^60]: Dankar et al., “Dynamic-Informed Consent.”\n\n[^61]: Barnes et al., “Enabling Demonstrated Consent for Biobanking with Blockchain and Generative AI,” 97\\.\n\n[^62]: Natalie Ram, “Tiered Consent And The Tyranny Of Choice,” _Jurimetrics_, Spring 2008, http://scholarworks.law.ubalt.edu/all\\_fac/873.\n\n[^63]: Thomas Ploug and Søren Holm, “Meta Consent – A Flexible Solution to the Problem of Secondary Use of Health Data,” _Bioethics_ 30, no. 9 (2016): 721–32, https://doi.org/10.1111/bioe.12286.\n\n[^64]: Bernal, “Collaborative Consent.”\n\n[^65]: Schmidt et al., “Challenges and Solutions in Implementing Informed Consent in Digital Environments,” 71977\\.\n\n[^66]: Schmidt et al., “Challenges and Solutions in Implementing Informed Consent in Digital Environments.”\n\n[^67]: Jones, _The Character of Consent_.\n\n[^68]: Barocas and Nissenbaum, “Big Data’s End Run around Anonymity and Consent,” 45\\.\n\n[^69]: Helen Nissenbaum, “The Meaning of Anonymity in an Information Age,” _The Information Society_ 15, no. 2 (1999): 141–44, https://doi.org/10.1080/019722499128592.\n\n[^70]: Alan F. Karr and Jerome P. Reiter, “Using Statistics to Protect Privacy,” in _Privacy, Big Data, and the Public Good: Frameworks for Engagement_, ed. Julia Lane et al. (Cambridge University Press, 2014).\n\n[^71]: Karr and Reiter, “Using Statistics to Protect Privacy,” 278–79.\n\n[^72]: Solove, “Privacy Self-Management and the Consent Dilemma,” 1901\\.\n\n[^73]: Richards and Hartzog, “The Pathologies of Digital Consent,” 1494\\.\n\n[^74]: Ana Caraban et al., “23 Ways to Nudge: A Review of Technology-Mediated Nudging in Human-Computer Interaction,” _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_, May 2, 2019, 1–15, https://doi.org/10.1145/3290605.3300733.\n\n[^75]: Kim, “Clicking and Cringing.”\n\n[^76]: Dankar et al., “Dynamic-Informed Consent.”\n\n[^77]: Froomkin, “Big Data: Destroyer of Informed Consent.”\n\n[^78]: Scott Berinato, “‘Stop Thinking About Consent: It Isn’t Possible and It Isn’t Right,’” _Harvard Business Review_, The Big Idea Series  /  Tracked, September 24, 2018, https://www5.qa.hbr.org/2018/09/stop-thinking-about-consent-it-isnt-possible-and-it-isnt-right.","sidebarKey":null,"isDrawer":false},{"id":"privacy/consent#sidebar-explicit-tacit-implicit","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"explicit-tacit-implicit","title":"Explicit, tacit, or implicit","content":"To illustrate, consider what explicit, tacit, and implicit consent might look like when interacting with cookies:\n\n- When a user opens webpage A, a banner appears asking if they want to opt in to third-party cookies. By selecting the ‘yes’ option they are explicitly consenting to such tracking.\n\n- When a user opens a webpage B, a banner appears asking if they want to opt out of third-party cookies. By not unchecking the pre-selection ‘yes’ option, they are tacitly consenting to such tracking.\n\n- When a user opens webpage C, no banner appears. The footer at the bottom of the page says that by browsing the website they are agreeing to the terms available at the provided link. Buried in these terms is a provision stating that third-party cookies are employed. By continuing to use the website, whether or not the user has read the terms, they are tacitly consenting to such tracking.\n\nFor more information about what cookies are and how they work, check out the article “[Understanding Cookies in Web Browsers](https://www.geeksforgeeks.org/websites-apps/understanding-cookies-in-web-browsers/).”\n\nTo learn more about opt-in and opt-out consent in the context of digital privacy, see the article “[What is Opt-Out and Opt-In Consent?](https://www.datagrail.io/blog/data-privacy/opt-out-and-opt-in-consent-explained/)” by DataGrail.","sidebarKey":"explicit-tacit-implicit","isDrawer":true},{"id":"privacy/consent#sidebar-warren-and-brandeis","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"warren-and-brandeis","title":"Warren and Brandeis","content":"See [What is Privacy?](/privacy/whatIsPrivacy) for a more detailed discussion of Warren and Brandeis’ contributions to our thinking about privacy.","sidebarKey":"warren-and-brandeis","isDrawer":true},{"id":"privacy/consent#sidebar-data-subjects","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"data-subjects","title":"Data Subjects","content":"Rather than adopting a concept like the data subject that transcends legal domains, the US protects data privacy through a patchwork of laws that address specific situations. For example, the Fair Credit Reporting Act (FCRA) concerns consumers, the Family Educational Rights and Privacy Act (FERPA) concerns students, and the Health Insurance Portability and Accountability Act (HIPAA) concerns patients.[^7]","sidebarKey":"data-subjects","isDrawer":true},{"id":"privacy/consent#sidebar-adapt-fries","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"adapt-fries","title":"Adapt","content":"FRIES The Consentful Tech Project, an initiative of Allied Media Projects, also applies FRIES to digital consent. You can check out their version at [ConsentfulTech.io](https://www.consentfultech.io/).","sidebarKey":"adapt-fries","isDrawer":true},{"id":"privacy/consent#sidebar-conditionality","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"conditionality","title":"Conditionality","content":"Conditionality undermines one’s ability to freely give consent because one must agree to all the terms or else not use the product or service, which is unrealistic in the absence of true alternatives.\n\nFor example, university students access all sorts of essentials—like housing, dining, and healthcare—via their ID cards. Receiving an ID card and unlocking this access is conditional upon agreement to certain data processing and sharing by the university. Consent is take-it-or-leave-it as there is no option to enroll as a student without assenting to the university’s terms. Since most institutions have similar terms, the option to attend a different school does not provide a student with meaningful freedom of choice.","sidebarKey":"conditionality","isDrawer":true},{"id":"privacy/consent#sidebar-limit-access","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"limit-access","title":"Limit access or entirely remove","content":"There is no universally-accepted definition of reversing consent. Different jurisdictions, industries, and data processors take different stances on what should happen when a user withdraws consent. An example of a lower standard is not using research participants’ data for future studies, while an example of a higher standard is the right to erasure (also called the ‘right to be forgotten’) encoded in [GDPR Article 17](https://gdpr.eu/article-17-right-to-be-forgotten/).","sidebarKey":"limit-access","isDrawer":true},{"id":"privacy/consent#sidebar-users-preferences","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"users-preferences","title":"Users' preferences","content":"One recent trend in users’ preferences is growing opposition to ‘mommy bloggers,’ family vloggers, and other parent influencers who publicly share details of their children’s lives. As explored in the article “[Influencer Parents and The Kids Who Had Their Childhood Made Into Content](https://www.teenvogue.com/story/influencer-parents-children-social-media-impact),” some content creators are changing course and not posting their kids anymore after learning, sometimes firsthand, about [the risks ‘sharenting’ poses](https://www.newyorker.com/culture/cultural-comment/instagram-facebook-and-the-perils-of-sharenting). For further reading, check out this [blog post](https://www.laurenbeechingpr.com/blog/is-family-vlogging-on-its-way-out-the-ethical-dilemma-of-monetising-childhood) examining consent and the ethics of family vlogging.\n\nAnother example of user preferences shifting due to privacy and safety concerns is a transgender person wanting to erase their pre-transition digital footprint to avoid being outed or discriminated against. The articles “[Taking Control of Your Digital Identity](https://translifeline.org/taking-control-of-your-digital-identity/)” by Trans Lifeline and “[Standing Up for LGBTQ+ Digital Safety this International Day Against Homophobia](https://www.eff.org/uk/deeplinks/2025/05/standing-lgbtq-digital-safety-international-day-against-homophobia?language=uk)” from the Electronic Frontier Foundation (EFF) offer digital harm prevention tips and resources.","sidebarKey":"users-preferences","isDrawer":true},{"id":"privacy/consent#sidebar-behavior-consent","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"behavior-consent","title":"Behavior of those who solicited consent","content":"Mozilla and Zoom are two data processors that recently received public backlash for changing their products’ Terms of Service. Both the [Mozilla Firefox](https://consumerrights.wiki/w/Firefox_introduces_TOS) and [Zoom](https://themarkup.org/hello-world/2023/08/12/this-is-what-happens-when-people-start-actually-reading-privacy-policies) examples concern policy updates that left the door open for training artificial intelligence (AI) on user data.","sidebarKey":"behavior-consent","isDrawer":true},{"id":"privacy/consent#sidebar-clickwrap-browsewrap-shrinkwrap","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"clickwrap-browsewrap-shrinkwrap","title":"Clickwrap, browsewrap, and shrinkwrap","content":"**Clickwrap** requires a user to click a button, check box, or link to agree to a website or software’s terms of service. Those terms may or may not be displayed in full for the user to peruse before consenting and may be structured as opt-in or opt-out.\n\n**Browsewrap** is a tacit consent mechanism whereby users are presumed to agree to a website’s terms of service simply by browsing the site.\n\n**Shrinkwrap**, like browsewrap, is a tacit consent mechanism but one that applies to physical products (e.g., data or software on discs). A shrinkwrap agreement considers the opening of the package consent to the terms of service attached, even if those terms are not accessible without opening the package. See the article “[Clickwrap vs Browsewrap vs Shrinkwrap](https://privacyterms.io/terms/clickwrap-vs-shrinkwrap-vs-browsewrap/)” for an in-depth comparison.","sidebarKey":"clickwrap-browsewrap-shrinkwrap","isDrawer":true},{"id":"privacy/consent#sidebar-behavioral-economics","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"behavioral-economics","title":"Behavioral economics","content":"Behavioral economics combines psychology and economics to try to understand why people sometimes act in ways neoclassical economics would consider irrational.[^16] Richard Thaler and Cass Sunstein brought mainstream attention to behavioral economics with their 2008 book [_Nudge: Improving Decisions about Health, Wealth, and Happiness_](https://psycnet.apa.org/record/2008-03730-000). Nudge theory posits that the decision environment, or **choice architecture**, can be intentionally designed to promote certain behaviors. Thaler and Sunstein define a **nudge** as “any aspect of the choice architecture that alters people’s behavior in a predictable way without forbidding any option or significantly changing their economic incentive.”","sidebarKey":"behavioral-economics","isDrawer":true},{"id":"privacy/consent#sidebar-majority-of-users","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"majority-of-users","title":"the majority of users do not change privacy settings","content":"For instance, a [2019 study](https://doi.org/10.1145/3319535.3354212) on cookies found that most webpage visitors will accept a privacy-invasive default allowing cookies to be set for all purposes (i.e., will not opt out) while less than 0.1% of visitors will opt in to those settings when given the choice.","sidebarKey":"majority-of-users","isDrawer":true},{"id":"privacy/consent#sidebar-deceptive-functionality","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"deceptive-functionality","title":"Deceptive functionality","content":"Dark pattern strategies as defined by [Gray et al. (2018)](https://doi.org/10.1145/3173574.3174108):\n\n- **Nagging** – “a minor redirection of expected functionality that may persist over one or more interactions” (e.g., pop-ups with options to either postpone or accept but not to permanently decline)\n\n- **Obstruction** – “impeding a task flow, making an interaction more difficult than it inherently needs to be with the intent to dissuade an action” (e.g., preventing users from easily comparing prices by not allowing website text to by copied and pasted)\n\n- **Sneaking** – “an attempt to hide, disguise, or delay the divulging of information that has relevance to the user” (e.g., requiring consent to a privacy statement that permits the sale of the users data in order to unsubscribe from an email newsletter)\n\n- **Interface interference** – “any manipulation of the user interface that privileges specific actions over others, thereby confusing the user or limiting discoverability of important action possibilities” (e.g., hiding information in the fine print; preselecting options)\n\n- **Forced action** – “any situation in which users are required to perform a specific action to access (or continue to access) specific functionality” (e.g., making levels of a mobile game impossible without purchasing power ups)","sidebarKey":"deceptive-functionality","isDrawer":true},{"id":"privacy/consent#sidebar-frameworks-practice","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"frameworks-practice","title":"Frameworks","content":"Ethically aware design approaches:\n\n- **Value sensitive design** – a methodological framework for incorporating “both moral and technical imagination” throughout the stages of design[^21]\n\n- **Values at play** – “a theoretical and practical guide to integrating human values into the conception and design of digital games”[^22]\n\n- **Value levers** – “practices that open new conversations about social values and encourage consensus around those values as design criteria”[^23]\n\n- **Critical design** – “a research through design methodology that foregrounds the ethics of design practice, reveals potentially hidden agendas and values, and explores alternative design values”[^24]\n\n- **Reflective design** – reflection on “the ways in which technologies reflect and perpetuate unconscious cultural assumptions” and the creation of alternatives[^25]","sidebarKey":"frameworks-practice","isDrawer":true},{"id":"privacy/consent#sidebar-privacy-selling-point","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"privacy-selling-point","title":"Privacy features as selling points","content":"Apple regularly highlights its products’ privacy features in marketing materials under the slogan: “Privacy. That’s iPhone.” [A 2022 advertisement](https://www.forbes.com/sites/kateoflahertyuk/2022/05/25/apple-slams-facebook-and-google-with-bold-new-privacy-ad/) depicts a user’s data on auction—until the protagonist activates ‘Mail Privacy Protection.’ [A series of billboards](https://www.dandad.org/work/d-ad-awards-archive/privacy-thats-iphone) in 2023 show people holding up iPhones to cover their faces. And in 2024, Apple released [a two-minute Hitchcockian film](https://lbbonline.com/news/apples-privacy-features-on-iphone-fend-off-bird-like-surveillance-cameras-in-fantastical-film) in which bird-like security cameras hunt people’s browsing data before iPhone users open Safari, causing the cameras to explode in midair.","sidebarKey":"privacy-selling-point","isDrawer":true},{"id":"privacy/consent#sidebar-privacy-add-on","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"privacy-add-on","title":"Privacy as an optional, paid add-on","content":"Some companies have business models wherein users must purchase the right to control how their data is treated. You may hear such models referred to as **‘pay-for-privacy,’ ‘consent-or-pay,’** or **‘freemium pricing’** (from the words ‘free’ and ‘premium’). In her [Columbia Law Review article](https://www.columbialawreview.org/content/paying-for-privacy-and-the-personal-data-economy/), Stacy-Ann Elvy argues that pay-for-privacy models “facilitate the transformation of privacy into a tradable product, may engender or worsen unequal access to privacy, and could further enable predatory and discriminatory behavior.” In [comments to the UK’s Information Commissioner's Office (ICO)](https://epic.org/documents/epic-comments-to-uk-ico-call-for-views-on-consent-or-pay-business-models/), EPIC asserts, “Consent is a facially invalid basis for processing data under ‘consent or pay’ business models.” Other authors explore [the move to paid subscriptions on social networks](https://www.techpolicy.press/pay-or-okay-the-move-to-paid-subscriptions-on-social-networks/?ref=theethicaltechdigest.org#) and [what freemium pricing looks like in the realm of generative AI](https://www.safehire.ai/blog-posts/free-ai-versus-paid-ai-is-your-data-really-safe-what-you-need-to-know).","sidebarKey":"privacy-add-on","isDrawer":true},{"id":"privacy/consent#sidebar-personal-data-removal","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"personal-data-removal","title":"personal data removal Personal data removal services are in the business of reducing customers’ digital footprints by requesting that data brokers and people-finder sites remove their information. Popular options include Aura, DeleteMe, Incogni, and Optery. Check out this article from PCMag or this one from CNET to learn more about how they work.","content":"","sidebarKey":"personal-data-removal","isDrawer":true},{"id":"privacy/consent#sidebar-virtual-private-network","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"virtual-private-network","title":"Virtual private networks (VPNs)","content":"VPNs work by encrypting the user’s data and routing their internet connection through a remote server, masking their real IP address and location. The Center for Democracy and Technology (CDT) offers a helpful [primer](https://cdt.org/insights/techsplanations-part-5-virtual-private-networks/) on VPNs and how they relate to privacy.","sidebarKey":"virtual-private-network","isDrawer":true},{"id":"privacy/consent#sidebar-for-profit-companies","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"for-profit-companies","title":"For-profit companies","content":"Microsoft, for example, has [instructed its employees to prioritize security](https://www.thurrott.com/microsoft/306687/microsoft-asks-all-of-its-employees-to-prioritize-security) and offers extensive resources on data privacy through the [Microsoft Trust Center](https://www.microsoft.com/en-us/trust-center/privacy).\n\nIt should be noted that with for-profit companies, it is nearly impossible to disentangle motivations rooted in brand image and profit from those rooted in genuine concern for user privacy.","sidebarKey":"for-profit-companies","isDrawer":true},{"id":"privacy/consent#sidebar-non-profit-initiatives","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"non-profit-initiatives","title":"Non-profit initiatives","content":"**The Tor Project**, established as a nonprofit in 2006, was established to maintain development of the Tor network. Tor is an acronym of ‘The Onion Router.’ **Onion routing** protects users’ privacy by directing their internet traffic through multiple servers and encrypting it at each step. Click the links to learn more about The Tor Project’s [history](https://www.torproject.org/about/history/), [principles](https://blog.torproject.org/tor-social-contract/), and [plans for advancing digital rights in 2026](https://blog.torproject.org/advancing-digital-rights-in-2026/).\n\nThe **Open Source Initiative** (OSI) was formed in 1998 to promote “development based on the sharing and collaborative improvement of software source code.” Open source products are potentially subject to high levels of public scrutiny, which encourages good privacy-respecting behavior by developers. You can read more about the history of OSI [here](https://opensource.org/about/history-of-the-open-source-initiative).\n\n[**Signal**](https://signal.org/) is an encrypted messaging app funded through grants and donations and based on open source code. It is designed to minimize data collection and protect user metadata.","sidebarKey":"non-profit-initiatives","isDrawer":true},{"id":"privacy/consent#sidebar-browser-extensions","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"browser-extensions","title":"Browser extensions","content":"Browser extensions are software add-ons that offer a specific functionality or feature within a web browser.\n\nA popular type of browser extension are ad blockers, which prevent advertisements from appearing when users load webpages. [Privacy Badger](https://privacybadger.org/), from the digital rights nonprofit Electronic Frontier Foundation (EFF), specifically blocks trackers and ads that violate user consent with the goal of incentivizing advertisers to adopt better privacy practices.\n\nDaniel C. Howe and Helen Nissenbaum have created several browser extensions to subvert the web tracking ecosystem. With Vincent Toubiana they developed TrackMeNot, which obfuscates a user’s search history by automatically adding random “ghost queries.” With Mushon Zer-Aviv they developed AdNauseam, which ‘clicks’ every ad behind the scenes to make engagement analytics useless. For more on [TrackMeNot](https://www.trackmenot.io/), see [Howe and Nissenbaum’s chapter](https://nissenbaum.tech.cornell.edu/papers/D.Howe_TrackMehot.pdf) in _Lessons from the Identity Trail_ (2008). For more on [AdNauseam](https://adnauseam.io/), check out the article “[Engineering Privacy and Protest: A Case Study of AdNauseam](https://nissenbaum.tech.cornell.edu/papers/EngineeringPrivacyandProtest-AdNauseam.pdf)” or watch [Nissenbaum’s 2019 talk](https://www.youtube.com/watch?v=vZpu3UrGeZg) at the Simons Institute for the Theory of Computing.","sidebarKey":"browser-extensions","isDrawer":true},{"id":"privacy/consent#sidebar-how-to-evade","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"how-to-evade","title":"How to evade insidious tracking techniques","content":"The following are examples of educational resources published by groups or individuals interested in helping people take control of their digital privacy:\n\n- The research and organizing initiative [Our Data Bodies](https://www.odbproject.org) offers the [_Digital Defense Playbook: Community Power Tools for Reclaiming Data_](https://www.odbproject.org/wp-content/uploads/2019/03/ODB_DDP_HighRes_Single.pdf).\n- Investigative tech reporter [Yael Grauer](https://yaelwrites.com) maintains the [Big Ass Data Broker Opt-Out List](https://github.com/yaelwrites/Big-Ass-Data-Broker-Opt-Out-List), a US-based volunteer project guiding users on how to remove their information from data broker sites.\n- The international digital rights nonprofit [Access Now](https://www.accessnow.org) offers [guides](https://www.accessnow.org/guides/) searchable by issue, region, and language. Access Now also has a [Digital Security Helpline](https://www.accessnow.org/help/) which provides 24/7 technical support for activists, journalists, and other human rights defenders.\n- [GLAAD's Social Media Safety Program](https://glaad.org/smsi/lgbtq-social-media-safety-program/) works to secure safer online spaces for LGBTQ+ individuals. Resources include the [We Keep Us Safe](https://glaad.org/smsi/lgbtq-digital-safety-guide/) digital safety guide and an annual [Social Media Safety Index & Platform Scorecard](https://glaad.org/smsi/social-media-safety-index-2025/).","sidebarKey":"how-to-evade","isDrawer":true},{"id":"privacy/consent#sidebar-government-regulation","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"government-regulation","title":"Government regulation","content":"The Electronic Frontier Foundation (EFF) article “[Digital Privacy Legislation is Civil Rights Legislation](https://www.eff.org/deeplinks/2023/04/digital-privacy-legislation-civil-rights-legislation)” offers a social justice perspective on the importance of government regulation. A brief [summary of laws governing privacy in the US](https://epic.org/issues/privacy-laws/united-states/) is available from the Electronic Privacy Information Center (EPIC). To follow the latest developments in state-level privacy law, check out the [State Privacy Legislation Tracker](https://iapp.org/resources/article/us-state-privacy-legislation-tracker/) from the IAPP.","sidebarKey":"government-regulation","isDrawer":true},{"id":"privacy/consent#sidebar-holds-accountable","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"holds-accountable","title":"Holds accountable","content":"EPIC’s webpage on [Enforcement of Privacy Laws](https://epic.org/issues/data-protection/enforcement-of-privacy-laws/) provides a useful summary of this topic in the US context. To learn more about private rights of action in US privacy law, check out [this](https://iapp.org/resources/article/private-rights-of-action-us-privacy-legislation/) IAPP resource and [this](https://www.eff.org/deeplinks/2019/01/you-should-have-right-sue-companies-violate-your-privacy) Electronic Frontier Foundation (EFF) article on the subject.","sidebarKey":"holds-accountable","isDrawer":true},{"id":"privacy/consent#sidebar-lobbying","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"lobbying","title":"Lobbying","content":"The Electronic Privacy Information Center (EPIC) explains the motivations behind its campaign for an American data protection agency in the article “[The U.S. Urgently Needs a Data Protection Agency](https://epic.org/campaigns/dpa/).”","sidebarKey":"lobbying","isDrawer":true},{"id":"privacy/consent#sidebar-dpas","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"dpas","title":"Data protection agencies (DPAs)","content":"You can read more about the missions and contributions of select federal data protection agencies at their English-language websites linked below:\n\n- Canada – [Office of the Privacy Commissioner of Canada (OPC)](https://www.priv.gc.ca/en/)\n- France – [Commission Nationale de l’Informatique et des Libertés (CNIL)](https://www.cnil.fr/en)\n- Singapore – [Personal Data Protection Commission](https://www.pdpc.gov.sg/) = South Africa – [The Information Regulator](https://inforegulator.org.za/)\n- United Kingdom – [Information Commissioner's Office (ICO)](https://ico.org.uk)\n\nFor a comprehensive list of data protection authorities around the world, visit the IAPP’s [Global Privacy Law and DPA Directory](https://iapp.org/resources/global-privacy-directory/).","sidebarKey":"dpas","isDrawer":true},{"id":"privacy/consent#sidebar-brussels-effect","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"brussels-effect","title":"Brussels effect","content":"To learn more about the Brussels effect, see the following reports: “[The Brussels Effect and the GDPR: EU Institutions as Catalysts for Global Data Protection Norms](https://edpi.eu/brussels-effect)” from the European Digital Policy Initiative (EDPI) and “[Mapping the Brussels Effect: The GDPR Goes Global](https://cepa.org/comprehensive-reports/mapping-the-brussels-effect-the-gdpr-goes-global/)” from the Center for European Policy Analysis (CEPA).","sidebarKey":"brussels-effect","isDrawer":true},{"id":"privacy/consent#sidebar-california-effect","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"california-effect","title":"California effect","content":"The [California Consumer Privacy Act of 2018 (CCPA)](https://oag.ca.gov/privacy/ccpa) and its 2020 expansion through the (CPRA) currently comprise the most stringent digital privacy laws in the US. For a summary of California’s recently passed privacy legislation, check out [this blog post](https://www.insideprivacy.com/state-privacy/california-enacts-new-privacy-laws/).\n\nLegal experts consider California the ‘the one to watch’ in the realm of [state privacy regulation](https://www.squirepattonboggs.com/insights/publications/2025-state-privacy-roundup-key-trends-and-california-developments-to-watch-in-2026/). For instance, California jurisprudence allows consumers to opt out of websites selling or sharing personal information using an automated signal, which has spurred the development of a [Global Privacy Control (GPC) specification](https://globalprivacycontrol.org). The development is a collaboration between privacy experts, large media companies, and non-profit organizations. The GPC specification is intended to replace the now-deprecated [Do Not Track (DNT)](https://en.wikipedia.org/wiki/Do_Not_Track) HTTP header field.[^30]","sidebarKey":"california-effect","isDrawer":true},{"id":"privacy/consent#sidebar-ngos","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"ngos","title":"Non-governmental organizations (NGOs) The following are examples of prominent NGOs that function as watchdogs for digital consent:","content":"- [Consumer Watchdog](https://consumerwatchdog.org/issues/privacy/) advocates for US consumer rights and frequently challenges the data-sharing practices of Big Tech.\n- [Distributed Denial of Secrets (DDoSecrets)](https://ddosecrets.com/) is a transparency collective that publishes leaked datasets of public interest to expose corporate and governmental corruption.\n- The [Electronic Frontier Foundation (EFF)](https://www.eff.org/issues/privacy) defends civil liberties in the digital world through high-impact litigation and the development of privacy-enhancing tools.\n- The [Electronic Privacy Information Center (EPIC)](https://epic.org/about/) utilizes public interest litigation and rigorous research to highlight emerging privacy threats, focusing on holding the government and commercial sectors to account for data misuse.\n- The [International Digital Accountability Council (IDAC)](https://digitalwatchdog.org/) monitors the mobile app ecosystem, conducting technical investigations to ensure platforms and developers are adhering to their stated privacy promises.\n- [Mozilla Foundation](https://www.mozillafoundation.org/en/) audits consumer technology through its [\\*Privacy Not Included](https://www.mozillafoundation.org/en/privacynotincluded/) consumer guides.\n- The Vienna-based NGO [noyb](https://noyb.eu/en) works to enforce European data laws, like the GDPR and the ePrivacy Directive, by filing strategic legal complaints against companies that fail to obtain valid user consent.\n- [Open Rights Group (ORG)](https://www.openrightsgroup.org/what-we-do/) is a UK-based organization that campaigns against mass surveillance and for the protection of free expression online, pushing for legislative reforms that put users back in control of their data.\n- [Privacy International](https://privacyinternational.org/) is a global organization that investigates how governments and corporations use surveillance technology and advocates for stronger legal protections.\n- The [Security Lab at Amnesty International](https://securitylab.amnesty.org/) conducts forensic investigations to expose state-sponsored spyware and digital attacks used against activists, journalists, and human rights defenders.\n- [World Privacy Forum](https://worldprivacyforum.org/) is a research-focused group that analyzes data practices and educates policymakers and consumers.","sidebarKey":"ngos","isDrawer":true},{"id":"privacy/consent#sidebar-intergovernmental-organizations","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"intergovernmental-organizations","title":"Intergovernmental organizations","content":"The [UN Special Rapporteur on the right to privacy](https://www.ohchr.org/en/special-procedures/sr-privacy) is an independent expert mandated to monitor, investigate, and report on privacy violations worldwide while advising governments on aligning their laws with international human rights standards.","sidebarKey":"intergovernmental-organizations","isDrawer":true},{"id":"privacy/consent#sidebar-media-outlets","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"media-outlets","title":"Media outlets","content":"The following are examples of media watchdogs in digital privacy:\n\n- [_404 Media_](https://www.404media.co/) is a reporter-owned investigative site that provides on-the-ground reporting on the tools used by data brokers, police, and hackers to bypass privacy protections and exploit user data.\n- [_IAPP News_](https://iapp.org/news?size=n_16_n) is the editorial arm of the International Association of Privacy Professionals. IAPP News acts as a specialized monitor that tracks global regulatory enforcement, legislative shifts, and industry compliance.\n- [_The Markup_](https://themarkup.org/) is a non-profit newsroom that uses transparent, data-driven investigative journalism to audit how powerful institutions use technology. The Markup’s slogan is “challenging technology for the public good.”\n- [_WIRED_](https://www.wired.com/tag/privacy/) is a leading global publication and mainstream watchdog that provides deep-dive reporting and cultural analysis on technology and the risks of the digital age.","sidebarKey":"media-outlets","isDrawer":true},{"id":"privacy/consent#sidebar-independent-journalists","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"independent-journalists","title":"Independent journalists","content":"- [Julia Angwin](https://juliaangwin.com), co-founder of [_The Markup_](https://themarkup.org/) and founder of [_Proof News_](https://www.proofnews.org), is known as a pioneer in algorithmic auditing and has developed forensic tools to expose how companies track users across the web.\n- [Joseph Cox](https://www.404media.co/author/joseph-cox/), co-founder of [_404 Media_](https://www.404media.co/), tracks the ‘gray market’ of location data and exposes how data brokers sell sensitive user info to law enforcement and government agencies.\n- Troy Hunt is a software developer, [blogger](https://www.troyhunt.com), [Microsoft Regional Director](https://mvp.microsoft.com/en-US/RD/profile/cfba70c0-3c9a-e411-93f2-9cb65495d3c4), and creator of [_Have I Been Pwned_](https://haveibeenpwned.com/), a service that aggregates data breaches and helps people establish if they’ve been impacted.\n- Brian Krebs, author of the blog [_Krebs on Security_](https://krebsonsecurity.com/), is a veteran investigator most famous for breaking news on massive retail data breaches.\n- [Casey Newton](https://en.wikipedia.org/wiki/Casey_Newton), publisher of the newsletter [_Platformer_](https://www.platformer.news), is an influential analyst who focuses on executive leadership at major social media companies.\n- [Zach Whittaker](https://zackwhittaker.com), a cybersecurity reporter and author of the newsletter [_this week in security_](https://this.weekinsecurity.com/), is known for finding unsecured databases and compelling companies to secure them before they are exploited.\n- [Kim Zetter](https://en.wikipedia.org/wiki/Kim_Zetter) is a forensic investigative reporter who breaks stories on the intersection of cybersecurity and national security.","sidebarKey":"independent-journalists","isDrawer":true},{"id":"privacy/consent#sidebar-we-have-not-even","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"we-have-not-even","title":"we have not even read them","content":"As an April Fools’ Day joke, UK videogame retailer Gamestation changed its terms and conditions so that “all of the 7500 people who bought from Gamestation online on 1 April 2010 effectively consented to the sale of their immortal souls.”[^39] See coverage from law firm [Pinsent Masons](https://www.pinsentmasons.com/out-law/news/nobody-reads-terms-and-conditions-its-official) for more.","sidebarKey":"we-have-not-even","isDrawer":true},{"id":"privacy/consent#sidebar-harms-are-distant","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"harms-are-distant","title":"Harms are distant","content":"This issue of people’s inability to accurately estimate the long-term risks of data sharing is sometimes called privacy myopia.[^41]","sidebarKey":"harms-are-distant","isDrawer":true},{"id":"privacy/consent#sidebar-cannot-be-anticipated","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"cannot-be-anticipated","title":"cannot be anticipated","content":"For example, data we once considered de-identified often becomes identifiable because of advances unpredicted at the time of ‘de-identification.’[^42] See the Re-identification primer to learn more.","sidebarKey":"cannot-be-anticipated","isDrawer":true},{"id":"privacy/consent#sidebar-done-little-harm","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"done-little-harm","title":"Big Tech’s bottom line","content":"Alphabet (parent company of Google), Apple, Meta, and Amazon were fined $7.8 billion in 2025, a sum which, with how much revenue they make, will take them less than a month to pay off. This finding comes from [Proton’s work tracking Big Tech fines](https://proton.me/tech-fines-tracker), a project the group taglines “Big Tech, small consequences.”","sidebarKey":"done-little-harm","isDrawer":true},{"id":"privacy/consent#sidebar-consent-theater","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"consent-theater","title":"Consent theater","content":"“Consent theater” is a variation on the term “security theater,” coined by privacy and computer security specialist Bruce Schneier in his 2003 book _Beyond Fear: Thinking Sensibly About Security in an Uncertain World_. Security theater refers to the practice of implementing toothless security procedures that give the appearance of improved safety. It is commonly used in reference to airport security as operated by the United States Transportation Security Administration (TSA). The term “consent theater” gained traction in 2021 through a [conference paper](https://doi.org/10.1145/3411763.3451230) by a team from the CISPA Helmholtz Center for Information Security and a [blog post](https://onezero.medium.com/consent-theater-a32b98cd8d96) by author and activist Cory Doctorow.","sidebarKey":"consent-theater","isDrawer":true},{"id":"privacy/consent#sidebar-privacy-nihilism","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"privacy-nihilism","title":"Privacy nihilism","content":"Professor Ian Bogost explores the topic in his article “[Welcome to the Age of Privacy Nihilism](https://www.theatlantic.com/technology/archive/2018/08/the-age-of-privacy-nihilism-is-here/568198/)” published in _The Atlantic_ in 2018.","sidebarKey":"privacy-nihilism","isDrawer":true},{"id":"privacy/consent#sidebar-powerless-to-control","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"powerless-to-control","title":"Powerless to control their data","content":"In [this article](https://www.consumerreports.org/electronics/personal-information/i-tried-to-get-my-name-off-peoplesearch-sites-it-was-nearly-a0741114794/) for _Consumer Reports_, Mara Hvistendahl describes the “herculean task” of trying to remove herself from data broker and people-search sites.","sidebarKey":"powerless-to-control","isDrawer":true},{"id":"privacy/consent#sidebar-signing-away","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"signing-away","title":"Signing away privacy rights in exchange for welfare","content":"For further reading on intersection between privacy and socioeconomic status, check out these resources:\n\n- “[Building Digital Benefits That Protect Privacy for All](https://www.georgetownpoverty.org/issues/building-digital-benefits-that-protect-privacy/)” (2025) – Georgetown Law Center on Poverty and Inequality blog post by affiliated scholar Jae June Lee\n- “[The Class Differential in Privacy Law](https://brooklynworks.brooklaw.edu/cgi/viewcontent.cgi?article=1140&context=blr)” (2012) – _Brooklyn Law Review_ article by Michele Estrin Gilman, Venerable Professor of Law at the University of Baltimore\n- [_The Poverty of Privacy Rights_](https://www.sup.org/books/law/poverty-privacy-rights) (2017) – book by Khiara M. Bridges, Professor of Law and Anthropology at Boston University","sidebarKey":"signing-away","isDrawer":true},{"id":"privacy/consent#sidebar-submitting-workplace","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"submitting-workplace","title":"Submitting to workplace surveillance for fear of losing one’s job","content":"The 2021 report “[The Constant Boss: Work Under Digital Surveillance](https://datasociety.net/wp-content/uploads/2021/05/The_Constant_Boss.pdf)” by Aiha Nguyen, director of _Data & Society’s_ Labor Futures Program, examines the consent-related issues with workplace surveillance, particularly of low-wage and hourly workers.\n\nA member of United for Respect, representative of the retail worker advocacy group, told Aiha Nguyen that Walmart managers asked employees to download an app on their personal devices to check inventory and scan misshelved items, neglecting to mention that the app requires camera access and location services, which are constantly active by default.[^47]","sidebarKey":"submitting-workplace","isDrawer":true},{"id":"privacy/consent#sidebar-smart-home-devices","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"smart-home-devices","title":"Smart home devices","content":"The articles “[Smart Locks Endanger Tenants’ Privacy and Should Be Regulated](https://www.eff.org/deeplinks/2023/04/smart-locks-endanger-tenants-privacy-and-should-be-regulated)” by Mario Trujillo and Adam Schwartz of the Electronic Frontier Foundation (EFF) and “[In smart apartments, is tenants’ privacy for rent?](https://www.bostonglobe.com/2020/02/11/business/smart-apartments-is-tenants-privacy-rent/)” by [Hiawatha Bray](https://www.bostonglobe.com/about/staff-list/staff/hiawatha-bray/?p1=Article_Byline) of the _Boston Globe_ dive into this issue.","sidebarKey":"smart-home-devices","isDrawer":true},{"id":"privacy/consent#sidebar-genetic-testing","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"genetic-testing","title":"Genetic testing","content":"Widely available DNA testing, including through direct-to-consumer services like 23andMe, raises challenging questions about individual and group privacy. If you are interested in exploring this topic, Martin Gomberg’s IAPP opinion piece (“[Privacy laws, ethics and the conundrum of DNA](https://iapp.org/news/a/privacy-laws-ethics-and-the-conundrum-of-dna-2)”) and Ethan Magistro’s _Princeton Legal Journal Forum_ article (“[It’s Not Just Me, It’s Also You: How Shared DNA Complicates Consent](https://legaljournal.princeton.edu/wp-content/uploads/sites/826/2024/04/1-Prin.L.J.F.-19.pdf)”) are good starting points.","sidebarKey":"genetic-testing","isDrawer":true},{"id":"privacy/consent#sidebar-social-media-ecosystems","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"social-media-ecosystems","title":"Social media ecosystems","content":"“In a digital age, individual consent is flawed and ineffectual when protected class data and social profiles can be easily inferred via our social networks,” writes complex systems researcher Juniper Lovato.[^49]\n\n[A 2010 study](https://doi.org/10.1145/1718487.1718519) of college students’ Facebook profiles found that “multiple attributes can be inferred globally when as few as 20% of the users reveal their attribute information.”[^50] More recently, [a 2019 study](%20https://doi.org/10.1038/s41562-018-0510-5) out of the University of Vermont concluded “that 95% of the potential predictive accuracy for an individual is achievable using their social ties only, without requiring that individual’s data.”[^51]\n\nFacebook has consistently taken advantage of consent by association. In the infamous Cambridge Analytica incident, “users took surveys that captured their data and, if they had not adjusted their privacy settings, data from users’ friends and contacts.”[^52] Around the same time, the phenomenon of Facebook **shadow profiles** gained attention. The _Vox_ article “[Facebook collects data on you even if you don’t have an account](https://www.vox.com/2018/4/20/17254312/facebook-shadow-profiles-data-collection-non-users-mark-zuckerberg)” explains how and why Facebook engages in the practice.","sidebarKey":"social-media-ecosystems","isDrawer":true},{"id":"privacy/consent#sidebar-human-subjects","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"human-subjects","title":"Human subjects research","content":"**Positive externalities** in human subjects research (i.e., spillover effects through which we learn about individuals or groups beyond the study population) are nothing new. In many cases, the whole point of the research endeavor is learning about people beyond those who consented to be studied. The problem identified by scholars like A. Michael Froomkin is that, in the era of **Big Data** and advanced **algorithmic learning**, “data derived from a large population can in some circumstances generate particularized predictions about small populations, or even individuals, outside the study group.”[^53] This has been referred to as the ‘**tyranny of the minority**,’ explained by Solon Barocas and Helen Nissenbaum as a situation in which “the volunteered information of the few can unlock the same information about the many.”[^54]\n\nTo learn more, check out the article “[Predictive privacy: Collective data protection in the context of artificial intelligence and big data](https://doi.org/10.1177/20539517231166886)” by Rainer Mühlhoff.\n\nThe Spanish Data Protection Agency (AEPD) offers a helpful [primer](https://www.aepd.es/en/prensa-y-comunicacion/blog/group-privacy) on the concept of **group privacy**.","sidebarKey":"human-subjects","isDrawer":true},{"id":"privacy/consent#sidebar-may-not-be-aware","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"may-not-be-aware","title":"May not be aware","content":"Back in 2013, Ewa Luger and Tom Rodden described living in an “era of ubiquity” when it came to **smart environments**. Like other researchers, they were concerned with data captured “below the line of user visibility.”[^55] This is highly relevant when considering **Internet of Things (IoT) devices**, physical objects equipped to sense and share information across networks. No version of consent—explicit, tacit, or implicit—is valid if people are unaware they are being tracked.\n\nThe Office of the Victorian Information Commissioner (OVIC) in Australia has a helpful primer called “[Internet of Things and Privacy – Issues and Challenges](https://ovic.vic.gov.au/privacy/resources-for-organisations/internet-of-things-and-privacy-issues-and-challenges/).” Another useful reference may be the Data & Society Research Institute’s [comments](https://datasociety.net/pubs/dcr/Data&Society_NTIA-comments_May2015.pdf) to the US National Telecommunication and Information Administration (NTIA) on cybersecurity in the digital ecosystem.","sidebarKey":"may-not-be-aware","isDrawer":true},{"id":"privacy/consent#sidebar-children","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"children","title":"Children","content":"Meaningful consent requires the consenter to have the appropriate cognitive capacity to engage in the decision-making process. Therefore, children categorically cannot provide valid consent. However, in practice, “whether legally or illegally, it has been trivially easy to circumvent the consent of **legally incapacitated** minors in ways that have led to serious financial and even physical harm.”[^56]\n\nTo learn more, check out the [resources](https://epic.org/issues/data-protection/childrens-privacy/) the Electronic Privacy Information Center (EPIC) provides on children’s privacy.\n\nIndividuals, including family members, frequently violate children’s consent by posting content of and about them online. The article “[Influencer Parents and The Kids Who Had Their Childhood Made Into Content](https://www.teenvogue.com/story/influencer-parents-children-social-media-impact)” highlights some examples.","sidebarKey":"children","isDrawer":true},{"id":"privacy/consent#sidebar-human-expertise","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"human-expertise","title":"Human expertise","content":"For example, [Terms of Service; Didn’t Read (ToS;DR)](https://tosdr.org/en/), founded by attorney Hugo Roy, programmer Michiel de Jong, and web designer Jan-Christoph Borchardt in 2012, is a community project that grades the privacy practices of major websites, apps, and services.","sidebarKey":"human-expertise","isDrawer":true},{"id":"privacy/consent#sidebar-artificial-intelligence","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"artificial-intelligence","title":"Artificial intelligence (AI) For example, the webXray Privacy Search Engine uses proprietary machine learning models to support legal and compliance professionals in identifying and addressing privacy violations.","content":"","sidebarKey":"artificial-intelligence","isDrawer":true},{"id":"privacy/consent#sidebar-automate-aspects","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"automate-aspects","title":"Automate aspects of the consent process","content":"The [Usable Privacy Policy Project](https://usableprivacy.org) and the [Personalized Privacy Assistant Project](https://privacyassistant.org) are two initiatives that have attempted to semi-automate digital consent. Back in 2018, the Usable Privacy Policy Project was able to match 80% of users to a “privacy profile” that aligned with their actual privacy choices; just 6% of users changed their settings from what the automated tool selected for them.[^59] The Personalized Privacy Assistant Project involves multiple research areas including applications for the **Internet of Things (IoT)**, modeling privacy preferences, and transparency for **Big Data**.","sidebarKey":"automate-aspects","isDrawer":true},{"id":"privacy/consent#sidebar-secondary-data-use","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"secondary-data-use","title":"Secondary data use","content":"Secondary data use refers to the re-analysis of existing data that was originally collected for another purpose. Classically, consent to secondary use has taken the form of either blanket consent (consent to any and all future uses of data) or broad consent (consent to all future uses of data within certain limits, e.g., for studies of a certain health condition).[^60]","sidebarKey":"secondary-data-use","isDrawer":true},{"id":"privacy/consent#sidebar-ordering-those-options","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"ordering-those-options","title":"Ordering those options by risk and sensitivity","content":"For an example of how tiered consent might be implemented, see the journal article “[A tiered-layered-staged model for informed consent in personal genome testing](https://www.nature.com/articles/ejhg2012237)” by Bunnik et al. (2012).","sidebarKey":"ordering-those-options","isDrawer":true},{"id":"privacy/consent#sidebar-consent-requests","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"consent-requests","title":"consent requests for future uses","content":"For an example of how meta-consent might be implemented, see the journal article “[Eliciting meta consent for future secondary research use of health data using a smartphone application \\- a proof of concept study in the Danish population](https://doi.org/10.1186/s12910-017-0209-6)” by Ploug & Holm (2017).","sidebarKey":"consent-requests","isDrawer":true},{"id":"privacy/consent#sidebar-dynamic-consent","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"dynamic-consent","title":"Dynamic consent","content":"[Kaye et al. (2015)](https://www.nature.com/articles/ejhg201471) and [Budin-Ljøsne et al. (2017)](https://link.springer.com/article/10.1186/s12910-016-0162-9) are good overviews of dynamic consent as a theory and project. [Steinsbekk et al. (2013)](https://www.nature.com/articles/ejhg2012282) refutes its characterization as an improvement on broad consent (consent to all future uses of data within certain limits, e.g., for studies of a certain health condition).","sidebarKey":"dynamic-consent","isDrawer":true},{"id":"privacy/consent#sidebar-communicate-investigators","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"communicate-investigators","title":"Communicate with investigators in real time","content":"Enabling two-way communication between subjects and investigators is a step towards reducing the power disparities that have historically existed between these groups. Coupled with greater bargaining power for potential participants, two-way communication could make the digital consent process less like an ultimatum and more like a negotiation. This is consistent with **collaborative consent** as described by Paul Bernal.[^64]","sidebarKey":"communicate-investigators","isDrawer":true},{"id":"privacy/consent#sidebar-assess-participant","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"assess-participant","title":"assess participant comprehension","content":"The National Institutes of Health (NIH) [All of Us Research Program](https://allofus.nih.gov) implements this kind of competency check in an attempt to ensure individuals consenting to contribute data have the cognitive capacity to make an informed and engaged decision. Check out [All of Us’ description](https://allofus.nih.gov/article/all-us-consent-process) of their consent process or [this assessment](https://pubmed.ncbi.nlm.nih.gov/33275082/) published in the journal _AJOB Empirical Bioethics_ to learn more.","sidebarKey":"assess-participant","isDrawer":true},{"id":"privacy/consent#sidebar-distributed-consent","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"distributed-consent","title":"Distributed consent","content":"For a more in-depth explanation of distributed consent, read “[Limits of Individual Consent and Models of Distributed Consent in Online Social Networks](https://dl.acm.org/doi/10.1145/3531146.3534640)” by Lovato et al. (2022). Using a decentralized model uncharacteristic of other platforms, the social network [Mastodon](https://docs.joinmastodon.org) implements some of the principles discussed by Lovato and colleagues.","sidebarKey":"distributed-consent","isDrawer":true},{"id":"privacy/consent#sidebar-consent-receipts","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"consent-receipts","title":"consent receipts","content":"To learn more, check out the article “[Consent Receipts for a Usable and Auditable Web of Personal Data](https://ieeexplore.ieee.org/document/9730898)” by Vitor Jesus and Harshvarhan J. Pandit. As a member of the [W3C Data Privacy Vocabularies and Controls Community Group](https://www.w3.org/groups/cg/dpvcg/), Pandit is a leading contributor to [technical guidance](https://w3c.github.io/dpv/guides/consent-27560) on the implementation of this technology.","sidebarKey":"consent-receipts","isDrawer":true},{"id":"privacy/consent#sidebar-binary-governance","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"binary-governance","title":"Binary governance Legal scholar Margot Kaminski is credited with coining the term binary governance. See her article “Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability” (2019) for more.","content":"","sidebarKey":"binary-governance","isDrawer":true},{"id":"privacy/consent#sidebar-risk-assessments","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"risk-assessments","title":"Risk assessments","content":"Learn more about the **data protection impact assessments (DPIA)** required for high-risk processing under article 35 of the GDPR [here](https://gdpr.eu/data-protection-impact-assessment-template/). |","sidebarKey":"risk-assessments","isDrawer":true},{"id":"privacy/consent#sidebar-fiduciary-duty","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"fiduciary-duty","title":"Fiduciary duty","content":"The idea of data fiduciaries is explored in the article “[Data Controllers as Data Fiduciaries: Theory, Definitions & Burdens of Proof](https://scholar.law.colorado.edu/lawreview/vol95/iss1/4/)” (2024) by Noelle Wilson and Amanda Reid of the University of North Carolina Center for Information, Technology, and Public Life.","sidebarKey":"fiduciary-duty","isDrawer":true},{"id":"privacy/consent#sidebar-while-analysis","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"while-analysis","title":"While still allowing for useful statistical analysis","content":"Economists Ian M. Schmutte and Lars Vilhuber provide an overview of balancing privacy and usability in their [chapter](https://admindatahandbook.mit.edu/book/v1.1/discavoid.html) of the _Handbook on Using Administrative Data for Research and Evidence-based Policy_ published by the Massachusetts Institute of Technology. The work of the [United State Census Bureau](https://www.census.gov/about/policies/privacy/statistical_safeguards.html) is an excellent case study of how SDL techniques are applied to protect individual identities while still delivering valuable demographic insights.","sidebarKey":"while-analysis","isDrawer":true},{"id":"privacy/consent#sidebar-coarsening","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"coarsening","title":"Coarsening","content":"For example, shifting the scale of analysis from a census tract to a larger geographic area such as a county or state makes it more difficult to identify someone based on details like their month and day of birth. It is common to **top code** or **bottom code** entries beyond a certain threshold into a single category (e.g., ‘90 years or older’) when values on one or both extremes are rare.[^71]\n\nOne way to formalize such indistinguishability is **k-anonymity**, which requires that the values of each individual record released match at least *k*−1 other records from the batch. To learn more about k-anonymity, check out Latanya Sweeney’s paper “[Achieving K-Anonymity Privacy Protection Using Generalization and Suppression](https://doi.org/10.1142/s021848850200165x)” (2002).","sidebarKey":"coarsening","isDrawer":true},{"id":"privacy/consent#sidebar-data-swapping","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"data-swapping","title":"Data swapping","content":"To learn more about data swapping, see [Dalenius & Reiss (1982)](<https://doi.org/10.1016/0378-3758(82)90058-1>) and [Fienberg & McIntyre (2005)](https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/data-swapping-variations-on-a-theme-by-dalenius-and-reiss.pdf).","sidebarKey":"data-swapping","isDrawer":true},{"id":"privacy/consent#sidebar-synthetic-data","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"synthetic-data","title":"Synthetic data","content":"For further reading on synthetic data, see the explainer “[Synthetic Data \\- what, why and how?](https://royalsociety.org/-/media/policy/projects/privacy-enhancing-technologies/Synthetic_Data_Survey-24.pdf)” by scholars of the Alan Turing Institute (the United Kingdom’s national institute for data science) or the post “[When what is old is new again – The reality of synthetic data](https://www.priv.gc.ca/en/blog/20221012)” published on the Office of the Privacy Commissioner of Canada’s _Privacy Tech-Know_ blog.","sidebarKey":"synthetic-data","isDrawer":true},{"id":"privacy/consent#sidebar-choice-architecture","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"choice-architecture","title":"Choice architecture","content":"See the [‘Consent in Practice’](#consent-in-practice) section for more background on behavioral economics, choice architecture, nudges, and dark patterns. Alessandro Acquisti is a leading scholar on the digital privacy applications of behavioral economics. His articles “[Nudging Privacy: The Behavioral Economics of Personal Information](https://doi.org/10.1109/MSP.2009.163)” (2009) and “[Nudges for Privacy and Security: Understanding and Assisting Users’ Choices Online](https://doi.org/10.1145/3054926)” (2017) are recommended reading on the topic.","sidebarKey":"choice-architecture","isDrawer":true},{"id":"privacy/consent#sidebar-incorporating-personalized","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"incorporating-personalized","title":"Incorporating personalized scenarios into permissions dialogues","content":"Marian Harbach and colleagues from the Usable Security and Privacy Lab at Leibniz University Hannover found that people made more privacy-conscious choices when presented with a Google Play Store permissions dialogue that told users in text that the app could see and delete their photos and showed a random selection of images from the user’s camera roll to illustrate this point. For the full details of their experiment, see their paper “[Using Personal Examples to Improve Risk Communication for Security and Privacy Decisions](http://dx.doi.org/10.1145/2556288.2556978)” (2014).","sidebarKey":"incorporating-personalized","isDrawer":true},{"id":"privacy/consent#sidebar-tradeoffs","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"tradeoffs","title":"Tradeoffs","content":"(The Privacy Tradeoffs)[/privacy/tradeoffs] primer explores in depth the issue of balancing privacy against other goods.","sidebarKey":"tradeoffs","isDrawer":true},{"id":"privacy/consent#sidebar-sci-and-med","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"sci-and-med","title":"scientific and medical progress","content":"Some scientists believe data sharing for research purposes should be non-optional since advancing human health is in the public interest, and they propose strong anti-discrimination laws to safeguard individuals’ privacy.[^76] One issue such a proposal addresses is **consent bias**, a hindrance to generalizability which occurs because people who consent to participate in studies are almost always systematically different from those who do not consent to participate.[^77] These ideas are explored in the _New York Times_ article “[Balancing Privacy With Data Sharing for the Public Good](https://www.nytimes.com/2021/02/19/business/privacy-open-data-public.html)” by Harvard economist David Deming.","sidebarKey":"sci-and-med","isDrawer":true},{"id":"privacy/consent#sidebar-contextual-integrity","section":"privacy","sectionTitle":"Privacy","subsection":"consent","subsectionTitle":"Consent","anchor":"contextual-integrity","title":"Contextual integrity","content":"For an overview of contextual integrity, watch and/or read Voices of VR’s “[Primer on the Contextual Integrity Theory of Privacy with Philosopher Helen Nissenbaum](https://voicesofvr.com/998-primer-on-the-contextual-integrity-theory-of-privacy-with-philosopher-helen-nissenbaum/)”.\n\nInformation flow control (IFC) models are one way of operationalizing the tenets of contextual integrity, and some cybersecurity researchers are working on systems that enable more nuanced control. The 2024 paper “[Sesame: Practical End-to-End Privacy Compliance with Policy Containers and Privacy Regions](https://doi.org/10.1145/3694715.3695984)” from Brown’s Efficient and Trustworthy Operating Systems (ETOS) Group is an example of such work.","sidebarKey":"contextual-integrity","isDrawer":true},{"id":"accessibility/whatIsAccessibility#intro","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"intro","title":"Introduction","content":"Entire populations are affected by the computational choices made in technology design. When thinking about how to incorporate accessibility into computer science pedagogy, it is crucial to understand that accessibility cannot exist under a single definition. For the purpose of this primer, accessibility will be framed through three categories.\n\n1. Universality\n2. Disability\n3. Equity\n\nAccessibility is often regarded as a priority that is superseded by functionality and efficiency. But accessibility is inseparable from the systems we are familiar with {we-are-familiar-with}, and is seamlessly integrated into our daily lives.\n\nTechnological accessibility is necessary in the creation of well-designed systems. How a system can be accessible will vary, but for its use in this primer, we will define something as accessible when **it takes into account the needs of its users, and then is designed with the intention of minimizing the barriers associated with those needs.**","sidebarKey":null,"isDrawer":false},{"id":"accessibility/whatIsAccessibility#accessibility-through-universality","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"accessibility-through-universality","title":"Accessibility through Universality","content":"Universal design is a utilitarian approach to technological accessibility. While there exists contestation as to the exact definition of universal design, most definitions converge on the idea that it necessitates the design of an environment, be that digital or physical, constructed in a manner that makes it usable for all (or at the least most) people.[^4] {In-practice} this looks like systems that require little to no adaptation from the user because the design took into account, and catered itself to, the similarities between most people.\n\nBecause universal design often considers the most people that can be included in a design, it runs the risk of not addressing people with needs at the margins.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/whatIsAccessibility#accessibility-through-disability","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"accessibility-through-disability","title":"Accessibility through Disability","content":"There exist four main categories of disability: visual, hearing, mobility, and cognitive. Individuals with disability may experience disability permanently, temporarily, and situationally.[^3] In the context of disability, accessibility can mean taking an explicit look at the needs of those with disability and designing in collaboration with those {perspectives}.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/whatIsAccessibility#accessibility-through-equity","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"accessibility-through-equity","title":"Accessibility through Equity","content":"Technological equity can be understood as the narrowed application of social equity. The Internet Society Foundation defines digital equity as \"the state where every person and community has the necessary information technology resources to participate in society, democracy, and the economy fully.\"[^9] The barriers that keep people from equitable access to technology follow patterns of social disparity, meaning that factors such as income, gender, and race often correlate with digital inequity.\n\nDisparities in digital access (referred to as the \"digital divide\") exacerbate disparate outcomes between marginalized and privileged groups in a tech-centered world.[^7] Digital equity is accessibility because it centers how different groups are able to access technology, and thus {can-be-addressed} in product design, computation, policy, and beyond.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/whatIsAccessibility#a-dynamic-approach-to-accessibility","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"a-dynamic-approach-to-accessibility","title":"A Dynamic Approach to Accessibility","content":"These definitions of accessibility are just three conceptions amid an entire field of scholarship. What makes digital accessibility exciting to think about is the way it is constantly changing and redefining itself given the context in which it is being evaluated. By using the framework of Universality, Disability, and Equity, we hope to clarify that there are facets to accessibility that one _must_ consider when interacting with, designing, and creating in the technological space.\n\n[^1]: Aimi Hamraie and Kelly Fritsch, \"Crip technoscience manifesto,\" _Catalyst: Feminism, Theory, Technoscience_ 5, no. 1 (2019): 4.\n\n[^2]: Akhilesh Ganti, “Unified Payments Interface (UPI),” Investopedia, March 7, 2024, https://www.investopedia.com/terms/u/unified-payment-interface-upi.asp.\n\n[^3]: “An Introduction to Digital Accessibility,” Mass.gov, accessed April 14, 2025, https://www.mass.gov/info-details/an-introduction-to-digital-accessibility#what-is-digital-accessibility?.\n\n[^4]: “Definition and Overview of Universal Design (UD),” Centre for Excellence in Universal Design, accessed April 14, 2025, https://universaldesign.ie/about-universal-design/definition-and-overview.\n\n[^5]: Divyam Raj Meng, “India’s Unified Payments Interface Has Revolutionized Its Digital Payments Market,” Cornell SC Johnson College of Business, December 20, 2024, https://business.cornell.edu/hub/2024/12/20/indias-unified-payments-interface-has-revolutionized-its-digital-payments-market/.\n\n[^6]: Neal Broverman, “Only 2% of Tech Jobs Are Held by Black Women. Cristina Mancini Knows That’s Unacceptable,” _Mashable_, February 23, 2025, https://mashable.com/article/black-girls-who-code-representation.\n\n[^7]: UN-Habitat, _Addressing the Digital Divide: Taking Action Towards Digital Inclusion_, United Nations Human Settlements Programme, accessed April 14, 2025, https://unhabitat.org/sites/default/files/2021/11/addressing_the_digital_divide.pdf.\n\n[^8]: Web Content Accessibility Guidelines, Level A Checklist, 1.2.1.\n\n[^9]: “What Is Digital Equity?” Internet Society Foundation, June 26, 2023, https://www.isocfoundation.org/2023/06/what-is-digital-equity/.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/whatIsAccessibility#sidebar-we-are-familiar-with","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"we-are-familiar-with","title":"Familiar Case Studies","content":"#### Case Study: WCAG 1.2.1 Audio-only and Video-only (Pre-recorded)\n\n“A transcript is provided for audio-only content and a transcript or audio description is provided for video-only content.”[^8]\n\nWhile this feature undoubtedly aids users with visual or hearing impairment, it also has the ability to benefit every user. Consider being in a library, and needing to listen to an audio clip for class. If you did not have headphones available, or the internet connection was making the video lag, you would be able to use the transcript instead.\n\n#### Case Study: High Contrast Screen Settings\n\nThe option to change one’s screen to high contrast is available on many of the most common operating systems—i.e. Linux, MacOS, and Windows. Students themselves use this feature since it is a default when they work on IDEs such as VSCode. While we may not think of high contrast as an explicit accessibility feature, it is built to make character distinction easier and less visually taxing than standard white or light screens with dark text.","sidebarKey":"we-are-familiar-with","isDrawer":true},{"id":"accessibility/whatIsAccessibility#sidebar-in-practice","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"in-practice","title":"Universal Design Case Study: iPhone Home Screens","content":"iPhone home screens are both easy to navigate (they are compatible with screen readers and other assistive technology) and customizable. That way, there exists a convention users are familiar with (icons corresponding to apps that are clickable), while giving agency to users to format their device as they best see fit.","sidebarKey":"in-practice","isDrawer":true},{"id":"accessibility/whatIsAccessibility#sidebar-perspectives","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"perspectives","title":"Disability Case Study: Hackathons","content":"Hackathons hosted by organizations where the prompt is a well-intentioned call to create some technology meant to aid disabled people can be more harmful than helpful. These events often frame the users of this technology as fixed and separate from those designing the technology. Further, by framing the invention of assistive technology (or otherwise) in the brief span of a Hackathon, these events hide the extended, exhaustive process that goes into creating technology by and for disabled people.","sidebarKey":"perspectives","isDrawer":true},{"id":"accessibility/whatIsAccessibility#sidebar-can-be-addressed","section":"accessibility","sectionTitle":"Accessibility","subsection":"whatIsAccessibility","subsectionTitle":"What is Accessibility?","anchor":"can-be-addressed","title":"Equity Case Studies","content":"#### Case Study: Unified Payments Interface[^5] [^2]\n\nIn 2016, the National Payments Corporation of India established the Unified Payments Interface (UPI), a single page mobile application that facilitates payment between parties in an efficient, accessible manner. UPI is managed by the Reserve Bank of India and partners with a variety of banks to allow users to access the service. UPI has been credited for including marginalized parties in a digital landscape with its ease and convenience of use, bolstering the Indian economy and reducing the cash-digital divide.\n\n#### Case Study: Black Girls Code [^6]\n\nBlack Girls Code is an organization that promotes the education of Black girls and young women in computation and technology. They recognize that only 2% of jobs in the tech sector are held by Black women and aim to improve those numbers.","sidebarKey":"can-be-addressed","isDrawer":true},{"id":"accessibility/intersectionsWithOtherValues#intro","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"intro","title":"Introduction","content":"The development of inclusive technology requires thoughtful consideration of how accessibility intersects with other values such as privacy, security, and usability. While these values are often treated as separate goals or controlled by separate teams, managing them does not have to be mutually exclusive with accessibility.[^27] In fact, prioritizing accessible features can create powerful positive externalities, ultimately leading to more innovative and equitable products.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/intersectionsWithOtherValues#why-accessibility-gets-overlooked","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"why-accessibility-gets-overlooked","title":"Why Accessibility Gets Overlooked","content":"Companies often fail to prioritize accessibility due to a combination of organizational culture and prejudices. A common barrier is the mistaken belief that accessibility features only serve a small fraction of the population, which leads to them getting deprioritized.[^28] In reality, this overlooks a large share of the population: the World Health Organization estimates that about 1.3 billion people globally experience significant disability,[^34] and the Center for Disease Control reports that more than 1 in 4 US adults have a disability.[^13] These figures also exclude many temporary and situational impairments (e.g., injury or pregnancy) that can affect usability.\n\nAnother barrier is the belief that accessibility is too expensive or too time-consuming, especially when it is treated as a retrofit rather than a core design requirement. However, accessible features can be more costly to implement or lower quality when added at the end of production rather than integrated into the design process from the start.[^7]\n\nThese misconceptions are particularly impactful in environments where nobody is directly responsible for implementing accessibility features. Often embedded in HR departments, Compliance, or Product teams, accountability for accessibility is not always clearly communicated or adequately funded. This negligence can lead to failure on three distinct levels:\n\n1. **Ethical:** Building inaccessible products actively excludes people with disabilities from society.[^25]\n2. **Legal:** Exposing companies to lawsuits under frameworks like the Americans with Disabilities Act (ADA) or the European Accessibility Act can be costly and time-consuming.[^11]\n3. **Business:** Ignoring accessibility alienates a massive global market while preventing the design of better experiences for everyone.[^24]","sidebarKey":null,"isDrawer":false},{"id":"accessibility/intersectionsWithOtherValues#the-framework","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"the-framework","title":"The Framework","content":"To understand the complexity of these intersections, developers should look beyond compliance checklists and instead utilize three analytical lenses: Disability, Equity, and Universality.\n\n1. **Disability:** This lens centers on the lived experiences of people with disabilities and the social structures that contextualize them. It specifically asks how a feature acts as a barrier or an enabler for someone with visual, motor, cognitive, or auditory disabilities.\n2. **Equity:** This lens examines power dynamics and fairness. This perspective forces us to ask who is disproportionately burdened by a design decision and whether the technology reinforces existing systemic inequalities or instead helps address them.\n3. **Universality:** This lens broadens the scope to the general population. It reflects the “{curb-cut-effect}”: features originally designed to improve access for disabled users often end up benefiting many others as well. In digital contexts, captions, keyboard navigation, and clear layouts can help not only disabled users but also people in noisy environments, with temporary injuries, or in high-distraction settings.\n\nFor more information on these lenses, please refer to the [What is Accessibility](/accessibility/whatIsAccessibility) primer.\n\n![Venn diagram showing Disability, Equity, and Universality converging in inclusive design.](/assets/primer-photos/accessibility/intersections-with-other-values/accessibility-intersections-venn.png)\n\n_Figure: Venn diagram showing the three lenses used in this section (Disability, Equity, and Universality) and their convergence in inclusive design._","sidebarKey":null,"isDrawer":false},{"id":"accessibility/intersectionsWithOtherValues#intersections-with-accessibility-synergies-and-tensions","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"intersections-with-accessibility-synergies-and-tensions","title":"Intersections with Accessibility: Synergies and Tensions","content":"The following sections explore how accessibility overlaps with values like privacy, security, and usability. Each intersection involves both synergies and tensions. While the examples are not exhaustive, they highlight the key trade-offs and opportunities that technologists must navigate.\n\n### Accessibility & Privacy\n\nMany assistive technologies function by collecting sensitive data to adapt to a user’s needs, leading to a clash between accessibility and the user’s privacy. This intersection thus raises a question: \"What is the privacy cost for access?\"\n\nThrough the Disability lens, we see that many assistive technologies require personal data to work as intended. This creates an equity imbalance, forcing users with disabilities to give up more privacy than the general population just to participate. Accessible privacy controls can reduce some of this burden by making consent, permissions, and settings easier to understand and manage. However, they do not fully eliminate the tradeoff, because some assistive tools still depend on collecting sensitive data that non-disabled users may never have to disclose.\n\n- **Tension:** Some assistive technologies depend on highly sensitive inputs. Voice assistants rely on microphone access and wake-word systems, which raise concerns about audio capture and downstream data handling. Eye-tracking and other adaptive systems can also involve biometric or behavioral data that may reveal sensitive information. This surveillance model also appears in personalized learning platforms, which monitor reading speed and error frequency to provide cognitive accessibility.[^5]\n  - Accessibility settings can also become part of a distinctive device profile, which means that even when a user’s name is unknown, combinations of fonts, settings, and device characteristics may still increase trackability.[^9]\n\n- **Synergy:** Privacy features and accessibility can reinforce each other when controls are understandable and usable. Clear permission prompts, readable privacy notices, and keyboard-accessible settings can improve cognitive accessibility of privacy controls and thus {promote-privacy-protection}.[^6]\n\n![Diagram showing a privacy-by-design model for a voice-control feature, including user choices about data collection, storage, and processing.](/assets/primer-photos/accessibility/intersections-with-other-values/privacy-by-design-voice-control.png)\n\n_Figure: This figure shows an idealized privacy-by-design model in which users can make clearer choices about how a voice-control feature collects, stores, and processes data._\n\n- **Takeaway:** Accessibility and privacy can be in genuine tension because some disabled users must share more data to access the same service. Better privacy design can reduce this burden, but it does not always erase the underlying tradeoff.\n\n### Accessibility & Security\n\nThis intersection describes the divergence and convergence between system security and user access. From a Disability lens, security features can cause friction for users with visual, motor, or cognitive impairments. These barriers create equity issues when disabled users face disproportionate difficulty accessing essential services in the name of protecting systems from fraud or abuse. Meanwhile, within the context of Universality, when inaccessible or confusing security features are redesigned to be intuitive and easy to use, they improve usability for disabled users and others, thus strengthening security for everyone.\n\n- **Tension:** Security features are frequently prime examples of inaccessible design. {CAPTCHA} (Completely Automated Public Turing test to tell Computers and Humans Apart) is a well-known example: visual challenges can be unusable for blind or low-vision users, while audio alternatives may still exclude some users or remain difficult in practice.[^33] Biometric authentication can also create barriers. Some users cannot reliably use fingerprint or facial recognition systems, which sheds light on the need for increasingly non-biometric alternatives.[^3]\n\n![Example of a CAPTCHA challenge interface.](/assets/primer-photos/accessibility/intersections-with-other-values/captcha-example.png)\n\n_Figure: Example of CAPTCHA._[^12]\n\n- **Synergy:** Good security design is accessible. For example, password managers are a powerful tool for security and accessibility, helping people to use more secure passwords without introducing extra cognitive strain. Security becomes more accessible when warnings, authentication steps, and recovery flows use plain language as well as screen-reader-compatible interfaces. This ensures all users, regardless of ability, understand risks and ultimately protect themselves.[^10] The same principle applies to newer, {recaptcha|less interaction-heavy bot defenses}: when systems decrease reliance on puzzle-style challenges and provide accessible fallback options, they can better protect both security and access.\n\n![Before-and-after comparison of a survey page, showing accessibility improvements such as clearer headings, grouped form fields, explicit labels, and a more readable results table.](/assets/primer-photos/accessibility/intersections-with-other-values/accessible-survey-before-after.png)\n\n_Figure: Example of a before and after of an accessible survey page. The latter shows accessibility improvements through clearer headings, grouped form fields, explicit labels, and a more readable results table._[^32]\n\n- **Takeaway:** When security tools are designed with accessibility in mind, they stop being barriers for disabled users and become clearer, safer protections for everyone.\n\n### Accessibility & Sustainability\n\nThis intersection connects environmental goals like low energy consumption with equitable access. From the Equity lens, sustainable design for low-bandwidth environments equitably benefits users in developing regions or low-income areas, who are disproportionately impacted by such infrastructure. From the Disability lens, we can see that this lightweight design is critical for assistive tech compatibility. However, some energy-saving features can harm access, creating tensions we must carefully design around. The Universality lens reveals a synergy as well, since efficient code leads to pages that load more quickly, which then increases usability.\n\n- **Tension:** Features designed to save energy can create accessibility barriers. A common example is power-saving modes on devices, which often work by lowering screen brightness and contrast. While this saves battery life (a sustainability goal), it can make the device completely unusable for users with low vision, forcing them to choose between a usable device and a charged one.[^26]\n\n![iPhone Low Power Mode screen showing dimmed brightness.](/assets/primer-photos/accessibility/intersections-with-other-values/iphone-low-power-mode.png)\n\n_Figure: Low power mode dims brightness on iPhone._[^23]\n\n- **Synergy:** Sustainable design can be inherently more accessible. Efficient, lightweight code is a sustainable practice that leads to faster-loading pages and reduced data consumption. This is an accessibility win for users on slow internet connections and benefits users with certain cognitive disabilities by providing a responsive experience.[^30]\n\n- **Takeaway:** Sustainability and accessibility can reinforce each other when we design lightweight, efficient systems. However, we must avoid choices that worsen visibility or usability for disabled users.\n\n### Accessibility & Usability\n\nAccessibility and usability are deeply intertwined. From the Disability lens, accessibility is a prerequisite for usability; a product that cannot be perceived or operated is, by definition, unusable for that person. The Equity lens demands we prioritize this, ensuring usability is not only measured by what makes technology usable for the majority. The Universality lens shows the payoff of how features designed for a specific disability enhance usability for all users in different contexts (e.g., bright sunlight or quiet environments), exemplifying the curb cut effect.[^8]\n\n- **Tension:** In some cases, the challenge is not only between accessibility and general usability, but also between different accessibility needs themselves. For example, over-labeling every element for screen readers can create verbose, cluttered audio output that is difficult to navigate.[^1] Similarly, adding too many specialized modes or controls for different disabilities into a single menu can be overwhelming for users with cognitive disabilities, paradoxically making the product less usable. In these cases, the issue is usually not accessibility itself, but inaccessible implementation or poor organization of accessibility features.\n\n**Case Study:** {accessibility-is-not-one-size-fits-all|Accessibility vs. Accessibility}\n\n- **Synergy:** Nearly all accessibility best practices are also usability best practices. High-contrast mode, designed for low vision, is now a universal feature for reducing eye strain or improving visibility in bright light. Clear layouts and plain language (for cognitive accessibility) help all users navigate more efficiently.[^21] Keyboard-only navigation (for mobility impairments) also benefits users who prefer not to use a mouse.\n\n- **Takeaway:** In most cases, designing for accessibility simply is designing for usability. Tensions usually arise only when accessible features are implemented in a cluttered or unstructured way.\n\n### Accessibility & Transparency\n\nThis intersection treats transparency not simply as disclosure, but as a form of comprehensibility. Information is only transparent if users can access it, perceive it, and actually understand what it means. From the disability lens, transparency fails when information is technically available but functionally inaccessible—for example, when a chart is visual-only, a video lacks captions or transcripts, or a permission request is written in dense technical language. In such cases, disclosure exists in form but not in practice.[^19] That is, transparency that is not comprehensible is not transparency at all. The Universality lens, furthermore, shows that designing for cognitive accessibility makes systems more transparent and trustworthy for all users, not just specialists.\n\n- **Tension:** A push for radical transparency can be at odds with cognitive accessibility. For instance, algorithmic explainability reports that output complex statistical models or raw data are transparent but completely inaccessible to most users, especially those with cognitive disabilities. In this case, transparency without interpretation creates confusion, not clarity.\n- **Synergy:** Transparency, when designed for the general public, is a core accessibility feature. Alt text for images makes visual information transparent to screen reader users. Video transcripts and captions do the same for audio content, making it transparent, searchable, and accessible. Using plain language to explain why a system needs a certain permission or how a feature works represents a strong synergy between transparency and cognitive accessibility.[^17]\n\n- **Takeaway:** True transparency is not just about exposing information. Rather, it requires accessible formats and plain language so that disabled and non-expert users can actually understand what is happening.\n\n### Accessibility & Accountability\n\nIntegrating accountability can enable users to support greater accessibility through opportunities for recourse. From the Disability lens, if a reporting form or grievance process is inaccessible (e.g., it uses complex jargon or has a visual CAPTCHA), accountability becomes impossible in practice. This is a failure of equity, as it systemically silences the users most likely to be harmed by design flaws. The Universality lens shows that building an accessible accountability process—using simple language, multi-modal inputs, and clear feedback—makes it easier for all users to report issues, strengthening the feedback loop for the entire system.\n\n- **Tension:** Mechanisms for accountability can themselves create barriers when formal reporting systems prioritize rigid procedures or inaccessible verification steps in the name of institutional reliability. Complex forms, hidden contact information, or grievance processes that require navigating legal jargon can, furthermore, make a system more procedurally defensible while simultaneously making it harder for disabled users to seek recourse.[^31]\n\n- **Synergy:** Accessible design is essential for enabling accountability. A clear, simple, and easy-to-find \"Report a Problem\" button is an accountability mechanism. Providing multi-modal options for reports (e.g., voice, text, or video) ensures that users with different disabilities can submit feedback. Furthermore, providing clear confirmation and follow-up messages in plain language (e.g., \"We received your report and are reviewing it\") makes the accountability loop itself accessible.[^2]\n\n- **Takeaway:** Without accessible feedback channels, accountability is hollow. Accessible reporting tools empower disabled users and improve systems for everyone.\n\n### Accessibility & Safety\n\nThis intersection balances protection from harm with open access. From the Disability lens, tensions run both ways. A safety feature, like content moderation, can censor vital disability discourse, while an accessibility feature can create a physical safety risk by exposing private data. The Equity lens asks who is being protected—automated safety tools often fail to protect marginalized users from harm while simultaneously censoring their non-normative content. The Universality lens reveals synergies, where safety tools designed for accessibility (like content warnings) give all users greater control over their experience.[^18]\n\n- **Tension:** Safety features can create significant accessibility barriers. Content moderation may erroneously flag content from disability communities (e.g., discussions of medical conditions or anatomy) as sensitive or harmful, censoring them. Conversely, an accessibility feature like \"read aloud\" for notifications could compromise a user's physical safety by exposing a private message in a public or unsafe space.[^14]\n\n- **Synergy:** Many safety features can also function as accessibility features. Content filters or “SafeSearch” modes can serve as important accessibility tools for some users with cognitive disabilities who want more control over what appears on screen. Clear, accessible block or mute buttons are also central safety tools, allowing users to avoid harassment. Trigger warnings are another example: they are a safety feature that can also support cognitive accessibility by allowing users to prepare for or skip distressing material.[^22]\n\n- **Takeaway:** When safety tools are transparent, adjustable, and accessible, features like content filters, block buttons, and warnings can better protect users while also giving them more control over their experiences.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/intersectionsWithOtherValues#conclusion","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"conclusion","title":"Conclusion","content":"By analyzing these intersections, we see that the tensions between accessibility and other values are often caused by resource constraints or oversights rooted in systemic barriers rather than inherent conflict. Though other values are sometimes prioritized over accessibility, more intentional design choices can help accomplish greater synergy between goals. When we apply the lenses of Equity, Disability, and Universality, we find that accessibility is not only compatible with privacy, security, sustainability, transparency, accountability, and safety—it is often the catalyst for achieving them more effectively.\n\nThese intersections demonstrate that accessibility is a foundational design principle that strengthens, rather than competes with, other values when it is built in from the start. Designers and policymakers who adopt the three-lens framework can better anticipate trade-offs, build in safeguards, and uncover synergies that benefit not only users with disabilities first, but also everyone else through the curb cut effect. Ultimately, accessibility is not only a legal or ethical requirement, but also a strategic pathway toward more human-centered technologies.\n\n[^1]: Abstracta. “5 Accessibility Heuristic Principles for Better UX.” _Abstracta_, n.d. https://abstracta.us/blog/accessibility-testing/accessibility-heuristic/.\n\n[^2]: Access Board, U.S. “Revised 508 Standards and 255 Guidelines.” _U.S. Access Board_, n.d. https://www.access-board.gov/ict/.\n\n[^3]: Alexiou, Gus. “Disfigurement Charity Exposes Rampant Exclusion By AI Facial Recognition Tools.” _Forbes_, February 27, 2025. https://www.forbes.com/sites/gusalexiou/2025/02/27/disfigurement-charity-exposes-rampant-exclusion-by-ai-facial-recognition-tools/.\n\n[^4]: Alharbi, Rahaf, John Tang, and Karl Henderson. \"Accessibility Barriers, Conflicts, and Repairs: Understanding the Experience of Professionals with Disabilities in Hybrid Meetings.\" In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 2023. https://doi.org/10.1145/3544548.3581541.\n\n[^5]: American Civil Liberties Union. “The Privacy-Invading Potential of Eye Tracking Technology.” _ACLU_, n.d. https://www.aclu.org/news/national-security/privacy-invading-potential-eye-tracking-technology.\n\n[^6]: American Foundation for the Blind. “The Talking Book.” _AFB_, n.d. https://afb.org/online-library/unseen-minority-0/chapter-10.\n\n[^7]: AudioEye. “Website Accessibility vs. Lawsuit Costs: Save Money Early.” _AudioEye_, n.d. https://www.audioeye.com/post/website-accessibility-vs-lawsuit-costs/.\n\n[^8]: Bai, Yang. “The Relationship between Website Accessibility and Usability: An Examination of U.S. County Government Online Portals.” _The Electronic Journal of e-Government_ 17, no. 1 (2019). https://academic-publishing.org/index.php/ejeg/article/view/666.\n\n[^9]: BrowserLeaks. “Font Fingerprinting.” _BrowserLeaks_, n.d. https://browserleaks.com/fonts.\n\n[^10]: Bureau of Internet Accessibility. “Ditch the Fancy Vocabulary for Accessible Language.” _BOIA_, n.d. https://www.boia.org/blog/ditch-the-fancy-vocabulary-for-accessible-language.\n\n[^11]: Bureau of Internet Accessibility. “The Robles v. Domino’s Settlement (And Why It Matters).” _BOIA_, n.d. https://www.boia.org/blog/the-robles-v.-dominos-settlement-and-why-it-matters.\n\n[^12]: Cloudflare. “How CAPTCHAs Work | What Does CAPTCHA Mean?” _Cloudflare_, n.d. https://www.cloudflare.com/learning/bots/how-captchas-work/.\n\n[^13]: Centers for Disease Control and Prevention. “Disability Impacts All of Us Infographic.” _CDC Disability and Health_, n.d. https://www.cdc.gov/disability-and-health/articles-documents/disability-impacts-all-of-us-infographic.html.\n\n[^14]: Future of Privacy Forum. “Contextualizing the Kids Online Safety and Privacy Act: A Deep Dive into the Federal Kids Bill.” _Future of Privacy Forum_, n.d. https://fpf.org/blog/contextualizing-the-kids-online-safety-and-privacy-act-a-deep-dive-into-the-federal-kids-bill/.\n\n[^15]: Gaggi, Ombretta. \"A Study on Accessibility of Google ReCAPTCHA Systems.\" In _Proceedings of the 2022 Workshop on Open Challenges in Online Social Networks_, 25–30. 2022. https://doi.org/10.1145/3524010.3539498.\n\n[^16]: Hofmann, Megan, Devva Kasnitz, Jennifer Mankoff, and Cynthia L. Bennett. \"Living Disability Theory: Reflections on Access, Research, and Design.\" In _Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility_. 2020. https://doi.org/10.1145/3373625.3416996.\n\n[^17]: Iwarsson, Susanne, and Agneta Ståhl. “Accessibility, Usability and Universal Design—Positioning and Definition of Concepts Describing Person-Environment Relationships.” _Disability and Rehabilitation_ 25, no. 2 (2003): 57–66. https://doi.org/10.1080/0963828021000007969.\n\n[^18]: Leça, Matheus de Morais, and Ronnie de Souza Santos. “Towards User-Focused Cross-Domain Testing: Disentangling Accessibility, Usability, and Fairness.” _arXiv_ (2025). https://arxiv.org/html/2501.06424v1.\n\n[^19]: LLYC. “Radical Transparency: How to Make the Most of Technology and Boost Stakeholder Dialogue.” _LLYC IDEAS_, n.d. https://llyc.global/en/ideas/radical-transparency-how-to-make-the-most-of-technology-and-boost-stakeholder-dialogue/.\n\n[^20]: Martín, Adriana, Alejandra Cechich, and Gustavo Rossi. “Accessibility at Early Stages: Insights from the Designer Perspective.” In _Proceedings of the International Cross-Disciplinary Conference on Web Accessibility (W4A ’11)_. New York: Association for Computing Machinery, 2011. https://doi.org/10.1145/1969289.1969302.\n\n[^21]: Nielsen, Jakob. “10 Usability Heuristics for User Interface Design.” _Nielsen Norman Group_, April 24, 1994. Updated January 30, 2024. https://www.nngroup.com/articles/ten-usability-heuristics/.\n\n[^22]: Palo Alto Networks. “Safe Search Enforcement.” _Palo Alto Networks Documentation_, n.d. https://docs.paloaltonetworks.com/advanced-url-filtering/administration/url-filtering-features/safe-search-enforcement.\n\n[^23]: Payette Forward. “Why Does My iPhone Keep Dimming? Here’s The Truth!” _Payette Forward_, n.d. https://www.payetteforward.com/why-does-my-iphone-keep-dimming-heres-truth/.\n\n[^24]: Retail TouchPoints. “The Cost of Inaccessibility: Businesses Lose More Than $6.9 Billion Annually.” _Retail TouchPoints_, n.d. https://www.retailtouchpoints.com/executive-viewpoints/the-cost-of-inaccessibility-businesses-lose-more-than-6-9-billion-annually/145764/.\n\n[^25]: Sustainability Directory. “What Are the Ethical Trade-Offs between Data Accessibility and the Environmental Cost of Storage?” _Sustainability Directory_, n.d. https://lifestyle.sustainability-directory.com/learn/what-are-the-ethical-trade-offs-between-data-accessibility-and-the-environmental-cost-of-storage/.\n\n[^26]: Sustainability Directory. “Why Is Accessibility Important in Design for Sustainability?” _Sustainability Directory_, n.d. https://lifestyle.sustainability-directory.com/question/why-is-accessibility-important-in-design-for-sustainability/.\n\n[^27]: TetraLogical. “Sustainable Accessibility in Complex Organisations: Organisational Realities.” _TetraLogical_, November 7, 2025. https://tetralogical.com/blog/2025/11/07/sustainable-accessibility-in-complex-organisations-organisational-realities/.\n\n[^28]: Texas A&M University. “Myths about Accessibility.” _Texas A&M University Accessibility Resources_, n.d. https://itaccessibility.tamu.edu/resources/myths_about_accessibility.html.\n\n[^29]: Thinking Autism Guide. “Understanding Competing Accessibility Needs.” _Thinking Autism Guide_, October 2018. https://thinkingautismguide.com/2018/10/acknowledging-and-accepting-competing-accessibility.html.\n\n[^30]: ThoughtLab. “Sustainable Web Design: How Your Website Can Help Save the Planet.” _ThoughtLab_, n.d. https://www.thoughtlab.com/blog/sustainable-web-design-how-your-website-can-help-s/.\n\n[^31]: W3C. “Accessibility, Usability, and Inclusion.” _Web Accessibility Initiative (WAI)_, n.d. https://www.w3.org/WAI/fundamentals/accessibility-usability-inclusion/.\n\n[^32]: W3C. “Before and After Demonstration: Overview.” _Web Accessibility Initiative (WAI)_, n.d. https://www.w3.org/WAI/demos/bad/.\n\n[^33]: W3C. “Inaccessibility of CAPTCHA: Alternatives to Visual Turing Tests on the Web.” _W3C Group Draft Note_, December 16, 2021. https://www.w3.org/TR/turingtest/.\n\n[^34]: World Health Organization. “Disability.” _WHO_, March 7, 2023. https://www.who.int/news-room/fact-sheets/detail/disability-and-health.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/intersectionsWithOtherValues#sidebar-curb-cut-effect","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"curb-cut-effect","title":"Curb Cut Effect","content":"A curb cut is the sloped section built into a sidewalk curb to make it easier for wheelchair users to move between the sidewalk and the street. The idea is that a design created for disabled users often ends up helping many others, too. In this guide, that same logic applies to digital design.\n\nFor more, please refer to the [Design Processes](/accessibility/designProcesses) primer.","sidebarKey":"curb-cut-effect","isDrawer":true},{"id":"accessibility/intersectionsWithOtherValues#sidebar-captcha","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"captcha","title":"Why Websites Use CAPTCHAs","content":"CAPTCHAs are typically used to reduce spam, credential stuffing, fake account creation, bulk scraping, and other forms of automated abuse.\n\nCAPTCHAs are designed to be specifically hard for bots to complete, which becomes increasingly more difficult as artificial intelligence improves. Many CAPTCHAs have historically been based on distorted text because this is an open problem in AI. CAPTCHAs have also been used to help improve technologies: [digitizing books and providing labels to datasets for machine learning](https://blog.goodaudience.com/how-we-all-helped-unknowingly-google-to-digitize-books-acb45bc65084).\n\nHere, the problem is not the security goal itself. Rather, the problem is that many CAPTCHA implementations block legitimate users, which is why more accessible alternatives and fallback options matter.\n\nPlease see [captcha.net](http://www.captcha.net/) for more about CAPTCHAs.","sidebarKey":"captcha","isDrawer":true},{"id":"accessibility/intersectionsWithOtherValues#sidebar-accessibility-is-not-one-size-fits-all","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"accessibility-is-not-one-size-fits-all","title":"Accessibility Is Not Always One-Size-Fits-All","content":"Accessibility is not always a matter of finding one solution that works for everyone. In some cases, different disabled users may have genuinely conflicting access needs. A feature that improves access for one group may create barriers for another. These are known as **access conflicts**.[^4]\n\nFor example, highly detailed screen-reader to make content accessible to blind users may produce overwhelming or inefficient audio output for others. Or, in a more specific example, the sound of a bus's door ramp being lowered to let on a wheel-chair user could cause someone with a syncope (fainting) disorder triggered by loud noises to faint, as Hofmann et al. describe from their lives in \"[Living Disability Theory: Reflections on Access, Research, and Design.](https://dl.acm.org/doi/10.1145/3373625.3416996)\"[^16]\n\nAs such, accessibility should not be understood as a single fixed standard or a universal design choice that automatically serves all disabled people equally. Instead, accessible design requires acknowledging trade-offs, offering meaningful alternatives where possible, and ultimately avoiding the assumption that one user profile represents disability as a whole.\n\nAs one disability advocate notes, efforts to make something “fully accessible” can become overly prescriptive and may actually end up erasing disabled people whose needs do not fit that model. Indeed, designers should approach it as an ongoing process of negotiating competing access needs with humility and care.[^29]","sidebarKey":"accessibility-is-not-one-size-fits-all","isDrawer":true},{"id":"accessibility/intersectionsWithOtherValues#sidebar-promote-privacy-protection","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"promote-privacy-protection","title":"Promote Privacy Protection","content":"For more, please see the [Consent](/privacy/consent) primer in the privacy section.","sidebarKey":"promote-privacy-protection","isDrawer":true},{"id":"accessibility/intersectionsWithOtherValues#sidebar-recaptcha","section":"accessibility","sectionTitle":"Accessibility","subsection":"intersectionsWithOtherValues","subsectionTitle":"Intersections with Other Values","anchor":"recaptcha","title":"Improved CAPTCHA Techniques","content":"Due to a number of problems with existing CAPTCHAs, including inaccessibility and poor efficacy, Google has been working to develop [improved forms of CAPTCHAs](https://developers.google.com/recaptcha/docs/versions) that record and analyze the behavior of website users to determine if they are human. Some of these upgrades to the system have been improvements to accessibility, allowing users with visual and/or hearing impairments to easily complete CAPTCHA tasks.\n\nHowever, these upgraded CAPTCHAs can continue to raise accessibility concerns. For example, Google's reCAPTCHA v2 asked users to click a checkbox, recording their actions as they did so. This task still renders these CAPTCHAs inaccessible for some groups. For example, visually impaired users often prefer keyboard navigation to the use of a mouse, which is more likely to be detected by the test as automated behavior. In addition, the fallback to these systems remains traditional CAPTCHAs, adding more layers of barriers rather than removing them.\n\nMore recent work by Google with their 'reCAPTCHA v3,' which tracks and analyzes user behavior _without_ the additional checkbox-marking challenge has been more successful in making CAPTCHA accessible. In their 2022 study, Gaggi found that there was no clear influence of visual impairment on ability to pass these new tests, with a success rate of 99.42% across all groups.[^15]","sidebarKey":"recaptcha","isDrawer":true},{"id":"accessibility/biasesInDesign#intro","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"intro","title":"Introduction","content":"One of the biggest challenges to accessible technology is cognitive biases, which are systematic patterns in how we think that can lead us to overlook certain users’ needs or make assumptions about how people interact with systems. As discussed in [What is Accessibility?](/accessibility/whatIsAccessibility), accessibility encompasses three interconnected dimensions: Universality, Disability, and Equity. Cognitive bias undermines accessibility across all three dimensions and can cause even well-meaning designers to unintentionally create barriers for users.\n\nCognitive biases shape design decisions by influencing how designers interpret user needs and system requirements. More generally, biases shape what we notice, prioritize, and whose needs we overlook. Accessibility requires recognizing how our instincts about what constitutes normal, intuitive, or good design are shaped by our own abilities and experiences. Understanding what bias is, which specific biases might appear, and how biases manifest can help us question those instincts. Bias recognition must precede bias prevention, because we cannot design accessible technology without first understanding how our assumptions shape design decisions.\n\nTo understand where these biases enter the design process, it is important to first understand three sets of conceptual ideas that explain why biases are persistent and harmful in design: 1. Mental models and familiarity, 2. The invisibility of bias, and 3. Exclusion.\n\n### Mental Models and Familiarity\n\nIndividuals create {mental-models} for how they expect technology to function based on past experiences. These expectations are judgmental heuristics, mental shortcuts that help people make quick decisions by drawing on what has worked before. Heuristics allow us to function efficiently when encountering large amounts of information and stimuli. In design contexts, aligning with well-established patterns and intuitive interactions can minimize confusion and improve usability.[^4] Because we inevitably draw from our own experiences when creating these patterns, however, we can unconsciously treat our own mental models as universal and risk creating systems that only work for people with backgrounds and abilities similar to our own. A hypothetical example of this mental model can be found {mouse|in most designers’ reliance on the mouse for navigation}.\n\nFor an activity that demonstrates how mental models vary even for simple everyday tasks, see the {draw-toast|Draw Toast exercise}.\n\n### The Invisibility of Bias\n\nBiases often operate unconsciously, making flawed decisions feel intuitively correct to the decision-maker. These mental shortcuts developed as evolutionary adaptations, helping early humans make quick judgments for survival.[^3] Cognitive psychologists have established that biases stem from unconscious, automatic processes that can enable fast and efficient decision-making.[^2] When we rely on familiar patterns or make assumptions about user capabilities, these choices are made {system|rapidly and intuitively}, rarely triggering self-doubt. In design contexts, studies show that designers frequently commit to concrete design concepts from the beginning, possibly ignoring alternative options.[^2]\n\nAdditionally, individuals often under-detect their own biases compared to those of others, causing a “bias blind spot”[^10] where people recognize bias in others more readily than in themselves. This creates a challenge in design contexts: we may acknowledge that the bias affects the field generally while remaining unaware of how it shapes our decisions. Even when we become aware that we have made an assumption, we may not recognize it as bias if the assumption aligns with professional training, industry standards, or widely accepted design principles.\n\n### Exclusion\n\nWhen biases go unexamined, designs will naturally center dominant groups—those who are most represented in design teams, user research, product marketing, and so on. The preferences, experiences, and assumptions of these groups can become the default, causing marginalized groups to be systematically overlooked over time. Inaccessible designs extend beyond individual inconvenience—they can prevent people from accessing essential services[^11], participating in civic life[^7], pursuing education[^6], or being able to work[^12]. These oversights compound into systemic barriers that reinforce existing inequities.\n\nThe biases described in the following section illustrate specific ways this exclusion may manifest in design decisions, from the patterns designers rely on to the assumptions they make about their users.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/biasesInDesign#understanding-key-biases-in-design","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"understanding-key-biases-in-design","title":"Understanding Key Biases in Design","content":"Here are some key cognitive biases that commonly affect design decisions, along with examples illustrating their impact. The term bias here refers to ones that a designer might be subject to that leads to inaccessible and poorly designed products. Each subsection includes a definition and the potential impact. Notably, the following biases is not a comprehensive list of biases—for more, see [this comprehensive list of biases](https://thedecisionlab.com/biases).\n\n### Automation Bias\n\nAutomation bias is the tendency for people to favor suggestions from automated systems over their own judgment, especially as these technologies become more integrated into daily decision-making. This bias can lead people to overlook errors, ignore contradictory information, or fail to question flawed outputs simply because they trust the perceived authority or objectivity of the system.\n\n### Availability Bias\n\nAvailability bias is a cognitive shortcut in which people often judge the likelihood or importance of an event based on how easily examples come to mind. This usually means that vivid, recent, or emotionally charged memories can disproportionately influence decision-making regardless of their relevance.\n\n### Choice Overload\n\nChoice overload occurs when individuals are confronted with too many options, causing them to experience decision paralysis. The abundance of alternatives can make it difficult to evaluate each option thoroughly, causing cognitive fatigue. As a result, users may struggle to make decisions, often leading to a sense of regret or doubt about their decision, even if the choice made was objectively good. If a poor choice was made due to the difficulty of evaluating alternatives, the user may be less likely to engage with the system in the future.\n\n#### Confirmation Bias\n\nConfirmation bias is when individuals seek out, interpret, or remember information in a way that supports their existing beliefs or opinions, often ignoring information that challenges them. This reduces objectivity and may perpetuate errors in judgment or evaluation.\n\n### Framing Effect\n\nThe framing effect describes how the choice of language, context, and perspective presented alongside information changes how it is perceived, often without the individual being aware of its influence. Situations might be presented with a focus on positive (gain) or negative (loss) aspects. Framing may cause individuals to make decisions based on context cues rather than the information itself, leading them to be dissatisfied with choices or even feel deceived by an interface.\n\n### Implicit Bias\n\nImplicit bias describes the subconscious associations made between certain attributes—such as race, gender, age, or ability—and particular traits or behaviors. Implicit biases can influence decisions in ways that perpetuate inequities, even among individuals who consciously reject discriminatory attitudes.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/biasesInDesign#important-notes-on-biases","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"important-notes-on-biases","title":"Important Notes on Biases","content":"### Biases rarely work in isolation.\n\nIt is often difficult to distinguish where one type of bias ends and another begins. For example, implicit bias about who uses technology might lead a designer to seek out certain user research participants (availability bias), which then reinforces their initial assumptions (confirmation bias). This overlap makes identifying specific biases in real-world scenarios challenging.\n\n### Biases aggregate across levels.\n\nThese biases can aggregate across levels, creating a compound effect.[^1] For example:\n\n- Individual level: a designer experiences availability bias and gets research from a specific demographic.\n- Team level: confirmation bias leads the team to interpret positive feedback as proof that the design works.\n- Organizational level: the company’s hiring practices result in homogeneous design teams who have similar blind spots.\n- Industry level: the success of the product shapes industry standards for “what works.”\n\nThis compounding effect helps explain why accessibility problems persist systematically. Individual awareness, while necessary, is not sufficient without addressing team processes, organizational structures, and industry norms. Biases, including ones beyond what we mention in this primer, may operate simultaneously across these levels.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/biasesInDesign#conclusion","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"conclusion","title":"Conclusion","content":"Inaccessible design due to cognitive biases is not inevitable. Rather, accessibility can be achieved by questioning what is “obvious” and addressing our biases. Although bias cannot be eliminated entirely, it is vital to encourage critical awareness and accountability in how design decisions can include or exclude different user groups. The goal is to build the habit of asking whose needs a decision may overlook. Recognizing bias is the first step; the next requires understanding how to translate this awareness into concrete design processes.\n\n_An update with real-world case study examples for the biases discussed in this primer will be available Fall ‘26._\n\n[^1]: Balarezo, Jose D., Nicolai J. Foss, and Bo Bernhard Nielsen. \"Organizational Learning: Understanding Cognitive Barriers and What Organizations Can Do about Them.\" _Management Learning_ 55, no. 5 (2024): 741–68. https://doi.org/10.1177/13505076231210635.\n\n[^2]: Bellman, Eric. “Amazon, to Win in Booming Rural India, Reinvents Itself.” 2018. _Wall Street Journal_. https://www.wsj.com/articles/amazon-to-win-in-booming-rural-india-reinvents-itself-11546196176.\n\n[^3]: “Charting the Evolutionary Roots of Cognitive Biases.” 2025. _Vanderbilt Law School_. https://law.vanderbilt.edu/charting-the-evolutionary-roots-of-cognitive-biases/.\n\n[^4]: “Cognitive Biases and Design Research: Using Insights from Behavioral Economics and Cognitive Psychology to Re-evaluate Design Research Methods.” In _Design and Complexity - DRS International Conference 2010_. 2010. https://dl.designresearchsociety.org/drs-conference-papers/drs2010/researchpapers/95/\n\n[^5]: \"Draw How To Make Toast: A Simple and Fun Introduction to Systems Thinking.\" n.d. http://www.drawtoast.com/.\n\n[^6]: “Inclusive Schools: Designing for Disability in Classrooms.” 2022. _HMC Architects_. http://hmcarchitects.com/blog/2020/06/12/inclusive-schools-designing-for-disability-in-classrooms/.\n\n[^7]: “Innovations in Accessible Elections – Final Report.” n.d. _Center for Civic Design_. https://civicdesign.org/avti/innovations-in-accessible-elections/.\n\n[^8]: Kahneman, Daniel. _Thinking, Fast and Slow_. 1st ed. New York: Farrar, Straus and Giroux, 2011.\n\n[^9]: Kannengiesser, Udo, and John S. Gero. “Design Thinking, Fast and Slow: A Framework for Kahneman’s Dual-System Theory in Design.” _Design Science_ 5 (2019): e10. https://doi.org/10.1017/dsj.2019.9.\n\n[^10]: Pronin, Emily, Daniel Y. Lin, and Lee Ross. \"The Bias Blind Spot: Perceptions of Bias in Self Versus Others.\" _Personality and Social Psychology Bulletin_ 28, no. 3 (2002): 369–81. https://doi.org/10.1177/0146167202286008.\n\n[^11]: Ravensbergen, Léa, Mathilde Van Liefferinge, Jimenez Isabella, Zhang Merrina, and Ahmed El-Geneidy. \"Accessibility by Public Transport for Older Adults: A Systematic Review.\" _Journal of Transport Geography_ 103 (2022): 103408. https://doi.org/10.1016/j.jtrangeo.2022.103408.\n\n[^12]: “Systemic Barriers in the Workplace: Disability Inclusion.” n.d. _Invisible Condition_. https://www.invisiblecondition.com/blog/systemic-barriers-in-the-workplace-disability-inclusion.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/biasesInDesign#further-reading","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"further-reading","title":"Further Reading","content":"- [Addressing Structural, Social, and Symbolic Exclusion of Disabled People \\- Anna Cechony, Ahmmad Brown, 2025](https://journals.sagepub.com/doi/10.1177/19367244251344530?int.sj-full-text.similar-articles.5)\n- [Considering cognitive biases in design: an integrated approach \\- ScienceDirect](https://www.sciencedirect.com/science/article/pii/S1877050924002746)\n- [Decolonising design in peacebuilding contexts \\- ScienceDirect](https://www.sciencedirect.com/science/article/pii/S0142694X21000120)\n- [Digital Accessibility by WeCo](https://theweco.com/)\n- [Invisible Women: Data Bias in a World Designed for Men by Caroline Criado Pérez | Goodreads](https://www.goodreads.com/book/show/41104077-invisible-women)\n- [Mitigating Cognitive Bias to Improve Organizational Decisions: An Integrative Review, Framework, and Research Agenda](https://www.researchgate.net/publication/385156312_Mitigating_Cognitive_Bias_to_Improve_Organizational_Decisions_An_Integrative_Review_Framework_and_Research_Agenda)\n- [Retention and Transfer of Cognitive Bias Mitigation Interventions: A Systematic Literature Study \\- PMC](https://pmc.ncbi.nlm.nih.gov/articles/PMC8397507/)\n- [Unpacking Dominant Design: A critical analysis of power and dominant discourse in Design](https://dl.designresearchsociety.org/cgi/viewcontent.cgi?article=1338&context=iasdr)\n- [What are Cognitive Biases? | IxDF](https://www.interaction-design.org/literature/topics/cognitive-biases)","sidebarKey":null,"isDrawer":false},{"id":"accessibility/biasesInDesign#sidebar-mental-models","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"mental-models","title":"Mental Models","content":"Internal representations of how individuals expect something to function based on past experiences. We may treat our own mental models as universal and therefore risk creating systems that only work for people with similar backgrounds and abilities to ours.","sidebarKey":"mental-models","isDrawer":true},{"id":"accessibility/biasesInDesign#sidebar-mouse","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"mouse","title":"Mouse Reliance for Navigation","content":"A designer who primarily uses a mouse may create an interface with hover-dependent menus and drag-and-drop functionality, assuming these interactions are intuitive. When they rely solely on their own mental model of interaction, they risk excluding users who navigate with keyboards, screen readers, or voice commands. This design solution is only effective for people with similar motor abilities and preferences, reflecting a bias in the designer’s mental model and thus overlooking users with needs that differ from the designer’s.","sidebarKey":"mouse","isDrawer":true},{"id":"accessibility/biasesInDesign#sidebar-draw-toast","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"draw-toast","title":"Draw Toast: A Mental Model Exercise","content":"Want to see mental models in action?\nTry this exercise: take 30 seconds to sketch how you make toast. Now compare your drawing with someone else’s. You’ll likely find significant variation, showing how differently we might conceptualize the same routine task.\n\nFrom the Draw Toast website: “\\[drawings range from\\] crisp and clear to cluttered and confused and from those that look at the whole system to those that pick out a single component. The point is to highlight the biases. No single drawing is complete or comprehensive. Each simply represents a point of view.”[^5]\n\nIn design, we might assume our toast-making process is universal, but users might have completely different mental models. Get step-by-step instructions for the activity at [http://www.drawtoast.com/](http://www.drawtoast.com/). ![Examples of the toast-making process](/assets/primer-photos/accessibility/biases/drawToast.png)\nFigure 1\\. Examples of the toast-making process.","sidebarKey":"draw-toast","isDrawer":true},{"id":"accessibility/biasesInDesign#sidebar-system","section":"accessibility","sectionTitle":"Accessibility","subsection":"biasesInDesign","subsectionTitle":"Biases in Design","anchor":"system","title":"System 1 and System 2 Thinking","content":"To understand why these automatic processes may dominate our thinking, we can look to Kahneman’s dual-system theory. The theory provides a two-system basis for human decision-making: System 1 for fast, intuitive thinking and System 2 for slow, deliberate reasoning. For further elaboration on the mechanisms of these two systems, see Kahneman’s _Thinking Fast and Slow_.[^8]\n\nIn design practices, System 1 drives efficiency but also fuels design fixation, where teams jump directly from “what this should do” to “how it should look,” relying on familiar solutions that feel intuitive.[^9] Studies show that design processes often institutionalize this speed through design catalogs, selection charts, and pattern libraries, which codify past solutions for rapid reuse.\n\n![Google’s Material Design icon library showing common UI icons.](/assets/primer-photos/accessibility/biases/iconLibrary.png)\nFigure 2\\. Google’s Material Design icon library showing common UI icons.\n\nThe figure above shows Google’s Material Design library, a famous example of a pattern library. Its Figma library has over 3.5 million users. Notice the search icon in the upper left. While widely recognized in the West, UX research has documented that users in some regions of India interpreted this icon as a ping-pong paddle instead of a search tool, highlighting how even standard library icons can lead to miscommunication across cultures.[^2] This demonstrates how accessibility is contextual, and the biases leading to a single design solution may not hold for all users.\n\nTools that effectively engage biases strengthen _reflexive_ and _reactive_ reasoning: fast, experience-based modes that can reduce cognitive effort but can also embed assumptions over time. Engaging System 2, the slower and reflective mode of reasoning, encourages designers to move beyond automatic responses and reassess past assumptions, decreasing (but not eradicating) the chance that design decisions lead to exclusion or oversights of diverse user needs.","sidebarKey":"system","isDrawer":true},{"id":"accessibility/designProcesses#intro","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"intro","title":"Introduction","content":"When designing a product, there are many possible routes from ideation to implementation. Design processes provide structure along these routes, offering methods for gathering user needs, testing ideas, and iterating on products through feedback.\n\nThoughtful design processes that take users' needs into account support **usability** and **accessibility**. Usability refers to the ease of access and use of a product or service, and accessibility includes the evaluation and implementation of accessibility standards.\nUnderstanding Accessibility Standards, which will be published in May 2026, will introduce key standards and legal requirements that designers consider. Integrating these standards throughout the iterative design process helps us discover accessibility barriers earlier and reflect critically on these products through an accessibility lens.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/designProcesses#core-design-concepts","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"core-design-concepts","title":"Core Design Concepts","content":"[Universal Design Principles](https://udl.berkeley.edu/universal-design) ensure that a product is designed for diverse users in terms of accessibility and usability. Its principles offer designers a lens to evaluate their design decisions. It is not a strict checklist that designers must follow every time, but a framework that identifies where a product may unintentionally exclude users. These principles include equitable use, flexibility in use, simple and intuitive use, perceptible information, tolerance for error, low physical effort, and size and space for approach and use.[^14]\n\nThe concepts below build on this foundation at the level of individual design decisions. The following concepts are not drawn directly from the Universal Design framework's principles, which describe what accessible design should achieve, but each connects to the principles in practice: Hick's Law operationalizes simple and intuitive use, while affordances and feedback translate equitable and error-tolerant design into concrete decisions.\n\n### Simplifying Choices (Hick’s Law)\n\nHick’s Law (or the Hick-Hyman Law) states that the time it takes to make a decision increases with the number and complexity of the choices.[^6]\n\nSimplifying choices improves efficiency and reduces cognitive load for users. When interfaces minimize unnecessary decisions, users can act more quickly and confidently. For accessibility, reducing cognitive demands especially supports users with ADHD, memory differences, or cognitive impairments.\n\n**Case Study:** {Simplifying-Choices}\n\nApplying Hick's Law reduces confusion and creates interfaces easier for users with cognitive differences. Thoughtful simplification prevents oversimplification—removing too many cues (e.g., physical buttons) can harm users who rely on tactile or patterned input.\n\n### Affordances and Signifiers\n\nAn **affordance** is a property of an object that, based on its design, suggests how it can be used. A **signifier** is a perceivable indicator that communicates affordances—it indicates how to perform those actions.[^9]\n\nClear affordances and signifiers make an interface intuitive and easier to use. Accessibility is improved when users can immediately perceive what actions are possible and how to execute them, even when their attention or sensory abilities vary.\n\n**Case Study:** {Universal-Symbols}\n\nStrong affordances and signifiers reduce learning time and prevent user errors, especially for disabled users, as clear cues ensure they can use the product confidently without trial-and-error.\n\n### Mapping\n\n**Mapping** is the connection between controls and functions, which should clearly match so that users can easily predict outcomes.[^9]\n\nIntuitive, well-mapped interfaces reduce errors and improve learnability. Mapping supports accessibility by allowing users to understand a system’s functionality without trial-and-error, whereas bad mapping may significantly confuse users.\n\n**Case Study:** {Mapping}\n\nGood mapping supports users with limited reasoning or memory by making systems more predictable.\n\n### Feedback\n\n**Feedback** is the communication of the results of an action. Systems should provide immediate and meaningful responses to user actions.\n\nFeedback confirms to users that actions have been received, so that users understand the status of both the system and their action. Strong feedback is an important part of accessible design, as it makes sure that users with cognitive or sensory differences can still perceive outcomes clearly.\n\n**Case Study:** {Feedback}\n\nClear multimodal feedback prevents user errors and supports accessibility across sensory differences.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/designProcesses#design-orientations-defining-who-were-designing-for","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"design-orientations-defining-who-were-designing-for","title":"Design Orientations: Defining Who We’re Designing For","content":"**Design orientations** describe how a project defines and prioritizes its intended users. Each orientation invites us to think about accessibility and inclusion in different ways, whether by focusing on a specific user group, aiming for universal design, or designing with marginalized communities first. These orientations shape what \"success\" means and whose experiences are centered.\n\nThese design orientations are not mutually exclusive. However, designers may begin creating a product for a specific group while following universal design principles, ultimately working toward a universally usable product. The intersections of these orientations and processes allow designers to address both specialized and wider user needs.\n\nThe following orientations describe different ways designers define and prioritize their users. **Target group**, **universal design**, and **inclusive design** are approaches a designer can deliberately choose. The **curb cut effect**, in contrast, is not an orientation but a potential outcome of these orientations, describing how designing for a specific group can end up benefiting a broader audience.\n\n### Target Group\n\nA **target group** requires a prioritization of who you are designing for based on their needs and contexts, with a specific **user population** in mind.\n\nThe importance of designing for a target group lies in meeting specialized needs more effectively than general-purpose design. While this approach focuses resources on specific users, it can also create innovations (i.e., the curb cut effect) that improve usability for broader audiences—though this benefit is not guaranteed.\n\nThis orientation helps ensure that a clearly defined group’s needs are met more effectively than they would be through general-purpose design.\n\n**Case Study:** {Target-Group}\n\n### Universal Design\n\n**Universal design** is a design that can be accessed, understood, and used to the greatest extent possible by all people regardless of their age, size, ability, or disability.[^4]\n\nUniversal design helps reduce the likelihood of excluding users by anticipating any potential variations between users at the early stages of design, ensuring that it is a consistent and accessible experience for everyone.\n\n**Case Study:** {Universal-Design}\n\n### Inclusive Design\n\nWhere universal design aims to create one solution that works for everyone from the start, **inclusive design** begins by identifying who is currently excluded and designing with them first. The key distinction from a target group approach is not just who is centered, but how: target group defines a population and designs for them, while inclusive design treats excluded users as active participants in the design process itself.\n\nGoing more in depth, inclusive design is a **human-centered** approach that considers the full range of human diversity—including ability, language, culture, gender, age, and other forms of difference.[^4] It often begins by focusing on a specific group whose needs are not fully met by existing systems, then extends benefits outward to create solutions usable by a broader range of users.\n| Design Approach | Description |\n| :--- | :--- |\n| User&#8209;centered&nbsp;design | Focuses on understanding the goals, preferences, wants, and needs of target users. This data is captured through market research and used to formulate product ideas and improve design iterations. |\n| Human&#8209;centered&nbsp;design | Moves beyond individual needs and considers how a product can impact society and the environment. Inclusive design draws from this tradition because it asks not just \"how does this user interact with the product\" but \"what broader conditions are excluding this person in the first place.\"[^13] |\n\n<!-- The weird &#8209; and &nbsp; are to make those stay on one line -->\n\n**Case Study:** {Inclusive-Design}\n\n### Curb Cut Effect\n\nThe **curb cut effect** is not a design orientation itself, but an outcome that can result from any of the above approaches. It illustrates how accessibility-focused design can extend benefits far beyond its target audience, showing how designing for a perceived edge case can improve experiences universally.  \n![Sketch of the curb-cut effect](/assets/primer-photos/accessibility/designProcesses/the-curb-cut-effect.png)  \n**Figure 1:** Sketch of the curb-cut effect (by Jono Hey, [sketchplanations.com](https://sketchplanations.com/the-curb-cut-effect)[^5])\n\nAfter World War II, disabled veterans in the United States advocated for curb cuts (short ramps from sidewalks to adjoining streets) to increase their access in the built environment. Advocacy eventually led to legislation mandating curb cuts in federally funded facilities.[^10] Curb cuts also benefit people pushing strollers or luggage, demonstrating that when we design for those at the margins, we often create better experiences for everyone.\n\n**Case Study:** {Closed-Captions}\n\nHowever, the curb cut effect is not always guaranteed. If a design solves a barrier that only applies to a very specific context, the benefits may not extend beyond that group. This is why understanding design orientations and using inclusive methods, as well as following accessibility standards, increases the likelihood of improving usability.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/designProcesses#design-process","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"design-process","title":"Design Process","content":"The next step is understanding how those core design concepts come to life in practice with **Design Processes**. The following design processes reflect varying degrees of user involvement, from conducting research on users to engaging users as collaborators in participatory design.\n\nIn the US, laws such as the ADA require that physical and digital environments be usable by people with disabilities, while standards like WCAG extend those requirements into more specific technical guidance. These represent the floor of accessible design, which is necessary but not sufficient for meaningful inclusion. For detailed information on specific legal requirements and accessibility standards like WCAG, see the Accessibility Standards primer.\n\nBeyond these standards, designers can engage a range of processes that bring users increasingly into the center of the design process.\n\n### User Research\n\n**User research** involves systematically studying target users to learn more about their behaviors, goals, and contexts. Some common methods of user research include:\n\n- **User Persona:** A fictional representation of a target user, based on research into their behaviors, attitudes, and goals.\n- **User Stories:** A short, informal description of a product feature from the perspective of the end-user, explaining who the user is, what they want to do, and why they want to do it.\n- **Contextual Inquiry:** Observing and interviewing users in their natural environments to understand how context is shaping their interactions.\n\nUser research moves assumptions into evidence-based insights, ensuring the designed product is empathy-driven and grounded in users' actual lived experiences.\n\n### Participatory Design\n\n**Participatory design** practices engage users as active collaborators throughout the design process by placing them in power over when and how systems are created for them. A key component of this practice is the continued involvement of **stakeholders**—users are not only consulted for feedback on a completed product, but are included throughout the design process.\n\nParticipatory design incorporates meaningful user participation and perspectives, especially for **underserved groups**, without granting full decision-making power.\n\n**Case Study:** {Participatory-Design}\n\n### Co-Design\n\n**Co-design** extends participatory design in user involvement by treating users not just as contributors, but as **equal partners** in shaping the solution. Users are actively engaged in defining scope and problems, generating prototypes and ideas, and making decisions along with designers and developers. Co-design allows participants to define what “problem,” “success,” and “meaningful use” look like—something participatory design may not fully involve users in.\n\nCo-design helps designers better understand user needs, increasing user engagement and promoting collaboration among stakeholders, all leading to more user-centered, engaged, creative, and innovative outcomes. Co-design is especially valuable in contexts where standard research methods may exclude participants, such as **children** or older adults with complex communication or cognitive needs.\n\n**Case Study:** {Co-designing-with-Kids-with-Complex-Needs|Co-designing with Kids with Complex Needs}\n\n### Arnstein’s Ladder of Participation\n\n{Arnsteins-Ladder|Arnstein’s Ladder} is a framework that clarifies the difference between token participation and real user power in decision-making, helping designers assess the depth and authenticity of their user involvement.","sidebarKey":null,"isDrawer":false},{"id":"accessibility/designProcesses#conclusion","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"conclusion","title":"Conclusion","content":"Design processes shape not only how a product is made, but also whose needs and experiences guide the product’s design. Throughout this primer, we saw that meeting legal requirements is only a starting point to designing for meaningful accessibility; it requires clear design orientations, grounding decisions in user research, and incorporating participatory and co-design approaches.\n\nNo single design process guarantees accessibility. Instead, accessibility grows out of an ongoing commitment to involve users early and iteratively. Ultimately, accessible design is not a technical add-on but a reflection of whose voices and experiences are included in shaping our products and systems.\n\n[^1]: Arnstein, S. R. (1969). \"A Ladder of Citizen Participation.\" _Journal of the American Institute of Planners_, 35(4), 216–224. https://doi.org/10.1080/01944366908977225\n\n[^2]: Birkeland, J. (2023, June 5). \"Behind the design: Headspace.\" _Apple Developer_. https://developer.apple.com/news/?id=fkfnhq8u\n\n[^3]: Center for Universal Design, NC State University College of Design. (1997). _The Principles of Universal Design_. https://design.ncsu.edu/research/center-for-universal-design/\n\n[^4]: Centre for Inclusive Design & PwC Australia. (2019). _The Benefit of Designing for Everyone_. https://centreforinclusivedesign.org.au/wp-content/uploads/2021/05/inclusive-design-report-digital-160519.pdf\n\n[^5]: Hey, Jono. (2021). \"The Curb-Cut Effect.\" Sketchplanations. https://sketchplanations.com/the-curb-cut-effect\n\n[^6]: Hick, W. E. (1952). \"On the Rate of Gain of Information.\" _Quarterly Journal of Experimental Psychology, 4_(1), 11–26. https://lawsofux.com/hicks-law/\n\n[^7]: Microsoft. (2021). \"Seeing AI App Launches on Android.\" https://blogs.microsoft.com/accessibility/seeing-ai-app-launches-on-android-including-new-and-updated-features-and-new-languages/\n\n[^8]: Microsoft. (n.d.). \"Xbox Adaptive Controller.\" https://www.xbox.com/en-US/accessories/controllers/xbox-adaptive-controller\n\n[^9]: Norman, D. A. (2013). _The Design of Everyday Things: Revised and Expanded Edition_. Basic Books. https://jnd.org/affordances-and-design/\n\n[^10]: Peterson, Julia. (2015). \"Smashing Barriers to Access: Disability Activism and Curb Cuts.\" _National Museum of American History_. https://americanhistory.si.edu/explore/stories/smashing-barriers-access-disability-activism-and-curb-cuts\n\n[^11]: Scribewire. (2021). _The History of Closed Captioning_. https://scribewire.ca/blog/the-history-of-closed-captioning\n\n[^12]: Shahi, S. (2019). _Co-designing with kids with complex needs_. Inclusive Design Research Centre, OCAD University. https://community-led.design/case-studies/co-designing-with-kids-with-complex-needs/\n\n[^13]: Shark Design. (2024). _User-centered design vs. human-centered design: Which approach is right for you?_ https://sharkdesign.com/blog/user-centered-design-vs-human-centered-design-which-approach-is-right-for-you/\n\n[^14]: UC Berkeley Center for Teaching & Learning. (n.d.). _Universal design_. https://udl.berkeley.edu/universal-design\n\n[^15]: Ullal, A., Tauseef, M., Watkins, A., Juckett, L., Maxwell, C., Tate, J., Mion, L. C., & Sarkar, N. (2024). An iterative participatory design approach to develop collaborative augmented reality activities for older adults in long-term care facilities. _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. https://doi.org/10.1145/3613904.3642595","sidebarKey":null,"isDrawer":false},{"id":"accessibility/designProcesses#sidebar-simplifying-choices","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"simplifying-choices","title":"Simplifying Choices","content":"The settings menus on the Nintendo Switch group options into clear categories (e.g., “Controllers and Sensors,” “System,” “Accessibility Features”). The interface shows only a manageable set of choices at each step. By reducing choice overload, the system supports users who may struggle with dense information or small motor movements.","sidebarKey":"simplifying-choices","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-target-group","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"target-group","title":"Target Group","content":"[Seeing AI](https://blogs.microsoft.com/accessibility/seeing-ai-app-launches-on-android-including-new-and-updated-features-and-new-languages/), a mobile app developed by Microsoft, was designed specifically for people with visual impairments, blindness, or low vision.\n\n**Implementation**: Seeing AI uses a smartphone’s camera and on-device AI to narrate the world to its users.[^7] There are different channels for tasks, from short text (reading texts as they appear), documents (guiding the user to frame a full page, then reading it with structure), product barcodes (using beeps to help locate the barcode, then announcing product information), currency recognition, color detection, and even a scene mode that describes the overall scene.\n\n**The impact**: The focused design orientation ensures its target group, blind and low-vision users, can gain greater independence, from reading printed documents and menus to identifying products and understanding their surroundings. Although optimized specifically for this group, many of its features benefit a broader audience. For example, anyone can use it to read text in low-light conditions or quickly scan labels. Designing for a target group is therefore not only essential for meeting specialized needs but can also create innovations that improve usability for all users.","sidebarKey":"target-group","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-inclusive-design","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"inclusive-design","title":"Inclusive Design","content":"The [Microsoft Xbox Adaptive Controller](https://www.xbox.com/en-US/accessories/controllers/xbox-adaptive-controller), a controller that supports gamers with limited mobility, exemplifies inclusive design as it dives into identifying flaws in traditional controllers for its specific audience and designing a more inclusive alternative.\n\n**Implementation**: Microsoft partnered directly with disabled gamers and occupational therapists. Through participatory design sessions, they identified barriers in traditional controllers—button reach, grip requirements, and simultaneous input coordination. The Adaptive Controller introduced programmable large-format buttons, multiple input ports for assistive devices, and flexible configurations based on individual needs.[^8]\n\n**Impact**: The product dramatically expanded access to gaming for players with diverse motor abilities. At the same time, the modular, customizable design improved gaming experiences for users without disabilities who benefit from alternative control schemes. This demonstrates inclusive design’s core approach: centering marginalized users creates improvements that scale outward.","sidebarKey":"inclusive-design","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-mapping","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"mapping","title":"Mapping","content":"![iPhone volume and brightness sliders](/assets/primer-photos/accessibility/designProcesses/mappingExample.png)\n\nThe iPhone volume and brightness sliders are an example of Natural Mapping—sliding up increases volume while sliding down reduces it A vertical slider for volume control provides natural mapping—sliding up increases the volume, sliding down reduces it. This mapping corresponds to spatial metaphors users already understand.\n\nProviding clear visual labels and multimodal cues (e.g., haptic or audio feedback) helps ensure users with varying cultural or sensory backgrounds can understand the control’s behavior.","sidebarKey":"mapping","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-feedback","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"feedback","title":"Example of Feedback","content":"![WhatsApp Messages showing different checkmarks for sent, delievered, and read](/assets/primer-photos/accessibility/designProcesses/feedbackExample.png)\n\nOn Whatsapp, Visual checkmark indicators serve as feedback for different states of the text after the user sends it. When a user sends a message, the app provides multimodal feedback, including a tactile vibration, a visual checkmark indicator, and an auditory notification that the user can turn on. This multimodal feedback makes sure that users, including those who may be blind, hard of hearing, or easily distracted, can confirm that their text is sent through and understand the system status clearly.","sidebarKey":"feedback","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-universal-symbols","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"universal-symbols","title":"Examples of Affordance and Signifiers","content":"![on and off power switch symbols](/assets/primer-photos/accessibility/designProcesses/affordanceExample.png)\nExample of an Affordance—showing that the button can be turned on and off\n\n![Active green submit button and inactive grey submit button](/assets/primer-photos/accessibility/designProcesses/signifierExample.png) Example of Signifiers—showing that a green button means Submit, while a gray button means an action is not yet available Consider the universal “power” symbol—a circle (O) for “off” and a line (I) for “on.”\n\nThese shapes indicate the affordance (a button can be pressed) and serve as signifiers that communicate its state. Similarly, a green button on a digital form signifies “Submit” or “Continue,” while a gray, disabled button signifies that an action is not yet available. These digital affordances help users with limited vision or cognitive differences when paired with text labels or screen reader cues.","sidebarKey":"universal-symbols","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-universal-design","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"universal-design","title":"Universal Design","content":"[Headspace](https://organizations.headspace.com/blog/leading-in-disability-inclusion-headspace-recognized-as-a-2024-best-place-to-work-by-dei), a meditation and mental health app, was designed with universal accessibility in mind from the start. The design team prioritized creating an experience that would work for users with varying digital literacy, cognitive abilities, and stress levels—recognizing that their target audience (people seeking mental health support) includes many who experience cognitive overload.\n\n**Implementation**: Headspace uses a calm, minimalist interface—soft colors, simple typography, generous white space, and clean iconography—combined with intuitive, consistent navigation patterns. The app's design accommodates a wide range of users, from individuals with low digital literacy to neurodivergent users to people managing stress, by making meditation and mindfulness content easy to find and use.\n\n**Impact**: [Headspace’s design](https://developer.apple.com/news/?id=fkfnhq8u) intentionally minimizes friction and maximizes usability for everyone. A universally accessible experience benefits all users—not just a select group—and reduces the likelihood of excluding potential users with varying needs or limitations.[^2]","sidebarKey":"universal-design","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-closed-captions","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"closed-captions","title":"Closed Captions","content":"Closed captions were originally designed for people who are deaf or hard of hearing. But today, they benefit a much broader audience: people watching videos in noisy environments (like gyms or airports), non-native speakers learning a language, viewers in quiet spaces (like libraries or late at night), and anyone who prefers to read along while watching. This demonstrates the curb cut effect—a feature designed for accessibility becomes useful for many.[^11]","sidebarKey":"closed-captions","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-participatory-design","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"participatory-design","title":"Participatory Design Approach to Develop Collaborative Augmented Reality Activities for Older Adults in Long-Term Care Facilities","content":"This study explores head-mounted display augmented reality (HMD-AR) as a potential strategy to facilitate social connectedness and mitigate loneliness among older adults in Long Term Care (LTC) residences. Using a participatory design approach, the feedback from the focus group is incorporated through rounds of requirements identification, prototyping, and evaluation. The focus group involves older adults, family members, and LTC staff. Through participatory design, developers designed AR activities such as checkers and virtual fireplace decoration, and tested iterative prototypes.\n\nThe process demonstrated how active user participation and feedback improve adaptability, accessibility, and engagement for underserved groups. While participatory design has meaningful participation of its user group, they do not co-own the design decisions.[^15]","sidebarKey":"participatory-design","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-co-designing-with-kids-with-complex-needs","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"co-designing-with-kids-with-complex-needs","title":"Co designing with Kids with Complex Needs","content":"In this case, researchers collaborated with children with multiple disabilities to design digital learning tools that reflect their daily realities. The co-design process included using tactile materials, visual storytelling, and simplified interaction models to help children express preferences and test early prototypes. Rather than interpreting the children’s needs from observation alone, designers co-created alongside them and their caregivers. The co-design process allowed designers to better match users’ sensory and cognitive needs while also fostering more engagement and a sense of pride among participants.\n\nThis shared creative authority distinguishes co-design from participatory design. Users here, the children and caregivers, are not just informing and testing the design to improve it, but also shaping the problem, the prototypes, and their meaning together with researchers.[^12]","sidebarKey":"co-designing-with-kids-with-complex-needs","isDrawer":true},{"id":"accessibility/designProcesses#sidebar-arnsteins-ladder","section":"accessibility","sectionTitle":"Accessibility","subsection":"designProcesses","subsectionTitle":"Design Processes","anchor":"arnsteins-ladder","title":"More on Arnstein’s Ladder of Participation","content":"Sherry R. Arnstein’s 1969 Ladder of Citizen Participation highlights a key idea in the design process and accessibility: “There is a critical difference between going through the empty ritual of participation and having the real power needed to affect the outcome of the process.\"[^1] Arnstein outlines eight layers of participation, grouped into three levels:\n\n- **Nonparticipation**: Manipulation, Educating\n- **Tokenism**: Informing, Consultation, Placation\n- **Citizen Power**: Partnership, Delegated Power, Citizen Control\n\nTokenism describes situations where users are asked what they think, but decision-making remains in the power of someone else. This allows power-holders to claim that “all sides were considered,” even though only some sides ultimately benefit.\n\nArnstein’s Ladder helps us assess where our design processes actually sit on the ladder. This model is helpful for distinguishing the intentions behind user research, participatory design, and co-design—ensuring that participation is actually impactful rather than symbolic.","sidebarKey":"arnsteins-ladder","isDrawer":true},{"id":"automatedDecisionMaking/bias#what-is-bias","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"what-is-bias","title":"What Is Bias?","content":"Automated decision-making (ADM) systems are increasingly embedded in day-to-day life: the Stanford AI index reports that as of 2024, 78% of businesses were working to incorporate AI into operations, that GPT 4 can outperform human doctors in key clinical tasks, and that as of 2025, Waymo provides over 150,000 rides in self-driving cars per week.[^5]\n\nDecisions made by ADM systems, like human decision-making, can be flawed. One of the primary ways in which decisions can be flawed is due to bias. **Bias** is any systemic process that discriminates against or favors a person or group over another based on stereotypical or inaccurate assumptions of the person or group.\n\nUltimately, society benefits from processes and decision makers that minimize bias. Understanding the ways in which ADM systems can be biased and the harms that may arise from those biases allows one to make informed decisions about whether imperfect ADM outcomes or imperfect human decision-making would ultimately lead to minimizing bias.\n\nFor example, judges are imperfect decision makers. They often let high-risk defendants out on bail; some are prone to being overly strict, all get hungry. Legal scholar Cass Sunstein argues that when evaluating the use of an AI risk prediction model in the justice system, the model should be evaluated against the imperfection of the current system.[^11] Evaluating systems of decision-making and attempting to implement systems that minimize biases can help reduce harms caused by bias.\n\n**Types of Potential Harm from Bias in Automated Decision-Making Systems:**  \nBiases in ADM systems can lead to material harm to people's health and livelihood making it essential that ADM system creators and implementers are aware of the potential biases that may arise.\n\n**Allocative Harm** occurs when biased systems deny individuals or groups access to resources, opportunities, or benefits.\n\n**Representational Harm** arises when certain people or groups are stigmatized, stereotyped, or underrepresented, shaping how they are perceived and valued in society.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/bias#sources-of-bias-in-automated-decision-making-systems","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"sources-of-bias-in-automated-decision-making-systems","title":"Sources of Bias in Automated Decision-Making Systems","content":"We can organize sources of bias in ADM systems into three categories based on where they occur in the ADM system development life cycle: bias in data curation, bias in model creation and bias in model testing and deployment.\n\nData Curation occurs when data is gathered, labeled and collected into data sets. Data sets are then employed in model creation to help train models to have proper responses to stimuli. Once models are created, they are then tested on new data sets to ensure that they respond appropriately and accurately before they are then deployed for their intended use. Biases can be introduced throughout all the stages of this process.\n\n![Figure 1 description: A flowchart diagram illustrating sources of bias in machine learning data pipelines, flowing left to right. It begins with a \"data generation\" box feeding into a globe icon labeled \"world,\" where red bold text marks \"HISTORICAL BIAS.\" An arrow leads to a \"population definition & sampling\" step connecting to a crowd icon labeled \"sample,\" marked with red bold text \"REPRESENTATION BIAS.\" From there, a \"measurement\" step feeds into a cylinder labeled \"dataset,\" marked with red bold text \"MEASUREMENT BIAS.\" The dataset then flows through \"preprocessing, train/test split\" and branches into two outputs: \"training data\" and \"test data.\" A second parallel lower pathway runs from \"population definition & sampling\" through \"measurement\" and \"preprocessing, train/test split\" to a final output labeled \"benchmarks.\" The three bias types — Historical Bias, Representation Bias, and Measurement Bias — are highlighted in red throughout to indicate key failure points in the pipeline.](/assets/primer-photos/ADM/bias/biasOverSystemLifecycle.png)\n\n**Figure 1:** Bias in Data Curation and Model Generation (adapted from Suresh et al.)_[^12]_","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/bias#bias-in-data-curation","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"bias-in-data-curation","title":"Bias in Data Curation","content":"Bias can be introduced at the very start of the ADM system lifecycle during **data curation**. Data curation is how data is generated, collected, represented, and measured. **Figure 2** below is a visual representation of bias in data curation, the first half of the ADM system lifecycle shown in **Figure 1**.\n\n![Figure 2 description: A left-to-right flowchart showing sources of bias in machine learning data pipelines. Starting from \"data generation\" and a globe (\"world\"), the flow passes through population sampling, measurement, and preprocessing stages before splitting into \"training data,\" \"test data,\" and \"benchmarks\" outputs. Three bias types are labeled in red at key stages: \"Historical Bias\" at data generation, \"Representation Bias\" at population sampling, and \"Measurement Bias\" at measurement.](/assets/primer-photos/ADM/bias/biasInDataGeneration.png)\n\n**Figure 2:** Bias in Data Generation (adapted from Suresh et al.)[^12]\n\nDuring data curation, developers' decisions about what data to include, how to define variables, and which populations to represent all reflect **structural** factors, such as existing social hierarchies, institutional priorities, and historical inequities. The \"structure\" that shapes bias can be defined as the broader social, political, and economic context in which the data is produced.\n\nBias in **data curation** displays itself through **historical bias, representation bias, and measurement bias.** Each of these mechanisms describes a different way that social inequality, exclusion, or distortion can enter a dataset.\n\n### Historical Bias\n\n**Historical bias** occurs when a model's data reflects societal stereotypes. Even if a dataset perfectly reflects reality, it can still be historically biased because it reflects prejudices that exist in the real world.\n\nOne example of historical bias is gender bias due to large language models being trained on {data-that-reflects-historical-sexism}.[^4]\n\n#### Statistical vs. Taste-Based Discrimination\n\nHistorical bias creates the distinction between statistical discrimination and taste-based discrimination.\n\n**Statistical discrimination** occurs in a limited-information environment where agents form expectations based on factors often associated with membership in a certain group.\n\n**Taste-based discrimination** occurs due to an agent's prejudiced group preferences.[^3]\n\n### Representation Bias\n\n**Representation bias** occurs when a model's dataset underrepresents or omits certain groups or attributes. This representation disparity can occur due to skewed sampling, unbalanced inclusion criteria, or limited diversity in the data sources.\n\n**Sampling Bias:** When the method of data collection systematically excludes members of the target population.\n\n- **Example:** A dataset that does not include hyphenated last names being used to train a name-generation model\n\n**{Underrepresentation-bias}:** When the data available on the target population is insufficient to properly train the model.\n\n- **Example:** A dataset being used to train a model identifying cancer from MRI scans that does not have enough scans of rare cancers.\n\n### Measurement Bias\n\n**Measurement bias** arises when a variable is chosen to act as a proxy for prediction of an abstract concept, but the proxy is inherently biased. For example, past credit scores are used as a proxy for fiscal responsibility, but minority groups have been historically denied opportunities to build good credit. If the proxies being used by a model are based on a history of discrimination, they can introduce bias into the model.\n\n- **Example:** A case study on {healthcare-risk-assessment} algorithms demonstrates how using money spent on healthcare as a proxy variable for health need can be biased against groups that systematically spend less on healthcare.[^10]","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/bias#bias-in-model-creation","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"bias-in-model-creation","title":"Bias in Model Creation","content":"### Model Creation\n\nBias can also emerge in model creation: the definition of the model architecture and the training of the weights. Engineers may bring their own biases into the design process, deciding which trade-offs to prioritize and which populations to optimize their model for. Figure 3 below is a visual representation of bias in model creation, the second half of the ADM system lifecycle shown in Figure 1\\.\n\n![Figure 3 description: A left-to-right flowchart showing sources of bias in the model training and deployment stage of a machine learning pipeline. Training data feeds into \"model learning\" (labeled \"Learning Bias\" in red), while test data and benchmarks feed into \"evaluation\" (labeled \"Evaluation Bias\" in red). Both, along with a \"model definition\" step (labeled \"Aggregation Bias\" in red), feed into a central \"model\" block. The model then runs to produce a grid-style \"model output,\" which passes through post-processing and human interpretation before reaching a globe icon representing the \"world,\" with this final stage labeled \"Deployment Bias\" in red.](/assets/primer-photos/ADM/bias/biasInModelTraining.png)\n\n**Figure 3:** Bias in Model Creation (adapted from Suresh et al.)[^12]\n\n### How Does Bias Appear in Model Creation?\n\nThe modeling stage involves countless design choices, so each of these choices can introduce or amplify inequities depending on what the model prioritizes and who it is optimized to serve. Bias in model creation typically appears through four main mechanisms: **aggregation bias, learning bias, evaluation bias, and deployment bias**.\n\n### Aggregation Bias\n\n**Aggregation bias** occurs when distinct groups are treated as homogeneous in a model, even though these populations differ in meaningful ways. Aggregation bias will often lead the model's performance to be unequal across subgroups. In some cases, it can lead to a model that is not optimal for _any_ group, or a model that only works properly for the dominant population.\n\n### Learning Bias\n\n**Learning bias** occurs when a model's encoded priorities make it practical to overlook a minority group. An objective function to evaluate a model's performance, such as accuracy or mean squared error, might be programmed so that a model can succeed by ignoring or excluding a minority group while still performing well according to the objective function.\n\n- **Example:** say a model is given the objective to get the highest percent correct answers guessing people's ages from photographs. If the model is given 100 subjects to attempt to guess their ages and guesses all 90 White people correctly and all 10 Black people incorrectly, the overall level of accuracy would be 90%. Looking just at the model's accuracy, it may seem like the model is performing well. But, when you look closer, it becomes clear that the model is unable to correctly predict age for _any_ Black people. This model is optimizing for the majority population, ignoring minority group errors and exhibiting learning bias.\n\n- **Example:** One {case-study-by-MIT-researchers} found learning bias and historical bias to be present in a now-defunct Amazon hiring software.[^8] This demonstrates how two or more biases can occur simultaneously.\n\n### Evaluation Bias\n\n**Evaluation bias** occurs _after_ model creation, when evaluation benchmark datasets are input to train a model, but they do not represent the user population. Many ADM systems are evaluated on narrow or sanitized datasets, which creates a misleadingly positive picture of model performance. When models perform positively on sanitized datasets, engineers do not work to further improve model performance. When such models are then deployed in more diverse, uncontrolled environments, they perform worse than developers intended.\n\nResearchers have found that facial expression recognition datasets used in model evaluation do not contain the diversity of faces that exist in the target population. This has caused companies to output models that do not correctly recognize diverse faces with a high level of accuracy.[^2] This was discovered by Joy Buolamwini in her MIT thesis {Gender-Shades}.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/bias#bias-in-model-deployment","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"bias-in-model-deployment","title":"Bias in Model Deployment","content":"### Deployment Bias\n\n**Deployment bias** arises after _model creation_, when models are used to solve problems they were never designed to address or towards a population on whose data they were never trained. Even if the model is well-trained and fairly evaluated, it can still produce harmful and discriminatory outcomes when used in an improper way or in a population it was never intended to be used on.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/bias#conclusion","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"conclusion","title":"Conclusion","content":"| Type of Bias        | Definition                                                                                                                                    | When It Occurs                               |\n| :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------- |\n| Historical Bias     | When data reflects pre-existing societal inequalities or stereotypes                                                                          | Before data collection and sampling          |\n| Representation Bias | When the dataset fails to represent the target population                                                                                     | During data collection and dataset formation |\n| Measurement Bias    | When a variable is chosen to act as a proxy for the prediction of an abstract concept, but the proxy does not properly represent the concept. | During data labeling/feature engineering     |\n| Aggregation Bias    | When distinct groups are treated as homogeneous                                                                                               | During model design and feature aggregation  |\n| Learning Bias       | When a model's encoded priorities make it practical to overlook a minority group                                                              | During model learning                        |\n| Evaluation Bias     | When evaluation datasets don't reflect the diversity of user population                                                                       | During testing/validation                    |\n| Deployment Bias     | When models are used in contexts different from those they were designed for                                                                  | During deployment/implementation             |\n\nBias can occur at all stages of the automated decision-making system lifecycle, ranging from how data is generated and measured, how models are optimized and evaluated, to where models are deployed. When creating a model, it is essential to critically examine the data input into a model during training and evaluation, as well as the assumptions you approach the project with. By having an awareness of the ways in which ADM systems become biased and examining our own assumptions, we can build ADM systems that serve target populations effectively and fairly.\n\n[^1]: Barocas, Solon, and Andrew D. Selbst. \"Big Data's Disparate Impact.\" _California Law Review_ 104, no. 3 (2016): 671\\. [https://doi.org/10.15779/Z38BG31](https://doi.org/10.15779/Z38BG31).\n\n[^2]: Buolamwini, Joy, and Timnit Gebru. \"Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.\" _Proceedings of Machine Learning Research_ 81 (2018): 1–15. [https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf](https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf).\n\n[^3]: Ewens, Michael, Bryan Tomlin, and Liang Choon Wang. \"Statistical Discrimination or Prejudice? A Large Sample Field Experiment.\" _Review of Economics and Statistics_ 96, no. 1 (March 2014): 119–134. [https://doi.org/10.1162/REST_a_00365](https://doi.org/10.1162/REST_a_00365).\n\n[^4]: Garg, Nikhil, Londa Schiebinger, Dan Jurafsky, and James Zou. \"Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes.\" _Proceedings of the National Academy of Sciences_ 115, no. 16 (April 2018): E3635–E3644. [https://doi.org/10.1073/pnas.1720347115](https://doi.org/10.1073/pnas.1720347115).\n\n[^5]: Gil, Yolanda, and Raymond Perrault, co-dirs. _Artificial Intelligence Index Report 2025_. Stanford, CA: Stanford University Human-Centered Artificial Intelligence, 2025\\. [https://hai.stanford.edu/assets/files/hai_ai_index_report_2025.pdf](https://hai.stanford.edu/assets/files/hai_ai_index_report_2025.pdf).\n\n[^6]: Hort, Max, Zhenpeng Chen, Jie M. Zhang, Mark Harman, and Federica Sarro. \"Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey.\" _ACM Journal on Responsible Computing_ 1, no. 2 (2024): 1–52. [https://doi.org/10.1145/3631326](https://doi.org/10.1145/3631326).\n\n[^7]: Huang, Linus Ta-Lun, and Tsung-Ren Huang. \"Generative Bias: Widespread, Unexpected, and Uninterpretable Biases in Generative Models and Their Implications.\" _AI & Society_. Published ahead of print, 2025\\. [https://doi.org/10.1007/s00146-025-02533-1](https://doi.org/10.1007/s00146-025-02533-1).\n\n[^8]: Langenkamp, Max, Allan Costa, and Chris Cheung. \"Hiring Fairly in the Age of Algorithms.\" arXiv preprint, arXiv:2004.07132. April 15, 2020\\. [https://arxiv.org/abs/2004.07132](https://arxiv.org/abs/2004.07132).\n\n[^9]: Leavy, Susan, Barry O'Sullivan, and Eugenia Siapera. \"Data, Power and Bias in Artificial Intelligence.\" Paper presented at the AI for Social Good Workshop, Harvard CRCS, online, July 20–21, 2020\\. [https://crcs.seas.harvard.edu/sites/g/files/omnuum6171/files/crcs/files/ai4sg_2020_paper_81.pdf](https://crcs.seas.harvard.edu/sites/g/files/omnuum6171/files/crcs/files/ai4sg_2020_paper_81.pdf).\n\n[^10]: Obermeyer, Ziad, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. \"Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations.\" _Science_ 366, no. 6464 (October 25, 2019): 447–453. [https://doi.org/10.1126/science.aax2342](https://doi.org/10.1126/science.aax2342).\n\n[^11]: Sunstein, Cass R. \"Algorithms, Correcting Biases.\" _Social Research: An International Quarterly_ 86, no. 2 (Summer 2019): 499–511. [https://doi.org/10.1353/sor.2019.0024](https://doi.org/10.1353/sor.2019.0024).\n\n[^12]: Suresh, Harini, and John Guttag. \"A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle.\" _EAAMO '21: Equity and Access in Algorithms, Mechanisms, and Optimization_, October 5–9, 2021\\. [https://doi.org/10.1145/3465416.3483305](https://doi.org/10.1145/3465416.3483305).","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/bias#sidebar-data-that-reflects-historical-sexism","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"data-that-reflects-historical-sexism","title":"Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes","content":"When training data is input into a word embedding program, words are assigned vector representations within the multi-dimensional space. Associated words are positioned closer to each other in the high-dimensional vector space as a result of shared context.\n\n[Garg et. al. (2018)](https://doi.org/10.1073/pnas.1720347115) found that words associated with women are systematically closer to traditionally female occupations, adjectives, and traits than words associated with men are to traditionally female occupations, adjectives, and traits. Similar disparities in distance exist between racial groups and certain occupations, adjectives, and traits. Their work laid the groundwork for subsequent research demonstrating how statistical patterns in training data associated with societal biases can lead to models that encode bias and reproduce it in their output.","sidebarKey":"data-that-reflects-historical-sexism","isDrawer":true},{"id":"automatedDecisionMaking/bias#sidebar-underrepresentation-bias","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"underrepresentation-bias","title":"Representation Bias in Image-Generating Software","content":"[Huang and Huang (2025)](https://doi.org/10.1007/s00146-025-02533-1) found that with Generative Adversarial Networks (GANs), a type of image-generating AI software, models will exacerbate societal biases and improperly represent minority groups. For example, if a GAN is asked to create images of engineers, it will overwhelmingly generate images of White males. This is because images of White males were the primary images in the dataset that the model was trained on. In their analysis of AI image-generating software, the models were shown to underrepresent minority groups because the training datasets did not include sufficient representation of minorities.\n\nWhen datasets overrepresent some demographics and underrepresent others, the resulting model will learn unevenly, performing well for majority groups but poorly for individuals from underrepresented or minority groups.","sidebarKey":"underrepresentation-bias","isDrawer":true},{"id":"automatedDecisionMaking/bias#sidebar-healthcare-risk-assessment","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"healthcare-risk-assessment","title":"Dissecting racial bias in an algorithm used to manage the health of populations","content":"[Obermeyer et al. (2019)](https://www.science.org/doi/10.1126/science.aax2342) found that to assess patient need, many hospitals use risk assessment software. One variable imputed into risk assessment models is healthcare spending. In the software studied by Obermeyer et. al., healthcare spending was used as a proxy in an algorithm that predicts health need to identify patients who will benefit most from healthcare programs. Because Black patients spend less on healthcare due to socioeconomic and systemic barriers, Black patients at a given risk score are significantly sicker than White patients with the same score. This is an example of measurement bias because the proxy variable, cost, was a poor measure of the target variable, health need.","sidebarKey":"healthcare-risk-assessment","isDrawer":true},{"id":"automatedDecisionMaking/bias#sidebar-case-study-by-mit-researchers","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"case-study-by-mit-researchers","title":"Hiring Fairly in the Age of Algorithms","content":"[Langenkamp et. al. (2020)](https://arxiv.org/abs/2004.07132) found that starting in 2014, Amazon used an ADM system to filter through resumes and find the best candidates. However, since the model was optimized to choose the best candidates, and in past hiring data, the majority of good candidates had been male, the model learned to use maleness as an indicator that someone would be a good candidate. This led to a model making hiring decisions that systematically disadvantaged women.\n\nThis study demonstrates learning bias because the model was taught to find the best candidates, so the software came to understand maleness as a factor that contributed to being a good candidate. However, this is simultaneously an example of historical bias, because historical sexism in the field of engineering contributed to the fact that the majority of past engineers hired were male.","sidebarKey":"case-study-by-mit-researchers","isDrawer":true},{"id":"automatedDecisionMaking/bias#sidebar-gender-shades","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"bias","subsectionTitle":"Bias in Automated Decision-Making Systems","anchor":"gender-shades","title":"Intersectional Accuracy Disparities in Commercial Gender Classification","content":"[Buolamwini and Gebru (2018)](https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf) created a dataset of thousands of pictures of male and female faces drawn from parliament members from across the world with a range of skin tones. They then assessed the efficacy of facial recognition technologies from IBM, Microsoft, and Face++ at recognizing faces and genders. While they found that models had a high level of overall accuracy, when looking at model accuracy by race and gender, they found that the models were only about 65-80% accurate in detecting dark-skinned, female faces as opposed to light-skinned male faces, with which all three models had an over 99% accuracy rate.","sidebarKey":"gender-shades","isDrawer":true},{"id":"automatedDecisionMaking/fairness#intro","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"intro","title":"Introduction","content":"In cases where machines replace humans in making high-impact decisions, it is important for their outcomes to remain fair. But what does algorithmic fairness actually mean?\n\nIn social terms, fairness is an ideal about justice, equality, or due treatment. In computational settings, however, **fairness is also a formalized mathematical concept**. It aims to distill qualitative notions of equity into a quantitative framework. This framing allows for systematic evaluation but also exposes deep conceptual tensions: fairness has no single definition and often looks different across domains, reflecting the social values and constraints embedded in each.\n\nThus, fairness is also a design decision that encodes subjective choices about what interests to prioritize and trade-offs to accept. These choices are driven by values and context, and results in the field show that different fairness criteria cannot all be satisfied simultaneously—a fundamental impossibility that makes the pursuit of algorithmic fairness as much an ethical and political project as a technical one.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/fairness#foundations-of-fairness","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"foundations-of-fairness","title":"Foundations of Fairness","content":"To make fairness quantitatively auditable, systems are measured using different statistical metrics and constraints, developed over many years by many researchers. However, while these understandings of fairness bring structure and transparency, they also impose limits on what fairness really expresses. What can be tested mathematically is only a fragment of what fairness means in moral or social terms, and it relies on various assumptions about what data, labels, and decisions the system can represent. Additionally, evaluating fairness only at the model level cannot fully account for disparities in the outcomes when these models are used in the real world to make decisions. Algorithmic fairness cannot fully substitute for social, legal, or institutional equity interventions.\n\nAlgorithmic fairness is built on the notion of **comparative fairness**, which defines fairness through consistency of treatment or impact across individuals or groups. On an individual scale, one core idea is “treating like cases alike”: individuals who are similar according to some relevant criteria should receive similar predictions or decisions. For example, in credit card applications, two applicants with comparable credit histories and incomes should have similar approval probabilities. Group fairness definitions extend this comparative logic by enforcing parity conditions across groups rather than individuals.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/fairness#sensitive-attributes-and-the-proxy-problem","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"sensitive-attributes-and-the-proxy-problem","title":"Sensitive Attributes and the Proxy Problem","content":"**Sensitive attributes** distinguish the groups whose treatments one wants to assess and potentially correct, e.g., race, gender, or other data points considered personal or private. Many fairness definitions explicitly require knowledge of sensitive attributes, and they are necessary for identifying disparities, measuring fairness across groups, or implementing fairness interventions.\n\nExcluding sensitive attributes appears to help with fair automated-decision making. If our model doesn't receive any information about sensitive attributes, how could it treat groups defined by said attributes unfairly? However, sensitive attributes are often encoded within other variables, an issue known as the **{proxy-problem}**. Even when sensitive attributes are excluded from a model, other variables may act as proxies because they correlate strongly with protected characteristics. Example proxies for sensitive attributes include:\n\n- Zip codes as a proxy for race or income due to residential segregation\n- Credit history as a proxy for socioeconomic status\n- Linguistic patterns or name features as proxies for ethnicity or national origin\n- Education as a proxy for social class or parental background\n\nBecause machine learning models are adept at detecting subtle patterns, they can inadvertently reintroduce group distinctions that a fairness intervention was meant to remove. This dynamic illustrates the **“fairness through unawareness” fallacy**, which is the mistaken belief that removing sensitive attributes ensures fairness. In reality, proxy features can perpetuate discrimination, undermining fairness metrics by indirectly encoding sensitive group membership. Effective fairness strategies, therefore, require thoughtful engagement with multifaceted and sensitive data, especially regarding systemic correlations and injustices.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/fairness#individual-fairness","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"individual-fairness","title":"Individual Fairness","content":"{Individual-fairness} is centered around the idea that **similar individuals should receive similar predictions.** A core component here is the **similarity metric**: which features count, how they are weighted, and how statistical distance is measured. For example, in automated résumé screening, similarity might be defined using education, experience, and skills. According to a determined similarity metric, candidates with similar degrees, work experience, and skill sets should receive similar scores or hiring decisions.\n\nA strength of individual fairness is that if two people have similar profiles, they will receive similar predictions, which helps prevent arbitrary discrimination on an individual level. However, it also brings trade-offs, primarily the assumption that there is a standard of “similarity” that is both fair and measurable. If the chosen metric reflects biased historical data or is a proxy variable for social disadvantage, individual fairness can legitimize and perpetuate existing structural inequities while still appearing formally fair. Because it only looks at an individual level, it might entrench group disparities for protected groups, placing it in conflict with the ideals of group fairness we’ll discuss below. It can also be computationally costly, making it hard to scale in complex systems with many interacting features.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/fairness#group-fairness","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"group-fairness","title":"Group Fairness","content":"Group fairness is based on the idea that groups of individuals should be treated similarly, acknowledging differences in historical and structural contexts. There are three key criteria: **independence, separation,** and **sufficiency.** Each offers a different interpretation of fairness and carries different assumptions and limitations.\n\nTo illustrate these concepts below, we can imagine an automated loan approval model for Groups A and B, where Group A marks a sensitive attribute and Group B is a baseline group. There are two types of predicted outputs:\n\n- A probability score that reflects how recommended loan approval is for any input, ranging from 0 to 1\n- The binary label of approved/not approved, which we can split by scores of \\>=0.5 and \\<0.5\n\n### Independence\n\n{Independence} requires that **predicted outcomes are statistically independent of sensitive attributes**; in other words, model predictions should not change depending on a person’s sensitive attributes.\n\nMetrics such as **demographic parity** compare positive outcomes across groups, which, in the loan approval example, would be the outcome of getting approved or getting a high score. To satisfy independence, the approval rates should be very similar across groups.\n\n![Bar graph showing independence](/assets/primer-photos/ADM/fairness/independence.png)\n\nIn the example above, there is a clear skew of Group A towards lower scores. Group A is less likely to receive a score above the positive threshold of 0.5, and therefore less likely to be approved, which violates independence. If independence were fully upheld, both groups would have equal rates of positive outcomes.\n\nIndependence is a helpful tool in looking at predicted outcomes. However, it assumes that differences in outcome rates across groups reflect unfairness, not legitimate disparities in qualifications or risk, which isn’t always the case. Even if Group A is actually less likely to pay off their loans, independence still mandates that they receive equal score distributions to Group B. It also ignores errors in false positive and false negative outcomes, which can significantly impact a model’s performance across groups.\n\n### Separation\n\n{Separation} requires that **predictions are conditionally independent of group membership given the true outcome**. In other words, individuals with the same actual outcome—in this case, whether an applicant is “qualified” according to ground truth data—should receive similar prediction behavior across groups. Truly unqualified people should have similar outcomes, and truly qualified people should as well. This is measured using group-specific outcome rates, including:\n\n|                 | Approved / Scored High    | Not Approved / Scored Low                                         |\n| :-------------- | :------------------------ | :---------------------------------------------------------------- |\n| **Qualified**   | True Positive Rate (TPR)  | False Negative Rate (FNR): (complement of TPR, equal to 1 \\- TPR) |\n| **Unqualified** | False Positive Rate (FPR) | True Negative Rate (TPR) (complement of FPR, equal to 1 \\- FPR)   |\n\nThe primary separation metrics assess parity across true positive rates and false positive rates to determine the difference between various groups, often summarized as an overall **equalized-odds gap**. To satisfy separation, these rates must be consistent across groups.\n\n![Bar graphs showing separation](/assets/primer-photos/ADM/fairness/separation.png)\n\nAs depicted above, Group B has a much higher rate of approval among unqualified applicants, suggesting a worse false positive rate. Unqualified Group B applicants were more likely to be approved than similarly unqualified applicants from Group A, which shows a fairness disparity. The score distribution among qualified applicants follows similar patterns, with qualified applicants from Group A less likely to be approved than qualified applicants from Group B, suggesting a disparity in the true positive rate.\n\nSeparation discerns predictions across groups given their true outcome, which can be helpful in scenarios where there is correlation between the sensitive characteristic and target variable. However, separation assumes that classification errors should be equally distributed across groups even if base rates (the ground-truth proportion of positive cases in each group) differ, which may conflict with accuracy or sufficiency. It also requires access to a “ground truth” label like the applicant qualifications label above, which may itself embed historical or measurement biases in deciding what exactly “qualified” means.\n\n### Sufficiency\n\n{Sufficiency} requires that the **true outcome is conditionally independent of group membership given the predicted outcome**. In practice, this means that for individuals who receive the same predicted score or label, their probability of the actual outcome should be the same across groups. In other words, a given score must represent the same level of underlying qualification regardless of group membership.\n\nSufficiency is evaluated using group-wise calibration metrics, including **positive predictive value (PPV)**, which is the proportion of approved or high-scoring applicants who are truly qualified, and **negative predictive value (NPV)**, which is the proportion of denied or low-scoring applicants who are truly unqualified. These are computed separately by group and compared for parity. Perfect sufficiency implies that calibration curves overlap across groups or that PPV and NPV gaps are near zero.\n\n![Bar graphs showing sufficiency](/assets/primer-photos/ADM/fairness/sufficiency.png)\n\nAs depicted above, among applicants who were predicted to be approved, Group B exhibits a much higher rate of true qualification than Group A, indicating a disparity in positive predictive value (PPV). In other words, an approval or high score corresponds to a stronger signal of true qualification for Group B than it does for Group A, meaning that identical model predictions do not have the same interpretive meaning across groups. The model therefore fails to satisfy sufficiency: conditional on receiving the same prediction, applicants from different groups experience systematically different probabilities of being truly qualified or unqualified.\n\nThe trade-offs of sufficiency are that it often conflicts with separation when base rates differ, meaning that achieving equal predictive meaning across groups can produce unequal error rates such as mismatched TPRs or FPRs. While sufficiency preserves interpretability and decision consistency, ensuring that a score communicates equal information for all individuals, it may still permit substantial differences in approval rates or error burdens across groups. Finally, like separation, it depends on ground-truth labels that may inherently reflect biases.\n\n### “Impossibility of Fairness” and Trade-offs\n\nExtensive research shows that independence, separation, and sufficiency are largely incompatible in practice. This result, often referred to as **{the-impossibility-theorem}**, shows that unless base rates are equal across groups or predictions are perfect, it is impossible for a model to perfectly satisfy all three fairness conditions simultaneously. Thus, any operational uses of fairness must select which definitions to relax and why.\n\nBecause the criteria are mutually exclusive, choosing which to enforce is not just a purely technical matter, but a value decision. Each fairness definition encodes a different moral or policy stance as written above, and selecting among them means deciding which conceptions of equity best fit the context.\n\nTheir fundamental incompatibility manifests as practical trade-offs among fairness goals and other system objectives. Common trade-offs include:\n\n- {Independence-vs-Separation}: equalizing predicted outcomes across groups (independence) typically requires accepting unequal error rates, while equalizing error rates (separation) typically produces unequal outcomes — and when base rates differ between groups, satisfying both simultaneously is mathematically impossible.\n\n- {Separation-vs-Sufficiency}: Equalizing error rates (separation) may conflict with ensuring that prediction scores mean the same thing across groups (sufficiency).\n\n- {Accuracy-vs-Fairness}: Accuracy focuses on prediction performance (e.g., minimizing overall error). Fairness adjustments may require introducing constraints that degrade performance for some groups to satisfy fairness criteria.\n\n- {Individual-vs-Group-Fairness}: Satisfying group fairness often requires treating individuals differently to compensate for historical inequality, which violates individual fairness principles. Enforcing individual fairness can perpetuate group disparities if the similarity metric is derived from biased data.\n\nThe impossibility theorem reframes fairness as a space of constrained choices. While {recent-research} focuses on reconciling and satisfying approximate fairness across multiple metrics, every fairness intervention still prioritizes some interpretation of equality over another. Understanding these trade-offs is essential for principled design: it clarifies that algorithmic fairness is less about finding a universal solution and more about making explicit, defensible decisions about which kinds of fairness to pursue and why.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/fairness#importance-of-fairness-research","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"importance-of-fairness-research","title":"Importance of Fairness Research","content":"Beyond the obvious goal of making algorithms more fair, formal research has transformed how automated decision systems are developed and evaluated. Its impact can be traced across multiple domains of technical and institutional practice:\n\n- Quantitative evaluation and benchmarking: Fairness metrics established quantitative baselines for model evaluation. Tools such as fairness scorecards, disparity ratios, and equalized odds plots became standard diagnostic components in fairness audits and model documentation.\n- Algorithmic methods and toolkits: Research produced concrete algorithmic techniques for enforcing fairness, such as reweighting, adversarial debiasing, constrained optimization, and post-processing calibration.\n- Regulatory translation: Formal fairness definitions have informed legal and policy discourse, including the Algorithmic Discrimination Protections section of the **AI Bill of Rights** [^10]. Statistical parity and disparate impact metrics appear in draft standards and regulatory frameworks across government and compliance frameworks.\n- Shift in machine learning practice: Fairness research reframed model performance as multi-objective: accuracy must now be evaluated alongside equity. This shift led to routine inclusion of fairness metrics in ML papers, competitions, and internal product evaluations at major AI companies.\n\nCollectively, these developments mark the field’s central contribution: turning fairness from a moral claim into a measurable property of computational systems. Any research or product that carries real-world impact also carries social responsibility, and algorithmic fairness begins to quantify and move towards more fair outcomes.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/fairness#conclusion","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"conclusion","title":"Conclusion","content":"Algorithmic fairness attempts to reconcile the technical logic of machine learning with the moral logic of equity. By translating social principles into quantitative constraints, it brings structure and accountability to automated systems, but also narrows what fairness can express. Formal definitions—namely independence, separation, and sufficiency—make fairness measurable, yet each captures only one dimension of equality, and the impossibility theorem shows that not all can perfectly coexist. Fairness is not a single technical goal or ideal, but a set of competing priorities and trade-offs that reflect intentional choices and values.\n\n<!-- BIBLIOGRRAPHY ----------------------------------------------------------------------------------------------------------------------- -->\n\n[^1]: Hardt, Moritz, Eric Price, Nati Srebro, et al. (eds.). “Classification.” In _Fairness and Machine Learning_. [https://fairmlbook.org/classification.html](https://fairmlbook.org/classification.html?utm_source)\n\n[^2]: Hardt, Moritz, Eric Price, and Nathan Srebro. \"Equality of Opportunity in Supervised Learning.\" In _Advances in Neural Information Processing Systems 29_, edited by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett. Curran Associates, Inc., 2016\\. [https://arxiv.org/abs/1610.02413](https://arxiv.org/abs/1610.02413)\n\n[^3]: Baumann, Joachim, and Michele Loi. \"Fairness and Risk: An Ethical Argument for a Group Fairness Definition Insurers Can Use.\" _Philosophy & Technology_ 36, no. 3 (2023): 45\\. [https://doi.org/10.1007/s13347-023-00624-9](https://doi.org/10.1007/s13347-023-00624-9).\n\n[^4]: Berk, Richard, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. \"Fairness in Criminal Justice Risk Assessments: The State of the Art.\" _Sociological Methods & Research_ 50, no. 1 (2021): 3–44. [https://doi.org/10.1177/0049124118782533](https://doi.org/10.1177/0049124118782533).\n\n[^5]: Speicher, Till, Hoda Heidari, Nina Grgic-Hlaca, Krishna P. Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. \"A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices.\" In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, edited by Yike Guo and Faisal Farooq, 2239–2248. New York: ACM, 2018\\. [https://doi.org/10.1145/3219819.3220046](https://doi.org/10.1145/3219819.3220046).\n\n[^6]: Bell, Andrew, Lucius Bynum, Nazarii Drushchak, Lucas Rosenblatt, Tetiana Zakharchenko, and Julia Stoyanovich. \"The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice.\" In _2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23)_, June 12–15, 2023, Chicago, IL. New York: ACM, 2023\\. [https://doi.org/10.1145/3593013.3594007](https://doi.org/10.1145/3593013.3594007).\n\n[^7]: Green, Ben. \"Escaping the Impossibility of Fairness: From Formal to Substantive Algorithmic Fairness.\" _Philosophy & Technology_ 35 (2022): 90\\. [https://doi.org/10.1007/s13347-022-00584-6](https://doi.org/10.1007/s13347-022-00584-6).\n\n[^8]: Hsu, Brian, Rahul Mazumder, Preetam Nandy, and Kinjal Basu. \"Pushing the Limits of Fairness Impossibility: Who's the Fairest of Them All?\" In _Advances in Neural Information Processing Systems 35_, edited by Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh. New Orleans, LA: NeurIPS, 2022\\. [https://arxiv.org/abs/2208.12606](https://arxiv.org/abs/2208.12606).\n\n[^9]: Sahlgren, Otto. \"What's Impossible about Algorithmic Fairness?\" _Philosophy & Technology_ 37 (2024): 124\\. [https://doi.org/10.1007/s13347-024-00814-z](https://doi.org/10.1007/s13347-024-00814-z).\n\n[^10]: Krantz, Tom, and Alexandra Jonker. \"What Is the AI Bill of Rights?\" IBM Think. Last modified February 26, 2026\\. [https://www.ibm.com/think/topics/ai-bill-of-rights](https://www.ibm.com/think/topics/ai-bill-of-rights).\n\n<!-- ALL MY SIDEBARS BELOW ----------------------------------------------------------------------------------------------------------------------- -->","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/fairness#sidebar-proxy-problem","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"proxy-problem","title":"The Proxy Problem","content":"The proxy problem arises when features in a dataset encode information about sensitive attributes indirectly through statistical correlation. Even if protected variables such as race, gender, or age are excluded, the joint distribution of other variables often contains enough signal to allow a model to infer them. In large datasets, these correlations can be subtle and nonlinear, amplified by the capacity of modern learning algorithms to detect complex dependencies. These proxies mean that prediction can be influenced by sensitive attributes, even if excluded.\n\nFormally, if a sensitive attribute _A_ is not included in the feature set _X_, but some subset of _X_ still provides information about _A_, then the model’s prediction function _f(X)_ will depend on _A_ indirectly.\n\nThis means that the assumption of “fairness through unawareness” fails: omitting A from the model does not prevent the algorithm from using it implicitly. The extent of this dependence can be measured by the statistical association between _A_ and _X_, for example, through mutual information or other measures of predictability.\n\nProxies can emerge through multiple mechanisms:\n\n- Explicit correlation between variables (e.g., postal code correlated with race due to residential segregation).\n- Derived or composite features that aggregate information from correlated inputs (e.g., credit utilization ratios reflecting socioeconomic status).\n- Representation learning in neural networks, where hidden layers encode latent attributes predictive of sensitive categories.\n\nDetecting proxies is difficult because sensitive information may be dispersed across many variables rather than isolated in one. One common diagnostic method is to train an auxiliary classifier that attempts to predict _A_ (the sensitive attribute) from _X_ (the feature set). If the classifier performs well, then the feature space encodes sensitive information. Statistical independence tests and conditional mutual-information analysis are also used to quantify proxy strength.\n\nVarious mitigation strategies aim to reduce this dependence:\n\n- Causal inference approaches try to block the causal paths from _A_ to the predicted outcome _Y_.\n- Adversarial debiasing methods train the model so that its representations are predictive of _Y_ but not of _A_.\n- Information-theoretic methods penalize the amount of information about _A_ contained in the prediction.\n\nHowever, these techniques rely on assumptions about the data-generating process and causal structure, and they rarely remove proxy influence completely. Because social and economic variables are deeply correlated, proxy effects are often unavoidable. Effective fairness assessment must therefore consider both explicit use of sensitive attributes and their indirect presence throughout the feature space.","sidebarKey":"proxy-problem","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-individual-fairness","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"individual-fairness","title":"Individual fairness","content":"### Formal definition:\n\nFor a prediction function _Ŷ_ \\= _f(X)_, and a task-specific similarity metric _d(·,·)_ over individuals, the model should satisfy a Lipschitz-style condition: | _f(xi​)−f(xj​)_ | ≤ L ⋅ _d (xi​,xj​)_ for all individuals _xi_, _xj_. The closer two individuals are according to the similarity metric, the more similar their predicted outcomes must be.\n\n### Common Metrics:\n\n1. Lipschitz violations: Proportion or magnitude of prediction differences exceeding the similarity bound.\n2. Pairwise consistency metrics: Frequency of near-neighbor pairs with large score gaps or conflicting decisions. x\n3. Ranking stability measures: Evaluations of whether individuals with similar feature profiles receive comparable rankings or thresholded decisions.\n\n### Key assumption:\n\nA valid, ethically defensible, and task-relevant similarity metric exists that captures which characteristics should matter for the decision and how much they should matter relative to one another.\n\n### Limitations:\n\n- Heavily depends on how similarity is defined; if the chosen metric is derived from historically biased data or includes proxy features correlated with protected characteristics, the fairness constraint can perpetuate structural inequities while appearing formally neutral.\n- Does not guarantee group-level parity and may entrench disparities across protected groups.\n- Potentially substantial computational overhead, as pairwise or neighborhood-based fairness checks scale poorly to large datasets.","sidebarKey":"individual-fairness","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-independence","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"independence","title":"Independence","content":"### Formal definition:\n\nWhere _X_ is the space of feature vectors, _Y_ is the true outcome, _A_ is a sensitive attribute, and _Ŷ_ \\= _f (X)_ is the prediction*,* independence holds when _Ŷ_ ⊥ _A._[^1]\n\n### Common Metrics:\n\n- Demographic parity difference: Difference in positive outcome rates across groups. These can be compared with minimum/maximum, pairwise differences, or using a reference group. Smaller values indicate greater parity.\n- Demographic parity ratio: Ratio of positive outcome rates across groups. Values closer to 1 indicate greater parity.\n- Mean score difference: Gap between average predicted scores across groups for continuous predictions.\n\n### Key Assumption:\n\nDifferences in outcome rates across groups reflect inherent unfairness, not legitimate differences in qualifications or risk.\n\n### Limitations:\n\n- Does not distinguish between disparities arising from bias and those arising from genuine differences in predictive features.\n- Allows false positive and negative rates to be unevenly distributed.\n- Can reduce predictive accuracy by forcing equal selection rates across dissimilar populations.","sidebarKey":"independence","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-separation","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"separation","title":"Separation","content":"### Formal definition:\n\nWhere _X_ is the space of feature vectors, _Y_ is the true outcome, _A_ is a sensitive attribute, and _Ŷ_ \\= _f (X)_ is the prediction*,* separation holds when _Ŷ_ ⊥ \\*A | Y.\\*[^1]\n\n### Common Metrics:\n\n- True Positive Rate (TPR) difference (equal opportunity): Difference in correctly approved qualified individuals across groups. Values closer to 0 indicate greater parity.\n- False Positive Rate (FPR) difference: Difference in incorrectly approved unqualified individuals across groups. Values closer to 0 indicate greater parity.\n- False Negative Rate (FNR) difference: Difference in incorrectly denied qualified individuals across groups. Values closer to 0 indicate greater parity.\n- Equalized odds difference: Combined measure of TPR and FPR differences, often taking their maximum. Values closer to 0 indicate greater parity.\n\n### Key assumption:\n\nDifferences in error rates across groups are unfair, even if true outcome rates differ between groups.\n\n### Limitations:\n\n- Requires access to accurate ground truth labels, which may themselves be biased or socially constructed.\n- Treats all misclassification errors as equivalent regardless of their real-world impact.\n- Conflicts with sufficiency when outcome base rates differ across groups.","sidebarKey":"separation","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-sufficiency","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"sufficiency","title":"Sufficiency","content":"### Formal definition:\n\nWhere _X_ is the space of feature vectors, _Y_ is the true outcome, _A_ is a sensitive attribute, and _Ŷ_ \\= _f (X)_ is the prediction*,* sufficiency holds when _Y_ ⊥ \\*A | Ŷ\\*[^1]\n\n### Common metrics:\n\n- Positive predictive value (PPV) gap: Difference in true qualification rates among approved or high-score individuals across groups.\n- Negative predictive value (NPV) gap: Difference in true unqualification rates among denied or low-score individuals across groups.\n- Expected calibration error (ECE): Average deviation between predicted probabilities and observed outcome frequencies, stratified by group.\n- Calibration curves / reliability diagrams: Visual checks of probability alignment across groups.\n- Brier score or log loss by group: Measures of probabilistic prediction quality across demographic strata.\n\n### Key assumption:\n\nPrediction scores represent true and comparable probabilities across all groups and can be meaningfully interpreted as equal estimates of underlying risk.\n\n### Limitations:\n\n- Assumes probability estimates are well-calibrated and comparable across groups.\n- Permits disparities in selection rates and error burdens across demographic groups.\n- Requires large sample sizes for reliable estimation.\n- Depends on accurate ground-truth labels, which may embed social, historical, or measurement bias.","sidebarKey":"sufficiency","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-the-impossibility-theorem","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"the-impossibility-theorem","title":"The Impossibility Theorem","content":"The impossibility theorem in algorithmic fairness formalizes the claim that standard group fairness criteria—independence, separation, and sufficiency—cannot all hold simultaneously except under restrictive or trivial conditions. The result follows from the probabilistic relationships among the sensitive attribute (_A_), the true outcome (_Y_), and the model prediction (_Ŷ_). Each fairness criterion specifies a conditional independence constraint, as defined above:\n\n- Independence: _Ŷ_ is independent of _A → Ŷ ⟂ A_\n- Separation: _Ŷ_ is independent of A given the true outcome \\*Y → Ŷ ⟂ A | Y\\*\n- Sufficiency: _Y_ is independent of A given the prediction _Ŷ → Y ⟂ A | Ŷ_\n\nEach expresses a different fairness intuition: equality of outcomes, equality of errors, and equality of interpretation.\n\nThe conflict arises because these conditional independence relations cannot all hold simultaneously unless group base rates _P(Y_\\=1 _ | A)_ are equal or the model is perfectly accurate (_Ŷ \\= Y_).\n\n### Mathematical outline:\n\nLet the model output a score _S \\= f(X)_, estimating _P(Y=1 | X)_.\n\n- Independence (demographic parity) requires _P(S | A=a)_ to be identical across groups.\n- Separation (equalized odds) requires _P(S | Y=y, A=a)_ to be identical across groups.\n- Sufficiency (calibration) requires _P(Y=1 | S=s, A=a) \\= s_ for all groups _a_.\n\nThese conditions overconstrain the joint distribution of _S, Y,_ and _A_. If sufficiency and separation both hold, then by the law of total probability _P(Y=1 | A=a) \\= Σs P(Y=1 | S=s, A=a) P(S=s | A=a)_ must be identical across groups. In other words, combining sufficiency and separation forces the group-level base rates _P(Y \\= 1 | A \\= a)_ to be identical. Unless the model is perfect or groups truly have identical Y distributions, these two conditions cannot both be satisfied.\n\nLikewise, enforcing independence alongside either sufficiency or separation generally leads to degenerate cases (such as constant predictions or identical label distributions). Demographic parity requires equal prediction rates regardless of differences in true outcomes. If we also require separation, we are demanding that both outcome rates and error rates match across groups—something that can occur only if the true label distributions are the same or the model makes no errors.\n\nSimilarly, independence and sufficiency can only hold together when the true outcome _Y_ carries no information about _A_. Because real datasets almost always exhibit different base rates and imperfect accuracy, these constraints cannot all be satisfied at once.","sidebarKey":"the-impossibility-theorem","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-independence-vs-separation","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"independence-vs-separation","title":"Independence vs. separation","content":"### Domain:\n\nLending (loan approvals)\n\n### Study:\n\n[“Equality of Opportunity in Supervised Learning”](https://arxiv.org/pdf/1610.02413) by Hardt, Price & Srebro (2016) [^2]\n\n### Summary:\n\nThe authors present a post-processing method for classifiers that adjusts decision thresholds group-by-group so as to equalize false-positive and false-negative rates across groups. Essentially, they enforce the separation criterion (equal error rates) rather than the independence criterion (equal approval rates irrespective of group).\n\n### Why separation was chosen:\n\nIn the lending context, the authors argue that it is unfair if one demographic group faces systematically higher error rates (e.g., being wrongly denied or wrongly approved) even if the overall approval rate is the same. Prioritizing equal error rates ensures more equitable treatment conditional on “would repay” vs “would default.”\n\n### Trade-off illustrated:\n\nBy choosing separation, the authors implicitly accept differences in approval rates across groups (thus violating independence). This shows how the technical choice of fairness metric reflects a policy decision about whether we care more about outcome-parity or error-parity.","sidebarKey":"independence-vs-separation","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-separation-vs-sufficiency","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"separation-vs-sufficiency","title":"Separation vs Sufficiency","content":"### Domain:\n\nInsurance (premium pricing / risk prediction)\n\n### Study:\n\n[“An Ethical Argument for a Group Fairness Definition Insurers Can Use”](https://link.springer.com/article/10.1007/s13347-023-00624-9) by Baumann & Loi (2023) [^3]\n\n### Summary:\n\nThe authors examine algorithmic risk scoring for insurance premiums and argue that the most appropriate fairness metric in this domain is sufficiency (calibration), so a predicted risk score should mean the same thing across demographic groups. They critique both independence and separation as normatively and mathematically ill-fitting for insurance. Why sufficiency was chosen: In insurance, predicted risk scores are used to set premiums or underwriting decisions; thus, fairness demands that those scores correspond to actual risk equivalently for all groups. Focusing on separation (equal error rates) could force distortions in the calibration of risk scores—undermining the actuarial validity of pricing. Trade-off illustrated: By choosing sufficiency, the authors accept that error rates or outcome distributions might differ across groups. This reinforces how the fairness criterion selection is a technical decision that encodes value judgments about what fairness means in that domain. |","sidebarKey":"separation-vs-sufficiency","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-accuracy-vs-fairness","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"accuracy-vs-fairness","title":"Accuracy vs. Fairness","content":"### Domain:\n\nCriminal Justice\n\n### Study:\n\n“[Fairness in Criminal Justice Risk Assessments: The State of the Art](https://arxiv.org/abs/1703.09207)” by Berk et al. (2017) [^4]\n\n### Summary:\n\nThis study examines how different definitions of algorithmic fairness apply to criminal justice risk assessment tools, including systems that predict reoffending risk to guide bail, parole, or sentencing decisions. They show empirically that when base rates differ across demographic groups, achieving statistical fairness typically requires reducing predictive accuracy. Enforcing fairness constraints can shift decision thresholds, altering false positive and false negative rates and producing less precise predictions overall. Trade-off illustrated: The study demonstrates that optimizing purely for accuracy tends to reproduce historical disparities present in the data, while optimizing for fairness may lower overall predictive performance. Balancing these goals requires explicitly deciding how much accuracy one is willing to sacrifice for fairer outcomes, again making fairness a normative and policy choice instead of a purely technical parameter. |","sidebarKey":"accuracy-vs-fairness","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-individual-vs-group-fairness","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"individual-vs-group-fairness","title":"Individual vs. Group Fairness","content":"## Domain:\n\nGeneral predictive classification tasks\nStudy: [“A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices”](https://doi.org/10.48550/arXiv.1807.00787) by Speicher et al. (2018) [^5]\n\n### Summary:\n\nThis paper proposes a quantitative framework that measures unfairness using economic inequality indices. They define “benefit” values for individuals based on algorithmic outcomes and decompose overall unfairness into between-group (group unfairness) and within-group (individual unfairness) components. Through experiments, they show that optimizing for group-level parity can unintentionally increase within-group inequality, meaning individuals inside the same group are treated less consistently. Trade-off illustrated: Pursuing group fairness improves equality between demographic groups but can worsen fairness within those groups. The study demonstrates that no algorithm can simultaneously optimize both. |","sidebarKey":"individual-vs-group-fairness","isDrawer":true},{"id":"automatedDecisionMaking/fairness#sidebar-recent-research","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"fairness","subsectionTitle":"Algorithmic Fairness","anchor":"recent-research","title":"Pushing The Boundaries of the Impossibility Theorem","content":"Recent works have sought to push the boundaries of the impossibility theorem:\n\nIn [“The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice,”](https://dl.acm.org/doi/pdf/10.1145/3593013.3594007) Bell et al. (2023)[^6] show that if fairness metrics are treated as approximate rather than exact constraints, many models can satisfy multiple fairness criteria simultaneously. Using analytical arguments and experiments, the authors argue that the impossibility theorem is less restrictive in practice, and approximate fairness along multiple metrics is often achievable.\n\nIn [“Escaping the Impossibility of Fairness: From Formal to Substantive Algorithmic Fairness,”](https://link.springer.com/article/10.1007/s13347-022-00584-6) Green (2022)[^7] argues for a shift from formal fairness metrics to substantive fairness, emphasizing that fairness must be grounded in social context, institutional practices, and normative reasoning.\n\nIn [“Pushing the Limits of Fairness Impossibility: Who’s the Fairest of Them All?,”](https://proceedings.neurips.cc/paper_files/paper/2022/file/d3222559698f41247261b7a6c2bbaedc-Paper-Conference.pdf) Hsu et al. (2022)[^8] propose a post-processing algorithm that optimizes across multiple fairness criteria simultaneously, mitigating incompatibilities between independence, separation, and sufficiency without requiring perfect balance.\n\nIn [“What’s Impossible about Algorithmic Fairness?,”](https://link.springer.com/article/10.1007/s13347-024-00814-z) Sahlgren (2024)[^9] provides a philosophical analysis of the impossibility results in algorithmic fairness, examining what kind of “impossibility” they establish and how this relates to broader questions about feasibility and the norms we attach to fairness criteria.","sidebarKey":"recent-research","isDrawer":true},{"id":"automatedDecisionMaking/justice#intro","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"intro","title":"Introduction","content":"Justice is the idea that individuals should be treated fairly. However, what that means when trying to create the best AI Systems presents many questions, diverging opinions, and diversity in implementation.\n\nWhile [fairness](/AI/fairness), as currently formalized in AI, relies on mathematical criteria, justice relies on an ever-evolving assumption about what counts as “just.” Justice reflects deeper philosophical tensions about what kind of justice AI systems should aim for: distributive, procedural, or other philosophical approaches. Justice requires tools that actively build on equality and access: Can AI be that tool?","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/justice#conceptions-of-justice","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"conceptions-of-justice","title":"Conceptions of Justice","content":"There are many efforts to build mathematically fair models and de-bias datasets as well as govern the outcomes of Automated Decision Systems (ADS). However, a philosophical notion of justice provides an intellectual framework for building models and tech. There is no one notion of justice that is accepted by everyone and some conceptions also conflict with each other, making all impossible to implement at once.[^11] The choice of guiding methodology largely depends on the outcome goal behind and values behind development of ADS. Principally, there are two big overarching conceptions that show different approaches to what justice should achieve:\n\n| Process-Focused Justice                                                                                                                                                                                                                                                                                                                                                                                                               | Outcome-Focused Justice                                                                                                                                                                                                                                                                                                                           |\n| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Process-focused approaches to justice focus only on the process through which the users go, not on the outcomes of that process Process-focused justice is the idea that all individuals and groups should be respected and valued in automated decision-making processes. It applies to interactions with all kinds of decision-makers, whether that be a legal authority, corporate system, community institution, or an algorithm. | Outcome-focused justice focuses on the fairness of the outcomes or results of decisions and actions taken. This is a philosophy of fair distribution of resources, opportunities, or benefits, which can also be applied to AI outputs. This concept emphasizes that outcomes must not perpetuate existing inequities or systemic discrimination. |\n\n#### Key Questions for Process-Focused Justice in AI:\n\n- Are individuals treated with dignity and respect in automated decision-making processes and outcomes?\n- Is there clear communication about why the system is making certain decisions?\n- Can individuals contest the AI system’s decisions, and are those processes accessible and fair?\n- Do appropriate mechanisms for accountability (e.g. effective auditing) exist?\n\nKey components of process-focused (or procedural) justice are **respect, trustworthiness**, granting everyone a **voice** to tell their side, and **neutrality.**\n\n**Respect** captures the idea that all individuals must be treated with dignity. This includes courteous communication, acknowledgment of people's concerns, and the absence of demeaning or dismissive behavior.\n\n**Trustworthiness** refers to whether decision makers convey sincere, benevolent, and fair motives to those impacted by their decisions. People evaluate not only the logic of a decision but also the intentions behind it. Trust is strengthened when decision makers demonstrate concern for stakeholders' well-being and explain their reasoning in ways that signal integrity and accountability. A key aspect of trustworthiness is transparency and right to request explanations for model's performance, like in the {youtube-case|YouTube moderation case}[^19].\n\n**Voice** emphasizes that everyone affected by a decision is given the opportunity to share their perspective. This does not guarantee that every viewpoint will determine the outcome, but it ensures that the process is participatory rather than unilateral.\n\n**Neutrality** refers to the expectation that decisions are unbiased. A procedurally just process requires that rules are applied consistently, that decision criteria are clearly explained, and that personal preferences or prejudices do not shape outcomes.\n\n#### Key Questions for Outcome-Focused Justice (Distributive Justice) in AI:\n\n- Do particular individuals or communities receive a disproportionate share of resources, opportunities, or risks because of the system’s design?\n- What trade-offs, if any, exist between the three tenets of distributive justice (i.e., sufficiency, priority, and equality of opportunity)?\n- Does the system cause the deepening of social, economic, or political advantages for certain groups while leaving other groups behind?\n\n**Sufficiency**, **priority**, and **equality of opportunity** are three distinct ways to assess automated decision-making systems through the lens of distributive justice. They are at times contradictory, but show diverse proposed ways of achieving outcome-focused justice.\n\n**Sufficiency** focuses on ensuring that all individuals and groups reach a minimum acceptable threshold of resources, protections, and opportunities. The core idea is that no one should fall below an ethically justifiable standard of well-being, even if outcomes are not perfectly equal. Rather than seeking to equalize all results, a sufficiency approach prioritizes meeting fundamental needs first and protecting the most vulnerable from serious harm. In AI systems, sufficiency asks whether automated decisions systematically push anyone below this threshold.\n\n**Priority** emphasizes directing resources and opportunities to those who are worst off or face the most urgent needs. It acknowledges that some individuals or groups require more intensive or immediate support in order to achieve genuinely fair outcomes. For automated decision-making, priority asks whether AI systems are capable of recognizing existing disparities and responding in ways that reduce them rather than entrench them. This lens focuses on how outcomes are distributed, especially for people who are already disadvantaged.\n\n**Equality of opportunity** ensures that all individuals have fair access to valuable resources and life chances, regardless of morally arbitrary characteristics such as race, gender, or socioeconomic background. Principally, it aims to remove any barriers that prevent equitable participation and advancement in society. In the context of AI, equality of opportunity involves designing systems that avoid reproducing historical biases and that support inclusive access to beneficial decisions. It evaluates not only who “wins” or “loses” in an algorithmic process, but whether people had a fair chance, given their starting position, to benefit from the technology.\n\n**Case Study:** {Generative-AI-Based-Tutoring-System-for-Upper-Egypt-Community-Schools|Generative AI-Based Tutoring System for Upper Egypt Community Schools}","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/justice#other-notions-of-justice","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"other-notions-of-justice","title":"Other Notions of Justice","content":"However, beyond the principally-different conceptions of justice (process vs outcome), many other ideas and philosophies have emerged to capture the breadth of inequalities and areas of life that get affected by possible injustices.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/justice#restorative-justice","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"restorative-justice","title":"Restorative Justice","content":"**Restorative justice is a set of principles and practices to deal with crimes or incidents, as violations of people and relationships. While it holds perpetrators accountable, it attempts to determine what can be done to repair that harm and rebuild the damaged relationships.**\nRestorative justice can help victims build a constructive relationship or dialogue with those who harmed them, but only through a voluntary, safe, and facilitated encounter in which offenders accept responsibility and all affected parties have a voice. Its goal is repair:[^20] supporting victims’ healing, giving offenders a chance to make amends, and helping the wider community restore trust, safety, and relational health. It also seeks transformation by addressing not only individual harm but also the deeper social or structural causes of wrongdoing, so that people, relationships, and systems can change. One example of these initiatives is the {dfBL-case|Data for Black Lives's effort to build data-based reparations for healthcare}.\n\n#### Key AI Questions for Restorative Justice:\n\n- Which people and communities most affected by this AI system have been harmed, excluded, or misrepresented by similar systems in the past?\n- Are harmed people and communities meaningfully involved in shaping how this system is designed, reviewed, and governed?\n- What concrete harms, exclusions, or injustices could this AI system perpetuate, and what specific steps are built in to acknowledge, prevent, and repair those harms?\n- Does this system merely avoid repeating past harm, or does it also address the deeper structural causes that made those harms possible?","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/justice#epistemic-justice","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"epistemic-justice","title":"Epistemic Justice","content":"**The key theorist of epistemic justice Miranda Fricker explains it through the prism of epistemic injustice. She theorized that epistemic injustice is “wrong done to someone in her capacity as a knower.”[^4]**\n\nAt the core of it, epistemic injustice occurs when certain people’s knowledge, experiences, or interpretive frameworks are systematically excluded or devalued. Along with Fricker, many theorists since then, like Revathi Krishnaswamy, Raewyn Connell, Boaventura De Sousa Santos, Victoria Zurita, Chen Bar-Itzhak, and others, have applied epistemic justice frameworks to literature, politics, science, and other domains where marginalization and delegitimating occur towards knowledge and ways of understanding come from non-dominant cultural centers.[^21] In AI, this manifests through {data-colonialism}, biased training sets, and {hegemonic-definitions-of-accuracy}. Algorithmic systems often silence marginalized perspectives by codifying only the dominant worldview, turning epistemic inequality into {technological-infrastructure}[^9][^8].\n\n#### Key AI Questions for Epistemic Justice:\n\n- Whose knowledge, perspectives, and lived experiences are included in the data and design and which are missing?\n- Does the AI system privilege one cultural or epistemic framework as “objective” or “neutral”?\n- Do marginalized communities have the power to contest how they are represented or categorized?\n- Does the system unintentionally silence, distort, or misinterpret certain groups?","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/justice#structural-justice","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"structural-justice","title":"Structural Justice","content":"**Stuctural bias examines not just what an AI system decides, but the wider web of institutions, norms, and power relations it may reinforce. Structural harm emerges when technological systems reproduce social processes that systematically expose some populations to domination, deprivation, or vulnerability while expanding the opportunities of others[^14].** For example, feminist theorists highlight that structural violence is sustained through policies, organizational routines, and cultural norms that constrain autonomy and reproduce inequality, even when no individual intends harm.[^2] Evaluating AI through this lens requires asking not only whether the model is biased, but whose power it extends, whose lives it becomes entangled with, and whose vulnerabilities it amplifies.\n\n#### Key AI Questions for Structural Justice:\n\n- Does the AI reproduce existing power hierarchies or actively challenge them?\n- Are there institutional practices, laws, or economic structures that the AI amplifies that disadvantage certain groups?\n- Does the AI system increase surveillance, monitoring, or control over already marginalized populations?\n- Does the AI rely on data collected through extraction, coercion, or [limited consent](/Privacy/Consent)?","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/justice#environmental-justice","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"environmental-justice","title":"Environmental Justice","content":"**Environmental justice examines how environmental benefits, burdens, and risks are distributed across communities, and whether some groups bear disproportionate harms from environmental degradation.** When applied to AI, environmental justice assesses how algorithmic systems influence environmental governance, resource allocation, climate risk management, and the ecological footprint of {AI-infrastructure} itself.\n\n#### Key AI Questions for Environmental Justice:\n\n- Who bears the environmental costs of AI development? Are these costs geographically or socio-economically concentrated?\n- Are Indigenous and local communities consulted when environmental models rely on land, cultural resources, or ecological knowledge?\n- How are environmental harms and benefits distributed across regions, particularly the Global South versus the Global North?","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/justice#conclusion","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"conclusion","title":"Conclusion","content":"ADS systems don’t exist in a vacuum but within the same social, political, and environmental systems that shape human institutions. As a result, questions of justice in AI cannot be reduced to a single definition of fairness or a single technical solution. Instead, they emerge at the intersection of multiple traditions of justice that illuminate different dimensions of how technological systems affect people and communities.\n\nDeciding which approach to justice to take will be driven by the community for whom the technology is created, the overall goals, and agency of the developer, but various frameworks laid out in this primer show that justice in AI is inherently multidimensional. A system may appear procedurally fair while producing unequal outcomes. It may reduce bias in predictions while still reinforcing structural inequalities or extracting environmental resources from vulnerable communities. Addressing justice in AI therefore requires looking beyond isolated technical metrics toward a more comprehensive understanding of how algorithmic systems operate within society.\n\n[^1]: Buccella, Alessandra. 2023\\. \"'AI for All' Is a Matter of Social Justice.\" AI and Ethics 3: 1143-1152.\n\n[^2]: Burnett, Camille, Michael Swanberg, Ashley Hudson, and Donna Schminkey. 2018\\. \"Structural Justice: A Critical Feminist Framework Exploring the Intersection between Justice, Equity and Structural Reconciliation.\" Journal of Health Disparities Research and Practice 11, no. 4, article 4\\.\n\n[^3]: Data for Black Lives. 2020\\. \"What Is Data for Black Lives?\" September 1, 2020\\. https://d4bl.org/videos/55-what-is-data-for-black-lives.\n\n[^4]: Fricker, Miranda. 2007\\. Epistemic Injustice: Power and the Ethics of Knowing. Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198237907.001.0001.\n\n[^5]: Gabriel, Iason. 2022\\. \"Toward a Theory of Justice for Artificial Intelligence.\" Daedalus 151, no. 2: 218-231.\n\n[^6]: Guo, Cindy X., Elizabeth X., and Monica Lange. 2023\\. \"Data Colonialism and Data Sets.\" Harvard Law Review Blog, June 22, 2023\\. https://harvardlawreview.org/blog/2023/06/data-colonialism-and-data-sets/.\n\n[^7]: Hao, Karen. 2022\\. \"A New Vision of Artificial Intelligence for the People.\" MIT Technology Review, April 22, 2022\\.\n\n[^8]: Kaur, Kirandeep, Ben Grama, Nairita Roy Chaudhuri, and Maria Jose Recalde-Vela. 2023\\. \"Ethics and Epistemic Injustice in the Global South: A Response to Hopman's Human Rights Exceptionalism as Justification for Covert Research.\" Journal of Human Rights Practice 15, no. 2: 347-373. https://doi.org/10.1093/jhuman/huad008.\n\n[^9]: Maddox, Raglan, and Melody E. Morton Ninomiya. 2025\\. \"Indigenous Sovereignty in Research and Epistemic Justice: Truth Telling through Research.\" Global Public Health 20, no. 1\\. https://doi.org/10.1080/17441692.2024.2436436.\n\n[^10]: Mejias, Ulises A., and Nick Couldry. 2024\\. Data Grab: The New Colonialism of Big Tech and How to Fight Back. N.p.: WH Allen.\n\n[^11]: Miller, David. 2017\\. \"Justice.\" In The Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/entries/justice/.\n\n[^12]: Newman v. Google LLC, No. 5:20-cv-04011 (N.D. Cal. filed June 16, 2020).\n\n[^13]: Newman v. Google LLC, No. 20-CV-04011-LHK, slip op. (N.D. Cal. June 25, 2021).\n\n[^14]: Nicholas, Jeffery. 2012\\. \"Structural Justice.\" Review of Responsibility for Justice, by Iris Marion Young. The Review of Politics 74: 521-524. https://doi.org/10.1017/S0034670512000678.\n\n[^15]: Rafanelli, Lucia M. 2022\\. \"Justice, Injustice, and Artificial Intelligence: Lessons from Political Theory and Philosophy.\" Big Data and Society 9, no. 1\\.\n\n[^16]: Santoni de Sio, Filippo, Txai Almeida, and Jeroen van den Hoven. 2024\\. \"The Future of Work: Freedom, Justice and Capital in the Age of Artificial Intelligence.\" Critical Review of International Social and Political Philosophy 27, no. 5: 659-683.\n\n[^17]: Soudi, Marwa, Esraa Ali, Maha Bali, and Nihal Mabrouk. “Generative AI-Based Tutoring System for Upper Egypt Community Schools.” _In Proceedings of the 2023 Conference on Human Centered Artificial Intelligence: Education and Practice (HCAIep ’23)_, 16–21. New York: ACM, 2023. https://doi.org/10.1145/3633083.3633085.\n\n[^18]: Stanford Humanities Center. 2024\\. \"Round Table: Epistemic Justice.\" YouTube video. Filmed January 20, 2022, posted August 29, 2024\\. https://www.youtube.com/watch?v=\\[insert video ID\\].\n\n[^19]: Stempel, Jonathan, and Rosalba O'Brien. 2023\\. \"YouTube Defeats Racial Bias Lawsuit by Black, Hispanic Content Creators.\" Reuters, August 17, 2023\\. https://www.reuters.com/legal/youtube-defeats-racial-bias-lawsuit-by-black-hispanic-content-creators-2023-08-17/.\n\n[^20]: \"Three Core Elements of Restorative Justice.\" n.d. Restorative Justice. Accessed April 20, 2026\\. https://restorativejustice.org/what-is-restorative-justice/three-core-elements-of-restorative-justice/.\n\n[^21]: Zurita, Victoria, and Chen Bar-Itzhak. 2024\\. \"In Search of Epistemic Justice.\" Arcade: A Digital Salon, Stanford Humanities Center, Spring 2024\\. https://shc.stanford.edu/arcade/colloquies/search-epistemic-justice.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/justice#sidebar-youtube-case","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"youtube-case","title":"YouTube Content Moderation Appeals (2023)","content":"In the [lawsuit _Newman v. Google_](https://www.courthousenews.com/wp-content/uploads/2020/06/newman-google.pdf)[^12], Black and Hispanic creators argued that YouTube disproportionately restricted, demonetized, and downranked their videos, and they specifically alleged that the platform interfered with, delayed, or ignored appeals, preventing timely manual review and leaving creators without effective recourse while they lost audience reach and revenue. This case can be understood primarily as a case about alleged unequal treatment in platform moderation and untransparent—procedurally unjust—appeals process. The case was not simply that creators lacked an appeal path, but the complaint centered on a broader claim that YouTube’s moderation systems were not race-neutral and instead treated creators from minority groups differently from similarly situated white creators. The creators ultimately [lost](https://www.govinfo.gov/content/pkg/USCOURTS-cand-3_20-cv-04011/pdf/USCOURTS-cand-3_20-cv-04011-0.pdf)[^13], and the case was dismissed with prejudice, meaning it can not be brought again. Although algorithmic discrimination by race was plausible in theory, the plaintiffs did not come close to showing that they had actually suffered discrimination on the record before the court, according to the judge. Newman v. Google illustrates how disputes over platform moderation can combine group fairness concerns with procedural justice concerns. The plaintiffs claimed that minority creators were treated unequally and that appeals were ineffective or obstructed, which raised questions about voice, transparency, and neutral review.","sidebarKey":"youtube-case","isDrawer":true},{"id":"automatedDecisionMaking/justice#sidebar-dfbl-case","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"dfbl-case","title":"Data for Black Lives's effort to build data-based reparations for healthcare","content":"[Data for Black Lives (D4BL)](https://d4bl.org/) is a movement of activists, organizers, and scientists that challenges the use of data and algorithms as tools of racial oppression and argues for data practices that serve Black communities instead. Its work spans abolition, political education, data governance, and algorithmic justice, and it produces campaigns, reports, events, and datasets. Rather than treating data harms as isolated technical flaws, D4BL places them in the longer history of systemic injustice, from redlining to predictive policing to unequal health outcomes.[^3] One concrete example is its COVID-19 racial disparity data work, including the [D4BL COVID-19 Disparities Tracker](https://d4bl.org/datasets/44-d4-bl-covid-19-disparities-tracker) and related datasets on Black COVID-19 cases and deaths. These projects consolidated and surfaced race-based public health data at a moment when many institutions were failing to make such disparities visible. In doing so, D4BL was not simply collecting data for its own sake. It was using data infrastructure to document harm, support public accountability, and direct attention toward communities bearing disproportionate health burdens. D4BL’s dataset initiatives show what a restorative orientation to data can look like. Its work suggests that just health AI requires more than reducing bias in models. It requires recognizing historical harm, involving affected communities in governance, and building datasets and analytic systems that help repair inequity rather than reproduce it. In that sense, D4BL reflects restorative justice through all three dimensions: creating space for affected communities to shape the conversation, using data to address concrete harm, and pushing for structural transformation in how data systems are governed. The dataset can be found [here](https://docs.google.com/spreadsheets/d/1NFViedF47p-P0MKKl8_O0mKAhba0Yqn200EfUR4GlcQ/edit?gid=0#gid=0).","sidebarKey":"dfbl-case","isDrawer":true},{"id":"automatedDecisionMaking/justice#sidebar-generative-ai-based-tutoring-system-for-upper-egypt-community-schools","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"generative-ai-based-tutoring-system-for-upper-egypt-community-schools","title":"Generative AI-Based Tutoring System for Upper Egypt Community Schools","content":"This tutoring system piloted in Egypt brought together teachers and AI experts to build a learning platform for under-resourced schools. The focus was on supporting mostly female students, and teachers were involved throughout the design process to make the platform fit real classroom needs. This tool hit all three pillars of distributive justice: Sufficiency: The platform provides a baseline level of educational support to learners who otherwise have insufficient access to tutoring or quality instruction. Priority: the design and distribution method prioritizes girls in low-resource schools, recognizing that they face urgent, systemic barriers. Equality of Opportunity: Teachers co-designed the system to ensure fair access to meaningful learning, preventing AI from reproducing gendered educational disparities.","sidebarKey":"generative-ai-based-tutoring-system-for-upper-egypt-community-schools","isDrawer":true},{"id":"automatedDecisionMaking/justice#sidebar-data-colonialism","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"data-colonialism","title":"data colonialism","content":"Data colonialism as defined by Nick Couldry and Ulises Mejias in their book _The Costs of Connection_ is “process by which governments, non-governmental organizations and corporations claim ownership of and privatize the data that is produced by their users and citizens.” For further reading and watching, see Professor Nick Couldry’s [explanation](https://www.lse.ac.uk/lse-player/what-is-data-colonialism) of data colonialism and how it shows up in tech practices.","sidebarKey":"data-colonialism","isDrawer":true},{"id":"automatedDecisionMaking/justice#sidebar-hegemonic-definitions-of-accuracy","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"hegemonic-definitions-of-accuracy","title":"hegemonic definitions of accuracy","content":"Hegemonic definitions of accuracy or epistemological hegemony refer to situations in which one dominant group’s standards for what counts as true, credible, valid, or “accurate” are treated as neutral and universal, while other ways of knowing or other speakers are discounted. In practice, this means that knowledge is judged not only by evidence, but also by who is speaking, which social perspective is treated as authoritative, and which histories of power shape credibility. An example would be an AI or institutional system that treats mainstream, Western, bureaucratic, or majority-group data categories as the most “accurate” way to describe reality, while dismissing the lived testimony of marginalized people or ignoring Indigenous and community-based understandings. For more information and discussion on this topic, see the “[Round Table: Epistemic Justice](https://youtu.be/S0uQ534Ox9g?si=4TRzMoARVxGvEbZQ)” from the Stanford Humanities Center.","sidebarKey":"hegemonic-definitions-of-accuracy","isDrawer":true},{"id":"automatedDecisionMaking/justice#sidebar-technological-infrastructure","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"technological-infrastructure","title":"Indigenous Data Sovereignty in Language AI (Papa Reo, Te Hiku Media)","content":"Te Hiku Media, a Māori-led media organization in Aotearoa New Zealand, has developed Papa Reo, a language AI platform grounded in Indigenous knowledge systems and principles of data sovereignty. The project emerged from decades of work to revitalize te reo Māori and responds directly to the limitations of Western-centered AI development, which typically relies on large-scale data extraction, standardization, and centralized control. In contrast, Indigenous approaches emphasize relationality, community accountability, and the right of communities to control how their knowledge is collected, used, and shared. Western data practices often treat knowledge as neutral “data” to be extracted and validated through dominant frameworks, sidelining local epistemologies and reducing communities to sources of input. This reproduces epistemic injustice by privileging Euro-Western standards of accuracy and authority, while silencing or distorting other ways of knowing. In AI, this logic is reflected in the “more is more” paradigm, where large datasets are scraped and aggregated with little regard for consent, context, or community benefit, often reinforcing historical patterns of exclusion and marginalization. Papa Reo offers a different model. Rather than extracting Māori language data into global systems, Te Hiku Media built its own infrastructure for speech recognition and natural language processing, using community-held archives and participatory data collection. Crucially, the project ensures that Māori communities retain ownership and governance over their data, and that the benefits of AI development flow back to them. This includes developing tools tailored to smaller datasets and supporting other Indigenous communities in building their own capabilities, rather than forcing them into dominant technological frameworks. The Papa Reo case demonstrates how Indigenous data governance can actively resist epistemic injustice. It reframes communities not as passive data subjects but as knowledge holders and decision-makers, restores control over meaning-making and representation, and challenges the assumption that openness and scale are inherently just. In doing so, it points toward a broader transformation of AI: from extractive systems that reproduce historical inequities to relational systems that respect sovereignty, enable participation, and sustain cultural knowledge in the digital age. The project can be found here: [https://papareo.io/](https://papareo.io/)","sidebarKey":"technological-infrastructure","isDrawer":true},{"id":"automatedDecisionMaking/justice#sidebar-ai-infrastructure","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"justice","subsectionTitle":"Justice in AI","anchor":"ai-infrastructure","title":"Thirsty for power and water, AI-crunching data centers sprout across the West","content":"[Several U.S. cities](https://andthewest.stanford.edu/2025/thirsty-for-power-and-water-ai-crunching-data-centers-sprout-across-the-west/), Phoenix, Las Vegas, The Dalles (Oregon), reported significant community conflict over the water consumption of hyperscale data centers used to train and host large AI models. Public records revealed that some centers used billions of gallons of potable water per year for cooling, disproportionately affecting local drought-prone or low-income regions. This case illustrates how AI infrastructure can produce environmental injustice by placing ecological burdens on communities with limited political power, while the benefits flow primarily to distant corporate and global users.","sidebarKey":"ai-infrastructure","isDrawer":true},{"id":"automatedDecisionMaking/governance#intro","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"intro","title":"Introduction","content":"As AI systems scale in power and impact, so does the urgency to ensure they are aligned with legal standards and public accountability. Therefore, the need for effective AI governance has become critical to direct how these systems are developed, deployed, and overseen.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/governance#defining-ai-governance","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"defining-ai-governance","title":"Defining AI Governance","content":"AI governance refers to the system of rules, practices, and processes employed to ensure an organization’s use of AI technologies aligns with its strategies, objectives, and values, complete with legal requirements, ethical principles, and the requirements set by stakeholders.[^4]\n\nAt its core, AI governance is about accountability – the principle that AI “should be developed, deployed, and utilized such that responsibility for bad outcomes can be assigned to liable parties.”[^3] Governance, therefore, concerns not only what AI systems do, but also who bears responsibility when they malfunction, discriminate, and cause harm.\n\nExisting approaches in AI governance can be framed around three key questions:\n\n1. **What** are we concerned about, and what are you measuring for, then\n2. **How** are we measuring these things, and finally\n3. **Who** are the key stakeholders shaping how AI is governed?\n\n### _What_ Is Being Regulated in AI Governance?\n\nAI governance targets both the **technical components** of AI systems and the **sociotechnical settings** in which they operate:\n\n- **Data**: how data is collected, cleaned, and labeled; privacy concerns and data provenance; bias mitigation.\n\n- **Compute**: control over computing power, environmental footprint, and access inequities.\n\n- **Models**: model architecture, training data provenance, and transparency of weights or parameters.\n\n- **Deployment Contexts**: specific applications (e.g., surveillance, hiring, healthcare) that raise domain-specific risks.\n\n- **Decision Making**: across various downstream application scenarios, decision-making processes that produce real-world impact on institutions and communities.\n\n### _How_ Is AI Being Governed?\n\nAI governance operates through multiple, overlapping mechanisms that translate accountability into practice. Generally, there are 5 major layers of governance approaches: technical design, industry standards, economic incentives, legislation and policies, and community conventions.\n\n#### Governance by Code: Technical Design\n\nGovernance by code embeds oversight directly into the technical stack, through how data is curated, how models are trained, and who controls access to compute.\n\n**{Data-governance}** encompasses filtering, anonymization, and bias mitigation at the dataset level.\n\n**{Algorithmic-governance}** influences model behavior through architectural choices, safety constraints, and transparency features.\n\n#### Governance by Coordination: Industry Standards\n\nGovernance by coordination refers to voluntary or semi-formal **{standards}** that create shared benchmarks for safety, documentation, and reliability without formal regulatory enforcement. These standards usually help establish what “responsible AI” looks like in practice.\n\n#### Governance by Market: Economic Incentives\n\n{Market-forces} and procurement choices pressure organizations to align with governance norms. Funding mechanisms and liability regimes determine who can afford responsible development, which means accountability is (always) also an economic question. In particular, {procurement-governance} uses the purchasing power of large institutions (especially governments) to enforce responsible AI practices by making them a condition of access to public contracts.\n\n#### Governance by Law: Legislation and Policies\n\nGovernments enforce accountability through {binding-rules}, ranging from national executive orders and state-level laws to international frameworks, which set thresholds for risk, transparency, and liability.\n\n#### Governance by Norm: Community Conventions\n\nProfessional, academic, and {civic-communities} establish informal expectations about responsible AI practice, and they often drive change with bottom-up legitimacy that can later be crystallized into formal policy.\n\nTogether, these mechanisms above express the “how” of governance. Accountability emerges not from any single layer, but from their interaction across technical infrastructure, regulatory bodies, market forces, and community norms.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/governance#stakeholders-in-ai-governance","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"stakeholders-in-ai-governance","title":"Stakeholders in AI Governance","content":"After defining the frameworks for key ingredients, governance approaches, and underlying values in AI governance, the next question is: **_Who_** _participates in shaping those frameworks?_\n\nThe major stakeholders involved in AI governance can be divided into four categories: governments, industry actors, academic institutions, and civil society organizations. Each player has distinct incentives, capacities, and claims to legitimacy. Understanding their roles and tensions is essential to grasping how AI governance is negotiated in practice.\n\n![stakeholders](/assets/primer-photos/ADM/governance/stakeholders.png)\n\n### Government Overview\n\nGovernments are central to AI governance, while the approach differs significantly by region. While the EU has adopted a comprehensive, horizontal strategy (a single cross-sector law applying to all AI uses), China has taken a more centralized, state-led regulatory approach. In contrast, in the United States, government involvement in AI governance has spanned multiple levels but remains highly _decentralized_. While dozens of AI-related bills have been introduced in Congress, there is no unified, horizontal AI law akin to the EU AI Act. Instead, the US has taken a more fragmented, sector-specific vertical approach (separate rules for each industry or use case, such as healthcare or hiring). This decentralization gives agencies and state governments room to experiment, but also creates ambiguity around enforcement and consistency.\n\nThe government stakeholder landscape can be segmented into **three areas**:\n\n1. **National/regional governments**: the national executive and legislative bodies.\n2. **Regulatory organizations:** independent agencies and commissions that enforce laws and have developed guidelines around some aspects of AI use\n3. **State entities:** state governments that create and enforce AI-related laws/policies within their own jurisdictions\n4. **Standard-setting bodies:** while not formally a part of government, the standard-setting bodies play a critical role in creating shared benchmarks and best practices that guide government regulation.\n\n#### National/Regional Legislation\n\nNational and regional {AI-legislation} represents the most direct form of governance by law. They usually define permissible uses, accountability mechanisms, and enforcement structures. Notably, these legislative approaches may vary dramatically across jurisdictions, reflecting different regulatory philosophies.\n\n#### Regulatory Organizations\n\n{Regulatory-organizations} usually play the role of translating legislation into enforceable practice through investigation, rulemaking, and adjudication. Unlike legislatures that set broad mandates, these agencies develop technical standards, conduct audits, and impose penalties for non-compliance.\n\n#### US State and Local Governments\n\nIn the absence of federal legislation, {US-state-governments} have become active AI governance actors. These interventions vary widely by state and often target specific use cases. See Understanding AI Legislation: The CNTR AISLE Framework[^61] and US State AI Governance Legislation Tracker[^63] for more information.\n\n#### Standard-Setting Bodies\n\nAlongside legislative actors and regulatory agencies, AI governance relies on {standard-setting-bodies} that develop technical benchmarks, auditing tools, and best-practice guidelines for AI systems. These bodies operate in the space between voluntary industry self-regulation and mandatory legal compliance.\n\n### Industry Overview\n\nIn the context of AI governance, companies are a primary industry stakeholder that needs to comply; therefore, they are usually one of the most powerful forces shaping the landscape.\n\nIndustry stakeholders can range from frontier model developers to infrastructure providers and model deployers. Unlike the government that operates through legislation, its decisions around model training, release, and deployment govern AI through code, scale, and market influence.\n\n[image 2]\n\nMapping the AI Supply Chain: An Analysis of the Complex Relationships in the AI Ecosystem[^40]\n\n| Category                      | Key Players                                                                                                                                                                     | Governance Role                                                                                                                                             |\n| :---------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Frontier Developers           | OpenAI, Anthropic, Google, Meta                                                                                                                                                 | Train and release cutting-edge general-purpose foundation models and often set de facto standards in the absence of regulation                              |\n| Infrastructure Providers      | Google Cloud, Microsoft Azure, AWS, NVIDIA                                                                                                                                      | Control access to critical infrastructure for building AI systems: compute, storage, and data                                                               |\n| Model Deployers               | Amazon (E-Commerce), Walmart (Retail); JPMorgan, Upstart (Finance); Tesla (Auto)                                                                                                | Integrate AI models into their services across domains, often without developing models in-house                                                            |\n| Venture Capitals and Startups | a16z, Sequoia, Y Combinator                                                                                                                                                     | Play an indirect but influential role in shaping governance norms by founding startups, lobbying for innovation-friendly policies, and resisting regulation |\n| Industrial Consortia          | U.S. Chamber of Commerce, Information Technology Industry Council (ITI), Chamber of Progress, TechNet, AI-Enabled ICT Workforce Consortium, MIT Generative AI Impact Consortium | Coordinate self-regulation, research, and policy dialogues; preempt regulation by standardizing responsible practices                                       |\n\n### Academia Overview\n\ncan\nAcademic institutions occupy a unique position in AI governance as they conduct independent research on AI. Their primary role in the AI governance space includes proposing best practices and governance methods, building relationships with other stakeholders, establishing projects, and serving as consultants and as {testimonials-in-policy-hearings}.\n\n| University                 | Lab/Center/Initiative                                                            |\n| :------------------------- | :------------------------------------------------------------------------------- |\n| Brown University           | Center for Technological Responsibility, Reimagination, and Redesign (CNTR)[^24] |\n| Harvard University         | Berkman Klein Center (BKC) For Internet & Society[^12]                           |\n| Georgetown University      | Center for Security and Emerging Technology (CSET)[^23]                          |\n| Stanford University        | Institute for Human-Centered Artificial Intelligence (HAI)[^39]                  |\n| Princeton University       | Center for Information Technology Policy (CITP)[^22]                             |\n| UC Berkeley                | Center for Human-Compatible AI (CHAI)[^21]                                       |\n| Carnegie Mellon University | Block Center for Technology and Society[^13]                                     |\n\n### Civil Society Overview\n\nCivil society stakeholders include any independent groups (separate from government or industry organizations, for example) that have a shared collective interest, which in this context is AI governance. These can include non-profit organizations, think tanks, and independent research institutes that have a variety of political perspectives and orientations.\n\nFor civil-society organizations, they range from broad civil rights groups like American Civil Liberties Union (ACLU)[^9] and Center for Democracy and Technology (CDT)[^20] to think tanks like the Brookings Institution[^54], the Cato Institute[^55], the American Enterprise Institute (AEI)[^53] and the Heritage Foundation[^57], to groups with a more specific technical focus like the AI Now Institute[^52], as well as university-housed think tanks like Center for Security and Emerging Technology (CSET)[^23] and Center for Information Technology Policy (CITP)[^22].\n\nMedia and journalism organizations serve as information intermediaries in AI governance. They investigate industry practices, translate technical developments for various audiences, and hold both companies and regulators accountable. They range from technology-focused publications like WIRED[^65] to policy-focused ones like Tech Policy Press[^51], to specialized independent outlets like 404 Media[^1].\n\nPhilanthropic foundations are entities that fund think tanks, researchers, and policy efforts in the space of AI governance. They usually shape AI governance through strategic grantmaking across the ecosystem. There is a wide range of foundations that also represent different agendas and political ideologies, such as the MacArthur Foundation[^58], the Ford Foundation[^56], Open Philanthropy[^47], and many others.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/governance#ai-governance-in-practice","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"ai-governance-in-practice","title":"AI Governance in Practice","content":"AI companion applications like Character.AI[^25] and Replika[^50] have scaled to millions of users, offering emotionally responsive chatbot interactions as friends, romantic partners, or mental health support. Critical incidents expose serious governance gaps. In October 2024[^44], a Florida mother sued Character.AI after her 14-year-old son died by suicide following months of interactions with a chatbot that allegedly encouraged self-harm and made romantic overtures.\n\nThis case matters because it involves vulnerable populations, extraordinarily sensitive data disclosures governed only by consumer privacy policies rather than health protections, platforms making health claims while avoiding FDA oversight, and business models optimizing for engagement over well-being.\n\nAs outlined previously, AI governance mechanisms operate across all five layers in this case, while each has its own limitations and challenges.\n\n- **By code**: platforms implemented crisis-detection classifiers, routing users to helplines and age-gated content restrictions, though users report easy circumvention.\n- **By market**: the Character.AI lawsuit creates liability pressure, but accountability comes only after demonstrable harm.\n- **By law**: California's SB 243[^18] now requires crisis intervention protocols, age verification, and AI disclosure, the first domain-specific regulation, while existing FTC and Children's Online Privacy Protection Act (COPPA) authority remains largely unenforced.\n- **By coordination**: voluntary industry codes of conduct exist, but see minimal adoption.\n- **By norm**: academic research documenting psychological harms and investigative journalism on tragic incidents build public pressure, though media attention is episodic rather than sustained.\n\nThis case reveals fundamental governance challenges that cut across the AI ecosystem.\n\n- First, there is deep regulatory ambiguity: AI companions don't fit neatly into existing categories like social media, healthcare, or entertainment, so no single regulatory framework clearly applies.\n- Second, the problem outpaces our ability to measure and respond to it: companies deploy these systems at scale before we understand their psychological impacts, which means evaluating whether a chatbot creates harmful emotional dependency is far harder than auditing a hiring algorithm for bias.\n- Third, the case exposes how governance operates as contested negotiation across misaligned stakeholders: industry self-regulation proves insufficient to prevent foreseeable harms, market accountability only kicks in after tragedies occur, legal frameworks remain fragmented across jurisdictions, and civil society can document problems and create public pressure but lacks the enforcement power to compel change.\n\nEffective AI governance requires coordination across all five mechanisms and meaningful participation from all four stakeholder groups: government, industry, academia, and civil society. No single layer or actor can establish accountability alone. As the AI companions case demonstrates, the most pressing governance challenges emerge at the intersections: when regulatory frameworks lag behind technical deployment, when market incentives misalign with public safety, when industry controls the information needed for independent oversight, and when vulnerable populations bear the costs of governance failures.\n\nMoving forward, successful AI governance will depend on building institutional capacity to respond at the speed of technological change, creating mechanisms for multi-stakeholder deliberation, and ensuring that those affected by AI systems have meaningful power in shaping how they are governed. The frameworks and examples in this primer provide a foundation for understanding how AI governance works in general, and where it still falls short.\n\n[^1]: \"404 Media.\" 404 Media. https://www.404media.co/.\n\n[^2]: \"AAU Response to OSTP RFI on AI Action Plan.\" Association of American Universities. https://www.aau.edu/key-issues/aau-responds-ostps-rfi-development-ai-action-plan.\n\n[^3]: \"AI Accountability.\" Carnegie Council. https://carnegiecouncil.org/explore-engage/key-terms/ai-accountability.\n\n[^4]: \"AI Governance: themes, knowledge gaps and future agendas.\" Emerald Publishing. https://www.emerald.com/insight/content/doi/10.1108/intr-01-2022-0042/full/html.\n\n[^5]: \"AI Impact Assessment (AIIA).\" IAPP. https://iapp.org/news/a/ai-assessments-how-and-when-to-conduct-them.\n\n[^6]: \"AI Security Institute (AISI).\" UK AI Security Institute. https://www.aisi.gov.uk/.\n\n[^7]: \"Algorithmic Justice League (AJL).\" Algorithmic Justice League. https://www.ajl.org/.\n\n[^8]: \"Amazon.\" MIT Technology Review. https://www.technologyreview.com/2020/06/12/1003482/amazon-stopped-selling-police-face-recognition-fight/.\n\n[^9]: \"American Civil Liberties Union (ACLU).\" American Civil Liberties Union. https://www.aclu.org/.\n\n[^10]: \"Andreessen Horowitz (a16z): Regulate AI Use, Not AI Development.\" Andreessen Horowitz. https://a16z.com/regulate-ai-use-not-ai-development/.\n\n[^11]: \"Anthropic's Constitutional AI.\" arXiv. https://arxiv.org/pdf/2212.08073.\n\n[^12]: \"Berkman Klein Center (BKC) For Internet & Society.\" Harvard Berkman Klein Center. https://cyber.harvard.edu/.\n\n[^13]: \"Block Center for Technology and Society.\" Carnegie Mellon University. https://www.cmu.edu/block-center.\n\n[^14]: \"Brussels Effect.\" Columbia Law School. https://scholarship.law.columbia.edu/books/232/.\n\n[^15]: \"California AB489 (AI in Healthcare).\" California Legislature. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB489.\n\n[^16]: \"California AB853 (AI Transparency).\" California Legislature. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB853.\n\n[^17]: \"California Civil Rights Council: AI Employment Discrimination Regulations.\" California Civil Rights Department. https://calcivilrights.ca.gov/2025/06/30/civil-rights-council-secures-approval-for-regulations-to-protect-against-employment-discrimination-related-to-artificial-intelligence/.\n\n[^18]: \"California SB 243.\" California Legislature. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB243.\n\n[^19]: \"California's GenAI procurement guidelines.\" California Department of Technology. https://cdt.ca.gov/wp-content/uploads/2024/07/3a-GenAI-Guidelines.pdf.\n\n[^20]: \"Center for Democracy and Technology (CDT).\" Center for Democracy and Technology. https://cdt.org/.\n\n[^21]: \"Center for Human-Compatible AI (CHAI).\" UC Berkeley CHAI. https://humancompatible.ai/.\n\n[^22]: \"Center for Information Technology Policy (CITP).\" Princeton CITP. https://citp.princeton.edu/.\n\n[^23]: \"Center for Security and Emerging Technology (CSET).\" Georgetown CSET. https://cset.georgetown.edu/.\n\n[^24]: \"Center for Technological Responsibility, Reimagination, and Redesign (CNTR).\" Brown University CNTR. https://cntr.brown.edu/.\n\n[^25]: \"Character.AI.\" Wikipedia. https://en.wikipedia.org/wiki/Character.ai.\n\n[^26]: \"Colorado's AI Act (SB24-205).\" Colorado Legislature. https://leg.colorado.gov/bills/sb24-205.\n\n[^27]: \"Cyberspace Administration of China (CAC).\" Cyberspace Administration of China. https://www.cac.gov.cn/.\n\n[^28]: \"Department of Justice (DOJ).\" US Department of Justice. https://www.justice.gov/.\n\n[^29]: \"DOJ Artificial Intelligence Strategy.\" US Department of Justice. https://www.justice.gov/d9/pages/attachments/2021/02/04/doj_artificial_intelligence_strategy_december_2020.pdf.\n\n[^30]: \"EEOC's role in AI.\" Equal Employment Opportunity Commission. https://www.eeoc.gov/sites/default/files/2024-04/20240429_What%20is%20the%20EEOCs%20role%20in%20AI.pdf.\n\n[^31]: \"Equal Employment Opportunity Commission (EEOC).\" Equal Employment Opportunity Commission. https://www.eeoc.gov/overview.\n\n[^32]: \"EU AI Office.\" European Commission. https://digital-strategy.ec.europa.eu/en/policies/ai-office.\n\n[^33]: \"EU Artificial Intelligence Act.\" EU AI Act. https://artificialintelligenceact.eu/.\n\n[^34]: \"EU Commission.\" European Commission. https://commission.europa.eu/index_en.\n\n[^35]: \"Executive Order 14110: Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.\" Federal Register. https://www.federalregister.gov/documents/2023/11/01/2023-24283/safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence.\n\n[^36]: \"Federal Trade Commission (FTC).\" Federal Trade Commission. https://www.ftc.gov/.\n\n[^37]: \"FTC: Artificial Intelligence tag page.\" Federal Trade Commission. https://www.ftc.gov/industry/technology/artificial-intelligence.\n\n[^38]: \"IBM.\" Medium. https://medium.com/@Joy.Buolamwini/ibm-leads-more-should-follow-racial-justice-requires-algorithmic-justice-and-funding-da47e07e5b58.\n\n[^39]: \"Institute for Human-Centered Artificial Intelligence (HAI).\" Stanford HAI. https://hai.stanford.edu/.\n\n[^40]: \"Mapping the AI Supply Chain: An Analysis of the Complex Relationships in the AI Ecosystem.\" AI Supply Chains. https://aisupplychains.org/.\n\n[^41]: \"Microsoft.\" CNBC. https://www.cnbc.com/2020/06/11/microsoft-says-will-not-sell-facial-recognition-software-to-police.html.\n\n[^42]: \"Ministry of Industry and Information Technology (MIIT).\" Ministry of Industry and Information Technology. https://www.miit.gov.cn/.\n\n[^43]: \"National Institute of Standards and Technology (NIST).\" National Institute of Standards and Technology. https://www.nist.gov/.\n\n[^44]: \"NBC News: Florida teen death lawsuit against Character.AI.\" NBC News. https://www.nbcnews.com/tech/characterai-lawsuit-florida-teen-death-rcna176791.\n\n[^45]: \"NYC AI Action Plan.\" NYC Government. https://www.nyc.gov/assets/oti/downloads/pdf/reports/artificial-intelligence-action-plan.pdf.\n\n[^46]: \"NYC Automated Employment Decision Tools (AEDT).\" NYC Government. https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page.\n\n[^47]: \"Open Philanthropy.\" Open Philanthropy. https://www.openphilanthropy.org/.\n\n[^48]: \"Overview of all AI Act National Implementation Plans.\" EU AI Act. https://artificialintelligenceact.eu/national-implementation-plans/.\n\n[^49]: \"Reinforcement Learning from Human Feedback (RLHF).\" arXiv. https://arxiv.org/pdf/2307.15217.\n\n[^50]: \"Replika.\" Wikipedia. https://en.wikipedia.org/wiki/Replika.\n\n[^51]: \"Tech Policy Press.\" Tech Policy Press. https://www.techpolicy.press/.\n\n[^52]: \"the AI Now Institute.\" AI Now Institute. https://ainowinstitute.org/.\n\n[^53]: \"the American Enterprise Institute (AEI).\" American Enterprise Institute. https://www.aei.org/.\n\n[^54]: \"the Brookings Institution.\" Brookings Institution. https://www.brookings.edu/.\n\n[^55]: \"the Cato Institute.\" Cato Institute. https://www.cato.org/.\n\n[^56]: \"the Ford Foundation.\" Ford Foundation. https://www.fordfoundation.org/.\n\n[^57]: \"the Heritage Foundation.\" Heritage Foundation. https://www.heritage.org/.\n\n[^58]: \"the MacArthur Foundation.\" MacArthur Foundation. https://www.macfound.org/.\n\n[^59]: \"the NIST AI Risk Management Framework (AI RMF).\" National Institute of Standards and Technology. https://www.nist.gov/itl/ai-risk-management-framework.\n\n[^60]: \"the OECD AI Principles.\" OECD. https://oecd.ai/en/ai-principles.\n\n[^61]: \"Understanding AI Legislation: The CNTR AISLE Framework.\" Brown University CNTR. https://cntr.brown.edu/news/2025-03-12/cntr-aisle-framework.\n\n[^62]: \"US Census Bureau: Differential Privacy for the 2020 Census.\" US Census Bureau. https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance/differential-privacy.html.\n\n[^63]: \"US State AI Governance Legislation Tracker.\" IAPP. https://iapp.org/resources/article/us-state-ai-governance-legislation-tracker/.\n\n[^64]: \"Virginia's HB2094.\" Virginia Legislature. https://lis.virginia.gov/bill-details/20251/HB2094/text/HB2094.\n\n[^65]: \"WIRED.\" WIRED. https://www.wired.com/.","sidebarKey":null,"isDrawer":false},{"id":"automatedDecisionMaking/governance#sidebar-data-governance","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"data-governance","title":"A case study of data governance","content":"**Privacy-enhancing technologies (PETs)** enable data analysis while protecting individual privacy through methods like differential privacy, federated learning, and homomorphic encryption. For example, the US Census Bureau adopted differential privacy for the 2020 Census[^62]. By adding carefully calibrated statistical noise to aggregate data releases, the Bureau ensured that no individual’s information could be reverse-engineered from public statistics.","sidebarKey":"data-governance","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-algorithmic-governance","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"algorithmic-governance","title":"A case study of algoirthmic governance","content":"**Reinforcement Learning from Human Feedback (RLHF)**[^49] and Anthropic’s Constitutional AI[^11] exemplify governance-by-design, encoding normative values into training objectives. In RLHF, models are aligned with human preferences by learning from ranked responses, usually with safety and helpfulness as guiding principles. In particular, Constitutional AI builds on this by replacing human raters with a “constitution” of ethical criteria for the model to critique and refine its own outputs. Overall, these design-based approaches operationalize governance goals into the model’s objective function, shifting oversight upstream into the development process.","sidebarKey":"algorithmic-governance","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-standards","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"standards","title":"Two types of standards in AI governance by coordination","content":"**Principles-based standards** usually articulate high-level values that organizations should aim to uphold. For instance, the OECD AI Principles[^60] established principal values for innovative and trustworthy AI, including human-centered values, transparency, robustness, accountability, and respect for human rights. **Procedural standards** offer structured guidance on how to identify, manage, and regulate risks and harms throughout the AI lifecycle. For example, the NIST AI Risk Management Framework (AI RMF)[^59] provides a structured, procedural guidance for organizations to identify, assess, and manage AI risks throughout the model lifecycle. It is distinguished from principle-based approaches by offering a concrete methodology that maps risks to organizational contexts. Among various procedural standards, AI Impact Assessment (AIIA)[^5] represents a special mechanism that systematically evaluates potential harms before deployment. Typically, AIIAs involve documenting the system’s purpose and context, identifying affected stakeholders, analyzing risks across dimensions, and establishing ongoing monitoring protocols.","sidebarKey":"standards","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-market-forces","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"market-forces","title":"A case study of market forces influencing AI governance","content":"Venture capital firm Andreessen Horowitz (a16z)[^10] shows how market actors shape governance through strategic advocacy against regulation. Through white papers, Congressional testimony, and public campaigns, a16z has lobbied for \"innovation-friendly\" policies that minimize compliance burdens on startups, arguing that heavy-handed regulation would disadvantage US competitiveness and concentrate power in incumbent tech giants who can afford compliance costs.","sidebarKey":"market-forces","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-procurement-governance","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"procurement-governance","title":"A case study of procurement governance","content":"Government procurement functions as a powerful market-based governance mechanism by making responsible AI practices a condition of accessing public contracts. For example, California’s GenAI procurement guidelines[^19] require risk assessments conducted by state agencies and vendor disclosures about AI capabilities. The mechanism shifts accountability upstream by requiring compliance during development rather than after deployment.","sidebarKey":"procurement-governance","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-binding-rules","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"binding-rules","title":"A case study of legally binding rules in AI governance","content":"The EU Artificial Intelligence Act[^33], which entered into force in August 2024, marks the world’s first comprehensive horizontal AI regulation. It introduces a risk-based framework that classifies AI systems into four risk tiers: minimal, limited, high, and prohibited. The Act creates enforcement through both national authorities and significant fines, which sets a potential “Brussels Effect[^14]” where global companies adopt EU standards internationally In the US, the legislative landscape is much more fragmented. Both Virginia’s HB2094[^64] and Colorado’s AI Act[^26] adopt a decision-centric consumer-protection approach, defining AI risk through consequential decision-making capabilities and concrete involvement.","sidebarKey":"binding-rules","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-civic-communities","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"civic-communities","title":"A case study of civic advocacy in AI governance","content":"Founded by Joy Buolamwini, the Algorithmic Justice League (AJL)[^7]’s campaign against facial recognition bias pushed both corporate reform and municipal legislation. AJL initially exposed severe accuracy disparities in commercial facial recognition systems through research, which catalyzed corporate responses from IBM[^38], Amazon[^8], and Microsoft[^41], as well as legislative actions, with multiple US cities banning government use of the technology.","sidebarKey":"civic-communities","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-ai-legislation","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"ai-legislation","title":"Case studies of AI legislation","content":"European Union The EU AI Act[^33] is the world’s first comprehensive horizontal AI regulation. It categorizes systems into risk tiers and imposes requirements accordingly, such as transparency, documentation, and liability structures. China In 2021, China introduced binding AI regulations through the Cyberspace Administration of China (CAC)[^27], targeting recommendation algorithms, deepfakes, and generative AI. United States No national AI law equivalent to the EU AI Act currently exists; instead, executive actions like EO 14110[^35] aim to coordinate federal agency efforts on AI safety, transparency, and equity.","sidebarKey":"ai-legislation","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-regulatory-organizations","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"regulatory-organizations","title":"Case studies of regulatory organizations","content":"United States Federal Trade Commission (FTC)[^36]: Addresses and prevents deceptive and unfair practices in AI systems through consumer protection and competition laws. See the FTC’s Artificial Intelligence tag page[^37]. Department of Justice (DOJ)[^28]: Enforces anti-discrimination and antitrust law as it intersects with AI use, combating criminal misuses of AI and ensuring its deployment aligns with legal and ethical principles. See the DOJ’s Artificial Intelligence Strategy[^29]. Equal Employment Opportunity Commission (EEOC)[^31]: Ensures that AI tools used in employment decisions do not create barriers or perpetuate bias based on protected characteristics, applying existing federal anti-discrimination laws. See the EEOC’s role in AI[^30]. Europe The EU Commission[^34], with support from the EU AI Office[^32], creates and enforces AI legislation like the AI Act, while national regulatory ss are responsible for implementing and overseeing the rules at the Member State level. See the Overview of all AI Act National Implementation Plans[^48]. China Cyberspace Administration of China (CAC)[^27]: Formulates and enforces rules on AI development and use, with a strong focus on content moderation, data security, and generative AI usage. Ministry of Industry and Information Technology (MIIT)[^42]: Guides national AI development through strategic initiatives: industry standards, computing infrastructure, ethical guidelines, fostering innovation ecosystems, etc.","sidebarKey":"regulatory-organizations","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-us-state-governments","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"us-state-governments","title":"Case studies of state-level AI legislation","content":"California Introduced several landmark bills focusing on high-risk AI systems, deepfake disclosure, and safety audits, most notably the Transparency in Frontier Artificial Intelligence Act (SB 53), making it the first US state to directly regulate the safety of powerful AI models. The state has implemented an extensive suite of laws focusing on transparency[^16], consumer protection[^18], and the use of AI in specific sectors like healthcare[^15] and employment[^17]. New York City Regulates AI through its landmark Automated Employment Decision Tools (AEDT)[^46] law, requiring employers to conduct annual bias audits, notify candidates about automated employment decision tool usage, and allow candidates to request alternative processing. The city also launched an AI Action Plan[^45] in 2023 to govern its own use of AI and support agency implementation.","sidebarKey":"us-state-governments","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-standard-setting-bodies","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"standard-setting-bodies","title":"Case studies of standard-setting bodies in AI governance","content":"The National Institute of Standards and Technology (NIST)[^43] introduced the AI Risk Management Framework (AI RMF)[^59] that helps organizations identify, assess, and manage risks associated with AI systems across sectors. The framework effectively translates abstract principles like “trustworthiness” into concrete, assessable organizational practices. The Center for AI Standards and Innovation (CAISI), established within NIST, represents a federal effort to build technical capacity for evaluating frontier AI models and developing safety standards. Similarly, in the UK, emerging bodies like the AI Security Institute (AISI)[^6] are tasked with model evaluation and ecosystem oversight. It is worth noting that AISI has secured voluntary commitments from major AI developers to provide early access to models for pre-deployment safety testing, which is a significant governance innovation that addresses the information asymmetry between industry and government stakeholders.","sidebarKey":"standard-setting-bodies","isDrawer":true},{"id":"automatedDecisionMaking/governance#sidebar-testimonials-in-policy-hearings","section":"automatedDecisionMaking","sectionTitle":"Automated Decision Making","subsection":"governance","subsectionTitle":"AI Governance","anchor":"testimonials-in-policy-hearings","title":"A case study of testimonials from academic institutions in AI governance","content":"This response[^2] from the Association of American Universities (AAU) to the Trump administration’s RFI on the AI Action plan shows what American universities/research institutions seek from the government in the context of AI development. AAU’s “overarching recommendation for the new AI Action Plan is to pursue a focused initiative to accelerate AI for discovery” and that “this initiative should seek to align government investments with industry, universities, and other stakeholders to develop the tools, practices, partnerships, and infrastructure to catalyze scientific progress using AI.”","sidebarKey":"testimonials-in-policy-hearings","isDrawer":true}]