Other Courses

Below we list some courses that are somewhat similar to ours (see the above for a more comprehensive list):

• A Course on Fairness, Accountability and Transparency in Machine Learning , crash course sponsored by the GIAN program of the Government of India.
• Ethical and Policy Dimensions of Information, Technology, and New Media , U. of Colorado.
• Fairness in Machine Learning , U. of Berkeley, 2017.
• Human-centered Machine Learning , U. of Colorado.
• Science of Data Ethics , UPenn..
• When Machines Decide: The Promise and Peril of Living in a Data-Driven Society , U. of Utah.

Technical Books

Below are some books that talk about the technical issues that we will consider in this course:

• Solon Barocas, Moritz Hardt and Arvind Narayanan, Fairness and machine learning . Online textbook (in progress).
• Michael Kearns and Aaron Roth, The Ethical Algorithm . This book is targeted at a non-technical audience though it does go into some of the high level ideas of the underlying algorithms.

Research Papers

Below is a list of research papers that we will refer to in this course roughly categorized by the broad topics we will cover in this course:

Discrimination and Fairness

• Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford and Hanna Wallach, A Reductions Approach to Fair Classification . In ICML 2018. [Full version ]
• Alexandra Chouldechova, Fair prediction with disparate impact: A study of bias in recidivism prediction instruments . In FAT/ML 2016.
• Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold and Rich Zemel, Fairness through awareness . In ITCS 2012. [Full version ]
• Cynthia Dwork and Christina Ilvento, Fairness Under Composition . In ITCS 2019. [More Details ]
• Sorelle A. Friedler, Carlos Scheidegger and Suresh Venkatasubramanian, On the (im)possibility of fairness . 2016.
• Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P. Hamilton and Derek Roth, A comparative study of fairness-enhancing interventions in machine learning . In FAT* 2019. [Arxiv version ]
• Michael Kearns, Seth Neel, Aaron Roth and Zhiwei Steven Wu Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness . In ICML 2018 [Arxiv version ]
• Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, Inherent Trade-Offs in the Fair Determination of Risk Scores . In ITCS 2017. [More Details ]
• Latanya Sweeney, Discrimination in Online Ad Delivery . In CACM, May 2013 . [Full version ]

Bias

• Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove and Aaron Rieke , Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes . In CSCW 2019.
• Joy Buolamwini and Timnit Gebru, Gender Shades: Intersectional Accuracy Disparities inCommercial Gender Classification . In FAT* Conference, 2018. [More Details ]
• Aylin Caliskan, Joanna J. Bryson and Arvind Narayanan, Semantics derived automatically from language corpora contain human-like biases . In Science April 2017.
• Le Chen, Anikó Hannák, Ruijin Ma and Christo Wilson, Investigating the Impact of Gender on Rank in Resume Search Engines . In CHI 2018. [Author copy of the paper ]
• Thomas Davidson, Debasmita Bhattacharya and Ingmar Weber, Racial Bias in Hate Speech and Abusive Language Detection Datasets . In AWL 2019.
• Batya Friedman and Helen Fay Nissenbaum, Bias in computer systems . In ACM Transactions on Information Systems 1996. [Full version ]
• Hila Gonen and Yoav Goldberg, Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them . In NAACL 2019.
• Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, Dissecting racial bias in an algorithm used to manage the health of populations . In Science 2019.
• Ronald E. Robertson, David Lazer and Christo Wilson, Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages . In WWW 2018. [Author copy of the paper ]
• Ronald E. Robertson and Shan Jiang and Kenneth Joseph and Lisa Friedland and David Lazer and Christo Wilson, Auditing Partisan Audience Bias within Google Search . In CSCW 2018. [Author copy of the paper ]
• Gabriel Stanovsky, Noah A. Smith and Luke Zettlemoyer, Evaluating Gender Bias in Machine Translation . In ACL 2019. [More Details ]
• Nathaniel Swinger, Maria De-Arteaga, Neil Thomas Heffernan IV, Mark DM Leiserson and Adam Tauman Kalai, What are the biases in my word embedding? . In AIES 2019. [More Details ]

Explainable AI

• Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Dino Pedreschi and Fosca Giannotti, A Survey of Methods for Explaining Black Box Models . In ACM Computing Surveys 2018. [More Details ]
• Zachary C. Lipton, The Mythos of Model Interpretability . In ICML 2016 Workshop on Human Interpretability in Machine Learning. [More Details ]
• Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin, "Why Should I Trust You?": Explaining the Predictions of Any Classifier . In KDD 2016.

Feedback loops

• Eszter Bokányi and Anikó Hannák, Ride-share matching algorithms generate income inequality . Manuscript 2019.
• Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger and Suresh Venkatasubramanian, Runaway Feedback Loops in Predictive Policing . In FAT* 2018.
• Lily Hu and Yiling Chen, Fairness at Equilibrium in the Labor Market . In FAT/ML 2017.
• Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz and Moritz Hardt, Delayed Impact of Fair Machine Learning . In ICML 2018. [More Details ]
• Kristian Lum and William Isaac, To predict and serve? . In Significance 2016.

AI and Law

• Jack M. Balkin, The Three Laws of Robotics in the Age of Big Data . 2017.
• Solon Barocas and Andrew D. Selbst, Big Data's Disparate Impact . In California Law Review 2016.
• Ryan Calo, Artificial Intelligence Policy: A Primer and Roadmap . 2017.
• Alice Xiang and Inioluwa Deborah Raji, On the Legal Compatibility of Fairness Definitions . In Workshop on Human-Centric Machine Learning 2019.

Interfacing with Society

• Susan Athey, Beyond prediction: Using big data for policy problems . In Science 2017.
• Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon and Iyad Rahwan, The Moral Machine experiment . In Nature 2018.
• Sebastian Benthall and Bruce D Haynes, Racial categories in machine learning . In FAT* 2019. [More Details ]
• David Danks and Alex J. London, Algorithmic bias in autonomous systems . In IJCAI 2017. [Author copy ]
• Sina Fazelpour and Zachary C. Lipton, Algorithmic Fairness from a Non-ideal Perspective . 2020.
• Marion Fourcade and Kieran Healy, Seeing like a market . In Socio-Economic Review 2016. [Author copy ]
• Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III and Kate Crawford, Datasheets for Datasets . In FAT/ML 2018.
• Bruce Glymour and Jonathan Herington, Measuring the Biases that Matter: The Ethical and Casual Foundations for Measures of Fairness in Algorithms . In FAT* 2019. [Author copy ]
• Ben Green, “Fair” Risk Assessments: A Precarious Approach for Criminal Justice Reform . In FAT/ML 2018.
• Anna Hoffmann, Where fairness fails: On data, algorithms, and the limits of antidiscrimination discourse.. In Information, Communication, and Society 2019.
• Zachary C. Lipton, The Mythos of Model Interpretability . In ICML 2016 Workshop on Human Interpretability in Machine Learning.
• Samir Passi and Solon Barocas, Problem Formulation and Fairness . In FAT* 2019. [More Details ]
• Andrew D. Selbst, danah boyd, Sorelle Friedler, Suresh Venkatasubramanian and Janet Vertesi, Fairness and Abstraction in Sociotechnical Systems . In FAT*2019. [More Details ]
• Harini Suresh and John V. Guttag, A Framework for Understanding Unintended Consequences of Machine Learning . 2019.