An overview

Recall that we have considered various notions of bias that can creep in when the ML pipeline interacts with society:

For simplicity assume that you as the ML pipeline designer has been able to get rid of all biases expect historical bias (because well, it is present in society to begin with and hence out of your control¹) and measurement bias (because at some point, one will have to measure a proxy variable for the actual variable that might not be directly measurable). In other words, the situation looks like this:

Now consider what happens when you deploy your ML pipeline in society and the deployment itself changes society in some way (and hence affects the next round of training of your model-- assuming you re-train your model to reflect changes in society, which most ML pipelines will do), like so (gif generated via ezgif ):

In other words, a model is developed that has some of the biases from society, which is then deployed back into society, where it changes the society (in particular, it can now contribute to historical and measurement bias in the future), which in turn changes the ML model for the future and the feedback loop continues:

Predictive Policing

We will use predictive policing as the "application area" to talk about feedback loops. In short, predictive policing is the use of an ML pipeline to predict when and where crime is going to happen next. Such a tool is used by police departments to figure out "crime hot spots" so that it can send more cops to areas with higher risk of crime happening--

Of course by now y'all should realize that this can potentially be problematic:

What we will see in these notes is that using predictive policing along with the feedback loop mechanism can actually make things worse:

Formalizing the intuition on feedback loops

So far we have seen intuitively that predictive policing can contribute to feedback loops, which can be harmful. But we need to go beyond intuition-- can we "prove" that predictive policing can lead to harmful feedback loops. We will see two ways of doing this-- one via simulations and another via theoretical modeling, which we do next.

Simulations for feedback loops

Our goal in this section is to run simulations to show that using PredPol (or similar predictive policing algorithms) can lead to feedback loops. We will walk through the process of "re-creating" the set of Lum and Isaac. We start off with the high-level question:

The question(s) we want to answer

As one might imagine even though we made two assumptions above (i.e. we have access to drug crime data in Buffalo in 2019 and we have access to PredPol), there are still many more things that we will need to figure. To direct the discussion, let us state the two main questions we want to answer. The question is to do with the initial state: i.e. what happens if one uses PredPol only once and the feedback loop state, we consider the case that the police acts on PredPol's suggestion and the resulting drug crime arrest data is fed back into PredPol and the cycle continues.

Initial state question

We will now focus on the initial state question. The big stumbling block is figuring out the actual number of occurrences of drug crimes in each grid cell in Buffalo. Here is one thing to try:

Hence we must use some other way to get our hand on the actual drug crime use. Of course, this is essentially impossible to do: how do you figure out the incidents of drug crime incidents in the past? You cannot go knock on doors and ask if the residents committed some drug crime on all the days in 2019? As an exercise figure out why this is not feasible!

With the aside out of the way, we now move on to actually figuring out the ground truth-- or more accurately a good approximation of the ground truth

Now with the most thorny question out of the way, let us go back to solving the initial state question:

Feedback loop state question

We will now focus on the feedback loop state question. We begin with the following exercise:

We are ready to try and answer the question:

Lum and Isaac present a fairly compelling case for feedback loop via their simulation results but we will still try and poke holes into their approach (especially since one of them leads to the next section!):

Theoretical modeling of feedback loops

One potential nitpick with simulation results is that they are simulations and e.g. do not guarantee that the observed result would necessarily hold in situations not covered in the simulations. In particular, we would like to argue that under some fairly general conditions feedback loops can exist. More specifically, we would like to argue that no matter what the specific population looks like as long as the general conditions holds, then a feedback loop exists. The above statement is not possible to prove using simulation results since simulations cannot cover all possibilities.

In this section, we will consider a theoretical model to mathematically prove that under some fairly generally conditions, feedback loops always exist. The catch is that to prove theorems one typically has to make simplifying assumptions in the theoretical model. Thus we being by stating the assumptions we will consider in this section (this is only one of the many settings that Ensign, Friedler, Neville, Scheidegger and Venkatasubramanian consider in their paper.

Now that we have listed all our assumptions, we actually have fully specified the model. Below we re-state the model as how the values $n^{(t)}_E$ and $n^{(t)}_W$ change over time:

OK, so we go into the weeds of the model for a bit. Let us go backup and make sure we understand what outcome could correspond to a feedback loop in this model:

Existence of feedback loop (and a fix)

We will very briefly argue why a feedback loop exists in the model above and how one might go about fixing it. We refer the interested reader to the paper by Ensign, Friedler, Neville, Scheidegger and Venkatasubramanian for more details (including they prove the existence of the feedback loop).

Next Up