Acknowledgments

The instantiations for the various steps in the ML pipeline from the first and second problem are (mostly) taken from Hal Duame III's blog post on the machine learning pipeline and the Medium article titled Black-Boxed Politics: Opacity is a Choice in AI Systems respectively. These resources also provide inspiration for some of the rest of the material in these notes.

Data collection mechanism: Example 1

Your group decides to log interactions with ads in the current system.

Data collection mechanism: Example 2

Your group decides to use the existing patient electronic health records (which includes details of the current care the patients receive in your hospital but possibly other details).

Data collection mechanism: In-class exercise

To be done in class.

Data collection mechanism: General thoughts

We first discuss potential issues with the choices made in the first two examples:

• One issue with the use of the existing system to log interactions is that this data cannot capture any potential changes that can happen when you actually apply the system. E.g. your new ad display system itself might change behavior of users and that would mean the data your model was trained on is no longer representative. We must note that this issue with using data is always a possibility and this in fact has its own name-- concept/distribution drift . Of course, if the historical data is indeed representative of future data then this might not be an issue.
• One big potential issue with using existing electronic health records is violation of the privacy of patient records. First, one will need consent from patients to use their records (either explicitly or via a blanket consent that the patients might have signed on when their data was entered in their electronic health records). However, even if the hospital has consent, the outcome of your learning system might leak information about patient data, which in some cases might be problematic (esp. if it becomes public even if indirectly).

One common aspect of both of the above examples is that your group had access to data that either already existed or could be logged from an existing system. However, in many scenarios such an access might not be possible and in such a case the data has to be collected somehow. Here is a non-exhaustive list of some possibilities:

• You could use third-party data. There are third party data vendors from whom you could buy data. Of course in this case you might not have much control over the quality or provenance of your data.
• Your group could decide to collect the data. Again, here is a non-exhaustive list of some possibilities:
  • You run a survey to collect data. The question then is whether you run an online survey or you literally have boots on the ground going around to collect data. Both options have their pros and cons.
  • In some cases you can install sensors to collect data (e.g. if you were trying to improve the UB stampede experience, you might want to put a senor that can signal to a central system where a certain bus is at any given time.
  • Rather than use specialized sensors, you could recruit people to allow the use of their smartphones to collect data (e.g. see the Street Bump , which uses data collected from volunteers while they drive on the road condition and then potentially give information to municipalities to improve road conditions.
  • Another way to "crowd-source" data collection is to create a game: e.g. the moral machine experiment that asked player to make moral decisions for self driving cars (do you kill old people vs. kids etc.). As part of the experiment, they were able to collect data on moral decisions that players made and then use it to make some statistical inferences.

Which data to collect?: Example 1

Even though you have access to the current system, you cannot log everything. This could be because e.g. sorting everything would need a lot of storage or perhaps if the system were to log every action it observes then just the act of logging everything can slow down the system (which is not desirable). For example, your group (as Hal suggests ) decides to log queries (for which ads are generated), ads and clicks.

Which data to collect?: Example 2

In this example,by restricting yourself to electronic health records, you are limiting yourself to what is logged into the electronic health records. One could e.g. try and use doctor's notes to glean more information but these are not necessarily standardized and its not clear how to extract information from doctor's notes. Further, there have been complaints from doctors on the usability of electronic health records , which raises issues about accuracy of data being collected. Finally, for the study that your group is planning will most probably need IRB approval from your hospital, which could in turn restrict which data can be collected/used for your system.

Which data to collect?: In-class discussion

To be done in class.

Which data to collect?: General thoughts

if you are not collecting the data yourself then this step would be moot for your group. However, if your group is actually collecting data than which data you can collect would be related to how expensive/feasible it would be. For example, when doing an offline survey, you cannot expect the respondent to talk for say an hour if a volunteer is knocking on their door asking for their opinion. Of course you can schedule longer offline surveys but in that case it would be expensive to collect a lot of data (you necessarily will need to hire more folks to do the interview)-- not to mention that respondents might not have the time to give detailed answers. (In some cases you might have to offer incentives for respondents to respond to question: e.g. monetary incentives in which case there is a much direct link to expenses and the amount of data collected. If your group is using an online survey then again, the time taken to complete a survey should be on the shorter side since a long survey might have a much lower completion rate.

What the above implies is that for survey (and other data collection methods), one would need to collect specific data/ask few questions that you as a group would deem most important.

In some cases, even though your group might want to collect certain data, it might be sensitive and e.g. survey responders might not be willing to give that information (e.g. date of birth). Even in cases where respondents might be willing to give some information, there might be other institutions constraints that prevent this-- see e.g. the discussion on IRB below.