The Case for Voter-Centered Audits of Search Engines During Political Elections

This page contains supplementary materials for the paper The Case for Voter-Centered Audits of Search Engines During Political Elections by Eni Mustafaraj, Emma Lurie, and Claire Devine. Published in the 3rd ACM Conference Proceedings in Fairness, Accountability, and Transparency (FAT*) 2020. Barcelona, Spain, Jan 27-30, 2020.

AMT Voter Survey

Our Amazon Mechanical Turker (AMT) survey was ran in two batches. The first survey ran between Oct 16-18, 2018 (three weeks before the date of the election). The second survey ran between Oct 30 - Nov 2, 2018, one week before the election date. According to the completion statistics, the median time for completing the survey was less than 3 minutes, with the mean being less than 4 minutes. We paid the respondents 50 U.S. cents for the successful completion, in par with an hourly wage in the interval $7.5-$10 per hour. We collected demographic data for various categories. Below we summarize the results.

This summary makes it clear that our sample is not representative of the population in most attributes: it skews more male, younger (only 12\% of the U.S. population is in the bracket 25-34y, while our sample has 45\%), more white (and it misses groups like Hispanics), and more educated. The only attribute that conforms to the U.S. population is the geographical distribution of participants by state, the proportions in our sample are very close to that of the U.S. population. We used statistics from this website for comparison.

The Audit Method

Two of our datasets, RS-Candidates and Virginia Election relied on automatically collecting Search Engine Result Pages (SERPs) in response to a set of queries. For RS-Candidates, we used the same laptop computer during the entire time period, with the same IP address, and the same blank-state browser in incognito mode (no search history). We installed the web browser automation tool Selenium, controlled through a Python program, to open a new browser instance every time we perform a search, in order to avoid query session interference. Once the search engine result page (SERP) is loaded, we save it as an HTML file. We keep all the original files (so that we can always verify the results of our automatic scraping). For the Virginia Election dataset, we use two laptops, one located in Virginia and one in Massachussets. With this exception, all other details for the data collection were the same.