Supplemental information about the pilot user study

The script for our study can be found here: Study procedure

Usage of Tranco in our study

Tranco is a research-oriented top sites ranking site that provides the latest list of domains obtained by averaging all rankings from Alexa, Cisco Umbrealla, Majestic, Quantcast, and Farsight over 30 days. Researchers used the ranking data from the Tranco list downloaded on June 30th 2022 to analyze the ranking range of the domains with the purpose of finding a search engine result page (SERP) with a variety of domain rankings from our data of 130 queries. These queries were collected by a previous user study of the Wellesley Cred Lab that focused on generating realistic search queries of two sensitive topics, domestic violence and gender identities. A larger ranking range indicated a mix of highly recognizable and uncommon domains. We wanted to find a SERP with a variety of domain rankings because our research objective included understanding user search behavior regarding unfamiliar sites. Using the Tranco list, we were able to conclude that the SERP for the query “How to learn to receive healthy love” had the best mix of popular and uncommon domains to use as our study stimulus. During our sessions, we would randomize the ranking of the domains to analyze participant search behavior when faced with a different rankings.

Study Stimuli

One of the goals of our study was to observe and understand user search behavior regarding unfamiliar sources. We created a search engine result page mock-up that consisted of search results that could be randomized. We used data collection methods such as Useberry, a codeless analytic tool and google forms.

mockup

Through literature review, we compiled a list of 9 W3C signals that were important for crediblity at a domain and article level. We presented the following signals to partcipants to get their opinion on the percieved usefulness of these signals in helping them distinguish credible sources. See Figure 9 in Findings.

signals