Page 20 -

1.4 Changes in HCI research methods over time 5

a citation analysis to understand trends in research, but most of that analysis is now
easily available using tools such as Google Scholar. On the other hand, automated
tools for testing interface accessibility, are still imperfect and have not yet replaced
the need for human evaluations (either with representative users or interface experts).
One important difference between HCI research and research in some of the other
social sciences (such as sociology and economics), is that, large entities or govern-
ment agencies collect, on an annual basis, national data sets, which are then open for
researchers to analyze. For instance, in the United States, the General Social Survey,
or government organizations such as the National Center on Health Statistics, the US
Census Bureau, or the Bureau of Labor Statistics, collect data using strict and well-
established methodological controls. Outside of the US, agencies such as Statistics
Canada, and EuroStat, collect excellent quality data, allowing researchers to, in many
cases, to focus less on data collection and more on data analysis. However, this prac-
tice of national and/or annual data sets, does not exist in the area of HCI. Most HCI
researchers must collect their own data. So that alone makes HCI research complex.
Typically, HCI research has utilized smaller size datasets, due to the need for re-
searchers to recruit their own participants and collect their own datasets. However, as
the use of big data approaches (sensors, text analysis, combining datasets collected
for other purposes) has recently increased, many researchers now utilize larger pools
of participant data in their research. Whereas, studies involving participants might
have had 50 or 100 users, it is common now to see data from 10,000–100,000 users.
That is not to say that researchers have actually been interacting with all of those us-
ers (which would be logistically impossible), but data has been collected from these
large data sets. Doing research involving 100,000 users versus 50 users provides an
interesting contrast. Those 100,000 users may never interact with the researchers or
even be aware that their data is being included in research (since the terms of service
of a social networking service, fitness tracking, or other device, may allow for data
collection). Also, those participants will never get to clarify the meaning of the data,
and the researchers, having no opportunity to interact with participants, may find it
hard to get a deeper understanding of the meaning of the data, from the participants
themselves. Put another way, big data can help us determine correlations (where
there are relationships), but might not help us determine causality (why there are re-
lationships) (Lehikoinen and Koistinen, 2014). On the other hand, by interacting with
participants in a smaller study of 50 participants, researchers may get a deeper under-
standing of the meaning of the data. Combining big data approaches with researcher
interaction with a small sampling of users (through interviews or focus groups) can
provide some of the benefits of both approaches to data collection, understanding not
only the correlations, but also the causality (Lehikoinen and Koistinen, 2014).
Another important difference between HCI research and research in some of the
other fields of study is that longitudinal studies in HCI are rare. Fields such as medicine
may track health outcomes over a period of decades. National census data collection can
occur over centuries. However, longitudinal data generally does not exist in the area of
HCI. There could possibly be a number of reasons for this. Technology in general, and
specific tools, change so rapidly that, a comparison of computer usage in 1990, or even