The course Research Methods deals with research methods relevant to Web Science and the design of interactive systems.
The Big Picture
Whether it is the evaluation of existing system, or the conceptualisation, design, or development of a new technology, conducting research is an essential part in this endeavour. The design of such a system "requires finding or generating data, analysing it and drawing conclusions" :9, i.e. doing research. An interdisciplinary field such as Web Science with multiple theoretical approaches opens a world of different research methods originated within the various disciplines like psychology, sociology, art and design, computer science, and so on. Since there is no such thing as the perfect method, methodologies and methods have to be chosen wisely based on the purpose of the research. Some methods might be combined, in order to bring together the strengths and compensate weaknesses of the particular methods. The crucial duty remains to choose between the research methods and combine and arrange them appropriately according to the problems and phenomena under study.
In order to be able to justify the choice and use of methodology and methods, it is crucial to understand the basic elements of any research process: methods, methodology, theoretical perspective, and epistemology :2ff.
The research process has to be planned and designed and starts with a carefully phrased research question, which guides both the research design and its conduct. Research questions are closely related to methodology, theory, and epistemology. While in quantitative research, the question is clear and fixed, qualitative research starts with a research question which can also evolve and be refined during the course of study :44.
Knowledge generation within social research is based on data, which can be distinguished between qualitative and quantitative data. Depending on the research design, this data can be archival (secondary sources, such as websites, blogs, newspapers, etc.) or elicited (collected by the researcher, such as interviews, surveys, observational field notes, etc). The collection and analysis of empirical data requires ethical considerations. Ethical implications in research are e.g. rights of research participants and the researcher's responsibilities towards directly and indirectly involved people :54ff.
Depending on the type of data generated and/or collected and the research question, different types of data analysis methods are appropriate (e.g. statistics, thematic analysis), and consequently different quality criteria apply for evaluation of the research. Finally, the results have to be communicated, i.e. the answer to the research question has to be presented (e.g. in a written conference paper, research report, thesis, and so forth).
Intended Learning Outcomes
The participants of this course
Structure of the Course
Introduction To Research and Research Methods
Oates defines research as: "the creation of new knowledge, using an appropriate process, to the satisfaction of the users of the research.":4 Depending on the academic discipline, criteria for what is considered good research might vary. Research can be carried out for many different reasons, e.g. :16
In many approaches for designing interactive systems (e.g. participatory design, value sensitive design, user-centred design, contextual design), some form of research is an essential part in order to understand e.g. the problem that the system is supposed to solve, or the prospective users and their needs which should be addressed in the design.
The research process consists of four elements as outlined by Crotty (1998):3
Basic Elements of the Research Process
Research questions are central in that they help to :65
A simple model of research involves to phrase research questions, determine what kind of data is necessary in order to answer these questions, design the research to collect and analyse the data, use the data to answer the questions, and finally to communicate the results:67.
According to Punch (2014), the criteria for good research questions are::76
Sampling and Recruitment
Both in qualitative and in quantitative research sampling is equally important, because no study can include everything:160f, 243f. Many decisions have to be made, e.g. which people to involve (e.g. whom to interview or to invite to a focus group), which situations to observe, the settings and processes. Even when it comes to analysis of documents, the researcher will be confronted with sampling issues:161f.
There is a difference between qualitative and quantitative research, when it comes to sampling. Quantitive research tends to focus on people sampling and often uses probability sampling directed at representativeness, while qualitative research use rather deliberate sampling ('purposive sampling'), with some purpose or focus in mind:161. In general, the sampling strategy should fit with the purposes, questions and overall strategy of the research :164.
The decision what kind of sample will be included is directly related to the recruitment, because potential participants have to be informed about the research through some form of advertisement:59ff. Different approaches may be applied in how the advertisement takes place, however, recruitment materials and strategies may have to be approved under the ethics protocol :59.
Due to scandals in research and horrific experiments on humans in the Second World War, ethical codes and regulations have been developed to define ethically acceptable and unacceptable practice :61ff:41ff. The British Psychological Society developed in their Code of Ethics and Conduct  four ethical principles, which have been elaborated by Braun and Clarke (2013):62ff:
Obtaining informed consent from participants necessitates that participants have been thoroughly informed and understand why the study is being conducted and what procedures are involved :381ff. Typically participants are provided with:67:
According to the American Psychological Association, participants have to be informed about: 
Informed Consent to Research
Several associations provide templates for informed consent forms, e.g. the World Health Organization (WHO) .
Although most of the common guidelines apply also to online research, some of the requirements relating to "offline" research are difficult to be met in Internet research :64ff, :391. Internet research poses difficulties in terms of how people are regarded: are they subjects of research or authors of works (e.g. websites, blog posts) :65. In this relation it is also questionable, whether the Internet can be seen as a text (which would lead to the adoption of certain procedures, e.g. credit their work) or a space (thus, took the communication place in public or private space) :140f. While some might argue, that information published online is already public knowledge and people should be aware of that, they might in fact "not expect that their remarks are going to be read by others outside of the community":141. The requirement for informed consent may be difficult to achieve when it comes to online studies :64. As mentioned before, one essential part of the informed consent form is the right to decline to participate and to withdraw from the research, which is not possible for the subjects if they are not informed that their remarks are being used in research.
In addition, ensuring anonymity for participants especially in qualitative research can be difficult, because direct quotes can easily be accessed through a full-text search in a search engine and hereby the original author might be identified:65:145.
In 2014 a study on 'emotional contagion' of Facebook users lead to a debate of the ethics involved in this research, especially in relation to informed consent, the right to withdraw participation and personal privacy :217ff. The researchers in that study conducted an experiment in which the news feeds of randomly selected users were adjusted to reduce the amount of emotional content presented in their feed; which led to the findings that "reducing positive emotional expressions let people to produce fewer positive and more negative posts":218. In reaction to this, APA issued a press release, in which they outlined the content of informed concent (outlined above), while adding:
APA reiterated its policy with respect to informed consent in light of a study involving approximately 700,000 Facebook users without their knowledge.
Qualitative and Quantitative Research Methods
Qualitative and quantitative research methods are presented and discussed with regard to their applicability and strengths and weaknesses. To give a brief overview, the connection between descriptive and relational investigations (for example ethnographic research or case studies) and experimental research will be outlined. According to Lazar et al. 2010 empirical investigations can be categorised into three groups with different goals and results. Descriptive investigations enable for an accurate description of what is observed, but don't allow for an explanation why this is happening. In order to reveal relations between different factors, relational investigations could be carried out. Using the correlation analysis helps the researcher to determine, if the result is significant but this also doesn't explain the cause of the relationship. Experimental research consists of controlled experiments which investigate causal relationship of different factors. :20ff
Different methods for data collection will be introduced and discussed: interactive (interviews, focus group), textual data collection (surveys, diaries, secondary source), and observation. Digital methods will also be discussed, which are not just another set of methods, since social research methods have had a computational dimension for a long time :ix
The terms survey and questionnaire are sometimes used interchangeably, however a survey can refer to a complete methodological approach (including sampling, reminders, incentives), in which a questionnaire (i.e. the list of questions) is one element :100.
Lazar et al. outline the process of the development of a survey :111f. It is important to determine early on, how the survey will be distributed. While online surveys may reach a large amount of potential participants, it may also exclude those who do not have access to computers or internet. After the goal and strategy for the survey has been decided upon, the survey questions have to be developed. The wording of the questions and the overall structure is crucial; the respondents have to be able to understand and use the questionnaire. Depending on the research purpose, there may already exist surveys that have already been tested. It is possible to make use of them in the current study; possibly with some modifications.
Survey questions can be structured as::112f
It is therefore crucial that survey questions are piloted in advance. Although qualitative surveys are not as widely used as quantitative, it can generate great data while not being as time-consuming as for example interviews and focus groups :134ff. Since this data is more focused on the topic than e.g. focus groups, the data can be useful for pattern-based analysis, while participants still provide their own answers in their own words (which is important for qualitative research):137.
Interview and Focus Groups
In qualitative research, one of the most important methods is the interview. The use of interviews are appropriate in order to get access to people's experiences, perceptions in their own language (their meanings, definitions, concepts), exploring their understanding about things they have a personal stake in. Interviews can be:78ff
Similar to the questionnaire design, it is very important to thoroughly prepare and pilot the interview guide. Important aspects to consider when conducting interviews are :81ff
Today interviews can also be conducted virtually (e.g. via telephone, email or video calls), which has certain advantages and disadvantages. Here the distinction between survey and interview methods can become blurred :36. While it might be convenient and empowering for participants who can participate from home, it might also be more time consuming for participants (e.g. writing takes longer than speaking), the researcher has less control over the interview and some forms of information are lost :97ff. Braun and Clarke (2013) outline important differences between virtual and face-to-face interviews in relation to the process:
Whereas interviews are usually conducted one-to-one, focus groups allow to collect data from multiple participants at the same time. Instead of having an interviewer asking questions, a moderator raises discussion points for the participants to discuss among themselves :107ff. Focus groups are a good way to elicit a wide range of views, perceptions, and understandings of an issue and have been considered suitable to conduct research with people from marginalised communities :110f. While the focus group discussions can be a reasonably effective and inexpensive tool for easily gathering a broad range of opinions:192 the transciption and anaylsis is very time-consuming while not allowing for in depth follow-up of individuals' views or experiences :113.
A diary is generated by participants and require participants to record their thoughts, feelings, experiences and/or practices over a specific period of time:147. They can take many forms (handwritten, hard-copy, digital, audio- or video recorded) and require regular entries over a certain period of time. Diaries can be useful where an observation or experimental design would not be appropriate; where the research question leads to data than cannot easily be observed or measured (e.g. feelings of frustration), and are useful in combination with other research methods (triangulation):138.
Another way to collect data is to make use of documents that already existed and is publicly available, thus the researcher had no role in the production of the data. Those secondary sources of data include different types of material in printed copy, electronic, and broadcast formats :152. Sources of documents can be :234
While the Internet is ideally suited for document-based research, certain problems should be considered:239
Observation has its tradition in social sciences and as a data collection method can be structured or unstructured (similar to interviews) :153ff. In research, observation is used as a data generation method "to find out what people actually do, rather than what they report they do when questioned":202. Quantitative approaches are highly structured and require predeveloped observation schedules, while qualitative approaches are much more unstructured and no predetermined categories and classifications are used :153. Two types of observation are relevant with regard to the role of the researcher:
In participant observation researchers not only observe and write notes, the aim is also to take part in the situation under study in order to experience it from the point of view of the others in that setting :208. Oates (2006) outlines four types of participation:
The roles as such form a continuum and roles might evolve throughout the course of a project :228. On both ends of the continuum is the researcher as being complete participant vs. being complete observer, inhibiting the risk of losing perspective (going native) vs. risking misinterpretation if one only observes :229.
According to Hewson et al (2016), the distinction between observational approaches and document analysis methods can sometimes be blurred in Internet-Mediated Research due to the new ways of interacting and publishing (e.g. blogs may be seen as interactively produced document with multiple authors, as individual diaries or something in between) :36. Kozinets (2010) emphasises, that a participative approach to the study of online culture and communities is at the core of netnography, thus netnography is an adaptation of participant-observational ethnographic procedures. :74
Analysis and Quality Criteria
Depending on the research question and the type of data collected, different analytical method have to be applied to analyse the data.
Quantitative Data Analysis
Quantitative data is based on numbers, typically generated by experiments or surveys and mainly used in the positivist paradigm :245ff. Quantitative data analysis looks for pattern in the data and to draw conclusion; a simple analysis would use tables, charts or graphs, while more complex analysis would make use of statistical techniques:245ff, e.g. descriptive statistics (central tendency; distribution); analysis of variance (ANOVA); identifying relationships (correlation coefficients).
Qualitative Data Analysis
Qualitative data includes all non-numerica data collected through interviews, diaries, etc. (e.g. words, images, sounds) :266ff. While some researchers make use of quantitative analysis on qualitative data (e.g. counting the occurrence of a particular word in a text), most qualitative data analysis involves abstracting themes and patterns from the data collected :266f. According to Braun and Clarke (2013) qualitative analyse can be :173ff
Different qualitative analysis approaches are for example Thematic Analysis, Grounded Theory, Discourse Analysis. Qualitative data analysis involves coding performed by the researcher, which can follow a :175:269
The quality criteria needed to evaluate the quality of research differ between quantitative and qualitative research. Quantitative research is evaluated in regards to :278ff
Didactic Concept, Schedule and Assignments
The course concept comprises lectures, online seminars with discussions, online and offline group work and an introductory and final on-site presence. After a first introductory lecture on site, the subject is treated in three online workshops, that are supplemented by a session on-site. Online workshops are held on three evenings with a duration of three hours each.
Introductory lecture on-site
During the introductory meeting the structure of the course and organisational issues will be outlined.
The lecturer will present an introduction to research and research methods, outlining the motivation for research and the basic elements of the research process. The development of research questions, as well as ethical considerations and issues relating to sampling and recruitment will be discussed.
1st Online Session
The first online meeting will deal with qualitative and quantitative research methods. Students will be separated in groups and work out a study plan for a given research question. The results will be presented afterwards in a plenary session. At the end, organisational issues will be discussed.
2nd Online Session
The second online meeting will deal with the analysis of qualitative and quantitative data, as well as quality criteria for research evaluation. During the session, the students will be separated in groups and apply certain methods. The results will be presented afterwards in a plenary session. At the end, organisational issues will be discussed. The session will end with instructions for the preparation of the next online session and the discussion of unresolved questions.
3rd Online Session
Prior to the third online session, students will submit an exposé for a possible research design for the problem they have identified in their Web and Society Project. During the third online session, the students will present their research design, which should comprise
Students and lecturer will discuss the research design and feedback will be provided.
Wrap-up Session on-site
Students will present their research design as part of their presentation of the Web and Society Project. Part of the presentation should be dedicated to the discussion of the research design (e.g. research question, design of the process, application, data collection, analysis).
The participants will work on an exposé comprising a specific research design for a specified research question. This exposé will be submitted in writing, is presented during the third online session and will contribute to the final grade. The main part of the final grade will be the appraisal of the method described and used in the Web and Society Project, as well as the presentation and discussion of the used methods at the on-site meeting.
Past Course Pages