Keywords

In this book, we have introduced and discussed ways in which experiments and the associated methods and techniques can be employed in the context of learning technology and CCI. The main aim is to clarify the various methods and techniques needed by a researcher to be able to design and conduct a research study efficiently. Understanding of the various methodological decisions will ensure that a learning technology and/or CCI researcher will be able to make optimal decisions that promote high internal validity, make it possible to attribute findings to treatment variations, and identify potential confounding or extraneous factors.

We have also elaborated on the reasons for focusing on both learning technology and CCI and why we put them together. Given the inherent connections between the learning technology and CCI fields and UIs (how children/learners interact with a technology) and the respective research designs (e.g., in most cases we are introducing a new technological innovation to the experimental groups), we have provided fundamental knowledge on the design of educational interfaces and visualizations, with a focus on learning dashboards. In preparation for our discussion of the common forms of experimentation, we also considered the role of the artefact in contemporary learning technology and CCI research, and we set out the fundamentals of treatment design and artefact-centered evaluations.

We then presented different research designs: between-subjects, within-subjects, and mixed research designs. We set out common decision factors when considering the use of between- and within- subjects designs, using examples from the CCI and learning technology research fields. We discussed four common types of experimentation and their qualities: randomized (or true) experiment, quasi-experiment, repeated measures, and time series experiment. We also discussed the importance of internal validity and identified typical threats to that validity, such as unequal treatments and confounding or extraneous factors.

Next, we provided fundamental guidance on how a learning technology/CCI researcher can identify the most appropriate data analysis approach for their study. Given the importance of disseminating research results and publishing them in high-quality venues, we also focused on writing quality, standards, and style conventions. We explained how to organize a typical article/report in the field, identifying the six important high-level sections (introduction, background and related work, methods, findings, discussion, and conclusion and further research), common criteria used by reviewers, pitfalls in learning technology/CCI research, and some useful practices for junior researchers.

Given contemporary advances in data science, AI and sensor data, and their impact on both learning technology and CCI research, we gave a brief overview of how those developments have affected learning technology and CCI research, suggesting potential worthwhile uses. In addition, we briefly discussed three important issues that a learning technology and CCI researcher needs to be aware of: the importance of the context, ethical considerations, and working with children. There is a growing literature on each of these areas (e.g., Shibani et al., 2019; Van Mechelen et al., 2020; Romero & Ventura, 2017; Luckin & Cukurova, 2019), and we by no means claim to have covered them in detail; nevertheless, this book would have been incomplete if we had not provided an introduction to these important issues.

Although this book does not focus exclusively on experimentation, it is a topic that has received much attention. The use of experimentation has been criticized (e.g., see Ross & Morrison, 2013) as force-fitting, as fixated on internal validity, or even as being in conflict with potential improvements in the use of technology for supporting learning, play, and our lives generally. However, we want to emphasize in this closing chapter that the purpose of this book is not to promote or criticize the experimental method, but rather to provide direction for its effective use in learning technology and CCI research. Like any other research method, experimental methods can be employed “badly”, and it is important for a learning technology/CCI researcher to be able to apply experimental methods in a way that aligns with the RQs and that takes into account the contextual particularities. Moreover, to improve sensemaking, experiments can (and in many cases should) be used in conjunction with other research methods and approaches.

As we close this book, we hope that learning technology and CCI researchers can benefit from it, and we emphasize the role that researchers play as research designers who employ, adapt, alter, and expand research methods to accommodate contextual complexion, relevant theories, and the scientific inquiry of focus. The ultimate goal is to increase understanding of how technological affordances and technology-mediated practices can enhance our capabilities (e.g., as learners and children) and support the mental processes involved in gaining knowledge and comprehension.