Keywords

1.1 Large-Scale Assessment in Education: Analysing PIAAC Data

To actively participate in modern society, skills such as literacy and numeracy are of utmost importance. Essential information is usually provided in written format—for example, in manuals, memos, and medication package inserts. To adequately process and react to such information, individuals require sufficient skills.

In 2008, the Organisation for Economic Co-operation and Development (OECD) initiated the Programme for the International Assessment of Adult Competencies (PIAAC). PIAAC aims to assess in an internationally comparable way basic adult skills such as literacy, numeracy, and problem solving. These skills are considered to be essential for successful participation in modern society and to be a foundation for developing numerous other, more specific, skills and competencies (OECD 2013).

PIAAC provides information about the skill levels of the adult population in the participating countries and the extent to which countries differ in terms of these skills. Moreover, and in particular, PIAAC provides information on factors associated with the acquisition, maintenance, and outcomes of these skills. Thus, it sheds light on effects of these basic skills on social and, in particular, economic participation.

Like the OECD Programme for International Student Assessment (PISA), PIAAC is designed as a cross-sectional study to be repeated at regular intervals. The first cycle of PIAAC started in 2008 and comprised three rounds, in which a total of 38 countries participated. The second cycle of PIAAC was launched in 2018 and will likely cover 33 countries (see Table 1.1).

Table 1.1 Data assessment of participating countries in the first and second cycles of PIAAC
Full size table

The OECD consistently pursues an open science strategy regarding the data resulting from PIAAC. To date, more than 60 PIAAC datasets have been published worldwide (see Chap. 4 in this volume); the first data were released in 2013 (OECD 2013). These datasets have been widely accessed and used by an interdisciplinary research community (for an overview, see Maehler et al. 2020). Furthermore, there are a large and increasing number of PIAAC-based publications.Footnote 1

As in the case of other international large-scale assessments (Rutkowski et al. 2014), analyses with the PIAAC data are very challenging for users due to the complex data structure (e.g. plausible values computed by imputation, complex sampling). To ensure the quality and significance of the data analyses, users require instruction in the correct handling of the data. This methodological textbook therefore aims to summarise existing techniques for analysing PIAAC data. It provides a standardised approach to successfully implement these data analyses. The present volume provides examples and tools for the analysis of PIAAC data using different statistical approaches and software and also offers perspectives from various disciplines. The textbook is designed for use by researchers and students from diverse fields (e.g. educational research, economics, sociology, and psychology) who are interested in working with large-scale educational assessment data.

This methodological textbook covers the following topics: (1) background information on PIAAC that is required for the analyses—for example, the design of PIAAC and the available datasets; (2) the (web) tools available for the analysis of PIAAC data, particularly the public use files; and (3) the analysis of cross-sectional PIAAC data with multidisciplinary methods (e.g. Stata or R) using public use files or scientific use files.

The next three chapters provide background information that serves as a basis for working with PIAAC data. Chapter 2 summarises the core features of the PIAAC survey design and briefly addresses sampling and data collection. In addition, it provides an overview of the background questionnaire and the competence domains assessed in PIAAC. The chapter concludes with a discussion of potential improvements to future PIAAC cycles.

Chapter 3 introduces item response theory and the principles of multiple imputations. It describes plausible values and explains how they can be used to address concerns regarding the introduction of bias when point estimates of latent indicators are used to estimate certain population parameters.

Chapter 4 provides an overview of the PIAAC datasets that are available for research purposes and outlines their structure, accessibility, and use. For example, public use files are accessible for public purposes and are thus highly anonymised, whereas scientific use files are available only for scientific research purposes and provide access to more detailed variables. Regarding the study design, most available datasets are cross-sectional, although some longitudinal data already exist. In addition to describing these longitudinal datasets, Chap. 4 presents PIAAC datasets that focus on specific population groups—for example, the population in Germany aged 65 years and over (Friebe et al. 2017) and the incarcerated adult population in the United States (Hogan et al. 2016).

The two subsequent chapters are devoted to the tools that are available for the analysis of PIAAC data. Chapter 5 presents PIAAC analyses using the web tool International Data Explorer (IDE), which is provided by the international PIAAC consortium. The IDE can be used to create tables and graphs that give an overview of the skills of adults aged 16 to 65 years in the areas of literacy, numeracy, and problem solving in technology-rich environments. It can also be used to calculate standard errors with complex designs, and it allows variables to be combined and indices to be created and validated. The data can be analysed both by country and by sociodemographic characteristics, such as education or employment status. The use of the tool to extract percentages, averages, benchmarks (proficiency levels), and percentiles is demonstrated, and limitations are outlined.

Chapter 6 introduces readers to the performance of both simple and complex analyses with PIAAC data using the International Database (IDB) Analyzer, a Windows-based tool that generates SPSS syntax. Using this syntax, corresponding analyses can be conducted in SPSS. The chapter presents the data-merging module and the analysis module. Potential analyses with the IDB Analyzer are demonstrated—for example, the calculation of percentages and percentiles, averages, benchmarks (proficiency levels), correlations, and regressions (linear only).

The final five chapters in this volume focus on the analysis of cross-sectional PIAAC data using multidisciplinary methods embedded in different disciplinary approaches. Chapter 7 is devoted to the analysis of PIAAC data using the statistical package Stata. Following an introduction to working with PIAAC data using Stata, it focuses on two features of the PIAAC data that present challenges to researchers: the availability of multiple plausible values for individual competence scores and the computation of statistics taking into account imputation and sampling errors. The chapter also presents repest, an OECD Stata module for running estimations with weighted replicate samples and plausible values.

Structural equation modelling (SEM) has become one of the most commonly applied statistical approaches to disentangling the relationships among latent variables across groups, over time, and at different analytical levels. Chapter 8 therefore provides an introduction to the principles and procedures of basic and more advanced SEM in Mplus using PIAAC data. Furthermore, it presents model specification and estimation by means of confirmatory factor analysis, showing approaches to testing measurement invariance across a few or many groups. Finally, the chapter introduces classes of structural models, such as path models, structural equation models, and multi-group versions thereof. The corresponding syntax files are provided for the reader.

Chapter 9 focuses on the analysis of PIAAC data using an R package that includes functions for importing data, performing data analysis, and visualising results. It describes the underlying methodology and provides examples based on PIAAC data. The data analysis functions presented take into account the complex sample design (with replicate weights) and plausible values in the calculation of point estimates and standard errors of means, standard deviations, regression coefficients, correlation coefficients, and frequency tables.

PIAAC Cycle 1 was the first fully computer-based large-scale assessment in education. The use of computers allowed not only for innovative item formats and an adaptive test design but also for the collection of a stream of user events (e.g. mouse clicks, text input) stored by the assessment system in log files. These data are interesting not only from a measurement point of view (e.g. to assess the quality of the response data) but also from the point of view of addressing substantive research questions (e.g. investigating the cognitive solution process). Chapter 10 introduces the accessibility, structure, and content of PIAAC log file data. It describes, in particular, the PIAAC LogDataAnalyzer, which allows log data to be extracted from PIAAC xml log files. The chapter includes a sample analysis in order to demonstrate how exported log data can be further processed using standard statistical software such as the R environment or Weka, a data mining software.

Finally, Chap. 11 addresses the linking of PIAAC data to administrative data, which are available, for instance, in the Nordic countries, such as Sweden (see Chap. 4 in this volume). The chapter presents the research procedure and exemplary analyses based on the linking of data from the German PIAAC-Longitudinal (PIAAC-L) study to administrative data provided by the German Institute for Employment Research (IAB) within the framework of a pilot project.

Although PIAAC itself is designed as a cross-sectional study, longitudinal data for PIAAC are available for some countries (e.g. Canada and Germany; see Chap. 4 in this volume). Unfortunately, in the present volume, we are unable to provide any chapters with exemplary longitudinal data analyses. However, we aim to cover this topic in the next edition.