Abstract
The aim of the paper is to compare two alternative survey designs in terms of resultant response rates, non response bias and cost. The first design is a simple postal survey with follow-up mailings; the second design is a two-phase multi-mode design, where the postal survey is followed at the second phase by a telephone survey of non-respondents. We present a case study based on a survey of employers. In this study we find evidence that the sample obtained using only postal methods is biased in important respects. Bias is not apparent in the demographic characteristics of the employees. But bias is observed in some of the employees’ employment characteristics and some of the characteristics of the firms in which they work. The multi mode design seems, overall, to have reduced or removed the bias of the postal sample. Only in marginal respects was some further bias introduced. We also compare costs of the two designs, to enable a comparison of cost-effectiveness at bias reduction.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Central Statistical Office: Standard Industrial Classification of Economic Activities 1992. HMSO, London (1992)
Cox B.G., Binder D.A., Chinnappa B.N., Christianson A., Colledge M.J., Kott P.S. (eds.) (1995). Business Survey Methods. John Wiley and Sons, New York
Curtin R., Presser S. and Singer E. (2000). The effects of response rate changes on the Index of Consumer Sentiment. Public Opin. Q. 64: 413–428
de Heer W. (1999). International response trends: results of an international survey. J. Off. Stat. 15(2): 129–142
de Leeuw E.D. (2005). To mix or not to mix data collection modes in surveys. J. Off. Stat. 21(2): 233–255
Dillman D.A. (2000). Mail and Internet Surveys: The Tailored Design Method, 2nd edn. John Wiley and Sons, New York
Dillman, D.A., Carley-Baxter, L.R.: Structural determinants of mail survey response rates over a 12-year period. In: Proceedings of the Survey Research Methods Section of the American Statistical Association 2000, pp. 394–399 (2000)
Employment Department Group and Office of Population Censuses and Surveys: Standard Occupational Classification, Volume 1: Structure and Definition of Major, Minor and Unit Groups. HMSO, London (1990)
Fisher, S., Bosley, J., Goldenberg, K., Mockovak, W., Tucker, C.: A qualitative study of nonresponse factors affecting BLS establishment surveys: results. In: Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 679–684 (2003)
Fox, J., Fisher, S., Tucker, C., Sangster, R., Rho, C.: A qualitative approach to the study of BLS establishment survey nonresponse. In: Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 1500–1505 (2003)
Gallagher P.M., Fowler F.J. and Stringfellow L.V. (2005). The nature of nonsponse in a Medicaid survey: causes and consequences. J. Off. Stat. 21(1): 73–87
Groves R.M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opin. Q. 70(5): 646–675
Haraldsen G. (2004). Identifying and reducing response burdens in internet business surveys. J. Off. Stat. 20(2): 393–410
Jäckle, A., Lynn, P.: Dependent interviewing and seam effects in work history data. J. Off. Stat (in press)
Jäckle, A., Sala, E., Jenkins, S.P., Lynn, P.: Validation of survey data on income and employment: the ISMIE experience. ISER Working Paper, 2004-14. University of Essex, Colchester. http://www.iser.essex.ac.uk/pubs/workpaps/pdf/2004-14.pdf (2004)
Jenkins S.P., Cappellari L., Lynn P., Jäckle A. and Sala E. (2006). Patterns of consent: evidence from a general household survey. J. Roy. Stat. Soc. Ser. A (Stat. Soc.) 169(4): 701–722
Jenkins, SP., Lynn, P., Jäckle, A., Sala, E. Feasibility of linking household survey and administrative record data: new evidence for Britain. Int. J. Social Res Methodol (in press)
Kaplan, A.F., Whiter, G.D.: Incentives in a business survey: a study in improving response rates. In: Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 1756–1761 (2002)
Keeter S., Miller C., Kohut A., Groves R.M. and Presser S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opin. Q. 64: 125–148
Luo, A. and White, G.D.: Exploring a new establishment survey incentive to improve response rates. In: Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 3915–3918 (2005)
Lynn, P., Sala, E.: The contact and response process in business surveys: lessons from a multimode survey of employers in the UK, ISER Working Paper, 2004-12. University of Essex, Colchester. http://www.iser.essex.ac.uk/pubs/workpaps/pdf/2004-12.pdf (2004)
Lynn P. and Sala E. (2009). Measuring change in employment characteristics: the effects of dependent interviewing. Int. J. Public Opin. Res. 18(4): 500–509
Lynn P., Jäckle A., Jenkins S.P. and Sala E. (2006). The effects of dependent interviewing on responses to questions on income sources. J. Off. Stat. 22(3): 357–384
Lynn, P., Jäckle, A., Jenkins, S.P., Sala, E.: The impact of interviewing method on measurement error in panel survey measures of benefit receipt: evidence from a validation study, ISER Working Paper, 2004-28. University of Essex, Colchester. http://www.iser.essex.ac.uk/pubs/workpaps/pdf/2004-28.pdf (2004)
Paxson, M.C., Dillman, D.A., Tarnai, J.: Improving response to business mail surveys. In: Cox, B., et al. (eds.) Business Survey Methods. John Wiley and Sons, New York (1995)
Petroni, R., Sigman, R., Willimack, D., Cohen, S., Tucker, C.: Response rates and nonresponse in bls and census bureau establishment surveys. In: Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 4159–4166 (2004)
Smith P., Pont M. and Jones T. (2003). Developments in business survey methodology in the Office for National Statistics, 1994–2000, with discussion. J. Roy. Stat. Soc. Ser. D (Statistician) 52(3): 257–296
Steeh C., Kirgis N., Cannon B. and DeWitt J. (2001). Are they really as bad as they seem? Nonresponse rates at the end of the twentieth century. J. Off. Stat. 17(2): 227–247
Tarnai, J., Paxson, M.C.: Survey mode preferences of business respondents. In: Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 4898–4904 (2004)
Willimack, D.K., Nichols, E., Sudman, S.: Understanding unit and item nonresponse in business surveys. In: Groves, R.M., et al. (eds.) Survey Nonresponse. John Wiley, New York (2002)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License ( https://creativecommons.org/licenses/by-nc/2.0 ), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Sala, E., Lynn, P. The potential of a multi-mode data collection design to reduce non response bias. The case of a survey of employers. Qual Quant 43, 123–136 (2009). https://doi.org/10.1007/s11135-007-9148-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-007-9148-5