Measuring and testing the interview mode effect in mixed mode surveys
DOI:
https://doi.org/10.6092/issn.1973-2201/4501Keywords:
self-selection, global imbalance, mixed interview mode, mode effectsAbstract
Many studies are showing an increased tendency to use more than one data collection mode for a particular survey. However, mixed data collection modes may influence responses given by interviewees and require researchers to verify if differences in responses, when present, are ascribable to the type of data collection mode. Often, random assignment is not feasible and requires researchers to solve an additional and not negligible problem, namely to verify if differences in responses are ascribable to the self selection or to the type of data collection mode being used. The aim of the present paper is to measure the mode effect on the answers using a new data driven multivariate approach, that allows to disentangle the interview mode effect on answers from the effect of self selection. We will work through the use of the new multivariate method with AlmaLaurea case concerning the evaluation of two different data collection methods: the CAWI (Computer Assisted Web Interviewing) and the CATI (Computer Assisted Telephone Interviewing). As with any new statistical method, the success of this method depends on its efficacy in relation to that of the existing methods. Therefore, results of the multivariate approach will be compared to the Propensity Score method that AlmaLaurea usually applies to identify the presence of an interview mode effect. Both methods produce similar results.References
R. E. BARLOW, D. J. BARTHOLOMEW, J. M. BREMNER, H. BRUNK (1972). Statistical Inference Under Order Restrictions. John Wiley & Sons, New York.
J. BETHLEHEM (2010). Selection in Web Surveys. International Statistical Review, 78(2), pp. 161–188.
J.M. BORKAN (2004). Mixed Methods Studies: A Fundation for Primary Care Research. Annals of Family Medicine, 2(1), pp. 4-6.
F. CAMILLO, V. CONTI, S. GHISELLI (2011). Integration of different data collection techniques using the propensity score. AlmaLAurea Working Papers n 4.
F. CAMILLO, I. D’ATTOMA (2010). A New Data Mining Approach to Estimate Causal Effects of Policy Interventions. Expert Systems with Applications, 37(2010), pp. 171-181.
F. CAMILLO, I. D’ATTOMA (2012). %GI: A SAS Macro for Measuring and Testing Global Imbalance of Covariates within Subgroups. Journal of Statistical Software, 51, Code Snippet 1, pp. 1-19.
A. CAMMELLI, G. ANTONELLI, F. CAMILLO, A. DI FRANCIA, S. GHISELLI, M. SGARZI (2011). Graduates’ employment and employability after the “Bologna Process” reform. Evidence from the Italian experience and methodological issues. AlmaLAurea Working Papers.
W.G. COCHRAN (1968). The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies. Biometrics, 24, pp. 205–213.
I. D’ATTOMA, F. CAMILLO (2011). A Multivariate Strategy to measure and test global imbalance in observational studies. Expert Systems with Applications, 38, pp. 3451-3460.
E. D. DE LEEUW (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, pp. 233–255.
D.A. DILLMAN, R.L. SANGSTER, J.TARNAI,T. H. ROCKWOOD (1996). Understanding Differences in People’s Answers to Telephone and Mail Surveys. New Direction for Evaluation. 70, pp. 45–61.
B. ESCOFIER (1988). Analyse des correspondances multiples conditionelle. In: E. Diday (Ed.), Data Analysis and Informatics. North Holland, Amsterdam: Elsevier Science, pp. 333–342.
J.D. ESTADELLA, T. ALUJA, S. THIÒ-HENESTROSA (2005). Distribution of the inter and intra inertia in conditional MCA. Computational Statistics. 20(3), pp. 449–463.
M.J. GREENACRE (1984). Theory and Applications of Correspondence Analysis. London: Academic Press.
D.E. HO, K. IMAI, G. KING, E. A. STUART (2007). Matching as nonparametric pre-processing for reducing model dependence in parametric causal inference. Political Analysis, 15, pp. 199-236.
B. JANSSEN (2006). Web data collection in a mixed mode approach: an experiment. Proceedings of Q2006. European Conference on Quality in Survey Statistics.
G. KING, L. ZENG (2006). The Dangers of Extreme Counterfactual. Political Analysis, 14, pp. 131-159.
F. KREUTER, S. PRESSER, R.TOURANGEAU (2008). Social desirability bias in CATI, IVR, and Web Surveys. The effects mode and question sensitivity. Public Opinion Quarterly, 72(5), pp. 847–865.
S. LEE (2006). Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Survey. Journal of Official Statistics. 22(2), pp. 329–349.
M. MORA (2011). Understanding the pros and cons of mixed-mode research. Quirk’s Marketing Research Review, 50 .
L.R. PECK, F. CAMILLO, I. D’ATTOMA (2010). A Promising New Approach to Eliminating Selection Bias. The Canadian Journal of Program Evaluation, 24(2), pp. 31-56.
L.R. PECK, I. D’ATTOMA, F. CAMILLO, C. GUO (2012). A new strategy for Eliminating Selection Bias in Non-Experimental Evaluations: The Case of Welfare Use’s Impact on Charitable Giving. Policy Studies Journal, 40(4), pp. 601-625.
P.R. ROSENBAUM, D.B. RUBIN (1983). The Central Role of Propensity Score in Observational Studies for Causal Effects. Biometrika. 70, pp. 41–55.
M. SCHONLAU, R. D. FRICKER, M. N. ELLIOT (2002). Conducting research survey via email and the web. Rand Corporation.
M. SCHONLAU, A. VAN SOEST, A. KAPTEYN , M. P. COUPER (2006). Selection Bias in Web Surveys and the Use of Propensity Score. Rand Corporation.
J. VANNIEUWENHUYZE, G. LOOSVELDT, G. MOLENBERGHS ( 2010). A Method for Evaluating Mode Effects in Mixed-Mode Surveys. Public Opinion Quartely. 74(5), pp. 1027–1045.
P.M. STEINER, D. COOK (2011). Matching and Propensity Scores. In Little, T.D. (Ed.), The Oxford Handbook of Quantitative Methods, Oxford: Oxford University Press.
H. F. WOLTMAN, A. G. TURNER, J. M. BUSHERY (1980). A comparison of three Mixed-Mode Interviewing Procedures in the National Crime Survey. Journal of the American Statistical Association, 75(371), pp. 534–543.
K. B. WRIGHT (2005). Researching Internet-Based Populations: Advantages and Disadvantages of Online Survey Research, Online Questionnaire authoring software packages, and web survey services. Journal Of Computer-Mediated Communication, 10(3).
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2013 Statistica
This journal is licensed under a Creative Commons Attribution 3.0 Unported License (full legal code).
Authors accept to transfer their copyrights to the journal.
See also our Open Access Policy.
