Events - 2014

New Statistical Adjustment for Mode Effects

    Release Date: 7/10/2014

    Survey researchers have been increasingly utilizing multiple interview modes to engage respondents
    by whatever mode respondents deem most desirable – mail, telephone, in-person,
    or mobile/web. The use of multiple modes has arisen because of the impact of computer/mobile
    technology and decreasing response rates to traditional survey modes, such as telephone.

    One major issue with multiple mode utilization is that data collected from different survey modes
    may introduce mode-related measurement error and reduce data quality. Interview mode itself
    may influence responses. In other words, respondents to a telephone survey may offer different
    answers than if they were completing the interview online.

    To address this challenge of adjusting for mode differences, two members of the Abt SRBI
    Advanced Methods Group, Stas Kolenikov and Courtney Kennedy, have just published an
    article in the Journal of Survey Statistics and Methodology (2014, 2, 126-158) assessing the
    strengths and weaknesses of various mode adjustment approaches . Adjustment approaches
    evaluated include:

    •  A Regression modeling approach, where adjustments are computed by regressing survey
       responses on mode,  demogrphics and other relevant variables.
    •   A multiple imputation approach, where mode effects are conceptualized as a missing- data
        problem.  
    •   And a proposed new imputation approach based on an econometric framework of
        implied utilities in logistic regression modeling.
                                                            
    They evaluated all three approaches using data from the second wave of the Portraits of
    American Life Survey sponsored by Rice University's Kinder Institute for Urban Research.
    This survey featured online and CATI interviewing with a national sample of adults, with
    random assignment to either CATI-only or web with CATI follow-up for non-respondents.

    To evaluate the three approaches, they developed a workflow to determine which variables
    require a mode effects adjustment based on standard false discovery rate multiple
    hypothesis testing procedures.  They detected a significant mode effect on four survey
    outcomes after controlling for demographics and risk of type I error. The mode effects
    adjustments were then applied to these variables.

    They concluded that the multiple imputation approach produced estimates with better
    apparent accuracy, as evidenced by better internal consistency of the estimates and a
    moderate increase in the standard errors. Unlike the regression adjustment approach, which
    can only produce aggregated estimates for the whole study, the multiple imputation
    approach can be used for disaggregated analysis with mode-adjusted estimates as well.

    About the authors

    Dr. Stas Kolenikov is Principal Survey Scientist in the Advanced Methods Group at Abt
    SRBI. His key responsibilities include sampling design, power analysis, weighting and
    imputation methods.

    Dr. Courtney Kennedy is Vice President in the Advanced Methods Group at Abt SRBI,
    specializing in complex survey designs, particularly those featuring multiple sampling
    frames.

    The full article is accessible free of charge at:
    http://jssam.oxfordjournals.org/content/2/2/126.full

    About ABT SRBI

    Abt SRBI, a leading public policy and market research organization, is a subsidiary of
    Abt Associates, a mission-driven, global leader in research and program implementation in the
    fields of health, social and environmental policy, and international development. Abt SRBI is
    headquartered in New York City, with offices in the Washington, D.C. area, Chicago, IL,
    Cambridge, MA, Fort Myers, FL, Durham, NC and other cities.www.srbi.com