Research Impacts - Articles

Insights into Abt SRBI's Contribution to HBS Survey of U.S. Competitiveness

Release Date: 9/29/2016
 
Written By: Benjamin Phillips, Ph.D., Vice President, Social Policy and Polling Group

Ben Phillips has directed Abt SRBI’s contribution to the HBS Survey of U.S. Competitiveness since the study’s inception in 2011.

Most years since 2011, a phone call or email from Manjari Raman, program director of the Harvard Business School’s U.S. Competitiveness Project, begins Abt SRBI’s work on that year’s HBS Survey of U.S. Competitiveness. The survey draws on the experience of HBS alumni to rate the strength (or otherwise) and trajectory of different aspects of the U.S. business environment, the overall strength and trajectory of the business environment, and ability of firms operating in the U.S. to compete in the global marketplace and offer high wages and benefits. The last factor is of great importance to the project’s co-chairs, Professors Michael E. Porter and Jan W. Rivkin, who define competitiveness as the ability of firms operating in the U.S. to compete successfully in the global marketplace while supporting high and rising standards of living for the average American. The ability of firms to operate successfully in the marketplace is not sustainable, the co-chairs argue, without shared prosperity. This dual agenda of sharpening the edge of the U.S. business environment while sharing the gains has shaped the study.

In addition to the core questions described above which have been repeated with limited changes over time, each year’s survey has a topical set of questions which probe deeper into the specific elements of competitiveness. These have included offshoring/‌onshoring and investment in local communities (2011), federal government policy (2012 and 2016), business actions to enhance U.S. competitiveness (2012), the K-12 education system (2013-14), middle-skills jobs (2013-14), transportation and infrastructure (2013-14), wealth distribution (2015), entrepreneurship (2015), taxation (2016), and the U.S. political system (2016). Parallel surveys have been undertaken of the general population in 2012 and 2016 with respect to the core items. In addition, a survey of school district superintendents in 2013 paralleled the focus on understanding the K-12 education system while a survey of members of collective impact organizations in 2015 addressed business support for public education. In 2016, for the first time, HBS also surveyed current MBA students, in both years of the program, on competitiveness.

Abt SRBI provides input into the design of topical survey items and any revisions to the core questions. Once the survey instrument has been developed, new items are tried out with small numbers of HBS students and alumni in a rapid-fire version of cognitive testing. The aim of cognitive testing is to ensure that questions are understood in the same way by all respondents and in the way that the question authors intended. Respondents are asked to think aloud as they answer questions and scripted and impromptu follow-up probes are used to better grasp how respondents understand the questions. To test new questions, we conduct interviews over the phone. These test interviews use a combination scripted probes along with impromptu probes as needed. A give-and-take approach prevails during the interview; we might ask whether alternative wording would be clearer or ask the respondents how he or she would phrase the question. At the conclusion of testing, I review responses with the interviewer and decide what recommendations for changes, if any, will be made. We then meet the HBS team to review the test results and work out any changes to be made to the instrument.

Once the instrument has been finalized, it is programmed on Abt SRBI’s web survey platform and tested, first by the Abt SRBI team and then by the HBS team. We use a custom template for the survey to match HBS branding, including incorporating Harvard Crimson and the U.S. Competitiveness Project logo.

Abt SRBI works in tandem with HBS to send email invitations and reminders. To enhance the legitimacy of the survey request, all study email is sent from hbs.edu accounts. Abt SRBI provides HBS with the list of recipients along with relevant read-in information (e.g., salutations, each person’s URL, custom wording for people who have begun but not completed the survey). Over the years, we have worked with HBS to conduct some experiments regarding email formatting. In the early years, the invitation was sent by the School’s Dean with reminders coming from Professors Porter and Rivkin. Now that HBS alumni are comfortable with the idea of an annual survey process, invitation and reminder emails go out only from Professor Porter and Professor Rivkin’s email accounts. Abt SRBI and the HBS team work to preserve the professors’ distinctive voices out of the belief that this more personal approach works better than overly scripted, impersonal text. To enhance this effect, personalized email accounts bearing each professor’s name, such as “Office of Michael Porter” or “Office of Jan Rivkin” are used to send the reminders.

Along with Abt SRBI, the HBS team closely monitors the progress of the survey while it is in the field. Managing alumni relationships and ensuring that alumni have a smooth experience is top priority for the HBS and Abt SRBI team when the survey is in the field. Typically, Abt SRBI sets up a password protected portal to allow HBS to see how many people have completed the survey in real time.

The final set of activities takes place after the close of the survey. Abt SRBI weights the survey data. This step includes an adjustment for the probability of selection for years in which a sample of alumni was drawn (as opposed to a census of all alumni with email address), an adjustment for nonresponse, and finally calibration to the population of HBS alumni with respect to alumni type (graduates of HBS degrees vs. those from qualifying Executive Education courses), location (U.S. vs. overseas), age, and sex. We also deliver longitudinal datasets that include responses from all years of the survey in HBS’s preferred Stata format. Finally, we provide analytical support to HBS as needed. This varies across the years and mostly consists of specific cross-tabulations, although we have also provided charts and statistical consultation in some cases.

This has been a rewarding study to lead for Abt SRBI across the years. The HBS team is lively, thoughtful, and deeply concerned about the integrity of the study, which is all one can ask for in a client. It is also always a thrill to see research one has worked on make it into the news, particularly where it is aimed at improving the economic well-being of average Americans.