Should you use a survey?
Survey studies are a popular way to conduct research studies and evaluations since you can gather data from a large number of participants with minimal time and expense. However, before you decide on a survey design, there are several issues to consider.
- Surveys do not always provide the best data to answer your research or evaluation questions because they rely on subjective self-reporting and perceptions. Subjective data can be useful if the purpose is to assess participant reactions to a program or intervention. However, the outcomes of interest often require more than the participants’ perceptions. For example, students’ self-reported perceptions of learning are not necessarily a reliable measure of actual student learning. While there may be situations in which a survey is the only feasible means to collect data, effort should be made to collect the best measure of outcomes.
- Survey fatigue: The prevalence of internet surveys has led to a dramatic increase in the number of surveys people are asked to complete. Schools, employers, and businesses are frequently sending out surveys. The result is “survey fatigue.” As people tire of receiving many surveys, fewer and fewer people are taking the time to respond. To reduce the impact of survey fatigue, and increase the power of your and others’ surveys, it is important only to use surveys when it is truly warranted.
Explore existing data sources
Oftentimes, the data needed to answer your research questions or evaluate a program or policy already exists. Tapping into existing data sources can make the research process more efficient and often provides the most valid measures. For example, obtaining student AP credits or GPAs from the registrar may provide more complete and reliable data than asking students to self-report this information on a survey. Three key types of existing data at MIT include administrative data, Institute surveys, and student work. Use of these data sources may require special permissions and must follow FERPA and COUHES guidelines/requirements.
Various offices collect and manage data as a part of their work for the Institute. For example, the admissions office has data on the demographics and certain pre-college academic indicators of the MIT student body. The registrar’s office has data on subject enrollments and grades given in those subjects. Various departments and program offices also have data related to program participation. Depending on the needs of your evaluation, these offices may be able to provide anonymized data to answer your research questions. Data in which individuals are identifiable can be accessed, but it may require COUHES review or informed consent from students. If you are looking for administrative data and are not sure where to access it, MIT Institutional Research can often help.
Institutional Research oversees the conduct of several Institute-wide surveys of undergraduates, graduate students, staff, and alumni. Summary data is available on their website for the past several years. You can also request access to data files that will allow you to conduct your own analyses. In some cases, including relevant questions from recent Institutional Research surveys on your own survey may allow for general comparisons between your participants and the broader MIT community. Before requesting access, you must complete Human Subjects training. For more information on available survey data, please visit the Institutional Research surveys webpage. If you do decide to conduct your own survey of a large group at MIT, it is important to consult with Institutional Research about the timing of your survey to ensure that your data collection is not compromised by overlap with other surveys.
Student work as data
Student work (i.e. exams, final projects) can provide some of the best measures of student learning. In addition to being more reliable than self-reported learning, using the work students do as a part of their regular educational program can reduce the burden placed on students when asking them to complete additional surveys or assessments. Basic scores on P-sets and exams provide one obvious and simple measure. For project-based learning, rubrics can be used to create a reliable measure of learning outcomes.