Selecting your research methods means choosing the type of data you will collect, how you will collect your data, from whom you will collect your data, and how you will analyze your data.
Methodology
Your choice of specific research methods may be guided by an overarching research methodology. Quantitative methodologies involve using numbers or statistics to describe, correlate, or compare variables and the relationships between them. Qualitative methodologies involve observing or describing phenomena using non-numerical data, such as human language or interactions.
There are many different types of both quantitative and qualitative research methods. At TLL, our R&E experts can help you identify the methodological approach most aligned with your research purpose(s).
Data sources
You can collect many different types of data in educational contexts. Researchers have historically relied on surveys (often student self-report surveys), interviews, and focus groups to assess the processes and outcomes of educational programs. But, depending on the context you are assessing, you might also consider some of the following data sources:
- Observations and field notes
- Student work (e.g., exams, final projects, reflections)
- Administrative data (e.g., student enrollment, grades, demographics) collected by MIT Institutional Research, the Registrar, or departments, labs, and centers (DLCs)
- Institute-wide surveys conducted by MIT Institutional Research
Each of these data sources has strengths and weaknesses, some of which are summarized in the chart below.
| DATA SOURCE | BENEFITS | CONSIDERATIONS/ CAVEATS |
|---|---|---|
| Surveys | -Time- and resource-efficient -Tailored to study objectives -Allows students to provide direct feedback | -May create survey fatigue -Less in-depth than interviews -Indirect measure of outcomes |
| Interviews & focus groups | -Allows students to provide direct feedback -Ability to collect in-depth information about context and experiences -Ability to ask follow-up questions | -Resource-intensive to collect and analyze -Requires deep engagement and time from students |
| Observations & field notes | -Reduces burden on students to complete extra assessments -Ability to collect in-depth information about context | -Time-intensive for researcher -May not represent student perspective and experience |
| Student work (e.g., enrollment, grades) from the Registrar, IR, or MIT DLCs | -Direct measure of outcomes -Can be qualitatively or quantitatively analyzed -Reduces burden on students to complete extra assessments | May not always be available, practical, or ethical to use May require scoring with a rubric (time-intensive) |
| Administrative data (e.g., enrollment, grades) from the Registrar, Institutional Research, or MIT DLCs | -Readily available (with permission to use) -Reduces burden on students to complete extra assessment -Can be more reliable | -Not tailored to study objectives -May require permission and/or training to use |
| Institute-wide surveys conducted by MIT Institutional Research | -Reduces burden on students to complete extra assessments -Allow students to provide direct feedback | -Not tailored to study objectives -Requires Human Subjects training to access |
It is important to carefully consider which data source(s) are best to achieve the purpose(s) of your assessment, evaluation, or research project, given the resources and time available. The R&E team at TLL can assist you with identifying and weighing the strengths and weaknesses of different data sources.
Sampling participants
The statistical population refers to the entire group of people you are interested in. The population of interest for one study might be all undergraduate students entering MIT in a given academic term, whereas, in another study, the population might be doctoral students in STEM programs. Defining the population, and understanding the characteristics of that population, is essential for ensuring that your study allows you to address your research objectives.
Sampling refers to collecting data from a subset of the population. We sample participants when we cannot collect data from the entire population, often because that population is too large or complex.
There are many different ways to sample participants. You can randomly sample, develop a stratified sample, or engage in purposeful sampling based on characteristics of interest. Some things that you will want to consider when determining how to sample participants include the following:
Will this sample be representative of the larger population of interest? That is, will it match the population on important characteristics, including demographics (e.g., racial and ethnic identification, citizenship, gender identity, age) and academics (e.g., major, year in school)?
Suppose you would like to draw comparisons between important subsets of the population. Will you have sufficient numbers of students in these subgroups to draw meaningful comparisons without asking a small number of students to represent a larger, more diverse group?
Which potential participants do you have access to, and what resources do you have for encouraging participation?
How you engage in sampling depends on your research objectives and answers to the above questions. Often, researchers choose a sample primarily based on their access to a particular group of students (e.g., undergraduate research pools). This convenience sampling allows researchers to collect data more efficiently but can limit representation.
In addition to defining who you would like to study, you must also consider how your data collection procedures might influence the characteristics of those who ultimately respond. Sampling bias occurs when data are collected in a way that makes some members in the population of interest less likely to participate than others.
Data collection procedures
There are several important considerations when deciding how to collect your data. These include:
Sampling bias
Sampling bias occurs when the sample of individuals participating in your study differs systematically from the population of interest because of how you collected your data. For example, suppose your population of interest is all undergraduate students at MIT, and you collect data by asking students to take a survey on an iPad as they exit a class. In that case, you may have sampling bias because your data will only reflect the experiences of students enrolled in that class.
Nonresponse bias is a specific type of sampling bias in which factors of interest may influence participants’ decision to participate in the study. For example, you may find that female students were more likely to respond to your study invitation than male students. This would make your sample less representative of the overall population.
To avoid sampling and nonresponse biases, carefully consider how you recruit participants and collect data to encourage broad participation among individuals in your chosen sample.
Selecting Research Methods
Inclusiveness and accessibility
You should ensure that how you collect data does not inadvertently exclude individuals from participating. A common mistake is collecting demographic data in a way that excludes underrepresented and marginalized groups. Researchers should structure demographic items such that participants can accurately record their identities.
In addition, researchers should ensure that their data collection methods are accessible to students with disabilities. If you are collecting data online (e.g., with an online survey), ensure that your data collection adheres to standards for digital accessibility (link to DAS: https://studentlife.mit.edu/das/accessibility/digital-accessibility). MIT’s Disability and Access Services can review and consult with researchers and instructors on digital accessibility. If you collect data in person, you will want to consider physical accessibility (link to DAS: https://studentlife.mit.edu/das/accessibility/physical-accessibility).
Ethics and informed consent
There are important ethical and legal issues to consider when collecting data. Depending on your study’s parameters and the data’s intended use, you may need approval from the Committee on the Use of Humans as Experimental Subjects (COUHES). Always check whether you need COUHES approval before collecting any data.
Whether or not you need approval from COUHES, you will generally want to request students’ informed consent before using their data for research purposes. Students should be informed of the intended use of their data and given the opportunity to provide consent or decline without any coercion from the researcher(s). Special considerations apply when instructors or other people in positions of authority or power over the potential participants are collecting data.
Privacy and confidentiality
When collecting and presenting data on humans, especially students, it is essential to be mindful of privacy law. For example, the Family Educational Rights and Privacy Act (FERPA) limits the use of students’ educational information for research without the informed consent of students or their parents. You may need to obtain permission from the students before using their educational information for research, even if you can access it for other purposes.
In addition, you will want to ensure that you collect and store data using secure methods that decrease the likelihood of data breaches. To learn more, see COUHES’ guidance on data protection (link: https://couhes.mit.edu/guidelines/data-protection).
Practical considerations
Researchers never have unlimited time and resources, so practicality is an important factor to consider. Some methods of data collection take less time and resources than others. However, practicality should never supersede ethical considerations. Ultimately, your goal as a researcher is to find the most practical methods that allow you to achieve your objectives in a reliable and valid manner.
If you need help weighing the various factors involved in data collection, or if you would like help identifying practical ways to achieve your study objectives, contact the R&E team at TLL for a consultation.
Data analysis and interpretation
Once you have collected and organized your data, you will want to have a plan for analyzing it, interpreting the results, and ultimately using it for its intended purpose(s). Just as there are many ways to collect data, there is a multitude of data analysis procedures and tools available to you.
How you analyze your data will depend heavily on your research questions, the type of data you have collected, and your overall methodological approach (qualitative and quantitative).