How to administer a survey at MIT
Often, MIT IR is approached by a student group or an office on campus who would like to administer a survey. The following is intended to assist those wishing to implement a survey efficiently and productively use their time and resources.
First Steps
When designing or planning a survey, the most important step is to clearly define the question to be answered by the survey results. A survey is most effective if its purpose can be clearly and succinctly stated. Surveys with vague or overly-broad motivations can become too lengthy or difficult to analyze. Also, a survey may not be the best way to answer your question; focus groups are an example of another research method that may be useful in certain contexts.
Once a target question or topic has been identified, we suggest reviewing existing information to see if it is possible to answer your question using data that has already been collected by another office or by the answers to questions on existing surveys. At this point in the process, we encourage you to contact our office for a consultation. We are happy to discuss how the data we have collected may be able to answer your question or to give you feedback on your survey. Also, if you do find that your question could be answered by an analysis of existing survey data (a particular question broken out by gender, for example), we welcome specific requests for data.
In general, the following list provides a rough guide of the survey process:
1. Authoring survey and creating the population profile
2. Designing incentives (optional)
3. Launching survey
4. Sending out invitations and reminders
5. Collecting the data
6. Analyzing
7. Archiving
Survey Options
If your question cannot be answered by existing sources of data, there are multiple options for using a survey to answer your question. We encourage you to come to us for advice on which option best suits your needs.
Administer the survey yourself. Commercial tools, (e.g., SurveyMonkey) can allow you to create and host a survey yourself.
Have IR administer and host the survey for you. Subject to our availability and existing survey schedule, we may be able to assist you with the design of a survey and then host and administer it for you. This service may be subject to a fee.
Include your questions on a survey already scheduled for administration. In the past, constituencies on campus have appended a limited number of targeted questions to surveys administered to various MIT populations by IR. In these circumstances, IR maintains editorial rights and the right to refuse questions.
Regardless of which option you choose, we are always happy to help you form your survey questions.
Considerations
Depending upon the population and the topic of your survey, we suggest considering the following issues:
Permissions
Before administering a survey to a population, and therefore emailing a large segment of that population, it is important to notify and get permission from the senior officer responsible for that population.
For Undergraduate students, that would be the Dean for Student Life, the Vice Chancellor, or the Chancellor
For Graduate students, the Vice Chancellor or the Chancellor
For Post-Docs, the Vice President for Research
For Faculty, the Chair of the Faculty or the Provost
For Staff, the Vice President for Human Resources, the Executive Vice President and Treasurer, or the Vice President for Institute Affairs
Anonymity and Confidentiality
There are different types of surveys: polls that ask for respondents to give input for the planning of an event, a data collection asking respondents to provide information to update their records, or engagement surveys that seek to gauge respondents’ attitudes toward a set of topics. Depending upon the type of survey and the subject matter, it is important to determine whether the survey should be anonymous, confidential, or neither.
Anonymous: It is difficult to guarantee anonymity. Many survey platforms log IP addresses, and many individuals can be uniquely identified by a combination of demographic data points that are commonly asked on surveys. . For example, if the combination of age, gender, ethnicity and major would allow the surveyor to know the identity of a specific individual, then the survey does not meet the standard of being anonymous. A downside to an anonymous survey is that it is difficult to restrict the responses to be one per individual. However, for sensitive subject matters, anonymity may yield more honest responses.
Confidential: In MIT Institutional Research, we most often conduct surveys that are confidential but not anonymous. We tend to authenticate our surveys (i.e. we know who is answering using unique links) and maintain some basic identifying information along with the responses. This allows us to analyze responses along a variety of demographic factors, but also gives us the responsibility of guarding sensitive information.
If the administrator of a survey promises or implies that the responses will be treated as confidential that means that the results are confidential to the research team; i.e., the results should never be reported at a level of granularity that would allow someone outside the research team to infer the identity of an individual. In no cases should individual survey responses be shared with a supervisor, an advisor, or anyone who is not part of the research team. Whenever possible identifiers should be separated from the dataset.
There will be cases where the research team includes people in authority or people who have a relationship with individuals being surveyed, such as the personnel administrator for a department or a housemaster. In these cases, it must be made very clear to those invited to take the survey who will have access to the individual responses
Occasionally, existing datasets will be requested for uses beyond the original intent of a survey. For example, a senior officer may want to have additional analysis done on a particular topic or an academic researcher may want to use data collected in a survey for their research. In these cases, the analyst will be bound by the same confidentiality requirements as the original team.
Incentives
Depending upon the population and needs of your survey, it may be desirable to offer incentives to survey respondents. Two guidelines to consider are that the incentives must be positive (i.e. no negative consequences for non-respondents) and should be relevant to the topic of the survey (e.g., if you are conducting a transportation survey with the purpose of encouraging public transit, free parking passes would not be an appropriate incentive).
Research with Human Subjects
If you or anyone working with the data will be analyzing data that identifies individual respondents, either by name, email address, MIT ID number, or other unique personal information, then those staff members working with the data will need to complete COUHES (Council on the Use of Human Experimental Subjects) training prior to handing sensitive data. This training can be done from any computer with MIT certificates enabled from the URL: http://couhes.mit.edu/
If the data are to be gathered and used as part of an academic research project, the project will need to be formally cleared with COUHES to ensure that it meets their requirements. If the data are to be used for administrative purposes, it may be exempt from a formal review by COUHES, but the survey must be voluntary and the results kept confidential. Please consult with IR if you have any questions about COUHES requirements.
Inclusive language
You may want to include demographic questions on your survey that ask respondents about various aspects of their identity so that you may analyze the results in a way that compares different groups of respondents. When doing so, it is important that the way you ask demographic questions is inclusive and respectful. This guide covers best practices for inclusive language in survey questions.
Testing the survey
It is useful to convene a small group from the population(s) you intend to survey to have them test the survey and survey questions. This can help gauge how the questions you have written will be perceived and answered, and whether the survey is too long or complicated.
Sample vs. Census
When planning a survey, it is important to decide if you will need to administer the survey to an entire population (all MIT seniors, for example), i.e. a census, or to a sample of the population (200 randomly-chosen MIT seniors). This choice can largely be informed by how you want to analyze the results. If you are interested in broad measures, such as overall satisfaction, a sample may suffice. If, however, you want to answer questions about students in Math versus students in Chemistry, you may need a larger sample or a census in order to net enough respondents for your analysis.
Small cell size
It is important to make sure that you do not report results for very small groups of people. For example, if your intended report would break out responses by department, but only 4 people responded from one department, then reporting those responses could jeopardize the privacy of those respondents’ answers. IR only reports summary results and uses the standard practice of only reporting cell sizes of 5 or greater.
Representative sample size
While a sample needs to be large enough to protect the privacy of the respondents (see above), it should also be large enough for your results to be considered representative. If you are interested in the responses of women in Math, and you only receive 5 responses from women in Math, (out of 20 women in Math, for example) you could not report with confidence that their summary responses as representative of all women in Math. To calculate the minimum sample size to be considered representative within a given margin of error, visit: http://www.surveysystem.com/sscalc.htm
Keep it simple
The simpler and shorter a survey, the more likely respondents are to answer all questions and answer them honestly. Longer, more complex surveys can annoy respondents and cause them to abandon the survey part-way or begin to pick arbitrary answers to speed to the end. Also, only ask questions that you need and intend to report.
Archiving Data
Once you have finished gathering your data, please consider archiving the survey instrument, administration details, and record-level responses with IR. IR maintains a secure server for MIT data and repots, and will be glad to archive data collected at MIT. This can be especially helpful, for example, when a student groups changes leadership. If that student group administers a survey and does not archive the data, the results can be lost instead of used to inform future decision making.
Reporting and Analyzing Results
After you have collected the responses to your survey, it will be necessary to analyze the results. Simple analyses can be done using Excel with Pivot Tables, histograms, and graphs, but more advanced statistical analyses will require the use of a statistical software package such as SPSS, SAS, Stata, or R. Stata is available to the MIT community on Athena, and R is open-source free software, but both systems have a learning curve. IR uses SPSS primarily and may be able to provide analysis of survey results for you, depending upon our schedule and availability. This service may have a fee associated.
Once your results have been analyzed, we urge you to share the results in a way that they are accessible to the community surveyed. We have found that by being transparent with our results, the populations we survey can see the value in answering our surveys. For example, see the IR Surveys page where we post overall summaries and results for each question on our surveys.
Conclusion
Our office is happy to provide guidance and assistance with answering your questions using a survey, designing a survey, or how to analyze or present results. Feel free to contact us with any questions.