Survey Guidelines
The First Steps
When planning or designing a survey, the most important step is to clearly define the question to be answered by the survey. A survey is most effective if its purpose can be clearly and succinctly stated. Surveys that are vague or overly-broad can become too long or difficult to analyze. Additionally, depending on the question(s) to be answered, a survey may not be the best method to answer the question. Focus groups and interviews are examples of other methods that may be useful in certain contexts.
Once a question or topic has been identified, OIRE suggests reviewing existing information to see if it is possible to answer your question using data that has already been collected by another office or an existing survey. We encourage you to contact us ([email]data@hmc.edu[/email]) for a consultation to find out what information is already available. OIRE staff can discuss how the data we have collected may be able to answer your question or to give you feedback on your survey. If you find that your question can be answered by an analysis of existing data, OIRE welcomes specific requests for data.
Survey Options
If your question cannot be answered by existing data, there are options for using a survey to answer your question. We encourage you to ask OIRE for advice on which option best suits your needs.
- Administer the survey yourself. Commercial tools (e.g., SurveyMonkey, Qualtrics, Google Forms) allow you to create, host, and administer a survey yourself.
- Have OIRE host and administer the survey for you. Subject to availability and existing survey schedule, OIRE can assist you with the design of a survey, host, administer and analyze it for you.
- Include your questions on a survey already scheduled for administration. In some cases, constituencies on campus have appended a limited number of targeted questions to surveys administered by OIRE. In these situations, OIRE maintains editorial rights and the right to refuse questions.
Important Considerations
Depending on the population and topic of your survey, the following issues should be carefully addressed before a survey is administered:
Institutional Review Board Approval
You and the College may incur legal liability if the treatment of survey participants is unethical, if data resulting from your survey are misused, or if any part of the survey violates certain protected rights of individuals. Survey researchers should be aware of their responsibilities and make every effort to protect the rights of survey respondents. For more information, please consult IRB.
Office for Human Research Protection (OHRP) Training
If the data to be gathered and used are part of an academic research project, the project will need to be reviewed and approved by the IRB to ensure it meets their requirements. If the data are to be used for administrative purposes, it may be exempt from a formal review by the IRB, but the survey must be voluntary and the results kept confidential. For more information, please consult IRB.
The Rights of Respondents
Our office adheres to the Association of Institutional Research’s Statement of Ethical Principles. Any survey must clearly identify the group or person who is conducting the survey and provide contact information (typically name, email address and telephone number) should respondents have any question about the content of the survey or about the use/publication of the survey results. This information is typically included in the instructions. Sample language is below:
“You are asked to complete this survey as part of the Engineering department’s annual assessment. If you have any questions or concerns about the survey, please contact the Director of Institutional Research and Effectiveness, Dr. Laura Palucki Blake, at [email]lpblake@hmc.edu[/email] or x78191”
All participants must be notified that their participation in the survey is voluntary. Voluntary participation means that participants have the right to decide how much information they share about themselves. This means they can decide whether or not to participate in the survey, but it also means they should not be required to answer any item on the survey. Sample language is below:
“Your participation is strictly voluntary, and you will be under no obligation to answer any questions that you are not inclined to answer. You may choose not to answer any specific questions you do not want to answer and still participate in the survey.”
or
“Your participation is voluntary and your responses will be confidential. Pseudonyms will be substituted for any identifying information, unless we explicitly obtain your permission to use your name. Faculty, staff, students, and others with permission or authority to see your study information will maintain its confidentiality to the extent permitted and required by laws and institutional policies. The names or personal identifiers of participants will not be published or presented. If you feel you have not been treated according to the descriptions in this form, or if you have any questions about your rights as a research subject, including questions, concerns, complaints, or to offer input, you may call the Institutional Research and Effectiveness at [tel]909.607.8191[/tel] or email us at [email]data@hmc.edu[/email].”
If your survey of data analysis will include academic or contact information for HMC students, you may be subject to Federal Family Educational Rights and Privacy Act (FERPA) regulations. Non-directory student data (e.g., GPA, Major(s), race/ethnicity) can be used without a student’s consent by college employees for “legitimate educational” purposes, provided the data are not reported in such a way that individual students can be identified. All other users must receive written consent from students to access non-directory student data. Learn more about student rights, or contact the Office of the Registrar.
Confidentiality and Anonymity
An anonymous survey means that the researcher cannot determine who participated in the survey and who has not. Anonymous surveys typically do not collect identifying information, and cannot restrict responses to one response per person. Keep in mind that even a survey does not collect identifying information, even collecting an IP address (as electronic survey tools do) can compromise anonymity. Anonymous surveys are often employed for sensitive subject matters (e.g., sexual assault, alcohol and drug use and abuse), because anonymity can yield more honest responses.
A confidential survey collects information that can identify participants either alone or in combination (e.g., race/ethnicity, gender, major, class year) and allows the researcher to analyze responses along a variety of demographic factors, but also means researchers have a responsibility to protect the privacy of the participant. When data is collected confidentially, it is helpful to let participants know that data will be analyzed at the group level in order to de-identify participants, and that results that potentially identify participants will not be presented.
Incentives
Depending on the population and the survey, you may offer incentives to survey respondents. Incentives should be positive (i.e., no negative consequences for failing to participate). They should be relevant to the topic of the survey, and small enough so as not to be coercive (e.g. gift cards to Amazon, Starbucks, or the bookstore, movie tickets, t-shirts, Claremont Cash, water bottles) Incentives above a certain dollar value can be construed as gifts: and potentially interfere with a student’s financial aid or violate NCAA rules. Raffles or lotteries can also be construed as gambling, and require special documentation. Please consult with OIRE if you have questions about incentives.
Simplicity
The shorter and simpler the survey, the more likely respondents are to answer all the questions and answer them honestly. Complex or long surveys can annoy respondents and cause them to stop responding part-way through or to pick arbitrary answers to get to the end quickly.
Permissions
Before administering a survey to a population, and therefore emailing a large segment of that population, it is important to notify and get permission from those responsible for that population. For example, before surveying all Computer Science majors, one would contact the chair of the Computer Science department. Before sending a survey to students-l one would contact the Vice President for Student Affairs. Before emailing a dorm list, one would contact the dorm proctor. OIRE can assist you in contacting the right individuals.
Sample vs Census
When planning a survey, it is important to decide if you will need to administer the survey to an entire population (All HMC seniors) or to a sample of the population (50 randomly chosen HMC seniors). This choice depends on how you want to analyze the results. If you are interested in broad measures, such as overall satisfaction, a sample may be sufficient. If you want to answer questions about seniors who are Math majors vs seniors who are CS majors, you may need a larger sample or a census in order to have enough respondents to complete your analysis.
Testing
It is useful to pilot test your survey and survey questions with a small group from the population you intend to survey. This can help you understand how the questions you have written will be perceived and answered, and whether the survey is too long or overly complicated. This is your chance to get the bugs out.
Timing
OIRE maintains a calendar of known surveys:
Please consult the calendar and OIRE for the timing of your survey, as concurrent surveys that target the same population can lower response rates and overburden respondents. OIRE will add your survey to the calendar.
Analyzing and Reporting Results
After your data have been collected, you will want to analyze the results. Simple analysis can be done using Excel or Google, and displayed with tables and graphs. More advanced statistical analysis will require the use of software such as R, SPSS, or Stata. All are available to the HMC community, but have a learning curve. OIRE uses primarily SPSS, and if OIRE has hosted the survey for you, can provide an analysis of the survey for you. Depending on schedule and availability, OIRE may be able to analyze surveys that are hosted independently.
Representativeness
A sample needs to be large enough to protect the privacy of respondents, and it also needs to be large enough for your results to be considered representative, but that does not mean you need everyone. If you are interested in the responses of women in Chemistry, for example, and you receive 5 responses from Women in Chemistry, out of a possible 50 women, you could not report with confidence that their responses are representative of women in Chemistry. To calculate the minimum sample size to be considered representative within a given margin of error, visit Sample Size Calculator.
Small Cell Size
In order to protect the privacy of your respondents, it is important that you do not report results for very small groups of people. For example, if your intended report would break responses out by department, but only 3 people responded from Engineering, then reporting those responses could jeopardize the privacy of those respondents. OIRE only reports summary results and uses the standard practice of only reporting cell sizes of 5 or greater.
Archiving Data
Once you have finished gathering data, you should consider archiving the survey instrument, administration details, and unit-level responses. OIRE maintains a secure server for HMC data and reports, and will be happy to archive data collected at HMC. This can be especially helpful, for example, when faculty go on sabbatical or if survey data need to be accessed during summer months/breaks, if a student group changes leadership, etc. If the group that administers the survey does not archive the data, the results can be lost instead of used to inform future discussions and decision-making at HMC.
Once your results have been analyzed and reported, we urge you to share your results in a way that is accessible to the community you surveyed. OIRE has found that by being transparent with results, the populations we survey can see the value in answering surveys. For examples, see Institutional Results by Instrument.