During conversations about program evaluation and survey design, we often about “what makes a good survey question?”  There is no one definitive set of rules about writing questions and questionnaires  But there are some general principles that are commonly accepted.  There is lots of advice focused on avoiding leading questions and avoiding cultural bias.  There is lots of discussion about choosing the best format for questions, comparing the merits of open-ended, rating scale, multiple choice questions or yes/no questions.  But there is no exact science.  Every survey-writer needs to apply their best judgment about their survey audience and the goals of their survey.

 

Here are some of my ideas for creating youth-friendly surveys and questionnaires for youth and families.

 

Questions Should Be:

Comfortable

Using questions for which a person can comfortably choose any answer, without feeling that they ought to give a particular answer.

Realistic

Using questions that realistically reflect the range of respondent interests and experiences and are likely to provide a variety of responses.

Respectful

Using questions that respect individual styles and cultures.

Reasonable length

Designing questions and surveys that are in-depth enough to be effective but not overly time-consuming.

Focused

Focusing questions on topics relevant to program goals, activities and necessary background.

Useful

Using questions that will produce information that can be can be analyzed, shared and discussed.

Integral to programming

Making information-gathering a natural and integral part of programming.

 

 

 

 

Principle #1: Comfortable. Using questions that a reasonable person can comfortably choose any answer, without feeling that they ought to give a particular answer.

 

Respondents should feel comfortable enough to respond honestly, without feeling that the question is leading them toward a particular conclusion or implying that they "ought to" respond a certain way.  Classic "leading questions" are phrased in a way that subtly or not-so-subtly lead the respondent toward a particular response.

 

Don’t you think that ____ is getting too old to be an effective mayor?” is an example of a not-at-all subtle leading question. "

 

Do you eat at least seven servings of vegetables daily?" or "Do you regularly eat dinner together as a family at the dining room table?" are examples of a what I call  ought-to" questions: questions which suggest that the respondent "ought to" say yes.

 

Yes/no type questions about healthy habits, family activities or many other lifestyle topics can take on an “ought-to” tone because schools, media and other sources regularly promote the idea that people ought to eat lots of vegetables, have a good breakfast daily. eat dinner together regularly, or other key habits.  The phrasing of the question and the juxtaposition of a series of "ought to" topics can also suggest that a respondent ought to answer a certain way.  Survey respondents should not be forced to say “no” explicitly to something they know ought to be a yes.  Instead, questions should be structured to allow respondents to choose from a variety of options on checklists or use rating scales that allow a range of answers, so they can feel comfortable assessing their relative strengths and weaknesses, preferences, opinions or interests related to a topic and answer comfortably and honestly.

 

Principle #2: Realistic. Using questions that realistically reflect the range of respondent interests and experiences and are likely to provide a variety of responses.

 

Questions should realistically reflect the likely interests and experiences of the population.  For example: a career interest survey asks a series of questions about interests, such as:

 

“Do you enjoy performing music?”

“Do you enjoy working with children?”
“Do you enjoy filling out income tax forms?”

 

For a younger population, the question about income tax forms is not realistic because young people probably haven’t filled out income tax forms yet. For an older population, income tax forms might not be something people realistically think of as “enjoyable” even if they enjoy working with information and filling out forms.

 

Any survey assessing interests, experiences and attitudes should suggest options that are realistic for the targeted population and age group.  It is okay if some questions are a “stretch,” but important that the overall range of options is realistic.

 

Principle #3: Respectful. Respecting individual styles and cultures.

 

It is helpful to have several people look over proposed questions to look for both cultural bias and for issues with regard to respecting variations in individual styles and values.

 

For example, different people or cultures may use different wording when talking about volunteering, service learning, community service, or helping out in the community.  The questions "Do you enjoy community service?" or "Do you enjoy volunteering in the community?" or "Do you enjoy helping out in your community?" may bring out different understanding and responses for different people.

 

Or, for example, different cultures (and individual families) may emphasize different aspects of family life, so questions about family life and parent involvement should recognize and respect a variety of approaches.

 

One important part of respectful questionnaire design is to avoid conveying biased or negative images of participants’ lives, families and neighborhoods.  For examples, surveys with a heavy focus on risk behaviors suggest to youth that the survey-writers expect these behaviors.  Or, for example, surveys that ask about possible deficits in the participants’ lives suggest that the survey-writers expect these deficits.  If it is important to survey youth about these issues, it can be helpful to balance the survey by asking about positive behaviors and assets as well, and perhaps to use rating scales, checklists with multiple options or other survey methods to allow a range of positive or mid-range responses.

 

It is important to remember that for youth especially (but also for individuals of all ages) taking a survey is a learning experience.  Survey takers form impressions about the world, about themselves, and about your program based on the nature of the questions posed to them.

 

Principle #4: Reasonable Length. Designing questions and surveys that are in-depth enough to be effective but not overly time-consuming.

 

It’s important to seek balance in determining the overall length of a questionnaire and the length and complexity of the questions.  You -- and your survey respondents -- want the survey to be long enough to gather useful and meaningful information.  But you want to be respectful of the respondents' time and attention.

 

Consider the quality of the survey-taking experience.  If answering a question provides respondents with a positive opportunity to reflect on their own experiences or interests or to share feedback about your program, then it feels worthwhile for the respondent to “invest” time in the survey.  But if questions feel repetitive or require too much effort, respondents will quickly start to feel that the survey is too long.

 

Principle #5: Focused. Focusing questions on topics relevant to program goals, activities and necessary background.

 

Questions should be clearly focused on the program goals and activities, including any background information that is relevant.

 

Questions about goals and activities should be asked using simple, clear language:

 

* using simple, fresh, everyday language (not 'program-specific' language if possible);

* very briefly defining any terms or concepts that may need clarification;

* avoiding jargon or slang;

* avoiding "compound" questions that combine more than one thought into a single question;

 

Ideally, any questions about demographics, such as gender, age group or other details, are saved until the end of a survey, and kept to a minimum, focusing only on the demographics that are required for reporting or likely to be relevant to analyzing the survey data.

 

Principle #6: Useful. Using questions that will produce useful information that can be analyzed, shared and discussed.

 

Whatever the format of the question, including open-ended questions, yes/no, multiple choice, rating scale, etc., it is important to gather information that can be summarized, analyzed and shared with staff and others for improving programming, describing activities and outcomes, evaluating the program, or seeking support and funding for future programming.  Questions should be designed so that the analysis can compare results over time (if the survey will be repeated in the future) and across groups of survey-takers.  Open-ended questions, if used, should provide insights that can be used as conversation-starters for shaping future strategies and that can be summarized and grouped in a way that provides summary-level data as well as individual insights.

 

Principle #7: Integral to programming. Making information-gathering a natural and integral part of programming.

 

Wherever possible, information gathering should use reflections, goal setting activities, career interest checklists, or other materials that are integral to the programming.

 

Staff and participants should get accustomed to a few different types of information-gathering activities such as pre-program and post-program questionnaires or tests, reflection sheets after an activity, quarterly parent, teacher or student surveys, goal-setting questionnaires, interest questionnaires or college/career planning checklists.

 

If possible, information-gathering should be based on activities that are primarily designed as learning experiences for the participants, and that also provide useful data for the program.  Goal-setting questionnaires, interest checklists, skills checklists or other types of reflections can provide an opportunity for valuable reflection and goal setting for participants while also providing data for program evaluation.

 

Whatever the formats used, data collection should be seen as an integral part of programming.  It may provide a valuable opportunity to reflect and learn -- and it always should be seen as a comfortable and pleasant part of the program.