During conversations about program evaluation and survey design, we often about “what makes a good survey question?” There is no one definitive set of rules about writing questions and questionnaires But there are some general principles that are commonly accepted. There is lots of advice focused on avoiding leading questions and avoiding cultural bias. There is lots of discussion about choosing the best format for questions, comparing the merits of open-ended, rating scale, multiple choice questions or yes/no questions. But there is no exact science. Every survey-writer needs to apply their best judgment about their survey audience and the goals of their survey.
Here are some of my ideas for creating youth-friendly surveys and questionnaires for youth and families.
Questions Should Be:
Using questions for which a person can comfortably choose any answer, without feeling that they ought to give a particular answer.
Using questions that realistically reflect the range of respondent interests and experiences and are likely to provide a variety of responses.
Using questions that respect individual styles and cultures.
Designing questions and surveys that are in-depth enough to be effective but not overly time-consuming.
Focusing questions on topics relevant to program goals, activities and necessary background.
Using questions that will produce information that can be can be analyzed, shared and discussed.
Integral to programming
Making information-gathering a natural and integral part of programming.
Principle #1: Comfortable. Using questions that a reasonable person can comfortably choose any answer, without feeling that they ought to give a particular answer.
Respondents should feel comfortable enough to respond honestly, without feeling that the question is leading them toward a particular conclusion or implying that they "ought to" respond a certain way. Classic "leading questions" are phrased in a way that subtly or not-so-subtly lead the respondent toward a particular response. “
Don’t you think that ____ is getting too old to be an effective mayor?” is an example of a not-at-all subtle leading question. "
Do you eat at least seven servings of vegetables daily?" or "Do you regularly eat dinner together as a family at the dining room table?" are examples of a what I call ought-to" questions: questions which suggest that the respondent "ought to" say yes.
Yes/no type questions about healthy habits, family activities or many other lifestyle topics can take on an “ought-to” tone because schools, media and other sources regularly promote the idea that people ought to eat lots of vegetables, have a good breakfast daily. eat dinner together regularly, or other key habits. The phrasing of the question and the juxtaposition of a series of "ought to" topics can also suggest that a respondent ought to answer a certain way. Survey respondents should not be forced to say “no” explicitly to something they know ought to be a yes. Instead, questions should be structured to allow respondents to choose from a variety of options on checklists or use rating scales that allow a range of answers, so they can feel comfortable assessing their relative strengths and weaknesses, preferences, opinions or interests related to a topic and answer comfortably and honestly.
Principle #2: Realistic. Using questions that realistically reflect the range of respondent interests and experiences and are likely to provide a variety of responses.
Questions should realistically reflect the likely interests and experiences of the population. For example: a career interest survey asks a series of questions about interests, such as:
“Do you enjoy performing music?”
“Do you enjoy working with children?”
“Do you enjoy filling out income tax forms?”
For a younger population, the question about income tax forms is not realistic because young people probably haven’t filled out income tax forms yet. For an older population, income tax forms might not be something people realistically think of as “enjoyable” even if they enjoy working with information and filling out forms.
Any survey assessing interests, experiences and attitudes should suggest options that are realistic for the targeted population and age group. It is okay if some questions are a “stretch,” but important that the overall range of options is realistic.
Principle #3: Respectful. Respecting individual styles and cultures.
It is helpful to have several people look over proposed questions to look for both cultural bias and for issues with regard to respecting variations in individual styles and values.
For example, different people or cultures may use different wording when talking about volunteering, service learning, community service, or helping out in the community. The questions "Do you enjoy community service?" or "Do you enjoy volunteering in the community?" or "Do you enjoy helping out in your community?" may bring out different understanding and responses for different people.
Or, for example, different cultures (and individual families) may emphasize different aspects of family life, so questions about family life and parent involvement should recognize and respect a variety of approaches.
One important part of respectful questionnaire design is to avoid conveying biased or negative images of participants’ lives, families and neighborhoods. For examples, surveys with a heavy focus on risk behaviors suggest to youth that the survey-writers expect these behaviors. Or, for example, surveys that ask about possible deficits in the participants’ lives suggest that the survey-writers expect these deficits. If it is important to survey youth about these issues, it can be helpful to balance the survey by asking about positive behaviors and assets as well, and perhaps to use rating scales, checklists with multiple options or other survey methods to allow a range of positive or mid-range responses.
It is important to remember that for youth especially (but also for individuals of all ages) taking a survey is a learning experience. Survey takers form impressions about the world, about themselves, and about your program based on the nature of the questions posed to them.
Principle #4: Reasonable Length. Designing questions and surveys that are in-depth enough to be effective but not overly time-consuming.
It’s important to seek balance in determining the overall length of a questionnaire and the length and complexity of the questions. You -- and your survey respondents -- want the survey to be long enough to gather useful and meaningful information. But you want to be respectful of the respondents' time and attention.
Consider the quality of the survey-taking experience. If answering a question provides respondents with a positive opportunity to reflect on their own experiences or interests or to share feedback about your program, then it feels worthwhile for the respondent to “invest” time in the survey. But if questions feel repetitive or require too much effort, respondents will quickly start to feel that the survey is too long.
Principle #5: Focused. Focusing questions on topics relevant to program goals, activities and necessary background.
Questions should be clearly focused on the program goals and activities, including any background information that is relevant.
Questions about goals and activities should be asked using simple, clear language:
* using simple, fresh, everyday language (not 'program-specific' language if possible);
* very briefly defining any terms or concepts that may need clarification;
* avoiding jargon or slang;
* avoiding "compound" questions that combine more than one thought into a single question;
Ideally, any questions about demographics, such as gender, age group or other details, are saved until the end of a survey, and kept to a minimum, focusing only on the demographics that are required for reporting or likely to be relevant to analyzing the survey data.
Principle #6: Useful. Using questions that will produce useful information that can be analyzed, shared and discussed.
Whatever the format of the question, including open-ended questions, yes/no, multiple choice, rating scale, etc., it is important to gather information that can be summarized, analyzed and shared with staff and others for improving programming, describing activities and outcomes, evaluating the program, or seeking support and funding for future programming. Questions should be designed so that the analysis can compare results over time (if the survey will be repeated in the future) and across groups of survey-takers. Open-ended questions, if used, should provide insights that can be used as conversation-starters for shaping future strategies and that can be summarized and grouped in a way that provides summary-level data as well as individual insights.
Principle #7: Integral to programming. Making information-gathering a natural and integral part of programming.
Wherever possible, information gathering should use reflections, goal setting activities, career interest checklists, or other materials that are integral to the programming.
Staff and participants should get accustomed to a few different types of information-gathering activities such as pre-program and post-program questionnaires or tests, reflection sheets after an activity, quarterly parent, teacher or student surveys, goal-setting questionnaires, interest questionnaires or college/career planning checklists.
If possible, information-gathering should be based on activities that are primarily designed as learning experiences for the participants, and that also provide useful data for the program. Goal-setting questionnaires, interest checklists, skills checklists or other types of reflections can provide an opportunity for valuable reflection and goal setting for participants while also providing data for program evaluation.
Whatever the formats used, data collection should be seen as an integral part of programming. It may provide a valuable opportunity to reflect and learn -- and it always should be seen as a comfortable and pleasant part of the program.
- Category: Resources
The concept of Youth Development is used to design youth programs through a positive lens, viewing youth as positive assets to the community and understanding the positive experiences that contribute to personal development. Youth development theory emphasizes the idea of focusing on assets rather than deficits, viewing youth employment programs and summer jobs programs not as “keeping kids off the streets” or “crime prevention”, but as positive opportunities to develop skills, make a contribution to the community, earn money, gain work experience, gain career awareness and build a resume.
In the spirit of youth development, it is useful to recognize the assets that youth bring to their jobs, as well as to understand the developmental needs of youth as they gain their first work experiences. Therefore, when supervising youth in youth employment programs, finding the right level of supervisory support is important. As teens and young adults begin their early work experiences, they bring a mixture of skills and readiness to these experiences. Teens and young adults have an emerging ability for complex reasoning and intellectual development. They are excited by opportunities to learn about the background and history of an organization, understand the larger context of their work, and understand how their work contributes to the goals of the organization. They appreciate settings in which they are respected and treated like adults. At the same time, while youth are starting to develop problem-solving skills, and starting to learn about careers and work, they generally need clear guidance about how to manage workplace expectations for time management, attendance and punctuality, workplace appearance, taking initiative and other basic skills.
Many youth employment programs provide orientation sessions and workshops to provide coaching about basic workplace expectations and skills as well as other topics. Programs also use informal, one-on-one coaching to support youth. In both formal and informal orientation, the Massachusetts Work-Based Learning Plan helps to open up conversations between supervisors and youth about a range of topics relevant to skill development.
- Use the Work-Based Learning Plan (WBLP) as a tool to open up conversations about the context of the work, the history and goals of the organization, and other topics of interest, as well as to outline the basic foundation skills needed on the job. Communicate excitement about the organization and its role in the community. Share information about the organizational history, current projects, the number of customers or visitors, etc.
- When practical, when writing a job description and list of skills/tasks, include information about “who, what, when, where, why, how” in order to share information about the context of the work.
- Emphasize workplace safety through Work-Based Learning Plans and through formal or informal orientation to the workplace.
- Throughout the WBLP and other program materials, use language that sets a positive, professional tone. Read more about language and youth development in the article Marketing and Messaging in Youth Employment Programs.
In workshops about the Work-Based Learning Plan for youth employment program staff, I often ask participants to think about the best summer job they ever had. Images of working at a favorite ice cream shop, a family-owned dress shop, a beach, a farm, a playground, a zoo and many other diverse workplace settings enter the conversation. Images of informal mentoring, a sense of “belonging” and a sense of accomplishment are important. These are the positive moments of summer for youth and young adults working in first jobs.
- Category: Resources
The most creative aspect of the work of staff in a youth program is the role of bringing together a variety of people - youth, employers, school personnel, parents, and community organizations. In a successful program, a positive synergy develops, and each aspect of programming reinforces the others. At any of our statewide or regional conferences or meetings, we hear success stories about ways that Connecting Activities staff and others have worked to set creative partnerships in motion in their communities.
Success stories may be very specific -- the parent of a student in your program volunteers to be a career speaker; a career speaker decides to offer a summer job or internship; a student intern makes a visible and positive contribution to a local community organization. Success stories may be community-wide -- the community mobilizes for a fundraiser or a summer jobs campaign or to open a new youth career center; a school organizes a school-wide strategy to raise gradaution rates; an employer group makes a commitment to mentoring students in your community.
In successful programs, participating youth are energized by their experiences. Youth develop new skills, gain valuable experiences, and see new connections between the work they are doing in school and their future careers. Through school, workplace and community activiites, they start to envision future career and educational paths and have confidence in their first steps toward career development.
This section of the website provides links related to:
- Youth development concepts
- Employer partnerships
- Connecting witih school and community initiatives
- Connecting with parents
- Marketing, messaging and media
- Category: Resources
Connecting Activities programs uses a set of Work and Learning Program Elements to guide programming and structure reporting. The program elements are designed to help us build connections in schools so that Connecting Activities programming is an integral part of school-wide efforts to support College and Career Readiness for all students.
Definitions of the "Work and Learning" Program Elements:
A = All Work Experiences. Total number of students participating in jobs/internships brokered by Connecting Activities. Brokering may include direct placement, managing Work-Based Learning Plans, providing workshop or classroom programming or other brokering roles.
G = Planning Goals. Number of students in jobs/internships linked to individual student formal planning goals, based on a College and Career Plan, Individual Education Plan, Educational Proficiency Plan or other formal planning process.
R = Raising Graduation Rates. Number of students in jobs/internships where the student was targeted or identified through school-wide initiatives to raise graduation rates. This may include, for example, students identified through a MassGrad program, the Early Warning Indicator System, MCAS results for students who need to re-test, and dropouts who have re-enrolled in school.
O = Workshop or Classroom Component. Number of students in jobs/internships with a classroom or workshop component. This may include an internship class, Academic Support program, CVTE program, career pathway program, or workplace readiness or career development classes and workshops.
W = Work-Based Learning Plans. Number of students in jobs/internships with Work-Based Learning Plans.
E = Number of Employers. Total employers providing jobs/internships through Connecting Activities. The Work and Learning Performance report also includes a count of private, nonprofit and public employers.
Details -- Work and Learning Program Elements, Reporting and Data Entry:
A = All Work-Based Learning Placements.
All Work-Based Learning Experiences placements count in the category "A = All Work-Based Learning Placements." The database reports will look at the field "Program Type" and count all of the "Work-Based Learning" experiences.... which is filled in by default on all entries in the database.
G=Connected to Planning Goals - Work-based learning experience is coordinated with a College and Career Plan, EPP, Transition Plan or IEP, WIA service plan or other planning process.
This defintion emphasizes that this element should highlight programs that are coordinated with college and career plans or other planning processes. "Coordinated" can mean that a student was referred to a job or internship program via a planning process, or that the student selected a career interest area for their job or internship via a planning process, or that students update their college and career plan after participating in a work-based learning experience, or that career specialists who coordinate the work-based learning program help students with the college and career plan, or other direct coordination. (It is not enough that the student happens to have a college and career plan, EPP, IEP, Transition Plan or other plan, if there is no connection with your program.)
In the database, choose one of the following:
Yes-College and Career Plan (use this answer when the student has a College and Career Plan and participation in a job or internship is one of the student's goals.)
Yes-Other (use this answer when the student has any other type of plan, such as an EPP, IEP, Transition Plan, WIA Inidvidual Service Plan, or other document, and participation in a job or internship is one of the student's goals.)
R=Raising Graduation Rates - Participant was targeted/identified through a school initiative to raise graduation rates. This includes students identified through MCAS results who need retesting, as well as students identified through an Early Warning Indicator Index, Dropout Recovery Program, or other school-wide initiatives to raise graduation rates. It is important that Connecting Activiities programs be a "natural" part of a school-wide initiative, and not "artificially" target students for participation based on any of these factors. Connecting Activities staff should work with school leaders to make jobs and internship and career development activities an integral part of the school's programming for students who need extra support to graduate.
In the database, this field is a simple Yes, No, Unknown or Not Applicable. There is a comment field that allows extra detail as desired.
O=Has Workshop or Classroom Component - A workshop, class or academic program is connected to this work-based learning experience.
In the database, choose one of the following:
Yes-Work and Learning Academic Support Program
Yes-Work and Learning Academic Support Program
Yes-Career Pathway Program
Yes-Work Readiness Workshop
Yes-Career Exploration Workshop
One-on-one coaching (this is just for information - does not count as a YES)
A comment field is available if you wish to enter the name of the class or workshop or other details.
W=Has Work-Based Learning Plan? This element has been part of Connecting Activities Reporting for many years. It is filled in automatically in the database when you start the job description in Step 2 of the database screens. Or you may enter a "YES" if the student has a Work-Based Learning Plan that is pen and paper or in the Microsoft Word document. The Work-Based Learning Plan counts as a "yes" for reporting as soon as it is started; with the expectation that during the summer or during the year the WBLP will include a job description and a list of skills/tasks and the student will have at least two performance reviews.
In the database, choose one of the following:
Yes - In Database
Yes - In Word
Yes - Pen and Paper
E=Employers As always, a count of employers who provide work and learning experiences is part of Connecting Activities reporting. In the database, new screens are being added to help us to manage information about employers.
- Category: Resources