What makes a good reflection question?? What makes a good survey question?

Youth-friendly questions should be:

Comfortable:  Questions that a person can comfortably more than one way, without feeling that they ought to give a particular answer.

Realistic: Questions that are realistically likely to provide a variety of responses.

Respectful: Respecting individual styles and cultures.

Reasonable length: In-depth enough to be meaningful but not overly time-consuming.

Focused: Questions focused on topics relevant to program goals and activities.

Useful: Questions that will provide participants with a valuable opportunity to reflect and that may also produce useful information for the program.

Integral to programming: Making journal writing, reflection and/or information-gathering a natural and integral part of programming.

Principle #1: Comfortable. Questions that a reasonable person can comfortably answer more than one way, without feeling that they ought to give a particular answer.

Discussion: It is well known that surveys/reflections should avoid explicitly leading questions (such as “Don’t you think that ____ is getting too old to be an effective mayor?”) but it is also possible that some questions have an “ought-to” tone even if they are phrased neutrally.

For example, questions about healthy habits or family activities can take on an “ought-to” tone because other media messages have said that families “ought to” eat dinner together regularly and that children should eat a good breakfast daily. Even asking a series of questions in a certain way can suggest that the survey-writer feels that you ought to do these things. Ideally, survey-takers should be able to choose from a variety of options on checklists, or use rating scales that allow a range of answers, so they aren’t forced to say “no” explicitly to something they know ought to be a yes.

Principle #2: Realistic: Questions that are likely to provide a variety of responses.

Discussion: Questions should realistically reflect the likely interests and experiences of the population. For example: a career interest survey asks a series of realistic questions like “do you enjoy performing music” or “do you enjoy working with children” but also asks questions like “do you enjoy filling out income tax forms.” This question is probably not realistic for a younger population because young people probably haven’t filled out income tax forms yet. (And for an older population, income tax forms might not be something people realistically think of as “enjoyable” even if they enjoy working with information and filling out forms.

Generally, questions should focus on activities that children/teens might have experience with, including some that may be a “stretch,” but not on activities that are unlikely to be relevant.

Principle #3: Respectful. Respecting individual styles and cultures.

Discussion: It is helpful to have several people look over proposed questions to look for potential cultural bias. For example, different cultures may use different wording when talking about volunteering, service learning, community service, or helping out in the community. Therefore, questions about community involvement have to be worded in a way that makes sense across cultures. Or, for example, different cultures (and individual families) may emphasize different aspects of family life, so questions about family life and parent involvement should recognize and respect a variety of approaches.

One important part of respectful question design is to avoid conveying biased or negative images of participants’ lives, families and neighborhoods. For examples, surveys with a heavy focus on risk behaviors suggest to youth that the survey-writers expect these behaviors. Or, for example, surveys that ask about possible deficits in the participants’ lives suggest that the survey-writers expect these deficits. If it is important to survey youth about these issues, it can be helpful to balance the survey by asking about positive behaviors and assets as well, and perhaps to use rating scales, checklists with multiple options or other survey methods to allow a range of positive or mid-range responses.

It is important to remember that for youth especially (but also for individuals of all ages) taking a survey or responding to a reflection question is a learning experience. Youth form impressions about the world, about themselves, and about your program based on the collection of questions posed to them.

Principle #4: Reasonable Length. In-depth enough to be meaningful but not overly time-consuming.

Discussion: It’s important to seek balance between “ease of collection” and “quality of information” in determining the length/complexity of questions. As a corollary, if answering a question provides people with a positive opportunity to reflect on experiences or interests, or provides an opportunity for staff to learn about something important, then it is often worthwhile to “invest” more time and use questions that are more in-depth.

Principle #5: Focused: Questions focused on topics relevant to program goals and activities.

Discussion: Questions should be focused on the program goals and activities.  This can include questions about relevant background and interests, reflections about future goals, and questions about current program activities.  This can also include questions that extend beyond the current program activities to explore applications of the concepts and skills that participants are learning.

Principle #6: Useful: Questions that will produce useful information that can be analyzed and used for reporting and/or improving programming.

Discussion: Whatever the format of the question, including open-ended questions, yes/no, multiple choice, rating scale, etc., it is valuable to gather information that can be summarized, analyzed and shared with staff and others for improving programming, describing activities and outcomes, evaluating the program, or seeking support for future programming.

Principle #7: Integral to programming: Making journal writing, reflection and information-gathering a natural and integral part of programming.

Discussion: Wherever possible, programs should use reflections, goal setting activities, career interest checklists, or other materials that are integral to the programming. Staff and participants should get accustomed to a few different types of information-gathering activities such as pre-program, mid-program and post-program questionnaires or tests, reflection questionnaires after an activity, or periodic goal-setting questionnaires, interest questionnaires or college/career planning checklists. Whatever the formats, data collection should be seen as an integral part of programming – and may be seen as a valuable opportunity to reflect and learn, or at least a reasonably comfortable and pleasant part of the program.

Students need a time and place to reflect and write.

Next week I am presenting a workshop on the new “Reflection Screen” in the Online Work-Based Learning Database at the CWC conference.  Excited to launch a dialogue about this tool and this topic, I am reflecting on the message I hope to share.

What are the necessary elements to help student reflect on learning experiences?  Reflection has many benefits, helping students to consolidate what they have learned and to expand on ideas and insights (see Reflection – Part 1).  Programs should consider tools, techniques and time:

Tools -
Journals |  Workbooks |  Reflection sheets | Portfolios |  Student learning plans (college and career plans) |
Online journaling (including reflection screens in the Online Work-Based Learning Database)

Techniques -
Effective prompting questions |   Supportive environment for reflection and dialogue | Open-minded approaches | Clear motivation (such as emphasizing the personal benefits of reflecting on experiences, value of reflection as preparation for future resumes and interviews, reflection as part of a grade, reflection essays to be shared as feedback to project coordinators, etc.)

Time -
Regular schedule for reflection and writing |  Time for teachers/staff to explain reflection activities | Time and place for students to complete reflection activities | Access to tools (i.e., journals, paper, online access)

Effective reflection is a gift to the student or participant — people are generally grateful for the opportunity to reflect on what they are doing if the reflection is affirming, open-minded and feels natural (not forced).  I continue to explore tools that facilitate reflection — online journaling, reflection sheets, surveys, portfolios — and techniques that make reflection effective. Incorporating reflection into work experiences, contextual learning projects and other learning experiences is an art worth perfecting!

This article continues the series of articles on 21st century skills, including Environmental Literacy, Media Literacy, Health Literacy and Economic Literacy and other “21st Century Literacies.”

Civic awareness refers to the foundation that makes people active participants in the fabric of their community and beyond. What makes a person decide to participate in elections, participate in community planning and decision-making, invest in the community,  volunteer in the community and feel connected to the community?

Youth development professionals agree that one of the essential positive developmental steps for youth is establishing strong connections with their communities and the wider world. Youth connect with their communities in many roles – student, neighborhood resident, sports participant, arts participant, volunteer, worker/entrepreneur, voter or soon-to-be voter, and perhaps as participant in town meetings, neighborhood groups and other community forums.

A combination of formal and informal learning supports this growth.  Formal learning supplies the concepts that underlie civic participation: understanding of democracy, voting and elections, the structure of local, state and federal government, the US Constitution, the history of democracy and civic participation.

Informal learning supplies the experiences that reinforce a desire to connect with the community and contribute to the community. Early experiences like elementary school field trips to local businesses, organizations, parks and historic sites show students how others invest in the community and contribute to the fabric of the community.   Participation in youth programs in the community — sports, arts, technology, recreation — communicate a sense of a caring community to children as they are growing up.  Community experiences that children have with their families — shopping in neighborhood stores, going to local parks and playgrounds and attending local events — shape a sense of community.  Community connections with police through day-to-day interaction, special events, and youth/police sports and other events shape a sense of community.  Career speakers and career-related fields trips and job shadows help youth begin to see themselves in various professional roles in the community and beyond.

Many contextual learning projects and youth leadership projects explore civic connections through a variety of hands-on projects in the community…. organizing food and clothing drives, community clean-up days, history walks, museum exhibits, veteran appreciation events and many other interesting projects. As I explore the idea of “how to create good reflection questions” to reinforce hands-on experiences with journal writing and reflective writing, I am considering the connections between formal and informal learning and the many experiences that contribute to a growing sense of community and civic connection.

I am presenting the new screen and reflective writing ideas in webinars on April 10 and 18 and in a workshop at the CWC Conference (http://cwc.cedac.org/2012Conference.html) on May 14th.  These opportunities are designed is to open up dialogue about what makes a good reflection question and what types of questions help writing to flow easily for students and others. The focus of the conversation is on reflective writing for youth employment program participants, but the concepts can apply to writing and reflecting about any type of experience or topic.

Ideally, the purpose of the reflective writing is to help program participants to consolidate what they have learned and to expand on their ideas and insights.

Reflective writing is valuable for consolidating your knowledge – reviewing and summarizing what you have learned about a job, about career options, and about skills you have obtained.  This type of reflection is valuable as a foundation for taking next steps in your career development: for setting goals, looking at postsecondary education options, writing college applications, writing resumes, answering job interview questions, and other next steps.

Reflective writing is also valuable for  expanding your ideas and insights –using your experience as a starting point for exploring ideas, thinking about trends, and working out your own opinions on various subjects.  The topics for these reflections may be very specific or very broad, including reflections on specific skills used during the work experience — such as leadership skills or listening skills or motivation – or reflections on larger trends in technology, job markets and other areas.

What makes a good reflection question that helps writing to flow easily and that helps participants to consolidate knowledge and expand ideas and insights?  Questions might include the following:

Summarize – What did you do?
Describe – Can you give an example of an experience, accomplishment, etc.?
List – Can you list 5-10 ideas, tips, questions….?
Predict – What do you think will happen?
Analyze – what are the specific skills involved in this job?
Evaluate – Evaluate, compare and contrast different points of view
Imagine – What skills/accomplishments would you like to attain?
Set Goals– What will your next steps be?

Here are some examples:

1.) Write a list….  One technique for getting writing to flow is to write a list.  For example, “list ten tips” or “list five questions.”

• Suppose you were creating a poster about “do’s and don’ts” for your workplace with safety rules, confidentiality rules, workplace ethics and other things.  List 5-10 things that should be included.
• List five questions you would ask if you were a manager interviewing someone to work at this organization. Why did you choose these questions?
• List ten tips for “interacting with others” in a workplace. (This can focus on any of the following — working with co-workers, customers, clients, patients in a hospital, children in a summer program, pets in a veterinary hospital, etc.)

2.) Express your opinion… Another approach for getting writing to flow is to answer questions that ask for your opinions, predictions or analysis; especially if the questions don’t imply that you “ought to” think a certain way!!

• For many people, gardening is a love-it-or-hate-it activity.  Some people love to garden; others just don’t like it at all.  With the rising popularity of gardening and healthy foods, do you think more “non-gardening” people will become gardeners? Why or why not?
• What new technology, tools and equipment do you think this workplace could have in one year, five years, or twenty years?

3.) Imagine a scenario…  Another technique for getting writing to flow is to imagine a particular scenario – such as a job interview, dialogue, writing project or presentation.

• Think of a controversial issue related to this field. Write a brief dialogue between two people who hold opposing viewpoints on this issue.
• Suppose that you are interviewing for a job. The interviewer says “We need people who are very reliable.” What can you say about your attendance and work style to tell the interviewer that you are a reliable worker?
• Suppose you were writing a guide to troubleshooting issues, problems or challenges that commonly come up in this work. What would be the three most important issues that you would write about?

4.) Empower yourself… Empowering questions are questions that naturally guide you to think about possibilities.  Questions often begin with “imagine” or “picture” or “what else.”

• Picture yourself in ten years.  What is your leadership style like now?
• Describe a problem that you have faced at work; one that you have not been able to solve yet, despite trying many options.  Briefly brainstorm – WHAT ELSE could you do to approach this problem?
• Imagine that you could have any job in this organization. What job would you choose? Why did you choose this job?
• [For insights on the idea of "empowering questions" see -- http://www.leadershipthatworks.com -- a source of inspiration for this category.]

JOIN THE CONVERSATION — What are some reflection questions and writing prompts that have worked for you? What questions sometimes fall flat?  Comment below or email with ideas….

Attending the CWC Conference on May 14?  Sign up for the workshop on “Incorporating Reflection into the Work-Based Learning Plan” to continue this conversation.

http://resources21.org/firstcareersteps/survey1.htm

Results showed that students are well-engaged in career development, expressing confidence and feeling ready (as well as some worry and not-so-readiness) and expressing interest in a variety of career areas and options for post-high-school first steps. Most have participated in several career development activities, including talking with parents, friends, teachers and counselors, career interest inventories, work experience, including summer jobs, after-school jobs, internships or volunteer work and a variety of career-related events and activities. Some highlights:

• Students described many positive things that their school or program does to support career development, and also provided suggestions for additional efforts.
• Survey results provide testimony to the value of youth employment experiences, with students who have had jobs, internships and volunteer work indicating that they have developed career skills through these experiences.
• Survey results suggest that students use the high school years to explore broadly and consider a variety of career and postsecondary education and training options.
• The survey results provide “food for thought” for exploring youth development issues and for establishing a vision of career readiness that embraces the confidence and concerns that youth bring to this important part of their development.

A preview: the first question asks “How do you feel when people ask you about your career plans? Do you feel confident, worried, ready, not-so-ready?”

This question served primarily as a “warm-up” question for the survey and to get a general sense about how students feel about career development. Answers were mostly positive – confident (but also worried), excited, ready. Answers varied somewhat by grade, with 9th grade students more confident than 10th grade students, and with confidence, worry and a combination of both rising and falling from grade 9 to 12.

Answers vary by grade, with 9th grade students more confident than 10th grade students, and with confidence, worry and a combination of both rising and falling from grade 9 to 12.

The most creative aspect of the work of a coordinator of a youth program is the role of bringing together a variety of people – youth, employers, school personnel, parents, and community organizations.

Success stories may be very specific — the parent of a student in your program volunteers to be a career speaker; a career speaker decides to offer a summer job or internship; a student intern makes a visible and positive contribution to a local community organization.  Success stories may be community-wide — the community mobilizes for a fundraiser or a summer jobs campaign or to open a new youth career center; a school organizes a school-wide strategy to raise gradaution rates; an employer group makes a commitment to mentoring students in your community.  

 In successful programs, participating youth are energized by their experiences.  Youth develop new skills, gain valuable experiences, and see new connections between the work they are doing in school and their future careers.  Through school, workplace and community activities, they start to envision future career and educational paths and have confidence in their first steps toward career development.

How does your work create good foundations and connections to allow these successes to happen? What are the things that energize you in your work? Share your thoughts or read more…

Economic literacy comes from an understanding of how markets work, how economic trends shape job opportunities, how businesses succeed and grow, how goods and services are produced and exchanged, and how money flows through the economy. 

“Economic literacy” refers to general knowledge about economic markets from the perspective of  workers, consumers, savers, investors, entrepreneurs, community volunteers and active citizens.  Students build economic literacy through experiences that expose them to the type of analytical thinking used in economic analysis and to the general knowledge  that underlies economic literacy.   Experiences can include:

• Looking at history through the lens of economic and social change;
• Studying geography, government and civic awareness;
• Learning concepts and vocabulary that underlie economic literacy, including: competition, markets, income, interest rates and entrepreneurship.
• Learning about the different sectors and institutions that interplay in the economy — businesses, consumers, government, financial institutions, labor unions and nonprofit organizations;
• Learning about how the role of government in the economy has evolved over time, as well as understanding different perspectives about the role of government in the economy;  
• Having a variety of experiences in finding, analyzing, presenting and talking about data;
• Drawing flowcharts or other models to learn how to apply analytical, “cause-and-effect” thinking to economic issues;
• Participating in in-depth classes, workshops and events about topics of immediate interest – particularly jobs and careers,  entrepreneurship and personal financial literacy.

Career development experiences that build a deeper understanding of economic literacy and job markets include visiting and touring local companies; hearing guest speakers about various industries; reading about local economic issues from different points of view; and gathering and analyzing data.  The Contextual Learning Portal provides examples of career exploration projects that draw on job market data — see http://resources21.org/cl/projects.asp?tags=careers

Examples of youth employment experiences that build economic literacy skills include:

• Working with a nonprofit organization that helps local growers to achieve and sustain economic success through marketing and technical assistance.
• Working with a local museum on projects related to local economic history.
• Writing business plans and starting small businesses through entrepreneurship programs.
• As a marketing intern, learning marketing skills such as identifying the target market and creating a marketing profile.
• As an intern in a restaurant, working with the restaurant manager to prepare marketing materials and to survey customers about food and dining experience.
• As a small business intern, learning to understand all aspects of a small business, including assisting customers, placing orders, handling all transactions and sales, taking inventory, pricing, and learning the financial cycle of a retail store.

Additional thoughts:  The 21st century skills and “literacies” in this series – environmental literacy, media literacy, health literacy and economic literacy – all draw on the use of analytical and critical thinking skills.  This type of higher-level thinking develops gradually, supported by a strong foundation of knowledge and experiences.   Ideally, a well-balanced blend of formal and informal learning, in classroom, community and workplace settings, builds the foundation for effective analytical and critical thinking and 21st century “literacies.”  Teachers  and program leaders can role model analytical thinking by demonstrating systematic ways of studying and analyzing topics and by showing students where they can find more data or read alternate points of view on a topic.

If I were asked to describe my hopes for career readiness work in Massachusetts, I would describe a continuum of academic + classroom, community and workplace experiences that build interests, passions, knowledge and skills that youth need to do well in first steps after high school and in long-term career management.

 How do students in Massachusetts attain the knowledge and skills needed for career readiness? 

1.) Strong academic foundation — having a strong foundation in core subject areas — with skills including literacy and communication skills, critical thinking, mathematical literacy, civic awareness, history and economic literacy, scientific literacy,  information skills, the arts, music and languages.

2.) Classroom, community and workplace experiences - with enrichment experiences starting in elementary school and continuing through all stages of education and including school-day and out-of-school-time experiences.  These experiences build students’ knowledge and skills and help them to develop potential interests and ignite passions.  These experiences build four areas of readiness:

• Applied academic skills – seeing how writing, math, research, information, critical thinking, creative thinking, scientific, design and technology skills are applied in classroom/community/career settings.  Having opportunities to “try-out” and demonstrate these skills.
• Essential career skills — understanding how basic foundation skills and higher order skills — professionalism, teamwork, goal setting, motivation, communication, project management, customer service, leadership,  entrepreneurial thinking — are used in classroom/community/career settings.  Having opportunities to build and demonstrate these skills.
• Career awareness and career management skills – understanding how to learn about career options, understanding how job markets evolve and change, knowing what types of careers people have, knowing how people prepare for and navigate various career paths.  Understanding how to set goals, navigate transitions, find mentors, seek out information and build a network of support.  Building personal resiliency and persistence.
• Interests and passions– having academic-subject-related and career-related interests and passions — as a starting point for further study, personal exploration and/or career development.  Opportunities to enjoy the arts, journalism, science, technology, engineering, design, environmental study, math, media and other interesting areas.  Opportunities to organize community events, participate in community service, work on leadership projects and participate in the arts.  Opportunities and encouragement to explore books and media on all types of subjects.   Opportunities to begin to explore in-depth and to develop skills and knowledge in areas of interest.

In Massachusetts we have lots of examples of how these elements currently happen in schools and communities.   Through Work-Based Learning Plans, Contextual Learning Portal entries, surveys and other forums, we can get a glimpse of what Massachusetts students, employers, community members, educators and workforce development professionals are doing to build career readiness experiences.

In today’s “mosaic economy” many businesses, organizations and career paths are focused on ways of promoting and preserving personal and community health.  From renewed attention on healthy eating to programs for exercise and relaxation to holistic approaches to healthcare and prevention, people are more interested then ever in ways of promoting health.

Communities and individuals are more interested then ever in ways to promote personal and community health.

“Health literacy” is an important skill for youth — not only because being health-savvy is important personally, but also because knowledge of health issues can be valuable for career success and career growth.   All kinds of organizations — grocery stores, fitness centers, youth sports programs, food banks, community gardening programs, schools and healthcare organizations — value employees who can guide customers and develop marketing and educational materials to promote health and wellness.

Youth Employment Experiences.  How do youth explore health literacy through their youth employment experiences?  Examples drawn from Work-Based Learning Plans (see http://skillspages.com/masswbl) of youth in jobs and internships in Massachusetts include a variety of approaches to working on health literacy issues:

• Understanding exercise and movement as applied in physical therapy clinics and fitness centers;
• Promoting good health through leadership in youth sports programs;
• Helping to plan and prepare nutritious meals in a culinary arts program, restaurant, youth program or nursing home;
• Working on public education and peer leadership campaigns in programs related to health and wellness;
• Working on a nutrition newsletter, articles and events for a supermarket chain (SEE SPOTLIGHT BELOW);
• Working with the school nurse to promote health through a school-wide newsletter and school web page;
• Maintaining a file of health information resources for the school nurse’s office;
• Working in medical research projects;
• Conducting surveys related to health and wellness issues.
• Working in internships in healthcare, pharmacy, dental care and other settings.

Classroom and Community Experiences.  Students explore health literacy through a variety of classroom and community projects. as described in the Contextual Learning Portal (http://resources21.org/cl).  For example:

• Elementary school students organized an Elementary School Health Fair  at Union 61, a school serving Brimfield, Brookfield, Holland, Sturbridge and Wales.  As part of this project, students selected topics, invited presenters, created exhibits, wrote press releases and developed publicity, managed the event and evaluated the success of the event.
• Students from Whitman-Hanson Regional School conducted a project about hunger in their community.  Students needed to do initial research on healthy eating recommendations, WIC and food stamp information, poverty income guidelines, as well as the cost of food items based on local supermarket circulars. Students participated in a panel symposium in which they taught each other what they had learned through their in-class study, and later heard from experts on the topic of local hunger. They were able to ask questions of the panel and gain a greater understanding of hunger facts within their communities.  
• Students from several schools have organized anti-bullying projects, creating healthier environments for their felllow students.
• Students from other schools and after-school programs have participated in exercise, nutrition, cooking, community gardening and other health-related programs.

How can an organization gain visibility through effective use of media?  How can an organization create an effective public education campaign using media? What are the characteristics of a high-quality website, video, newsletter, e-newsletter, or other media communication?  How can I organize and synthesize information to be presented via various media?  What are the technical steps involved in creating websites, videos, booklets, brochures, e-newsletters, podcasts and other media?

Communication skills, information skills, critical thinking and creative, artistic and technical skills come together when students use electronic, print and other media in their classroom, community and workplace projects.   Through exposure to a variety of projects, youth gain skills as thoughtful and effective producers, consumers and planners of a variety of media.

Youth Jobs/Internships:  A sample of Work-Based Learning Plans shows many examples of youth working with media as part of jobs and internships. 

• Writing press releases;
• Writing for newsletters and e-newsletters;
• Researching topics for newsletters and e-newsletters;
• Working on scripts for television, radio and in-store voice-over promotions for a supermarket;
• Posting news and announcements to local cable tv;
• Posting updates to social media;
• Setting up daily podcasts for school announcements;
• Creating or updating a website for an organization or project;
• Producing a promotional video;
• Researching media coverage for a media book for an organization;
• Participating in the development of social media strategies.

Classroom and Community Projects:  A look at projects in the Contextual Learning Portal (at http://resources21.org/cl) shows a variety of uses of media:

• Elementary school students research and produce a booklet about emergency preparedness, covering how to take care of people and pets during power loss;
• High school students research a local proposal to restore flood chutes on the Hoosic River and produce a documentary about the issue;
• After a technology class about cyberbullying, students conducted an online survey about bullying and used Excel and PowerPoint to present the results to the school community;
• Students research and analyze media coverage of a presidential election;
• Grade 8 students produce a video about the scientific process based on a project done by grade 6 and 7 students;
• Students create a video as part of an anti-bullying project;
• Elementary school students write, edit and print a school newspaper;
• Students use media and primary and secondary documents to research the life of a local artist, while learning about general local history and arts; students work with local artists to produce works for an art show.