Measuring Youth Development: How Out-of-School Time Programs Collect and Use Data

Out-of-school-time (OST) programs and their funders rely on sound data to make decisions about everything from professional development and student recruitment to the selection of activities to offer students. Programs operate at a range of times (before and after school, weekends, summer) and in a variety of locations (e.g., schools, community-based organizations, city parks and recreation centers), are run by a variety of entities (e.g., government agencies, private community organizations), and receive funding from a variety of sources (e.g., government, philanthropy)—each of which may be interested in a different set of data and come with its own reporting requirements. This means there is a great deal of variation in the types of data programs collect.

In 2019, The Wallace Foundation (Wallace) commissioned Child Trends to conduct a study of the kinds of youth outcomes OST programs are interested in measuring, the tools they use to measure those outcomes, and the challenges they experience in doing so. The study included a literature scan and interviews with leaders and staff members at 28 OST programs. Twelve of the 28 also completed surveys; a separate group of 10 provided information by survey only. The study expands on past research by a) focusing on programs that work in specific content areas (e.g., the arts, civic engagement and social justice, career and workforce development) and b) covering both quantitative approaches (i.e., tracking numerical data) and qualitative approaches (gathering descriptive information through surveys, interviews, etc.) to data collection.

While its findings apply to OST programs in general, the study focused on particular types of programs (i.e., afterschool, summer, online) and particular content areas, as well as programs that serve school-age children and adolescents from marginalized communities, those that support students’ social and emotional learning (SEL), those that serve systems-involved youth, and those that focus on promoting equity—for example, by training staff to recognize and overcome personal biases or by recruiting and retaining leaders and staff who reflect the diversity of the participants served.

Key findings

What outcomes do programs measure?

• Programs in the study measured outcomes that were closely related to the content they delivered (e.g., the arts, career and workforce readiness, civic engagement and social justice). They also generally measured SEL outcomes and other outcomes required by funders.
• Programs thought critically about how to measure progress toward achieving equitable outcomes for the youth they serve. Approaches included disaggregating data by race and other demographic variables and tracking the development of equity-related knowledge and skills.
• Programs used a number of criteria to decide what outcomes to track, including whether a given outcome was consistent with the program’s logic model or theory of change, usability and shareability of the data, the effort and capacity required to collect the data, availability of valid and reliable measures, youth interest, and whether measuring a given outcome would promote equity.
•  In addition to outcomes, programs consistently measured outputs (i.e., steps that lead to desired outcomes), particularly program participation and quality. This finding aligns with our literature scan that revealed that public and private funders have invested heavily in quality assessment tools and participation tracking systems. Comparatively few programs used qualitative methods to understand the factors that contribute to program participation and quality.

What methods do programs use to measure progress?

• Programs reported using a variety of traditional methods to measure outcomes, including quantitative methods—like administering surveys and questionnaires, conducting formal assessments, and tracking attendance rates—and qualitative methods such as conducting interviews and focus groups with young people. Their use of quantitative methods was well-documented, but their use of qualitative methods less so.
• Programs also used nontraditional methods, such as regular check-ins with participants and reviewing participant journals and portfolios, along with creative, informal ways to track SEL outcomes, including games and award systems. As with their use of qualitative measures, use of these nontraditional methods was not well-documented.

What measurement challenges do programs face?

• The programs we interviewed did not consistently identify specific outcomes of interest that they were unable to measure, although some said they lacked the tools they needed to track equity- or SEL-related outcomes or program quality.
• Programs pointed to broader types of outcomes that were challenging to measure, including longer-term outcomes like college matriculation, career attainment, and participation in civic life; behavior change (e.g., whether participants in a civics program exercise their right to vote); school outcomes like test scores and grades that require a data-sharing agreement with the district; and the relationship between improvements in staff members skills and knowledge and youth outcomes.
• Programs reported that the process of collecting data could be burdensome for both participants, who have their fill of testing at school, and staff, who in many cases do not have the training or time to do the work. Some pointed specifically to the burden of fulfilling the reporting requirements of various funders, which often involved recording duplicative information using multiple tools and databases.