To continue strengthening the CSforALL movement, it’s time to ask ourselves how we’re measuring impact
At Siegel Family Endowment, we’re invested in new ways about thinking about impact. For too long, philanthropic organizations have asked grantees to submit “impact” reports that ask nonprofits to list the number of program participants, or to calculate the percentage of participants who are from different targeted groups. Such stats are significant, but they do not get at the true impact that the program has.
What change did participants see in their lives? What relationships did they build? How are their experiences in a program affecting their larger communities? These are some of the questions that we’ve prompted our grantees to consider as they chart a new way forward for considering impact.
This post is the first in a series about how Siegel Family Endowment grantees have responded to that challenge. CSforALL, a nonprofit with the mission of making high-quality computer science education an integral part of K-12 education across the country, is taking that work to a new level. In this post, we explain how CSforALL is thoughtfully and powerfully offering a new vision for “impact”—how it is defined, assessed, measured, and explained.
***
Leigh Ann DeLyser, the executive director and co-founder of the nonprofit organization CSforALL, is tired of trying to fit impact into predetermined boxes on a form.
To be clear, it’s not that DeLyser objects to measuring impact. Far from it. It’s just that the standard paperwork is using the term “impact” incorrectly. DeLyser says, “In the nonprofit space, we’re often told, ‘Here’s this money. What or who are you going to count in the room? We call that ‘impact.’ But actually, that is only merely the output of our program – not the impact.”
Distinguishing Between Impact and Output
What does this distinction between impact and output look like in practice?
Let’s take an example. An educational nonprofit might run a professional development workshop for teachers. The output for that session might include the number of teachers served; the proportion of teachers from Title I schools; an assessment of what the teachers learned during the session; a survey of how the teachers felt about the session; and the number of students that are collectively reached by the teachers who attended the session.
Meanwhile, the session’s impact would include an examination of the actions that teachers took as a result of the session and the sustainability of those changes. It would take into account not only what students in participating teachers’ classes learned, but also if and how that learning reverberated through the communities of which those students were a part.
DeLyser explains, “When we talk about impact at CSforALL, we’re really trying to think about the permanent change we’ve made in the system.”
DeLyser argues that outputs are still important, but as a measure of what is currently occurring and as a means for marking progress toward long-term goals. Rethinking output in these terms allows an opportunity to better deliver on mission and to understand whether and in what ways a model is making a lasting change.
Truly distinguishing between the two would represent a seismic shift in the nonprofit and philanthropic sectors. And it’s one that isn’t easy.
What Should We Measure?
Different systems have different reasons for pursuing programs. For example, CSforALL has seen that schools and districts might choose to implement a CS education initiative as a workforce development opportunity. In other cases, CS education programs are designed to move the needle on civic engagement.
But DeLyser stresses that there’s also a common thread to CS education programs: providing high-quality, equitable, and inclusive computer science education. DeLyser ticks off a number of easy-to-measure questions that align to progress to this goal: Is computer science being offered in the school or district? Are students taking the classes? Are they learning things in those classes? Are there gaps for particular slices of the student population in participation, access, or experience?
That’s typically where measurement stops. That’s a problem, says DeLyser. “Too often we measure our progress [as an organization], not theirs [as students],” DeLyser says, quoting Diane Levitt, a colleague in the K-12 CS education space. By that, DeLyser means that nonprofit organizations typically measure their success on a narrow set of programmatic outputs, rather than student success in the myriad places and spaces where students learn.
To get closer to that vision, DeLyser urges supplementing the typical “butts-in-seats” measurements with additional questions that get beyond output, to real impact: What are the root causes of documented gaps? How can we investigate those root causes? Are we able to address those root causes with locally-driven approaches and solutions?
The Value of Developing Holistic Solutions
When we begin to ask those questions, we gain a much more sophisticated—and accurate—vision of the holistic impact that we are having or not having in spaces that are far beyond the classroom walls.
As an example, DeLyser points to a partnership in Miami between the public schools and a local nonprofit that is designed to address inequities in CS education. Rather than simply counting the number of students from historically excluded communities who are taking computer science courses and recruiting students from those groups to fill gaps, the program is designed around a basic question that seeks to identify and address root causes of the inequity: why are students from Spanish-speaking families not participating in opportunities for advanced CS learning?
Leaders in Miami found that many Spanish-speaking families did not know about or encourage their children to take CS courses in high school. DeLyser explains that this is a problem because research has shown that “parents are a really important part of encouraging kids to take courses and to see themselves in particular careers.”
As a result, CSforALL’s partners in Miami decided to offer sessions for family members and students to speak with Spanish-speaking software engineers. The program gave parents an understanding of what computer science-related jobs entailed and allowed students to see themselves in those roles.
As DeLyser explains it, these career panels for families had a “direct relationship to what the Miami team had seen as a challenge on the ground—not just the immediate challenge of under-enrollment, but a deeper understanding of the root cause of that systemic challenge.”
As a result, DeLyser expects enrollment in CS courses among students from Spanish-speaking families will increase, and that the program will have a number of “downstream effects” as well. “That’s going to trickle through into long term impacts in that community as those parents go and talk to their neighbors,” DeLyser says.
DeLyser’s vision of leveraging community assets and considering the ways in which learning permeates through entire ecosystems reflects Siegel Family Endowment’s “Schools as Community Infrastructure” work. “This work centers on the involvement of families and communities in students’ growth and trajectory,” says DeLyser. “The school is the hub, and there are opportunities to support opportunities for students in the community in a multi-year, multi-faceted system.”
How Should We Measure?
The challenge is twofold: how to measure these broad impacts, and how those data impact programs. DeLyser says, “At CSforALL, we look at data not only to collect evidence of impact and to see progress on the ground, but also, as an intervention.” By that, DeLyser means that we choose to measure things that we believe are important and that we rearrange our systems to respond to those areas of importance.
DeLyser cites the example from her past work as a teacher at a highly ranked public high school. The school’s national ranking was determined solely by students’ performance on Advanced Placement (AP) tests. As a result, the school decided to have its Spanish-speaking students take the Spanish AP exam, even though they were not enrolled in an AP Spanish class. This improved the school’s overall AP performance, but was not a good measure of student learning or equity in the system.
With funding from the National Science Foundation’s Broadening Participation program, and drawing on the National Equity Project’s model of equity work in schools, CSforALL is developing a dashboard that will help schools and districts align data points to different goals, and to help them capture and interpret those data. DeLyser sees huge potential in this work. She notes that there are already dashboards that track the number and demographics of students that are participating in CS learning opportunities.
What is missing is a way for educators to think about how the landscape of equitable opportunity for students of different backgrounds compare from locale to locale. Those opportunities exist both within the classroom walls, but also outside of it—through Girl Scouts or the Boys and Girls Club or through local companies or many other places.
DeLyser believes that this dashboard of holistic measurements is the first step toward viewing success and equity through the eyes of students, not just through the eyes of individual institutions.
