Beginning with The End in Mind What We Know about Performance Measurement for Vmf Programs

Start the webinar for you. Welcome to our webinar--Beginning With the End in Mind: What We Know about Performance Measurements for Veterans and Military Family Programs. It is our hope that, after this session, you will have learned about performance measurements for veterans and military family programs, discovered methods to measure outputs and outcomes with available tools, and also learned the difference between outputs and outcomes, learn about veterans and military family programming data sources to help measure outputs and outcomes, and then explore--along with Equal Justice Works--the best practice examples in measuring outcomes. Koby Langley, from the Corporation for National Community Service, is here to tell you how this session fits within the What We Know series that started earlier this month and to give you the official welcome from the Corporation. Koby? Thanks, Debbie. I really appreciate everybody taking the time to participate in this webinar. This is a very important series. We've now had over a dozen of these webinars over the course of the past year. We're in the middle of our series of webinars around performance measurement and outcomes. Very important webinar to talk to our stakeholders about principally because--as folks have been engaged with the national service community over the past few years have probably noticed--there is an increasingly important focus on performance measures and outcomes in the veteran and military family space. As some of you are aware, in 2009, the Corporation was directed by the Kennedy Serve America Act to expand services for the veteran and military family community through our programs and initiatives as well as to engage more veterans and military family members in national service. In addition to that, the Corporation decided to incorporate very specific objectives in the veteran and military family space in our strategic plan, more specifically that we would increase the number of veterans and military family members that we engage in national service as well as increase the numbers of veterans and military family members that were national service participants. We're now into the full second year of that strategic plan, and, as a part of expanding our own commitment to performance measures last year, the performance management team here internally looked at the way that the Corporation has been measuring success in this field and decided to realign a lot of the performance measures around the specific activities that are defined in the Kennedy Serve America Act. That's not to say that the performance measures, the broad and overarching performance measures, are no longer valid-- they are. We're still counting and encouraging our programs to identify the numbers of veterans and military family members that are in their programs as well as the number of activities serving the veteran and military family community. But we're being a little bit more specific. It's important to note that in the most recent Notice of Funding Opportunity, as well as the Notice of Funding Opportunities that the Corporation will be fielding in the future, you'll see more around the issue areas that are defined in the Kennedy Serve America Act. I'm briefly going to talk about those. But we've got Chris Spera on the line who is our Director of Evaluation and Strategy. He'll be able to go into a little bit more detail about what that means for potential grantees, for current grantees, and for the national service community and veteran and military family community at large that are interested in leveraging national service in order to both engage and expand services to our veteran and military family community. The legislation directed the Corporation to work with grantees providing national service opportunities in nine issue areas. I'll just run through those really quickly. The first one is to provide services and support to military families. Thank you. The second one is to increase volunteer opportunities for veterans and military families. The third one is to improve veteran and military families' education and certification opportunities. The fourth one is to improve veterans' and military families' employment prospects. Access to benefits. Military children. To improve transportation options, particularly for disabled veterans. To facilitate veterans' participation in disaster preparedness programs. And the last one is to improve wellness opportunities for veterans and military families. If you take a step back from those very specific activities you will see the three buckets that the Chairman of the Joint Chiefs of Staff have identified as key issue areas that communities can engage in in order to improve the lives of veterans and military families. These are published both in the QDR, Quadrennial Review of Military Compensation, as well as the white paper that was published by the Chairman of the Joint Chiefs of Staff called the Sea of Goodwill, as well as their joint military family strategy. All three of those documents focus on the three big buckets: improving employment opportunities for veterans and military families, improving educational opportunities for veterans and military family members, and improving health and/or the wellness of veterans and military family members. Those are the three buckets. What I've done in this graphic is put each one of those very specific Serve America Act activities into one of those buckets in order to visually show alignment with what our veteran and military family leaders have identified as social service need areas that can be met through community-based solutions. And, of course, national service is all about community-based solutions. On the side you'll see the kinds of activities that the actual grantees are encouraged to focus on. Those are the ones that we talked about a little bit earlier-- creating service and volunteer opportunities for veterans and military families, service directly to veterans and military families, and community coordination as a tactic or a strategy to implement any one of those nine specific strategies. So this is kind of an alignment slide. I hope it helps provide the context of this particular discussion and the data that you're going to be hearing about from both Chris Spera as well as Kerry O'Brien from Equal Justice Works. The data are really informed by two of our internal studies. One looked specifically at the emerging practices of grantees over the course of years in all of our program areas-- in Senior Corps and AmeriCorps State and National. One looked very specifically at those grantees that did well in our grant competition as well as an analysis of 100 of our current grantees in Senior Corps, AmeriCorps State and National, AmeriCorps VISTA and helped identify what I would call some of the emerging practices in outcomes and outputs. The folks that worked very diligently on improving these performance measures here at the Corporation headquarters really listened to the field and really tried to identify and align these activities in a way that would make sense for individuals that wanted to be competitive in our grant-making process. It's imperative that we share some of these ideas with you all in the hopes that we will be able to continue to improve the services that are provided through national service. I would like to say that a great number of opportunities still exist within the Corporation-- that the Corporation has maintained the veteran and military family priority focus area. It has also maintained its competitive preference for veterans and military family activities in their strategic focus areas as well as in their competitions to the extent that there is room to grow and the CEO is committed to growing our investment in the space. I would say the folks on the call should definitely take an opportunity to read her most recent testimony before the Senate Veterans Affairs Committee. I think folks will come away with a full understanding that the Corporation is fully behind the veteran and military family community. We look forward to continuing engagement, and appreciate your opportunity to participate in these sessions. So with that we're going to go ahead and introduce our two panelists, our two subject matter experts-- or our three subject matter experts--that will talk about these new performance measure areas in the veteran and military family space. The first one is Dr. Chris Spera. As I mentioned, he's the Director of Research and Evaluation here. He's a recent addition to the team, and has been really pushing the Corporation and our grantees to look at outputs and outcomes. He'll hopefully share some ideas about how our grantees--current, former, and potentially future--could improve their program design around these areas. We also have Kathryn Gravely and Kerry O'Brien who are working with Equal Justice Works. They're going to share with you some of their own ideas about how they were able to align their own activities around collecting data that would help inform some of the outcomes and outputs from their own program design. The more that we can do in this space--the better. The opportunity to expand our own investment in the national service community at large, I think, really depends upon our ability to prove impact. So we look forward to these continuing discussions and answering any questions you might have. So with that we're going to turn to Dr. Chris Spera who will talk a little bit more about program evaluation and the performance measures specifically. Chris. Great, thank you, Koby. Again, my name is Chris Spera, and I'm the Director of Research and Evaluation at CNCS. It's a real pleasure to be speaking with you today about this important topic of performance measures within the domain of veterans, military members and their families. As Koby mentioned, national service programs focus in three large topic areas of employment, health and education. Just to start off the conversation here, I wanted to give you a few slides that depict the current landscape of national service programs that involve military members, veterans, and families. As you know, and as Koby mentioned, national service provides opportunities to both serve the military and veteran population as well as enables military members and veterans to be volunteers in service programs. As you can see on this slide, about 58 programs provide opportunities to veterans and military members to serve. Another 21 current grantee programs serve veterans, military members and families in their local communities. And another 17 do community coordination. This slide also gives you a sense of how that distributes across our three core programs of Senior Corps (including both RSVP and the Foster Grandparent Program), AmeriCorps State and National, and VISTA. Next slide. Helping veterans transition into the workforce is a key component of national service programs with 43 programs focusing exclusively on this topic, and relatedly, 25 focusing on education and certification topics, such as the G.I. Bill. Furthermore, another 26 focus on expanding access and knowledge of VA benefits available to veterans. In my previous position--actually before I came to CNCS--as a contractor, I did a lot of research and evaluation for the Department of Veterans Affairs. One of the things that was real shocking to me when I started working with VA programs is the VA has really only contact with about 50 percent of veterans through access to their benefits. So there's about 26 million veterans in the country, and only about half of those have taken advantage of using VA benefits. So that just gives you a sense of how much need there is in really making sure veterans are aware what their benefits are and really leading them to water-- making sure they take advantage of those great sets of benefits they earned in providing service to our country through the military. In fact, the VBA (the Veterans Benefit Administration) is organized, around three benefit areas of education (including the new G.I. Bill, which is very central to what they do), vocational rehabilitation and employment (which is really the large rubric of helping veterans transition back into the workforce if they choose to do so), and loan guaranty (which allows veterans to use the VA to solicit a home loan). Also the Department of Labor, you may not know, has a Veterans Employment and Training Service that also provides programs and services to veterans, such as the Homeless Veterans Reintegration Program. As you know, wellness, including behavioral health, is a top area of concern for national service programs working in this area. And as you know, and as you've heard on the news likely, issues such as PTSD and traumatic brain injury have been central issues for our returning veterans. In fact, new research has shown increases, based on the DoD Health-Related Behavior Survey which was just fielded last year, on issues such as alcohol use and prescription drug misuse among this population. With that said, many veterans remain resilient and have successfully transitioned into the civilian life and their wellness is strong. I believe that national service can play an important role in augmenting and unifying federal resources provided by agencies such as DoD, VA, and DOL. Our reach is often at the local level in communities across the country, so I believe the national service presence can be felt. Next slide. I just wanted to mention this chart shows a little bit about how some of the wellness programs operate across national service. You can see there are programs focusing in areas such as housing, morale boosting activities, and behavioral support and you can see how those disperse across Senior Corps, AmeriCorps and VISTA. Next slide. So we're going to jump in to the main topic of conversation for today's webinar which is performance measurement. Koby mentioned how important performance measurement is for our veterans and military programs. So what we'll spend some time on, over the next few minutes, is talking a little bit about performance measures, trying to unpack the concepts, and hopefully leave you with a greater set of knowledge, skills and abilities than when you logged into this webinar. Performance measurement is the ongoing systematic process of tracking your program's outputs and outcomes. Outputs are the amount of service provided. They measure the completion of activities and document the fact that individuals received services, products were created, or programs were developed. Essentially they answer the questions of: How much service did we perform? What products did we develop? They do not answer the question: What changed as a result of service provided or product developed. Outcomes reflect the changes or benefits that occur. Outcomes can reflect changes in individuals, organizations, communities, or the environment. This may include changes in attitudes, knowledge, behavior, or condition. As an example, change within an organization may include an enhanced system to better serve military community members. Outcomes answer the questions: What difference did our service make for beneficiaries? How did the new system or product enhance the capacity of the organization to serve the community? In general, the most compelling outcomes address changes in behavior or a condition. However, what makes a change compelling is not just the type of change, but the magnitude--the amount of change-- and that's what performance measures is really all about. Part and parcel to performance measurement and program development is developing a solid logic model. I'll spend a little bit of time later in the presentation going over a logic model more in depth. Next slide. As you know, measuring the performance of our AmeriCorps programs and our Senior Corps programs is of critical importance to our mission and the broader field of national service. Strengthening the evidence base for national service is a priority outlined in the Agency's 2011 to 2015 strategic plan. As a result, CNCS has put rigorous program evaluation and performance measurement of the impacts of national service programs on members, beneficiaries, and communities as one of its highest priorities. So it's in the conversation that we have every day at CNCS about documenting good performance measures, improving them, providing robust technical assistance and training to the field, and ultimately using performance measures to tell the collective impact of national service. And you're such a big part of that as grantees and invested stakeholders in the community. Next slide. So I just want to spend a few minutes talking about why we measure performance and why it's so key. I think one reason is accountability. Performance measurement satisfies the need of funders and stakeholders to see that the program or product is getting results. Accountability is a key part of why we measure performance. Another reason to measure performance is to determine if the change you thought would happen with your intervention is actually occurring. It will tell you if you're making progress. Perhaps the third reason is that you could also use performance measurement data to spot and correct problems. As you know, well know, implementing programs and interventions on the ground to support military members and families is not easy. A lot of grantees are implementing sort of newer programs which don't have such a long history. So an important part of performance measurement is to allow you to do continuous quality improvement, look for things that work or are not working, and look for things that you can tweak along the way. So in short, performance measurement is a way to get feedback. That feedback is critical at the local level for your program, and it's critical for us at CNCS to be able to roll up to talk about collective impact of all of our programs. And that's really what performance measures are all about. Next slide. So a key to getting good performance measures data is having good resources. One of the things that we've been working on here at the Corporation over the last two years is developing good resources for our field of grantees to use. So if you haven't been on it before, I suggest you go--we have a website that has lots of resources around performance measures, including some slides, some tools for you including pre, mid-term and post-program surveys, different resources that you might find available. The website link is here on the slide. I'll provide it again towards the end of the briefing as well so you can make sure you have that at your fingertips. It's just a wealth of information, so I highly suggest you take advantage of it. Next slide. One thing I just wanted to make clear is there is a difference between performance measurement and evaluation. Performance measurement, as we talked about, is the process of systematically and regularly collecting and monitoring data related to the direction of observed changes in communities, participants, and beneficiaries. It is really intended to provide an indication of your program's operation and performance. In contrast to evaluation, it's not really able--it doesn't have the techniques or the ability--to establish a cause-and-effect relationship knowing that your program caused the outcome. That really can only be done with program evaluation. But performance measures can tell you an awful lot about how you're doing in terms of your operational performance and whether you're going in the right direction. So I think both are important parts of the story here, both performance measures and evaluation. Next slide. As I mentioned earlier, developing a logic model, I think, is a key part of really understanding your program, understanding what intervention you're implementing to serve any community. It's really in our lexicon here at CNCS. We're trying to promote and continue to use that language with all of our grantees in both AmeriCorps and Senior Corps. So this slide provides a nice graphic of a logic model. Basically a logic model, to really boil it down in simplest terms, a logic model is a graphical way to depict your program. The important thing is to have a clear program description and a logic model is the easiest way to get there. A logic model is a graphic representation of the intended relationships between a program's activities and their intended effects. It visually represents the program theory, why we expect the program to work, and helps identify any gaps in the program logic. A flow chart tries to describe the "what" of the program--what steps are required to complete this process. Logic models try to answer the question: Why is this going to work and what is going to lead to what? So you could really get a sense of how we think about logic models: in terms of inputs, these are the resources a program has to invest. The outputs (we talked about outputs) are really counting the service and the number of folks that you serve. The outcomes--and I would strongly suggest you think about outcomes in terms of time--so the short-term outcomes are things you can try to accomplish with your program in the short term, like six months. Medium, and longer-term outcomes. We know issues like successfully transitioning folks, military members as an example, back into the work force takes some time-- might be a trajectory of a couple of years out. There might be some short and medium-term outcomes to track along the way that will help you get there. Okay, next slide. So we've spent some time talking about outputs, which is the amount of service provided. As you may know, we have standard definitions of performance measures related to outputs for our veterans and military families programs. They include the number of veterans or active Service members and families served, the number of veterans and active service members engaged in service provision through CNCS programs. Next slide. This just gives you a sense of some of the programs that are currently measuring these outputs including the program called Veterans Helping Veterans Now, Metro Community College in Omaha, Nebraska, and the CA Conservation Corps. Next slide. I just want to talk a little bit more about outcomes because outcomes are obviously so key to documenting change. Outcomes regarding attitude and beliefs involve a change in thought or feeling. So, for example, a military spouse that feels she has more support after receiving services is an example of an outcome-related attitude and belief change. You could also measure changes in knowledge or skills, behaviors or conditions. An example of an outcome measuring knowledge or skill, for example, is that a military child may improve reading ability after attending a tutoring program. So essentially, performance measures allow you to document changes in attitude or belief, knowledge or skill, behavior or condition. Next slide. And, again, here are some examples from our current programs that are measuring outcomes. You can see the Veterans Helping Veterans Now program is doing this. The Metro Community College program is looking at the number of veterans succeeding in college due to the support of informed college staff and faculty. And the Conservation Corps in California is looking at those who completed the program and gained skills for future employment. So folks are out there doing this and learning along the way. We hope to share those best practices through our lens here at CNCS. And, later, you'll hear a little bit from Equal Justice Works which is doing a great job in documenting their performance and evaluating their program. Next slide. So you may be wondering--what resources are available to me? I mentioned the website that we host here at CNCS. It has lots of tools. I just want to talk a little bit about sources of data collection when you think about measuring performance of your program. There are different kinds of data that are accessible or that you can collect. One is, of course, self-reported data by doing surveys essentially of national service members and volunteers, whether that's, you know, if they are folks serving in national service programs, measuring them as members, or if they're being served as a beneficiary, gathering self-report data on them such as whether they have obtained employment post-service and so forth. There's also, and I think this is often overlooked, but there are a lot of public use data sources and administrative data sources out there that you could really use to get a sense of the population you're trying to serve and where they currently are with some of their outcomes. So I'll go through a couple of sources that I'm aware of, and you might want to do some of your own research on this. I think programs should always collect their own data on folks who serve and the beneficiaries that they're working with, but it's always important also to use available public use data sources at your fingertips. Next slide. In terms of self-report data, it could really help you answer the question of who are you serving, such as gathering data on the number of veterans served and the number of military families served. We often talk about the number of folks served, but one point I wanted to make on this webinar is it's important to both have a numerator and a denominator in your performance measures so you can look at what we call your penetration rate. As an example, the percent of veterans served in relation to the total size of the veteran population in your community? So it'll give you some sense of how much you're moving the needle in serving a certain population. There's a great resource on the Department of Veterans Affairs website called VetPop. I believe it stands for Veteran Population. It's a dynamic website where you could type in, I believe, a zip code or a town and it will give you the total number of veterans currently living in that area. It's a very valuable resource, I think, for local programs working on the ground to really just get a sense of how many veterans are in my area, how many military families are in my area. It's a very well used and well regarded resource that the Department of Veterans Affairs makes available. Next slide please. Another thing to think about is gathering self-report data from beneficiaries; that is, what outcomes are you achieving? Potential outcomes include increase in educational payment, increase in vocational skills and credentials, and an increase in employment outcome. And, of course, these data sources include self-report data from veterans and military members through surveys as well as, I mentioned, public use data sources. Next slide. I mentioned already the public use data source called VetPop available by the Department of Veterans Affairs. Another valuable resource is the Department of Defense's annual demographics report. I believe that data is available at the state and local level. That gives you a sense of members who are currently serving in the military and what they look like in terms of their different demographics. It's a very valuable resource. The other thing is that there's a lot of really good public use survey data out there that you can use for several reasons. One is to get a sense of the population you're trying to serve in your community. Another is to have a good comparison rate. So, if you can measure the performance of your program in terms of the outcome, you can get a sense of how those outcomes are being addressed at the national level. So there are just a couple of these available in the health and wellness area. The Army does a large-scale survey of military families called the Survey of Army Families which has a truckload of information in there about family wellbeing and family support and a lot of questions on deployment support for military families. It's a wonderful resource. The Air Force does its own similar kind of effort. It's called the Air Force Community Assessment. The VA does a national survey of veterans every five to ten years that gathers data from a representative set of veterans across the country. So, again, I would encourage you to take a look at these. I think it could be a very valuable resource for you. Another way to use it is to take survey questions from these surveys. They've been highly tested. And rather than develop your own questions, if you're going to gather data from your folks, this could be a really good way to develop a good instrument. Next slide. A couple other ones in the health arena include a survey of DoD health-related behaviors. There are other survey instruments on employment and homelessness that the Department of Labor and Veterans Benefits Administration put out. So, again, these may be resources for you. Next slide. Finally there's some good administrative data if veteran homelessness is an issue you're working on. The Department of Labor's Veterans Unemployment Training Service has good data on that available. The Veterans Benefit Administration puts out good data on G.I. Bill usage and usage in their voc rehab program around education. So that could be a good resource. And finally there's some really good employment data that's out there that you could potentially use. Next slide. This is going to conclude my part of the discussion, but I'll be available, obviously, towards the end to answer any questions. I just wanted to leave you with a reminder that we have a lot of good resources and tools related to performance measures on our website. Again the link is provided here, and I believe these slides will be made available to you after the completion of today's session so you'll have that available to you. It's been a pleasure to speak with you, and now I'm going to turn it over to our colleagues at Equal Justice Works who will tell you a little bit about their experience in collecting and managing performance measurement data. Great, thank you, Chris. This is Kathryn Gravely. I'm the Program Manager here at Equal Justice Works for our AmeriCorps Legal Fellowships Program. I'm also here with Kerry O'Brien, who's going to chime in as needed and definitely will be available for questions at the end of the presentations. Next slide. For those of you who don't know about Equal Justice Works and what we do, we are a national legal nonprofit based out of Washington, D.C. committed to social justice. We provide post-graduate legal fellowship opportunities to public interest attorneys wanting to embark on a public interest career. Currently we have four fellowship programs, one of which is our AmeriCorps Legal Fellowships Program, which consists of both full-time attorney members and minimum-time law student members that are placed at eligible legal aid organizations across the country. Our members primarily provide direct legal services to low-income communities including veterans. Next slide. We were asked to provide an overview of our veterans program and how we got to where we are as far as fine-tuning our theory of change and developing strong performance measures for AmeriCorps. And, let me tell you, it was hard. It was not easy! Our team had to put our heads together and summarize our theory of change, which is up on this slide. It's just a snapshot of our theory of change, and I'm going to now take the next few minutes to go into greater detail for each bucket. Next slide. So, when we thought about the need we were addressing in the community, we really had to think hard about what our attorney members were addressing for their veteran clients. We knew what they were doing. We knew that they were doing great work. But we had not yet articulated the need in writing. Basically, what we boiled it down to is that veterans face very complicated legal issues and our attorney members had a unique skill set to be able to address these complicated legal issues for their veteran clients. And these are legal issues that were severely affecting the quality of their veteran clients' lives. In documenting the need, we had to search for a lot of data out there-- data supporting that there was a need in each individual local community as well as on a national level, that veterans do have legal needs that need to be addressed. These are just a few examples of some of the research we were able to do including the VA Challenge Report, which documented a few of the top legal needs that veterans face, as well as information from various different government agencies such as HUD, and then studies on PTSD and TBI as well as homeless veteran studies. Next slide. Then we really had to think about, you know, what exactly are our attorney members doing? They were doing many different things for our veteran clients in serving their legal needs, but when we had to boil it down, we came to our attorney members were serving the legal needs of veterans to improve their lives by increasing income, promoting better health, and helping them to secure safe housing. Ultimately we wanted our attorneys to be improving the lives of veterans served, which was our intended outcome. When we had to evaluate the evidence, you know, try to understand how we know that what our attorney members are doing is actually working in addressing the legal needs of veterans. Well, we were fortunate enough to have a pilot going on since 2010 where we had attorney members serving the legal needs of veterans, and each year we began increasing the number of attorney members serving the legal needs of veterans. So, we had that pilot study as well as the evaluation results of that pilot study. In addition to that, we had studies that pointed to lawyers making a difference in general, not necessarily for the veteran population, but studies that showed having representation-- having a lawyer help you in your legal matter, you're more likely to have a positive outcome in your legal matter as opposed to representing yourself in a legal matter. So that's the evidence we had for our intervention. Next slide. So, after we came up with our, you know, articulated our need, the intervention, and our intended outcome, and we had all our evidence, we really had to break down the difference between outputs and outcomes that Chris so eloquently described. But, for us, it was hard at first really trying to understand what types of outputs we could collect, and then from those outputs what sort of outcomes we wanted to know from that information-- what outcomes we wanted to know from the service that was being provided to the veteran community. This is just an example of one of our outputs and then the aligned outcome that we collect in our data collection. Next slide. Here is how we broke down our specific buckets of outcomes. There are four different categories: income, employment, housing, and then we had an "other" category as a catchall because there were some other kind of very specific matters that our attorney members were helping to address for their veteran clients. But, in general, our attorney members were helping veterans with increasing their income, obtaining employment by helping to remove barriers, and then helping them obtain or retain housing. If you look at these, you should note that these were broken down even further to provide a detailed picture of each bucket of outcomes, and I will show some more detailed data outputss and outcomes we were able to collect. Next slide. So, what we could say in 2010 was the number of veterans we were serving through our program along with a lot of anecdotal evidence. And that was really it! We couldn't say anything about the outcomes of the cases that our attorney members were working on. We couldn't say anything about the actual impact of the veterans served. And we didn't really know whether the veterans we were serving were better off. Next slide. But today is a different story! Our data and outcomes are much more expansive and paint a clearer picture of our program and the impact in the veteran community. This is just a sampling of data and outcomes that we are now able to report to CNCS. Next slide. Some more examples. I'm not going to read them all out to you, but I just wanted to give you a sampling. You will be provided a copy of this presentation. Next slide. More examples. As you can see, it's much more expansive than 2010. Next slide. So, the big question you're probably asking is how we were able to get to a much more expansive data collection and outcome reporting. Next slide. This just shows our evolution from 2011 to today, 2013, of where we started and where we are at now. As you can see, we really just reached out to our subgrantees--or we call them our host sites-- and people doing work in the field-- people on the ground actually doing the veterans service--and talking to them about what types of outputs and outcomes we should be looking for. We also used our program evaluation from the pilot study as inspiration. In the end, our report to AmeriCorps is now substantially better than in years past. Next slide. We used to collect data using a very expensive web-based application module that was very hard to make any changes to measures we were trying to ask our subgrantees to report out on. If we wanted to adjust anything, it was extremely difficult and cost a lot of money. Then we decided to use Google Forms. It was really easy to create. It was easy to change measures that we wanted to refine or make clearer for our subgrantees that were reporting out to us. And best of all, it's free. So it did exactly--Google Forms does exactly the same thing that our web-based application module was doing, meaning it has the capability to dump data into an Excel spreadsheet, which is really easy for us to use. It creates a snapshot of the data for all of our subgrantees, and it makes it easier for us to then report out to CNCS for our GPRs. Next slide. This is just a screenshot of what our Google Form looks like. So, if one of our host sites wanted to input their data into our reporting form, this is how they would do it. They would go into the Google Form and input their numbers through here and then just click submit. I believe we have a sample to show you of what we send to them at the beginning of a reporting period. This is just the Word document form of the Google Form that host sites eventually end up submitting to us every reporting period. So, we like to provide them with the Word form so that they are able to keep a version for their own records. Because the one caveat about Google Forms is that once you click submit--and there are different options-- but we learned that once you click submit, the person that submitted it doesn't necessarily get a copy of their data. So we provide them with a Word document form so that they know what they're tracking and they're able to actually keep a version for their own records. Next slide. We also changed our Great Stories. For those of you that are grantees, you know that you have to submit Great Stories to CNCS every progress report period. Wwe made sure that our stories were more outcome oriented. Each member is now able to tell a story about a veteran client that they served that has its own little mini-theory of change. As you can see here, as an example, one of our Equal Justice Works AmeriCorps legal fellows, Rochelle Richardson, clearly was addressing a need of a veteran client. She was able to discuss in her story what exactly she did for that client. Then she had a positive outcome for that client, which, in this case, resulted in retroactive benefits as well as future benefits for her client. Next slide. Here are just some lessons that we learned through the growing pains of developing performance measures and data collection. I would say define your outcomes early and make sure they are aligned with funder requirements, in this case the Corporation for National Community Service. Get a group of experts together to help you define your performance measures. This is extremely helpful. Don't try to do it on your own. Get lots of input from folks on the ground actually doing the work. And then once you've created your measures, don't think that that's the end. You need to keep refining and improving your measures to ensure that they are truly reflecting your program's impact in the community that you're serving. Next. We would like to go even further than where we're at right now. Some big aspirations are figuring out ways to track long-term outcomes. As Chris mentioned, right now we're able to capture short and medium term outcomes, but we'd like to capture long term as well as we'd like to do some comparison evaluations with our program and other programs and national statistics. Of course, our biggest goal is possibly doing a randomized control trial within our program, which is truly the gold standard in the evidence world. That's kind of the big vision for our program in looking forward. With that, I will hand it off to Chris for a recap. Kathryn? Yes? This is Debbie. Just before you go on, we had a really good question in our chat panel that says, How do you make sure you do not post any personal information on the Google Docs site? Would you like to address that right now? Personal information, as far as clients served or veterans served? Because we don't--all that's reported are numbers and then through the anecdotal stories names are always changed and kept confidential. Google Forms is private to us. It's attached to our Google account and so it's not publicly available on the Internet. So, even if a site inadvertently included some personal information, it would only be known to us. But so far we haven't had any sites reveal or post any personal information that would be a problem in that regard. Thank you, I think that answers their question. Okay, this is Chris Spera again. Just wanted to give a quick recap on what you heard today. So, again, thanks for participating in today's webinar on performance measurement related to veterans and military family programs at CNCS. As you hopefully learned across the session, performance measurement is absolutely critical helping CNCS and the national service field tell its story about the impact we're having in the field so it could be used at the national level for our agency. It could also be used at the local level for you to see how you're doing in terms of your program and what it's trying to accomplish, hopefully through the development of a logic model. And then systematically tracking outputs and outcomes: outputs, again, being the amount of service provided, such as the number of veterans served within a geographic area; and outcomes that really more reflect changes, again, in knowledge, attitude, behaviors, or a circumstance such as veteran employment. Also, hopefully, you understand now that there's a little bit of a difference between performance measurement and evaluation with mostly the key difference being evaluation is really the only way to establish a cause and effect to really know that your program is causing the outcomes you're tracking. But performance measurement is such a key part of measuring performance of your program annually. So, thanks again for participating. Hopefully you learned a lot. As Koby mentioned at the beginning, we'll be having more of these sessions. We expect to be more heavily and more engaged in performance measurement as an organization with our grantees going forward. Webinars are one way to do it, but we have other tools we'll be using along the way. We welcome all ideas, suggestions and feedback as we move forward. So with that let me turn it back over to our host to facilitate the question and answer period. Thank you, Chris, very much. We did get one question here in the chat just recently. Kadie Schaeffer asked, Are some of the vets served in more than one area, and, if so, how do you get an unduplicated count of individuals? That's an excellent question. At the beginning of our growth toward a more expansive data collection list, that was something that we struggled with in the beginning and we're still--we're getting better at unduplicated data, but it's still a concern of ours. But to generally answer your question, some of the veterans are served in multiple different areas. So a veteran walking through the door of a legal aid organization might have a housing issue as well as needing to appeal a disability benefits claim as well as they needed some barriers to employment removed. So that would make them being served in all three different buckets of our outcome buckets. So what we try to do, we try to get global data and then start breaking it down from there to avoid duplicating veterans that are being served in more than one area. So globally we'll ask, how many veterans did you serve? And then globally we'll ask, how many veterans did you address for a housing matter? And then employment matter? Or a benefits matter? And we're able to discern from the sub-global data points using the total number of veterans that they served, that's how we try to figure out who's being duplicated and who's not. So it's tough, as you can hear. It's not an easy thing to do, but that's kind of how we try to resolve making sure we're not duplicating information. And it's how you present the information, right, if you're talking about how many veterans you served in a housing matter, you know, that's different from discussing how many veterans you served totally. So it's--Kerry, do you want to chime in? It's hard. It's hard to make sure you're not duplicating, but you start at a global number and then from there you separate out the different areas that you're serving and comparing that to your global number of veterans served to make sure you're not duplicating. Great, thank you. Kadie also followed up with, What kind of documentation do you require from sites to back up their specifics? Well, so what's great about our host sites is they're all legal aid organizations and most of them are required to use case management software. Each of the attorneys, they have a certain number of cases, they have a caseload, and they have to input tons of information and tons of data into their case management software for each client that they serve. So we're able to kind of piggyback onto the case management software systems that the host sites already have, and that's what they use to ultimately collect the data that they report out to us and then we report that out to the Corporation. So we are blessed with that, but that is how our host sites back up the information that they document. Thank you. We've got another question that came in from Jake Cummings. He asked, can you discuss some of the limitations of the county levels vet pop estimates (i.e. sampling errors, etc.), and why numbers reported by the VA differ from those reported by the Census Bureau? This is Chris. That's a good question. I'd probably have to investigate it more in order to address it specifically. I know that the methods that the VetPop uses to ascertain--basically they make projections on where veterans are a little bit different than the way the Census Bureau does it, so it could be related to the detailed methodology in terms of how they weight the data and so forth. But I can, probably the best thing for me to do is to research that and get back to the person who asked the question. Thanks, Chris. We could also post that answer on the edited chat, too, so that all the participants can get that answer. Also on our registration form and on the Veterans and Military Families Knowledge Network we asked if there were any questions or comments. Jake also had a couple of questions he posted there. He said he'd like to know what the outcomes of actions, such as referrals to outside agencies like the VA, VSO, etc. are. For example, it would be great to know if a veteran or military family member referred to the VA received the help they needed and whether additional assistance is required. He goes on to say, another concern is how to quantify outcomes resulting directly from program activities. I suppose this was somewhat connected to the first issue I mentioned. In addition, sometimes it can be difficult to isolate the true change agent. So can you address a couple of those things for Jake? Sure, sure. Yeah, the last couple of things he mentioned there in terms of isolating the real change agent is difficult, and that was the point I was trying to make that program evaluation is really the tool to do that. Performance measurement really doesn't allow you to measure cause and effect. I think Equal Justice Works mentioned that one of their longer-term priorities is to do a randomized control trial where they literally are randomly assigning folks to receiving their intervention or their program, and folks who don't, which will really get you a cause and effect. So I think it's important for folks out there in the field to recognize that performance measures is one piece of the puzzle, but longer term they should aspire to, and we would encourage them to at CNCS, do program evaluation. We do have program evaluation requirements in our AmeriCorps language. If you have a grant of $500,000 across a three-year grant window, you are required to do an independent evaluation. So perhaps in a separate webinar we'll have a little bit more discussion on the evaluation requirements and some of the techniques there, but really the way to isolate the change agent is to do a true program evaluation. I think you also mentioned getting data from VA on where the needs are and where some of the handoff needs to happen. I think that's a really good point. I think that piece of the puzzle hasn't really yet been figured out, but I would encourage folks to work with their local VA organizations and their veterans' service organizations to really identify what the needs are in their areas for veterans and military families and make sure they really understand those populations well in order to develop the right community-based solutions. So lots of good points by that question. If I may add something, this is Kerry from Equal Justice Works. Our evaluator has looked into the question of what data does the VA collect with regard to outcomes in its grant program, and I think there is really a lot of potential there. Just to give an example, in the grant per diem program, which is a housing stream of funding that the VA gives to hundreds of nonprofits throughout the country, a lot of our legal aid organizations are working with them very closely. An example of some of the outcome that those grantees are required to track are: obtained permanent housing within 12 months of finishing the program; retained that housing for three months, six months, or 12 months; placed in a job; and retained a job for three months, six months, or 12 months. That could be really powerful and our sites are able to get that sometimes where they have a strong working relationship and they have the veterans sign waivers of confidentiality at the beginning of services. It would be great to see our community come together to allow for more sharing of that type of data so that when our clients are participating in those types of programs, we could have that outcome data so that we could know what was happening to them after they left our specific programs. Thank you all very much. That was very helpful I think. I don't see any last questions in our chat, and that was all our questions from our forum and our registration form, so are there any last words that you presenters would like to say before we get to our final slides for the end of our session today? Just from Equal Justice Works, I would say that we now have, as we've shown, a much more robust set of data and we've been able to use it in all different kinds of ways including determining which host sites might need a little bit of additional assistance and which host sites we should be trying to place more members at. Just generally having this level of data has helped us not only to make the case that the services are valuable, but to help us to improve our program. I think Kathryn really gave the flavor that it was a real pain to get to the place where we are, but I think we absolutely have come to the conclusion that it was completely worth it for both ourselves and for the taxpayers that we're serving with these funds. Thank you. I think you were a really good program to showcase that because I loved the fact that you said that lawyers are so often not the kind of people that follow up with their clients. Once there's something that happens, they are done with the case and it's over and that you really had to put real intentionality into what you did to get these outcomes. I think you're a really good example for folks to be able to learn from about what needs to happen to go from outputs to outcomes. So thank you very much. Thank you. Chris, any last words? No, other than just to say thank you to folks for attending, and hopefully this is the beginning of a discussion and dialogue that will continue around performance measures. I look forward to working with folks in the field as we continue. All right, thanks, Chris, so much. What we'd like to show you here is this is our link to the Veterans and Military Families Knowledge Network. We've talked about it several different times throughout our session today. This is the place where all the information from this webinar will go. We save the recording and we get it closed captioning and create an edited transcript from it, and those will all be housed there as well as the PowerPoint slides and the handouts. We also put information from all our past webinars and any future webinars up here, so it's a great place to keep your eye out for any more learning opportunities from the Corporation for National Service and some of its other T/TA providers. It's also a wonderful place to join and get information about veterans and military families and other organizations and programs doing the same work out there. So you can connect with others and build your own support network and community of practice that can help you strengthen the processes that you're doing yourself. As Chris pointed out to you, this also houses the performance measure resources and data measurement tool that he gave you the link to a little bit earlier in the session. Next I'd like to remind you that today's session was just the third in this webinar series of What We Know about Veterans and Military Families Programming. Our next and final session for this series is June 12th-- Making a Difference, What We Know About Using An Evidence-Based Approach. Like today, you'll learn some of the nuts and bolts of an evidence-based approach, and then learn from a couple of real-world examples about how they've been using evidence-based approaches within their work. If you're available, please join us for this webinar to complete your set of information from the What We Know series. The registration link is here on your screen right now, and you'll be able to find that as well if you go onto the Knowledge Network. Lastly I'd like to say thank you. Koby, did you have any final words for the participants today? 768 ]01:05:03,000 --> 01:05:06,000 I just want to say thanks again for participating in these webinars. It's very important that we continue the community base of knowledge that can help improve and expand our programs and services in this area. Your participation in these conversations is critical to that conversation, so I would ask that folks go on the Knowledge Network and join up and continue the discussion there. So thank you everybody. Thank you in particular for our panelists from Equal Justice Works as well as Dr. Chris Spera. All right. Thank you so much. And from us here at Education Northwest, we'd like to thank all of our hosts and all of our panelists and participants for being on the webinar with us today and to remind everybody to please fill out the evaluation so that we can make each one of these sessions just a little bit better for you. Thank you and have a great afternoon. 781 01:05:50,000 Bye bye.