This month we sat down with our Director of Evaluation, Brian Schmotzer to learn more about how he puts together evaluation plans and the sources of data we use to execute program evaluations.
What does your typical day look like?
My typical day is dominated by carting my children around to their activities after school. My typical work day is somewhat calmer than that. Surprisingly, I’m not generally swamped by email, so I can usually batch my email effort and deal with it first thing in the morning and later in the afternoon (with occasional peeks during the rest of the day). Then it is a mix depending on the work that needs to be done. Often, I write or review evaluation plans that we are working on for clients. Other times I am working with data or consulting with my co-workers on the work they are doing.
What kind of data do you work with?
The data we work with comes from a variety of sources. Primarily, administrative data such as from a client’s database or from governmental agencies. Also, we generate a lot of data from surveys we conduct to answer questions that are hard to address from readily available data.
What goes into making an evaluation plan?
The details of an evaluation can be as varied as the project, but we use the same process to develop our plan for any project. First, we begin with a Logic Model, a map of the project flowing from the problems to be solved, through the mechanisms of the project to solve them, to the outcomes that will be measured to assess success. This allows us to list a set of Evaluation Questions that we will answer with the evaluation. Next, we develop a Data Plan that defines what data will be needed, how it will be obtained, and how it will be used to answer the questions. In parallel, we develop the qualitative methods that will inform the portion of the evaluation not reliant on quantitative data. Finally, we work with the client to refine our planning and prepare to implement the evaluation.
What have you learned about workforce development from your evaluations?
In some ways I feel like I’ve learned a tremendous amount, and in other ways as though I’ve learned very little. I guess that averages out to saying that workforce development is difficult. We have seen so many different strategies implemented to try to help workers. Ideas that we “know” work include apprenticeships, business-facing job placement, training and education, and barrier removal. But the word “know” needs to appear in “scare-quotes” because so much depends on the quality of the implementation. Maybe the one thing I’ve really learned is that a well thought out strategy that is targeted, locally-informed, and well executed is likely to succeed, whereas a strategy that is scattershot, makeshift, generic, and/or poorly executed, no matter how stridently the theoretical evidence suggests otherwise, is likely to underperform.
What is the best part of your job?
The camaraderie. I’m truly lucky to work with wonderful people, and it is a joy to spend the day with them.
New Growth Group are program and impact evaluation experts. We have conducted evaluations for over 50 organizations, in a variety of sectors. Contact us if you have an evaluation project you need help with.
