Reflections from the window - project evaluations

As one of the last steps of the summer program with AMIGOS (and any good project methodology), we took some time to evaluate how the program went. We had some interesting conversations among the staff team as to how to implement the evaluations, in addition to the topics discussed within the evaluations.

We used a participatory evaluation method, which succinctly, means to involve participants in the evaluation process. For AMIGOS, this meant to get the opinions of people from all aspects of involvement with the program: host families, youth group members who worked with our volunteers, partnering agency contacts, to even community members who weren't involved with the program much.

The evaluation process looked different based on the different levels at which we evaluated

• At the community level, supervisors might have coordinated a town meeting to evaluate the project in a group with community members. In some communities, a town meeting wasn't feasible; these supervisors might have conducted multiple interviews with various community members and key players, or even distributed surveys to community members.
• At the partnering agency level, directors and supervisors met with PLAN contacts that we had worked with. We evaluated how the summer went: successes, challenges, suggestions. Based on the AMIGOS mission statement, what did we do well? what difficulties did we encounter? what suggestions do we have for next year?

One of the interesting topics of discussion when we were planning out our evaluation methodologies was whether the presence of our youth volunteers (who had been living and working in the communities) would affect the responses in the evaluation. While communities might be more reticient to discuss the challenges of their volunteers when the vols are right there, there was also the consideration of how to teach or pass along knowledge about the evaluation process to volunteers. None of our project supervisors (responsible for directly overseeing volunteers in the communities) had ever done a participatory evaluation before, either because the eval process was different when they were a volunteer, or more likely, that the participatory evaluation had always been conducted post-volunteer departure from the communities.

Here's an interesting document on involving youth in the participatory evaluation process published by the Program for Youth and Community at the University of Michigan.

In the end, we came to a consensus on a two-part participatory evaluation. First, that there would be some sort of participatory evaluation in all communities while the volunteers were still there. In general, this was the meaty evaluation step, although in some communities supervisors encountered challenges that prevented them from doing a thorough evaluation at this point. Next, supervisors returned to each community after volunteers left the country in order to check that everything was indeed OK with the volunteers and to complete the participatory evaluation process as needed.

The final step of the evaluation process was to complete the documentation on these evaluations. We created a number of documents based on the multi-faceted evaluation process:

• Community-specific evaluation summaries - for next year's staff to review and learn from at the beginning of the summer
• Partnering Agency evaluation matrix - based on the notes from three different partnering agency evaluation meetings we held (with the three PLAN offices with which we worked)
• Final Evaluation documents - we created one in Spanish and a summary in English that narrated our evaluation of the project. I focused on the key issues we had encountered throughout the summer, as well as common threads from all of our participatory evaluation encounters. These documents also included a more quantitative component that enumerated all of the projects we worked on this summer (# of hours of children's camps, # of murals painted, etc.)

What have your experiences been on evaluations of projects? Self-evaluations?