Do you carry out regular monitoring during projects, or do you prefer ex post facto studies?

Thank you, Shehzaad, and the many others who have contributed; I would argue that we need both regular monitoring and ex post studies, for different purposes.

Just to clarify, many project designs are based on a logframe format. This may have been inherited from the project designers (rather than having been developed by the implementation team). It may also form the basis of a legal agreement between funders and implementers. It might also come with a longer list of required indicators.

The implementation team may then choose to sit down and 'unpack' the logic of their interventions. We live in a complex and iterative world, but every project is based implicitly on a somewhat mechanistic set of assumptions about what the activities will lead to. It is in the project's interest to make those as explicit as possible - not least because the indicators should only be those that the logic suggests will be changed by the project.

This is the proposed results chain exercise - which is more detailed than the logframe format. It should show every key change that is expected, between the project activities and the anticipated impacts. Most projects anticipate outcomes that build on each other (e.g. training leads to increased awareness leads to change in behaviour leads to improved performance leads to more jobs, for example) and it is very useful to spell these all out.

If those steps are all articulated, and the various results chains made explicit, the project can then validate whether the steps are unfolding as hoped, during implementation. It can also point ex-post studies in the right direction, because it will be clear where the impacts are expected.

The only real alternative to this is to treat the project as a black box, and randomise the treatment, form a control group etc. This does nothing for effectiveness during implementation (as it is usually too late) and risks 'contaminated' control groups, self-selection bias and various other challenges. Some argue that all these challenges can be overcome, but at least it takes very clever experts to do so.

So for people in the field, I would argue that articulating the logic or results chains in sufficient detail is the only way to ensure effectiveness, and to be measured in future on the right basis. Unfortunately, of course, the results chains may not match with that inherited logframe, and this is a matter of negotiation often with the donor. Or the logframe is hopefully vague and broad enough to accommodate the more detailed (and evolving) logic.

In summary, the sentiment behind the logframe and results chains is the same, but the format and applications are different. Donors would always like rigorous measurement (ex post studies) but meanwhile would also like effective monitoring and implementation too.

Thank you again to all of you who have shared your experiences so far, and I'd be very keen to hear from anyone else who has also worked in this sort of direction.

Cheers,

Jim

Dear Shehzaad,

Thanks for the point on indicators. The drive to have common indicators comes in essence from institutions, who need to aggregate their achievements (and also arguably to orient priorities and decision-making). Otherwise, as you suggest, the exercise can become rather superficial.

The DCED has proposed income and employment as common indicators, on the basis that most private sector development projects are aiming to increase these, in various formats. Some have found that useful, to focus on what is really important (rather than on outputs) while for others, it has generated more heat than light.

Measurement at the household level (poverty) is more challenging than measurement at the enterprise level (income and employment) because multiple income streams at household level make attribution tough. We have found few programmes that are able and willing to measure with reasonable attribution at household level.

Others may want to share different views, of course!

Cheers,

Jim

Dear all, the discussions are really very interesting.I assumed that as part of the Project Management Cycle,mid-term and terminal evaluations are supposed to be carried out as part of monitoring exercise to generate feed back during project implementation & inform future progress and programming and strategies during the course.My question is, how relevant are these evaluations for short term projects?.Is it possible to develop and set SMART indicators as a basis for ex-post evaluation and we leave  out these kinds of "process" evaluations?.I do feel that the mid-term and terminal evaluations are not necessay since the project could already be having good data from its ex-ante-evaluation/findings to provide a strong basis for monitoring and evaluation of the intervention comprehensively.

I also forsee alot of funds being used in the process of evaluation in the PMC instead of funds being channeled towards real interventions which are expected to generate impact and good outcomes to the community.

Thanks,

Moses Owiny

WOUGNET,currently in Batavia, NY

Dear All

Agriculture information need is dynamic and ICT fits the glove by being both flexible and dynamic in nature. In Katalyst, we work with ICT channels to provide access to accurate information to farmers in a timely manner. The monitoring process of our work starts from the very beginning and is carried out from time to time, which enables us to enjoy a number of positive benefits. We are in line with the monitoring and result measurement (MRM) team from the start which helps us to look at our initiatives from a MRM perspective. In addition regular monitoring helps us to respond to any changes in the service provider's and beneficiaries' environment. The benefits of regular monitoring is further reflected when ex post facto studies are carried out.

A similar situation was skecthed when e- krishok was scaled in 100 location in 2009. E- krishok was a promotional campaign targeting the rural farmers, aimed at increasing the farmer's footfall in Grameenphone Community Information Centre (GPCIC) A close monitoring was caried out both by MRM and ICT team.  In 2011, an impact analysis study was carried out where we captured the impact of our interventions on  e-krishok members.

Thus depending on type of intervention, a combination of regular monitoring and ex post facto studies has the potential to give rise to a substantial impact in the target group.