Jim Kim’s Bold Vision of Beneficiary Feedback in Development

At the October 2013 World Bank annual meetings bank president Jim Kim committed to an extraordinary vision of collecting beneficiary feedback for 100 percent of projects with clearly identified beneficiaries. The World Bank locates beneficiary feedback as a subset within the larger theme of citizen engagement. Other elements of citizen engagement have been part of the World Bank’s modus operandi for decades, including public consultations on national development policies and the like. Beneficiary feedback is different, however, in that it makes a direct connection between service provider and service recipient. Just as user feedback shakes up so many commercial service settings – like taxis, hotels, dentistry and autos – done properly user feedback can directly improve the many governmental services supported through World Bank loans.

It is clear from the recently published “Strategic Framework for Mainstreaming Citizen Engagement in the World Bank Group Operations” that the World Bank sees the potential here to enhance real life outcomes of ordinary citizens. The challenge now is to operationalize the citizen feedback promise in a way that avoids the pitfall of becoming a tokenistic check box exercise.

There are two worrying signs pointing in this direction. The World Bank’s voluminous published work around beneficiary feedback does not provide a clear model. The second comes from the approach to monitoring the beneficiary feedback promise as set out on the President’s Delivery Unit (PDU) website. The measures now proposed by the PDU are inadequate.

With respect to the first concern, it is important to acknowledge that the citizen engagement Strategic Framework report is in itself a significant contribution to the field of development that is sure to be an important reference source for years to come. But it is more like a survey of the landscape than a practical guide. The Bank clearly wishes to avoid being prescriptive. But a practical model would be the opposite of a straightjacket; it would be an enabler. It would provide the kind of tested craft that ensures that sensitivity to context does not become a cloak for mediocrity. It could borrow from the global experience of the last few years in beneficiary feedback, for example as captured in a few pages in Keystone Accountability’s “Constituent Voice: Technical Note 1.”

What of the measures of progress on the beneficiary feedback promise? Citizen engagement is one of four “institutional priorities” identified for operations monitoring by the PDU, the other three being process simplification, data for development, and quality and results. But the metrics as reported on the PDU tracker just don’t match the promise. Ironically for a monitoring framework that includes tracking progress against “a target to achieve and maintain satisfactory outcome ratings in at least 75% of World Bank operations by 2017,” (emphasis added) the measures set out to track beneficiary feedback are in the nature of outputs, not outcomes.

One published metric for the design phase looks to the percentage of project loans that “declare the intention to collect feedback; and, those that plan to use at least one beneficiary results indicator during implementation.” The closest measure to an outcome metric tracks the percentage of project loans that “report action on and/or results on at least one beneficiary feedback indicator three years post project approval”. This is so lagging and so open to interpretation as to be essentially unhelpful in monitoring for effectiveness or impact.

Stopping at output measures is totally unnecessary since measuring the results of feedback mechanisms is incredibly easy. We propose two complementary solutions.

One tracks the rate of fixes to problems identified through feedback. As noted in the working paper, “The Fix-Rate: A Key Metric for Transparency and Accountability,” the primary function of many types of feedback mechanisms is to identify problems. The same data collection tools used to identify the problems can also be used to identify whether satisfactory responses or solutions follow. One simply asks those who spotted the problem if it has been satisfactorily addressed. This is the fix rate.

Another approach looks forward, again using the feedback mechanism itself. One asks beneficiaries how likely they think it is that their feedback will result in a satisfactory response. This is similar to the “How likely would you be to recommend this company” question that is now dominant in the multibillion dollar customer satisfaction industry. The answers provide a very accurate sense of the health or effectiveness of the feedback mechanism, and of the extent to which the organization that is receiving the feedback is learning and improving from the feedback. Both of these are necessary preconditions for the World Bank’s stated primary purpose here to be achieved – for feedback to accelerate development outcomes.

These two outcome metrics are simple and easy to implement and could be added to the current set of output measures at little effort to those implementing feedback systems. The impact of these metrics – as we have seen in our own work – is nothing short of transformational. We recommend them to the World Bank, along with our congratulations for the leadership that the World Bank Group is taking to realize this historic opportunity.

Many thanks to both David Bonbright and Fredrik Galtung for their latest op-ed piece. If you would like to learn more, be sure to join them in an upcoming webinar, on Thursday, January 22, 2015 3PM GMT – 4PM GMT. You can also follow them on Twitter @davidbonbright and @galtung.

To stay up to date with the latest Markets For Good articles and news, sign up to our newsletter here. Make sure that you are also following us on Twitter.