One advantage of a Challenge Fund is that it encourages creativity, innovation, and learning. In its quest for timely management information to monitor the progress of its Beneficiary Households (BHH), EEP/Shiree developed an innovative, smartphone based monitoring tool known as CMS2. This was the first time that such a system had been tried in Bangladesh, and some expected and unexpected challenges were encountered. As a result, there was a significant amount of learning, which may be instructive to other programmes looking to utilise technology.
CMS2 was developed in June 2011 by EEP/Shiree with mPower, a Bangladesh IT software company, as a monthly, census-based survey of the Scale Fund BHHs designed to be completed in ten minutes. The intention of CMS2 was to assist EEP/Shiree’s Partner NGOs (PNGOs) to capture dynamic, real time information about the experiences of thousands of beneficiaries in order to help guide project implementation and address problems/sudden shocks preventing households from leaving extreme poverty, or causing them to fall back into extreme poverty. Following promising initial trials, the technology was scaled up to a national pilot, which ran from July 2012 until December 2015 when the system was closed.
Using a smartphone platform developed for EEP, data for CMS2 was collected by PNGO field officers based on a standard questionnaire which recorded any progress achieved by BHHs on their anticipated trajectory out of Extreme Poverty. The questions were based on a number of indicators such as socio-economic status, water and sanitation, social well-being (happiness), nutrition and food security, health and education, social safety nets, and significant life events (positive and negative).
BHHs were surveyed approximately monthly. Data were inputted into the smartphone by field officers – together with a field officer’s assessment of the condition of the household and sent electronically to the Dhaka based servers. Software was developed to visualise CMS2 data on a dashboard for a selected range of indicators. The intention was to use the dashboard to analyse spatial, temporal and socio-demographic trends for insights into the dynamics of extreme poverty based on households’ self-assessment of change, and to identify households requiring further support.
The CMS2 technology consisted of the following key components:
- 1,022 ‘Samsung Galaxy Y’ smart phones equipped with an Android operating system, internet connectivity and GPS capabilities. Field staff accessed a pre-loaded application to conduct the CMS2 questionnaire;
- Data was automatically aggregated at EEP/Shiree and cross tabulated with the CMS1 baseline survey. Results were uploaded in real-time to an online visualisation dashboard;
- Users (NGOs, academics, the development community or public) could access the visualisation (with varying levels of privacy controls). The intention was to allow users to make use of this information for their own purposes, for example, PNGO staff could use up-to-date information to guide evidence based management decisions;
- Comprehensive training and testing of PNGO staff and field officers was provided by mPower and EEP/Shiree programme staff. Continual refresher training, feedback and supervision was also provided by EEP/Shiree staff.
Review of CMS 2
As EEP/Shiree came to its final, closing phase, and following the recommendations of two DFID Annual Reviews, CMS2 was subject to a Review in 2015. This Review consulted multiple stakeholders involved in the design and use of CMS 2, and contributed to a holistic understanding of the experiences of:
- The beneficiaries who regularly answered the monthly survey questions;
- Implementers (frontline staff who administered survey);
- How and to what extent both EEP/Shiree and PNGO M&E staff and managers used the resulting data both operationally and strategically.
The lessons from CMS2 can inform potential future Extreme Poverty, livelihoods and sustainable graduation programmes and be of interest to those involved in real time data collection for monitoring and evaluation in development programmes.
Please see the 2015 CMS2 Review.
The overall conclusion was that CMS2 was an experimental, pilot programme initiated in the context of a ‘Challenge Fund’. It was methodologically innovative and interesting enough to have the potential for adaptation, which, to some extent did, indeed, happen. However, there were key challenges that were not identified during the development and usage of the system. Notable amongst these were the additional pressures that the system placed on front line staff to collect data alongside their day jobs. This led to a significant amount of problematic and incorrect data being captured on a regular basis. Furthermore, this led to an inability to follow good survey methodology practices, resulting in the collection of cross-sectional and longitudinal data simultaneously: this data was inextricably mixed in the CMS2 database, which significantly limited its value both to PNGOs and to the research community.