Developing Spiritual Impact Metrics

Context

ICM’s vision is to see ultrapoor families released from all forms of poverty, and one of the key facets is spiritual poverty. ICM recognizes that living in extreme poverty is often characterized by a lack of hope, which is why the organization aims to transform the spiritual lives of the ultrapoor and allow them to experience renewed hope for their future. Through the 15-week core Transform program, ICM partners with pastors to provide a Biblically-based values curriculum, allowing participants to discover who God is, how much He loves His people, and what He graciously offers to those who have faith in Him.

ICM places importance in cultivating the spiritual formation of ultrapoor families. Because of ICM’s desire to continuously grow and steward the families God has entrusted to the organization, ICM invests in implementing and developing monitoring and evaluation tools used to assess Transform’s spiritual values component.

Since 2011, over 140 questions have been employed to better understand where people are in their faith journeys and whether attending Transform has made a difference in their relationship with God and their understanding of salvation. In 2014, a scoping review of measures designed to assess spiritual health and maturity was conducted. In this review, Engel’s scale was identified as a promising tool and also recommended by other organizations. Engel’s scale provided a framework to develop specific questions for the Philippine setting. During this time ICM also partnered with Back to the Bible to develop questions on spiritual disciplines and Bible engagement. These questions were then trialed in ICM’s baseline and endline participant surveys from 2014 to 2018.

In 2018, ICM was introduced to the Kingdom Impact Framework developed by EIDO research, to help the organization measure spiritual impact. A short version of this tool was tested in subsequent surveys to give the organization a snapshot of the participants’ spiritual formation before and after the program.

After undergoing several different iterations of spiritual health metrics, a thorough review of all spiritual measures used in the past decade was conducted, initially in 2019 and then completed in 2021, to reach a concise module of spiritual health questions that address ICM’s core measurement needs.

Design and Results

Design

The process of the spiritual metrics review were as follows:

1. Gather anecdotal enumerator feedback on the usability, translation quality, and acceptability of the survey questions and constructs being evaluated;
2. Refresh ICM’s spiritual values theory of change (ToC) with the implementation team and review all spiritual metrics used against the latest theory of change;
   Identify patterns of change across the batch-by-batch pre and post survey results;
3. Assess all available survey data for high-value questions or metrics; and
4. Review analysis and discuss the results with key ICM leaders to finalize core spiritual impact measures.

Results

A total of 128 questions were examined over this period spanning from questions about the participants’ (1) understanding of God’s grace and salvations, (2) beliefs about prayer and the Bible, (3) beliefs about who God is and His sovereignty and love, (4) perceptions and attitudes towards the state of their religiosity and relationship with God, and (5) church engagement.

From these, questions that were scarcely used, questions that were generating inconclusive and inconsistent results, and questions that were difficult to understand based on enumerator feedback were removed from the roster of survey items being closely considered. Additionally, upon reviewing ICM’s spiritual values ToC with the implementing team and strategic leaders, a second round of questionnaire culling was done to generate a short list of questions that were aligned with the ToC.

The key outcomes identified through this ToC process are the following:

1. To understand salvation by grace,
2. To increase church engagement, and
3. To increase the practice of spiritual disciplines.

From 128 questions, a final list of 27 questions were identified to be most informative in measuring spiritual formation and aligned with the updated ToC.

Upon narrowing the selection down to 27 questions, a third round of thorough assessments was employed, involving batch-by-batch comparisons from 2011 to 2019 Transform data and short reports in order to further dissect the patterns of change and each question’s ability to measure the core spiritual outcomes identified from the ToC review process. In this review stage, 11 high-value questions were identified by key ICM strategy leaders due to their alignment with ICM’s core spiritual values constructs and their ability to elicit consistent responses and generate consistent pre to post trends across Transform batches. These 11 questions showed the most potential in serving as ICM’s core spiritual impact measures[1]: