Measuring Policy Influence

With such a complex knowledge and policy ecosystem, how can CSOs, think tanks and researchers ensure that their activities are responsible for influencing policy? How do they measure it, and what is the basis of this measurement? We can measure the influence of activities within our control, and at the same time conduct tracking of policy change, but proving the extent to which these two are connected is no walk in the park. It requires an integrated and proper monitoring and evaluation system to measure the influence of specific activities on policy change.

Measuring Policy Influence

By: Dini Rahim

 

Success in influencing policy is the dream for all donors, civil society organisations (CSOs), researchers and think tanks working in advocacy. This is because influencing policy is believed to be a strategic path to achieving the sustainability of a system. The number of opportunities to influence policy increases as participative and evidence-based policy approaches become more accepted. This provides room for all actors in the knowledge sector to improve their participation in influencing policy.

The nature of policy influence is ‘multifactorial’, ‘non-linear’, ‘complex’ (Ramalingam et al., 2008), or, to borrow a term from Jones (2011), ‘highly conflicting’. Its success is determined by many factors and actors in the ecosystem. The RAPID framework (Crewe & Young, 2002) reveals four components of influence: evidence, relationships with policyholders and other actors, socio-political context, and external influence. Just relying on one factor is not enough. Influencing policy is not easy work, but it is not impossible. Achieving it, according to Jones (2011), requires policy entrepreneurs who have expertise in building networks and creatively packaging research agendas into policy issues, who are strategic in conducting advocacy, and who understand the needs of policymakers. Also, understanding the structure and distribution of power, the value system in said context, and having the ability to seize the ‘window of opportunity’ are important parts of this success. Ed Laws and Heather Marquette, in their book Thinking and Working Politically (2018), said that no matter how good the proposed evidence is, if it does not take into account the political context in which it operates, it will be very difficult to ensure its uptake and use. They mentioned five factors that determine the success of Knowledge to Policy (K2P): understanding the political context, the speed of the response to this context, the adaptability of the program design and implementation, the ability to provide various credible sources of evidence for policymakers, and networking with other actors. From here we can see that understanding the political context for policy advocacy actors is crucial because, along with other factors, it determines the feasibility, appropriateness, and effectiveness of a program.

 

Challenges in monitoring

With such a complex knowledge and policy ecosystem, how can CSOs, think tanks and researchers ensure that their activities are responsible for influencing policy? How do they measure it, and what is the basis of this measurement? We can measure the influence of activities within our control, and at the same time conduct tracking of policy change, but proving the extent to which these two are connected is no walk in the park. It requires an integrated and proper monitoring and evaluation system to measure the influence of specific activities on policy change.

 

Program logic as a program planning and monitoring tool

One of the monitoring and evaluation approaches often used to ensure a logical causality relationship between input, activity, outcome, and context is program logic or program theory. There are multiple advantages to program logic. With the proper monitoring questions, it can be used to measure outcomes in every stage of a process (immediate, intermediate and long term), and it also becomes a systematic program planning tool that helps to build a story on the change-influencing process, ‘from here to there’ (Nash, Hudson & Lutrell, 2006), from inputs to activities to outputs to outcomes. Program logic provides the description of this journey using program assumptions in the form of ‘if’ and ‘so’, which are stated explicitly with regard to the available evidence.

KSI chose to use program logic as a strategic tool to plan, implement, and monitor the Strategic Partnership Grants (SPG) program, partnering with 16 public research institutes (PRIs) located in Jakarta, Jogjakarta, and Bandung (Hind, 2019). In developing the program logic, KSI and individual PRIs discussed the program together from the planning stage, established achievements, and ensured sufficient activities and quality outputs. PRIs were also asked to map and analyse stakeholders, including media organisations, that were influential to the program and develop an engagement strategy with them. This was a participative program, involving not only the program manager but also the M&E team from both parties. The KSI team played a role as a critical ally for PRIs in the discussion process to develop this program logic. Through this process, the monitoring framework and indicators from each stage (outputs, outcomes) that would be considered during the program period were agreed upon. As a part of the SPG monitoring activity, KSI conducted periodic meetings to follow-up on the program’s progress, and scheduled a special reflection session with each PRI to reflect on activities, examine lessons learned, and identify opportunities for improvement. This series of processes continued to be conducted even during the COVID-19 pandemic.

It was anticipated that adaptation would be necessary during the implementation of the program. Several PRIs experienced this need a few months after the program was rolled out. The planned objective stated in the program logic turned out to be difficult to achieve due to changes beyond their control, including the changed context, and the need to update relationships with policymakers due to promotions, restructuring, and so on. The program needed to adapt. During the COVID-19 pandemic, these changes became even more inevitable. Some PRIs still went down to the field using tight health protocols, some had to be flexible and use online methods, and some even had to postpone activities. As a result, outputs needed to be adjusted, timelines shifted, and outcomes re-stated. Program logic had the flexibility to accommodate these changes, and this became a part of our adaptive program management practice.

 

Working with PRIs

The experience of working together with 16 PRIs over the last three years has shown that KSI’s working partners have improved their understanding and capability when it comes to using program logic. Initially, introducing this concept was not an easy process. During the first year, it was apparent that all of the PRIs were finding it difficult to understand how this approach worked in their program. This was understandable. A lot of time was spent conducting back-and-forth discussions between the KSI and PRI teams to ensure that the program logic had captured the program objective and fulfilled feasibility, appropriateness, and effectiveness requirements. Implementing the program to achieve the intended outcomes was a challenge, as was visualising it in a program logic diagram to demonstrate the causal relationship between activities and outcomes. Some PRIs mentioned that the process to develop program logic under the SPG program was something new to them, thus needing more intensive and detailed thinking compared to the previous stage.[1]However, the program logic discussions helped planning to become more strategic and long-term, even making it possible for it to continue after KSI is completed.[2] Some of the other PRIs admitted that discussing program logic ‘made us feel challenged’, ‘made us see more logically’, ‘enabled us to exchange ideas about program possibilities’ and helped ‘accountability and transparency in implementing the program from early on’.[3] Some PRIs also mentioned that the stakeholder map and analysis helped them to identify stakeholders and make the program more targeted and focused.[4]The improved understanding and expertise of PRIs in developing program logic was seen in the second and third years of the partnership, when KSI asked each PRI to submit a proposal for the next year in the form of a diagram, before producing a written proposal. The use of terms such as ‘uptake and use’ has become more frequent when discussing the expected form of policy influence. This year, a lot more PRIs have shown confidence in using program logic as a program planning and management tool. From this experience, we can see that measuring policy influence in complex political conflicts is not an impossibility for CSOs, researchers and think tanks that want to convert their evidence-based scientific research into useful and sustainable policy advocacy. With a systematic and effective measurement tool, this process can be achieved, and going forward, we hope this strategy will be used by other advocacy programs.

 

Surabaya, April 2021

 

[1] From several discussion series with PRIs.

[2] Ibid 1.

[3] From reflection sessions with PRIs.

[4] From program logic discussions with PRIs.

  • Share: