Bruce Rowse

A key failure of many carbon abatement policies, a mistake that can be very costly, is the failure to adequately incorporate measurement and verification (M&V), also known as measurement reporting and verification (MRV) into policies that aim to reduce GHG emissions. You can’t manage what you don’t measure.

Most certainly you can’t manage what you don’t measure

Every business measures its income and expenses; but when the business fails to monitor them, the chances are that it will go out of business very soon.

The speed of cars is measured by a speedometer, and it is illegal to drive without this instrument functioning. Speedometers and the laws around speeding have saved countless lives around the world.

Every investor is deeply interested in returns and certainty. When a bank promises to pay an interest of 4%, an investor knows that there is extremely high certainty that they will earn 4% interest. Complex financial instruments that may provide higher returns also have less certainty and are not used by conservative investors. Gambling also has the promise of high returns, but on balance punters get less out than they put in.

Climate change is a serious concern and does not warrant a cavalier response. Yet when it comes to climate change policy, M&V to improve certainty of results and thus optimise policy or quickly ditch bad policy is done too infrequently. Policies are tuned too slowly. The scientific method that I was taught in high school and that dominated my university training – develop a hypothesis, prove the hypothesis, and when the hypothesis is proven, apply it elsewhere – often isn’t used with sufficient rigor. Instead, we have policy by modelling. A consultancy is contracted to model the policy. The consultancy will deem that this policy costs $X per tonne of carbon saved. If the numbers look good and public consultation shows industry support, then the policy is implemented. However, too often, there is inadequate M&V to verify the actual cost-effectiveness of the policy in reality.

As a result, inefficient policies can live on and may only change, or be ditched, when there is a change of government, substantially bad publicity or strong lobbying by industry heavyweights who think they are missing out on some of the pie. On the flip side, good policy may be ditched far too early for the same reasons. In addition, when the policy is debated, all sorts of figures may be thrown around, but often no one really has a clue what the actual cost per tonne of carbon abatement is because it hasn’t been robustly measured!

A typical common policy approach seems to be as follows:

1. Formulate the policy;
2. Develop an economic model;
3. Contract a consulting company to develop the policy detail and to model detailed cost–benefit analysis;
4. Implement the policy;
5. Review it once, or perhaps twice.

The critical flaw in such policy making is that it assumes that the models developed and the detailed cost–benefit analysis are correct. But often, the certainty around the modelling accuracy is low, and in fact, in a world where technological change is happening exponentially, it is extremely hard if not impossible to model with great confidence. The modelling should be considered as a hypothesis; rather, it is treated as the truth.

In other words, the modeller takes an “educated” bet on the outcome, but there is no guarantee of certainty.

The policy approach that I put forth is different in that M&V must be core to the policy, and that policies should initially be rolled out on a small scale with robust M&V before a large-scale roll-out.

The approach is then as follows:

1. Formulate the policy;
2. Model a hypothesis as to the likely costs and benefits of the policy;
3. Develop an M&V plan that shows how the results will be robustly measured and verified so as to provide a high level of certainty as to the short- and long-term carbon abatement that will be achieved;
4. Implement the policy on small scale;
5. Undertake measurement and verification and produce an M&V report;
6. Compare the results from the M&V report with what was originally hypothesised. In light of the M&V findings, then either completely abandon or tune the policy. In tuning the policy:
   1. Develop an M&V plan for the changed policy;
   2. Implement the policy;
   3. Undertake M&V and report on;
   4. Undertake further tuning of the policy on a periodic basis.

Policy by modelling is a linear approach with few feedback loops to gauge real policy impact or effectiveness, whereas policy with strong M&V is iterative with an emphasis on maximising real benefits.

Policy by modelling is similar to taking a bet at the races.

Policy with robust M&V is like putting your money on term deposit.

Modelled savings versus actual savings – and why modelling should be treated as a hypothesis

As an energy auditor, I have a great deal of experience in estimating or modelling the likely savings that will arise out of an investment in energy efficiency. I have audited over a thousand buildings and produced hundreds of energy audit reports.

An energy audit is essentially a business case for investment in energy efficiency. It lists a range of measures, and for each measure shows the estimated cost, annual savings, payback and annual greenhouse gas (GHG) savings. It could be considered as a form of micro-modelling from a policy perspective.

It is extremely hard to model with a great deal of accuracy. Experience helps. In the following, I list some of the difficult learning experiences that I have been through as an energy auditor. What I’ve gleaned from these learning experiences has no doubt helped me improve my accuracy, but nonetheless it is still extremely difficult to estimate energy and GHG savings from many energy-efficiency measures with a high degree of confidence:

• On one occasion, where I under-estimated the cost of a lighting controls upgrade in a school by 40%, I ended up doing a lot of work for free and paying contractors out of my own pocket to get the work complete.
• On another, I overestimated savings from a lighting upgrade by 50%.
• While in another instance, I guaranteed savings of 7% in a tender bid from installing a voltage optimisation unit. Fortunately, I lost that one. The company that won the job and put in technology similar to what I would have installed, only achieved a 5% saving. Phew! I was out by a factor of 40% in my estimate!
• On another occasion, the advice I approved resulted in a $350,000 investment in a cogeneration unit. The expected annual cost savings were $27,000 but the actual cost savings were $0. The carbon savings were closer to the estimate. But clearly, this was not cost-effective carbon abatement.

And I am not the only energy auditor who can get it wrong. A study by Texas A&M when evaluating the work of pre-qualified energy auditors 5 years after projects had been implemented found that measured cost savings on average, across 24 projects, were 25.1% lower than estimated. In some cases, savings were as little as 5.5% of what was estimated![1]

If energy auditors, who understand the details of technology and undertake site-specific investigations, can be out by 25% or more on individual projects, how accurate can policy modelling be? Modelling is usually based on a number of assumptions, possibly a small number of case studies, and the results are then assumed to apply to a large number of buildings. It is generally far less rigorous than an energy audit – an energy audit can account for the diversity in an individual building, but modelling needs to somehow account for the diversity across a whole range of buildings.

Policy modelling should only be treated as a hypothesis. Robust M&V is needed to determine the effectiveness of carbon policy with a high degree of certainty.

In my book Carbon Policy – How robust measurement and verification can improve policy effectiveness, I show how policy makers can effectively incorporate  M&V into carbon abatement policy to provide much greater certainty of policy outcomes.

---

[1] As reported in: Hansen S. and Brown J., Investment Grade Energy Audit: Making Smart Energy Choices, 2004, The Fairmont Press Inc.