Monthly Archives: June 2013

Why M&V is so important to carbon policy effectiveness

Bruce Rowse

A key failure of many carbon abatement policies, a mistake that can be very costly, is the failure to adequately incorporate measurement and verification (M&V), also known as measurement reporting and verification (MRV) into policies that aim to reduce GHG emissions. You can’t manage what you don’t measure.

Most certainly you can’t manage what you don’t measure

Every business measures its income and expenses; but when the business fails to monitor them, the chances are that it will go out of business very soon.

The speed of cars is measured by a speedometer, and it is illegal to drive without this instrument functioning. Speedometers and the laws around speeding have saved countless lives around the world.

Every investor is deeply interested in returns and certainty. When a bank promises to pay an interest of 4%, an investor knows that there is extremely high certainty that they will earn 4% interest. Complex financial instruments that may provide higher returns also have less certainty and are not used by conservative investors. Gambling also has the promise of high returns, but on balance punters get less out than they put in.

Climate change is a serious concern and does not warrant a cavalier response. Yet when it comes to climate change policy, M&V to improve certainty of results and thus optimise policy or quickly ditch bad policy is done too infrequently. Policies are tuned too slowly. The scientific method that I was taught in high school and that dominated my university training – develop a hypothesis, prove the hypothesis, and when the hypothesis is proven, apply it elsewhere – often isn’t used with sufficient rigor. Instead, we have policy by modelling. A consultancy is contracted to model the policy. The consultancy will deem that this policy costs $X per tonne of carbon saved. If the numbers look good and public consultation shows industry support, then the policy is implemented. However, too often, there is inadequate M&V to verify the actual cost-effectiveness of the policy in reality.

As a result, inefficient policies can live on and may only change, or be ditched, when there is a change of government, substantially bad publicity or strong lobbying by industry heavyweights who think they are missing out on some of the pie. On the flip side, good policy may be ditched far too early for the same reasons. In addition, when the policy is debated, all sorts of figures may be thrown around, but often no one really has a clue what the actual cost per tonne of carbon abatement is because it hasn’t been robustly measured!

A typical common policy approach seems to be as follows:

  1. Formulate the policy;
  2. Develop an economic model;
  3. Contract a consulting company to develop the policy detail and to model detailed cost–benefit analysis;
  4. Implement the policy;
  5. Review it once, or perhaps twice.

The critical flaw in such policy making is that it assumes that the models developed and the detailed cost–benefit analysis are correct. But often, the certainty around the modelling accuracy is low, and in fact, in a world where technological change is happening exponentially, it is extremely hard if not impossible to model with great confidence. The modelling should be considered as a hypothesis; rather, it is treated as the truth.

In other words, the modeller takes an “educated” bet on the outcome, but there is no guarantee of certainty.

The policy approach that I put forth is different in that M&V must be core to the policy, and that policies should initially be rolled out on a small scale with robust M&V before a large-scale roll-out.

The approach is then as follows:

  1. Formulate the policy;
  2. Model a hypothesis as to the likely costs and benefits of the policy;
  3. Develop an M&V plan that shows how the results will be robustly measured and verified so as to provide a high level of certainty as to the short- and long-term carbon abatement that will be achieved;
  4. Implement the policy on small scale;
  5. Undertake measurement and verification and produce an M&V report;
  6. Compare the results from the M&V report with what was originally hypothesised. In light of the M&V findings, then either completely abandon or tune the policy. In tuning the policy:
    1. Develop an M&V plan for the changed policy;
    2. Implement the policy;
    3. Undertake M&V and report on;
    4. Undertake further tuning of the policy on a periodic basis.

Policy by modelling is a linear approach with few feedback loops to gauge real policy impact or effectiveness, whereas policy with strong M&V is iterative with an emphasis on maximising real benefits.

Policy by modelling is similar to taking a bet at the races.

Policy with robust M&V is like putting your money on term deposit.

Modelled savings versus actual savings – and why modelling should be treated as a hypothesis

As an energy auditor, I have a great deal of experience in estimating or modelling the likely savings that will arise out of an investment in energy efficiency. I have audited over a thousand buildings and produced hundreds of energy audit reports.

An energy audit is essentially a business case for investment in energy efficiency. It lists a range of measures, and for each measure shows the estimated cost, annual savings, payback and annual greenhouse gas (GHG) savings. It could be considered as a form of micro-modelling from a policy perspective.

It is extremely hard to model with a great deal of accuracy. Experience helps. In the following, I list some of the difficult learning experiences that I have been through as an energy auditor. What I’ve gleaned from these learning experiences has no doubt helped me improve my accuracy, but nonetheless it is still extremely difficult to estimate energy and GHG savings from many energy-efficiency measures with a high degree of confidence:

  • On one occasion, where I under-estimated the cost of a lighting controls upgrade in a school by 40%, I ended up doing a lot of work for free and paying contractors out of my own pocket to get the work complete.
  • On another, I overestimated savings from a lighting upgrade by 50%.
  • While in another instance, I guaranteed savings of 7% in a tender bid from installing a voltage optimisation unit. Fortunately, I lost that one. The company that won the job and put in technology similar to what I would have installed, only achieved a 5% saving. Phew! I was out by a factor of 40% in my estimate!
  • On another occasion, the advice I approved resulted in a $350,000 investment in a cogeneration unit. The expected annual cost savings were $27,000 but the actual cost savings were $0. The carbon savings were closer to the estimate. But clearly, this was not cost-effective carbon abatement.

And I am not the only energy auditor who can get it wrong. A study by Texas A&M when evaluating the work of pre-qualified energy auditors 5 years after projects had been implemented found that measured cost savings on average, across 24 projects, were 25.1% lower than estimated. In some cases, savings were as little as 5.5% of what was estimated![1]

If energy auditors, who understand the details of technology and undertake site-specific investigations, can be out by 25% or more on individual projects, how accurate can policy modelling be? Modelling is usually based on a number of assumptions, possibly a small number of case studies, and the results are then assumed to apply to a large number of buildings. It is generally far less rigorous than an energy audit – an energy audit can account for the diversity in an individual building, but modelling needs to somehow account for the diversity across a whole range of buildings.

Policy modelling should only be treated as a hypothesis. Robust M&V is needed to determine the effectiveness of carbon policy with a high degree of certainty.

In my book Carbon Policy – How robust measurement and verification can improve policy effectiveness, I show how policy makers can effectively incorporate  M&V into carbon abatement policy to provide much greater certainty of policy outcomes.

[1] As reported in: Hansen S. and Brown J., Investment Grade Energy Audit: Making Smart Energy Choices, 2004, The Fairmont Press Inc.


carbon policy lists

Two sources providing listings of carbon policies are:

Why measure carbon policy effectiveness?

Exploring the topic of carbon policy effectiveness makes the rationale for measuring the effectiveness of carbon policy self evident.

The effectiveness of a policy is dependent on the environmental benefit with respect to the economic costs and benefits of the policy.

I consider the cost of a carbon policy the total cost of the policy to society, which includes government investment in developing and administrating the policy (and in some cases, actually financing the policy) and the costs imposed by the policy either directly or indirectly on households and business. Often policy seeks to leverage household or business investment.

The financial benefits of a carbon policy are the benefits to society, such as reduced power bills. The key environmental benefit is the reduction in GHG emissions, although there may be other environmental benefits also of high importance, such as reduced air pollution because of a reduction in the combustion of fossil fuels.

Can the transaction costs arising from a policy be considered a benefit? For example, under a white certificate scheme, electricity retailers may pass on their costs, with a mark-up, to consumers. These costs actually then reduce the financial benefits to end users – continuing the example of a white certificate scheme, the power bill savings from reduced energy consumption may be partially, wholly or more than offset by the increase in electricity tariffs. Can the profit made by the electricity retailer be considered a benefit? It could be considered an economic benefit (i.e. to shareholders of the retailer), but this economic benefit comes at the cost of others (energy consumers). It adds nothing to the environmental benefit. Whilst transaction costs are unavoidable, the larger the transaction costs for a given amount of carbon saved, the greater the cost of abatement.

Therefore, policy effectiveness, in my view, comes back to the total cost, including the transaction costs, per tonne of carbon saved. The lower the total cost per tonne of GHG emissions saved, the more effective the policy. With all other factors considered equal, this also translates into greater real economic benefit.

The expected or understood cost of carbon abatement may be misrepresented because the transaction cost of the carbon abatement is not included in the analysis. For example, marginal abatement cost curves, which have a focus on individual technologies, do not show the transaction costs of achieving the abatement. Different policy types may have different anticipated and unanticipated transaction costs. Thus the marginal abatement cost curve can be policy specific.

Policy effectiveness is important for a number of reasons.

First, carbon policy fundamentally aims to reduce carbon emissions, and the less this costs, the greater the amount of carbon that can be saved within a given investment.

Second, an ineffective policy does not generate the spin-off effects of an effective policy. For example, a policy that does not effectively reduce the energy used to heat and cool a home does not improve comfort as much as it otherwise would have, and in addition, does not create national competitive advantage in being able to export skills, services and products that effectively reduce home heating and cooling requirements.

Third, ineffective policy means carbon abatement is not an investment that can provide a financial return on investment, but rather becomes an expenese. Many carbon abatement technologies now exist that can, if applied efficiently, provide a reasonable financial return on investment. However, ineffective policy can result in poor or no financial return on investment.

Fourth, ineffective policy reduces the political capital that is so important to reducing carbon emissions. Ineffective policy damages credibility.