Step six: assess risks and adjust for optimism bias

Introduction to risk

2.6.1 In appraisals, there is always likely to be some difference between what is expected and what eventually happens, because of biases unwittingly inherent in the appraisal and material risks and uncertainties.

2.6.2 The analysis of risks and uncertainties is a key element in appraisal. The Northern Ireland Audit Office (NIAO) has stated that appraisals must be based on sound and realistic assumptions, and costings that address risks and uncertainties. (Grants Paid to Irish Sport Horse Genetic Testing Unit Ltd, NIAO report HC 396, para 52).

The analysis has four broad purposes:

1. to adjust assumptions about costs, benefits and timing to allow for optimism bias
2. to inform decisions on how best to manage risks, by drawing attention to risk factors which require particularly careful monitoring and management, and enabling suitable risk management measures to be built into the project plan
3. to decide how best to allocate risks between the public and private sectors
4. to inform the option selection decision, by examining how risks and uncertainties affect net present values (NPVs) and the balance of advantage between options

2.6.3 This section of NIGEAE concentrates on basic principles relating to items 1 and 2 above. It covers the following elements of risk analysis:

• identify and analyse risks
• adjust for optimism bias
• risk management and risk reduction strategies

Item 3 above is chiefly relevant to public private partnership (PPP) proposals. There is extensive separate guidance on PPP issues, including the treatment of risks. See section five of NIGEAE for more details. Item 4 is chiefly about sensitivity analysis, which is dealt with under step eight.

Identify and analyse risks

2.6.4 Risk arises because of the possibility of more than one outcome occurring. This possibility may exist because, for example:

• construction costs depend on ground conditions, or the weather
• operating costs depend on the success of a new technology
• the demand for an outcome or output depends on future incomes
• there are uncertainties about future wage or fuel costs, or changes in consumers' tastes, or competition from other suppliers

2.6.5 No matter how robust the assumptions about these and other factors, there will still generally be risks to consider, and there will be uncertainty over the range of possible outcomes.

2.6.6 A distinction may be drawn between a risk, which is measurable and has a known or estimable probability, and an uncertainty, which is more vague and of unknown probability. In practice, this distinction is not always clear cut however NIGEAE uses the term risk in step six and uncertainty in step eight.

2.6.7 Variability, or variance, is the spread of possible outcomes around an expected outcome. All projects have a range of possible outcomes, although the range will be wider, and variability more important, for some cases than for others. For instance, the range may be wider for cases involving new technology.

2.6.8 In general, for proposals where the benefits and costs accrue to the community as a whole, the cost of variability is so small relative to the margin of error in appraisal or evaluation as to be negligible, and government may be regarded as risk neutral. In other words, decisions should be based on expected outcomes, not their variability.

2.6.9 However, there may be some situations where it is appropriate to take account of variability as well as expected outcomes and be risk averse i.e. be inclined to reject certain risky investments despite positive expected outcomes. For instance, this may apply where risk is:

• concentrated: that is where risks are large relative to the income of the section of the population that must bear them
• systematic: that is where variability is correlated with income; for example, where better outcomes from a project or policy are likely to accrue in good times and worse in bad times

2.6.10 If a set of circumstances or course of action or inaction could lead to a very adverse outcome, even if with a very small probability, action to avoid that outcome may be appropriate. This is the 'precautionary principle'. It also applies where the probability attaching to a possible outcome involving significant harm is uncertain. In such circumstances, it may be appropriate to consider precautionary action specifically to mitigate or avoid the risk of the particular outcome. An example would be restrictions on movements of people and cattle during an outbreak of foot and mouth disease.

2.6.11 Where a project or programme risks potentially irreversible consequences (for example, it would rule out important subsequent investment opportunities or a use of resources that might subsequently be preferred) this should be carefully appraised. Examples of irreversibility are destruction of natural environments or historic buildings.

2.6.12 Some projects expose the government to contingent liabilities; that is commitments to future expenditure if certain events occur. These should be appraised (and monitored if the proposal goes ahead). One class of contingent liabilities is the cancellation costs for which the government body may be liable if it terminates a contract prematurely. Such liabilities, and the likelihood of their coming about, must be taken into account in appraising the initial proposal. Redundancy payments fall into this category, but as the wider social and economic consequences of these should also be assessed, advice from economists should be sought.

2.6.13 A vital first step in the analysis is to identify and analyse the important risks and uncertainties relevant to the case, and to show how they compare under each option. This risk analysis should help inform the adjustments for optimism bias and identification of risk management and reduction measures (see below). It is good practice to summarise the relevant information in a tabulation, called a risk log or risk register, which identifies each relevant risk and compares how it impacts upon each option. This should cover not only the economic risks and uncertainties, such as possible variations in cost/benefit assumptions, but also relevant managerial, legal, financing and other risks and uncertainties. Textual description should be accompanied by quantification where possible. A risk register produced at this stage should form part of an overall risk management strategy, which sets out how identified risks will be proactively controlled and what actions should be taken if risks materialise.

A risk log should contain the following information:

• risk number (unique within register)
• risk type
• author (who raised it)
• date identified
• date last updated
• description
• likelihood
• interdependencies with other sources of risk
• expected impact
• bearer of risk
• countermeasures
• risk status and risk action status

2.6.14 The NICS Centre of Expertise website provides a useful template for completing a risk log and contains further information on risk identification and management. In particular it recommends the use of the management of risk approach, which provides for a very comprehensive approach to risk management.

Adjusting for optimism bias

2.6.15 There is a demonstrated, systematic tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both the private and the public sectors. Many project parameters are affected by optimism; appraisers tend to overstate benefits and understate timescales and costs, both capital and operational. It may occur, for example, through failing to reflect fully the chances of cost underestimation or time overruns; or by including projections of demand that are too generous.

2.6.16 To redress this tendency, appraisers should make explicit adjustments and thus determine a suitably optimism bias-adjusted NPV for each option. These adjustments will have the effect of increasing the cost estimates, decreasing the projected benefits and extending the timescales over which the costs and benefits are assumed to accrue, compared to the initial unadjusted estimates for each option.

2.6.17 NI departments should follow the guidance in annex four of the Green Book on how to deal with optimism bias in relation to capital works, works duration, operating costs, and under-delivery of benefits; and on how to prevent or minimise optimism bias. Further Her Majesty's Treasury (HMT) guidance, including adjustment percentages for generic project categories based on a study by Mott MacDonald, is given in supplementary guidance on the treatment of optimism bias on the HMT website.

2.6.18 The principles in annex four of the Green Book (page 79-90 of document) and in the HMT supplementary guidance should be applied with proportionate effort in a manner that suits the circumstances of NI departments. Wherever possible, the relevant adjustments should reflect local experience in preference to use of the HMT generic figures. They should be based on data from past projects or similar projects elsewhere, and adjusted for the unique characteristics of the project in hand. When such information is not available, departments are encouraged to collect data to inform their estimates of optimism, and in the meantime use the best available data.

2.6.19 In any case, departments should be satisfied that the adjustments made are realistic and justifiable in relation to local experience. They should represent a meaningful effort to improve the quality of assumptions rather than arbitrary percentage adjustments.

Adjusting for optimism bias step by step

2.6.20 The HMT supplementary guidance provides recommended adjustment ranges and advocates a five step approach to adjustment:

1. decide which project type(s) to use, for example, is it a building or civil engineering project? Is it 'standard' or 'non-standard'?
2. always start with the upper bound ie select the appropriate upper value from the recommended range of adjustments
3. consider whether the optimism bias (OB) factor can be reduced according to the extent to which the contributory factors have been managed and mitigated
4. apply the OB factor ie multiply the relevant cost or benefit estimate by the OB factor and add the result to the estimate
5. review the OB adjustment and reduce it where this is justified by further mitigation of contributory factors

See the HM Treasury Supplementary Guidance on Optimism Bias for a detailed explanation of these steps including worked examples.

2.6.21 Adjustment for OB should not be a one-off event in the life of a project. In fact it should be regarded simply as a starting point for dealing with appraisal optimism. As a project proceeds, risk management should be applied to mitigate factors identified as contributing to appraisal optimism, allowing the adjustment factors to be amended accordingly. Thus the optimism bias-adjusted (or "OB-adjusted") NPV for each option should be refined over time.

2.6.22 It follows that business cases should show evidence of progressive reduction in the adjustments made for OB. In general, the allowances for OB should be largest in a high-level business case (for instance a strategic outline case), smaller in a more detailed business case (such as an outline business case), and smallest in a fully developed business case (a full business case). Large allowances for optimism bias will not normally be acceptable in a full business case. If optimism bias remains high at full business case stage, approval may be withheld pending further research, costing and risk management.

2.6.23 It is plausible that some high-level business cases might have fairly low allowances for OB, where for instance there is a long track record of successful projects (say, if building the 8th of 10 standard workspace units) or if the project is concerned with off-the-shelf products or where there are published prices, for example,  buying a fleet of cars or vans.

2.6.24 OB must be assessed individually for each option. Different options can be expected to be subject to different risks and as a result different levels of OB. For example, accommodation options involving new build on an unidentified site will have different risk factors than options involving, say, new build on a known site, refurbishment, rent and fit-out, purchase and fit-out, or temporary accommodation.

2.6.25 Double counting of risk allowances, by including both an OB adjustment and a contingency allowance to cover the same risk, should be avoided. For instance, optimism bias adjustments should normally be applied to capital costs excluding contingency allowances.

2.6.26 OB adjustments should be used only to allow for underestimates of costs and time, and overestimates of benefits. They should not be used to allow for enhancements to the project scope or output specification.

2.6.27 Where costs consist mostly of revenue costs (for instance, staff costs) and capital expenditure is low in both absolute terms and relative to the NPC then there may be little value in adjusting costs for OB. However, in such cases the appraisal should state the rationale behind any decision not to adjust for OB to demonstrate that it has at least been considered. This would not affect the need to consider OB adjustments for project duration and benefit shortfall.

2.6.28 Some costs and other assumptions may not be amenable to adjustment for OB because the uncertainty surrounding them is due more to market conditions or the general economic climate than optimism. For example, rents assessed by LPS are based upon current market conditions but rents may go up or down in the future due to demand and supply in the market. Such assumptions should be tested using sensitivity analysis (see step eight).

2.6.29 The funding provision for a project including contingency allowances should be sufficient to cover expenditure after adjustment for OB. As OB adjustments are reviewed and reduced during the development of a business case, the corresponding estimates of required funding should be revised accordingly.

2.6.30 Business cases should demonstrate that OB has been considered and that appropriate adjustments have been made. Full details of the adjustments made including all calculations should be shown. Where OB adjustments are reduced on account of mitigation of contributory factors, the reductions should be explained and clear evidence supporting the mitigations should be provided.

Risk management and risk reduction strategies

2.6.31 Active risk management strategies should be adopted for the appraisal and implementation of large policies, programmes or projects, and risk management principles should be applied to smaller proposals. Before and during implementation, steps should be taken to prevent and mitigate both risks and uncertainties. It is important to be transparent with sponsors about the potential impact of risks and biases in proposals.

2.6.32 Risk management is a structured approach to identifying, assessing and controlling risks that emerge during the course of the policy, programme or project lifecycle. It involves a series of well-defined steps to support better decision making through good understanding of the risks inherent in a proposal and their likely impact. Risk management includes:

• identifying possible risks in advance and putting mechanisms in place to minimize the likelihood of their materialising with adverse effects
• having processes in place to monitor risks and access to reliable up to date information about risks
• the right balance of control in place to mitigate the adverse consequences of the risks, if they should materialise
• decision-making processes supported by a framework of risk analysis and evaluation

2.6.33 Following identification and analysis of risks, generation of a OB adjusted expected NPV, and assessment of options' exposure to uncertainty, appraisers need to consider and adopt strategies to prevent and mitigate risks and uncertainties. Steps to be considered include:

• early consultation: to identify needs at the outset and avoid costs increasing later due to poor initial understanding of requirements
• deferring irreversible decisions: to allow more time to investigate mitigating measures or alternative ways to achieve objectives
• pilot studies: to acquire more information about risks and take steps to mitigate adverse consequences or increase benefits
• design flexibility: increasing the flexibility of designs to make proposals more robust against changes in future demand
• taking precautionary action: to reduce the risk of a very bad outcome, even where the probability is considered small
• transferring risk to the private sector: through contractual arrangements; insurance, for example. The Green Book elaborates on ways to transfer risk
• making less use of leading edge technology: where simpler methods can reduce risk considerably
• reinstating or developing different options: where risk analysis suggests this is worth doing
• commissioning research: to confirm or disprove the reliability of new technology, or to reassess the nature of a danger
• undertaking site investigations: to reduce risks from unforeseen ground conditions or refurbishment costs
• staging a project: so that it can be altered at successive review points
• abandoning the project: because it is too risky

2.6.34 Business cases should report on all activity undertaken regarding Identification and analysis of risks and uncertainties, adjustment for OB, assessment of uncertainty, risk management and risk reduction strategies.