Project evaluation, sustainability and accountability: combining cost-benefit analysis (CBA) and multi-criteria analysis (MCA)

General abstract Decision-makers in governments and businesses must choose among different project alternatives which, in varying degrees, contribute to sustainability. Decision-makers also have to account for their choices to a large audience or a broad range of stakeholders. This thesis is about the positive and negative aspects of using the main judgement-oriented evaluation tools of Cost-Benefit Analysis (CBA) and Multi Criteria Analysis (MCA) together in the context of sustainability and accountability. An integration of CBA and MCA as a fruitful addition to the evaluator’s toolbox within a sustainability context is presented in this thesis. A new tool, ‘MCCBA’, stands for ‘multi criteria cost benefit analysis’. The acronym reveals that the approach is a new combination of existing elements of CBA and MCA, rather than something entirely new. The common ground between the two methods, which allows for the integration of CBA and MCA, is that full judgement is often impossible in a sustainability context, and that ‘decision aid’ is therefore the realistic aim for any evaluation tool; structuring and framing decision-making is central to the MCCBA approach. After presenting CBA and MCA and their integration into MCCBA, the thesis discusses three MCCBA case studies. The main aim of the case studies is to provide insight into the applicability of MCCBA. The selection of the cases was made because, taken altogether they highlight the crucial strengths of MCCBA, which are not normally tackled by either CBA or MCA. Practical essentials The main practical essentials of the MCCBA approach can be encapsulated as advice to two target groups, experts/practitioners and laymen/users. The main points of advice to the expert group are: 1. Do not give ‘best’ solutions to problems and do not accept preferences too easily as ‘given.’ 2. Sustainability should not be about everything, it should be economic development of the planet and its people in different places, but above that, only issues about which there is some consensus that global level outcomes are greatly worrying should matter: environmental degradation and extreme poverty are definitely cases in point. 3. Work with two evaluation scale levels: a decision-making and a global level. 4. Involve stakeholders, but focus on involving stakeholders with strongly different views, rather than on involving as many stakeholders as possible. 5. Starting from the impact matrix, use both CBA and MCA techniques for aggregation. Do not decide alone how to aggregate: try to achieve a consensus among your stakeholders on which aggregation is easiest to understand. 6. Take pride in interpretation of sub-aggregated outcomes. 7. In the final stage of your evaluation, address explicitly the question of whether increased problem understanding stipulates that new project alternatives should be considered or old alternatives should be combined. The second target group consists of decision-makers and stakeholders. To this laymen ‘user’ group, the main points of advice arising out of the MCCBA approach are: 1. If you think that expert evaluators can judge easily whether a project is sustainable or not, that is a wrong assumption. You will have to think together. 2. You might think that judging sustainability is about looking into the distant future. Perhaps. But be aware; sustainability surely requires that you scrutinize your project from a high international or even global level – next to examining it from your ‘own’ level, certainly. Good courses of action should preferably be good at both levels. 3. It is normal, even essential, that you understand how the evaluation results developed – although you need not understand every single detail. 4. You may think that it is best to involve as many stakeholders as possible in an evaluation. However, achieving a consensus on the evaluation outcomes with all stakeholders is illusory. A consensus about easily understood ways to measure the most significant impacts is, however, a realistic goal. To that purpose, the involvement of a small group of stakeholders with widely differing views on a project seems more fruitful. Complete Summary (original) Introduction Sustainability has become a high profile objective. Decision-makers in governments and businesses must choose among different project alternatives which, in varying degrees, contribute to sustainability. Decision-makers also have to account for their choices to a large audience or a broad range of stakeholders. This thesis is about the positive and negative aspects of using the main judgement-oriented evaluation tools of Cost-Benefit Analysis (CBA) and Multi Criteria Analysis (MCA) together in the context of sustainability and accountability. These evaluation tools are used for what might be called “accountable project evaluation in a sustainability context” or, put more simply, using them in the ‘sustainability context’. Accountability presents a major challenge to the development of evaluation tools. If the demands of accountability are neglected in ‘open’ decision-making processes, the risk looms large that the evaluation will either be judged as irrelevant or unwarranted interpretation of results will occur. The main challenge of this thesis with regard to accountability is to improve the communicative quality of the judgement-oriented evaluation tools. Sustainability also requires unique qualities from evaluation tools. Sustainability has been defined in various – sometimes confusing and vague - ways. This thesis gives an operational content to sustainability by identifying key elements of the concept. Sustainability evaluation – which enhances accountability - requires: -Triple E assessment (that is, evaluation of impacts on Economic development, Extreme poverty and Environmental degradation); -evaluation at both global and decision-making levels; -evaluation over a (long period of) time; -evaluation which increases everyone’s understanding (including awareness about limitations). After careful examination of CBA and MCA performance within the ‘sustainability context’, the thesis provides both a theoretical basis for a combination of CBA and MCA, and insight into the practical applicability of this combination. MCCBA An integration of CBA and MCA as a fruitful addition to the evaluator’s toolbox within a sustainability context is presented in this thesis. A new tool, ‘MCCBA’, stands for ‘multi criteria cost benefit analysis’. The acronym reveals that the approach is a new combination of existing elements of CBA and MCA, rather than something entirely new. The common ground between the two methods, which allows for the integration of CBA and MCA, is that full judgement is often impossible in a sustainability context, and that ‘decision aid’ is therefore the realistic aim for any evaluation tool; structuring and framing decision-making is central to the MCCBA approach. Three major building blocks are presented to further specify the style of integration. The first is that stakeholder involvement is useful (mainly) for incorporating a spectrum of views and for assessing consensus. The second is that standardisation along a global long-term Triple E criteria structure is needed. The third is the recognition that judgement and measurement in evaluation are closely related. We will discuss the major elements of the MCCBA approach that offer direction to the evaluation at each stage; MCCBA has eight stages indicated below in Table ES-1. Table ES-1 The eight stages of the MCCBA approach MCCBA Stage one: Identify function, project alternatives and scale of the evaluation Stage two: Involve a broad group of stakeholders Stage three: Organise judgement criteria on Triple E impacts Stage four: Quantify impacts physically Stage five: Aggregate monetary scores, consensus based Stage six: Aggregate non-monetary scores, consensus based Stage seven: Interpret trade-offs Stage eight: Perform sensitivity analysis and reconsider project alternatives Stage one: Identify function, project alternatives and scale of the evaluation As to the function of the evaluation, the most the analyst-evaluator can do is to clear as much ground as possible. To undertake this, the analyst works with judgement concepts that are necessarily partial, but are acceptable and clear to stakeholders of the evaluation. As to the definition of project alternatives, the specification of a null-alternative against which project alternatives are assessed would be very useful were it to be standardised. Furthermore, given its functional limitations, MCCBA should by itself be seen as important in finding new alternatives. The MCCBA approach is strict in the evaluation at different spatial levels, as it will help clarify impacts in the sustainability context and prevent unnecessary and unwarranted perceptions of complexity among stakeholders. The global level is standard in the MCCBA approach; it acts as insurance against any tendency to overlook impacts. However, in most evaluations there will be at least one other spatial level closer to the decision-making, such as the regional, national or organisation level. In order to remain practical, the second scale of the analysis should stick closely to that which the decision-makers think is the natural scale for the analysis. Stage two: Involve a broad group of stakeholders In the MCCBA approach involvement of stakeholders should focus primarily on achieving a broad range of perspectives; doing this may act as a mechanism for exploring consensus among stakeholders. Seeking consensus has to start from lower-order measurement and judgement issues and work upwards. The highest-order judgement issues will usually be beyond consensus. Stage three: Organise judgement criteria on Triple E impacts MCCBA in the sustainability context has two standardised requirements that give direction to the organisation of criteria: The first is to define separate Triple E criteria without redundancy and double counting. As judgement and measurement are closely related, one should try to use or develop standardised measurement of the criteria; this standardisation preferably holds on different spatial scales. The second is to analyse and present short and medium-term impacts separately from long-term impacts. For the short and medium-term impacts on economic development, the CBA practice of discounting can be applied, but the other Triple E impacts are shown in their own right. All long-term Triple E impacts (including long-term impacts on economic development) are also shown in their own right. Because of their long-term relevance, these impacts are conceptually on a more equal footing without discounting. Stage four: Quantify impacts physically In this stage the measurement of impacts in their natural dimensions occurs. This ‘fact finding’ phase is probably the most important phase in any evaluation. Stage four ends with the performance matrix, which shows the scores of the alternatives on different criteria. In the sustainability context this performance matrix will usually be too large and thus too complicated to warrant clear interpretations. Stages five and six are therefore crucial, as they reduce the information to a smaller number of dimensions. Stage five: Aggregate monetary scores, consensus based A CBA is performed in this stage of MCCBA for the impacts that fit well within it. The limitation to impacts that ‘fit well’ is the reason why it is ‘consensus based’. What should be included in the CBA part? The legitimacy of the MCCBA approach rests on the notion that CBA is a powerful evaluation tool that relates to everyday economic notions such as efficiency and monetary measurement, thus staying close to market-related valuation. Only those impacts that broad groups of stakeholders regard as well captured within the monetary measurement possibilities of CBA should be fit into CBA. In order to clarify how MCCBA differs from mainstream CBA, the MCCBA approach uses the phrase ‘limited CBA’, which underlines the limited scope of this type of CBA. Nevertheless, this type of CBA has a robust quality. A pure MCA approach often lacks the crucial economic realism that is well captured in CBA. MCCBA’s consensus based valuation with CBA can to some extent use the straightforward but labour-intensive procedure of asking for willingness to pay or willingness to accept monetary values. The further the decision-maker is removed from common everyday economic valuations (e.g., consumer preferences), the less reliable WTP or WTA valuations will be, and the more that MCA-like measurement techniques should be preferred. Stage six: Aggregate non-monetary scores, consensus based In stage six a consensus based MCA is performed on the remaining impacts. The results of the CBA can be seen as a separate MCA branch to be combined with the other branches of remaining impacts. The MCCBA challenge here is twofold: 1) to reduce the number of criteria to a minimum – in a CBA style and 2) to use consensus based judgement criteria and measurement. Ad 1) In the sustainability context the clarity of the judgement concept will be easily blurred when stakeholder involvement leads to huge value trees. To reduce the number of criteria, the MCCBA approach above all derives inspiration from the CBA practice of avoiding double counting and incorporating causality. As with CBA practice, the MCCBA approach uses indicators situated at the end (or at one point) of a causal chain and avoids using intermediate indicators, which would easily lead to double counting. As a final method of reducing criteria, MCCBA captures the most important criteria or criterion and ignores criteria with low weight or unclear or small impacts. Naturally completeness suffers from this procedure but the gain is an increased likelihood of understanding for many stakeholders – including the grasp of the limitations. At this stage the selection of the most important criteria should certainly be contingent on the increased problem understanding brought forward by the evaluation. Ad 2) The second challenge of MCCBA at this stage is to perform an MCA which incorporates use of judgement concepts and measurements that have achieved consensus among broad groups of stakeholders, or about which consensus can be developed. Compared to the performance matrix, what happens here is that several sub-criteria can be aggregated to comprise an overall criterion. Because the higher order valuations are most often disputed, further aggregation is avoided as much as possible in the MCCBA approach. The end best result of the MCCBA is an aggregation of the performance matrix, based on a broad consensus among stakeholders. Stage seven: Interpret trade-offs The major difference between the end result of stages five and six of MCCBA and that of CBA and MCA, is that insufficient judgement is provided. This thesis argues that although this is regrettable, in many instances it will be the optimum and realistically feasible result, because accountability is best served with MCCBA, and the provision of more judgement is likely to become uncertain, thus subjective and contestable. However, although ultimately one may be unable to pass full judgement on the various project alternatives, there are more possibilities for judgement, while one also remains within the confines of an objective and accountable approach. Proceeding in small steps is recommended, as non-involved stakeholders need to understand the process, and the limitations of the analysis should remain clear to all. The goal of stage seven is to interpret the consensus based aggregation of the performance matrix with the aim to analyse trade-offs. Generally, an important method for analysing trade-offs is by means of ratio-analysis on the basis of the aggregated performance matrix. Thus one examines, for instance, net-CBA outcomes per outcome on another criterion (Stewart, 2003). This type of analysis may closely resemble cost-effectiveness analysis. Ratio-analysis may gain in strength if comparisons across projects can be made. Perhaps ratio-analysis standardisation of measurement in project evaluation enhances the possibility for comparison across projects. A clear noteworthy example of this standardised measurement is the Quality Adjusted Life Years (QALY) from health economics. A related type of analysis is stakeholder perspectives analysis, which is to explicitly adopt the perspectives of specific stakeholders in order to assess the importance of different criteria to them – (what if priority were given to this criterion rather than that one or to those?) – and to present a preferred option from the perspectives. Several elements from the MCA toolbox are useful in this analysis; one will sometimes realise that despite choice of perspective, – from a logical set – the result remains the same. Stage eight: Perform sensitivity analysis and reconsider project alternatives The overall aim of stage eight of MCCBA is to share the increased problem understanding developed during the evaluation through an application of sensitivity analysis. The sensitivity analysis may have an important function for accountability; stakeholders can assess how several individual assumptions and estimates work out on the overall outcomes. However, practical experience with sensitivity analysis shows that it can only have a limited role in sharing problem understanding. Therefore, MCCBA uses another powerful tool for sharing the increased problem understanding: it explicitly requests a reconsideration of the project alternatives. The following (types of) questions should be posed: Does the increased problem understanding show that new alternatives should be considered or that existing alternatives should be combined? Questions about the consistency and feasibility of the selected goals in relation to the different project alternatives are relevant: increased problem understanding may also shed light on this. Three MCCBA case studies After presenting CBA and MCA and their integration into MCCBA, the thesis discusses three MCCBA case studies. The main aim of the case studies is to provide insight into the applicability of MCCBA. The selection of the cases was made because, taken altogether they highlight the crucial strengths of MCCBA, which are not normally tackled by either CBA or MCA. Evaluation of the Dutch Ecologische Hoofdstructuur (EHS): The value of combining monetary and non-monetary measurement. This study, conducted in 1995, evaluated the implementation of the EHS (literally: Ecological Main Structure) in the Netherlands within the time period 1990-2020. The EHS implied that 244,000 hectares of new nature were to be developed on what was until then mainly conventional farmland. Evaluation of the German Emssperrwerk: The value of more scale levels The evaluation of this case study was carried out in 1998, a few months before the German government decided to build the Emssperrwerk. The Emsperrwerk had two official purposes, first as a (movable) dam in the event of extremely high water, thus preventing water from the sea threatening the hinterland in the EmsDollard region. Its second function was to facilitate large cruise ships built at the Meyer shipyard in Papenburg (approximately 30 kilometers upstream the river Ems) in order to reach the deep waters of the North Sea. Design and evaluation of Sustainable Corporate Performance (SCP) policy: The value of causal analysis. The final case study of this thesis performed in 2001-2002 discusses Corporate Social Responsibility and sustainability, the combination of which is called Sustainable Corporate Performance (SCP). The Ahold Company, a leading Dutch multinational, was then in the process of formulating its SCP policy. Two closely connected concerns were particularly challenging to this group of quality supermarket and foodservice companies: first, the confusion about what SCP was or should be, and second, the lack of standardisation in measurement of SCP. Lessons from the case studies (per stage) The case studies revealed that in stage one the most important key word is acceptance; that is, acceptance of a change in function and acceptance of the different regional or organisational scales of the evaluation. This acceptance need not be ‘automatic’; stakeholders may have to be persuaded to accept certain points and success is not guaranteed. As for the involvement of stakeholders in stage two, despite a ‘solution’ proposed in chapter 5 to focus merely on a broad group of stakeholders, case studies illustrate that even this type of involvement may remain difficult for various reasons. The Triple E structure, which is proposed as a standardised element of MCCBA in stage three, is only preliminary. If no significant impacts can be found, the structure should be changed and criteria skipped. The global level assessment needed in MCCBA in many cases requires existing expertise and data; otherwise, too much time may be required to estimate impacts. The case studies clearly and concretely acknowledge that stage four is indeed critical in many respects. One especially noteworthy aspect is the role of stakeholders in facilitating access to information and data. Much has been learnt from the case studies about the aggregation of impacts and how it can be made ‘consensus based’. They clearly show that potential consensus only concerns the minimum importance of criteria and ease of understanding measurements. This conclusion relates to stages five and six – to both monetary and non-monetary aggregation. From stage six alone it has become clear that simplicity of the analysis should/may overrule the demand of completeness. Stage seven proved to be very rewarding in the case studies and its value is difficult to overestimate. It may turn out that the the main results of the evaluation are generated during this stage, as it could strongly increase the judgement value of the results from stages five and six. The main conclusion from stage eight is that although the sensitivity analysis can show increased problem understanding – a stumbling block may be the attention it receives when presenting main results. The explicit demand to reconsider project alternatives can potentially remove this stumbling block, but was not yet practiced in the case studies. The main building blocks of the MCCBA approach as identified above are that: 1) stakeholder involvement is useful for obtaining broad views and for checking on consensus; 2) standardisation of a global and long-term Triple E criteria structure is necessary; 3) judgement and measurement are closely related, and in an MCCBA both should be understandable to broad groups of stakeholders. Having discussed the case studies, a general conclusion is that mainly the aspect of the stakeholders is in need of ‘qualification’: stakeholder involvement may also be important for acquiring relevant data, but difficulties remain around fruitful involvement. The other building blocks have proven to be valid in practice. In conclusion, we return to the title of this thesis: ‘Project evaluation, sustainability and accountability – Combining cost-benefit analysis (CBA) and multi-criteria analysis (MCA)’. The MCCBA combination developed in this thesis has proven to be a useful project evaluation tool in the sustainability context. Through its consensus based aggregation, MCCBA improves the communicative quality of the evaluation, thereby enhancing accountability. Through its decision aid style and its emphasis on standardisation, tracking causality, and allowance for simplicity to (here and there) overrule completeness, MCCBA acts as a safeguard against over-ambition, and is a facilitator for clarity, utility and practicality in a sustainability evaluation. To end this summary we will give an overview of the main practical essentials of MCCBA. Main practical essentials of MCCBA How does the MCCBA approach evaluate whether a project is sustainable? How does it assure that accountability is well served? The main practical essentials of the MCCBA approach can be encapsulated as advice to two target groups, experts/practitioners and laymen/users. The first target group comprises the analysts/evaluators, and it is to them that this thesis is primarily aimed. The main points of advice to this expert group are: 1. Do not give ‘best’ solutions to problems and do not accept preferences too easily as ‘given.’ Be keen towards finding ‘possibly better’ solutions; be content to only frame and structure decision-making and remember that in sustainability matters most people will be ‘learning how to think.’ 2. Sustainability should not be about everything. Surely sustainability entails the economic development of the planet and its people in different places, but above that, only issues about which there is some consensus that global level outcomes are greatly worrying should matter: environmental degradation and extreme poverty are definitely cases in point. 3. Work with two evaluation scale levels: a decision-making and a global level. Evaluate the project at both levels (of thinking). In practice you may work with a lower than global level; the global level is merely an indication – to be on the safe side – of that highest level where economic, environmental, or extreme poverty impacts can be found. 4. Involve stakeholders, but focus on involving stakeholders with strongly different views, rather than on involving as many stakeholders as possible. 5. Starting from the impact matrix, use both CBA and MCA techniques for aggregation. Do not decide alone how to aggregate: try to achieve a consensus among your stakeholders on which aggregation is easiest to understand. Furthermore, when aggregating: - Use the consumer-citizen divide of preferences, the first are usually well-captured in CBA, the second best delineated in MCA. - Very reluctantly discount long-term impacts, even if they are economic. Long-term economic impacts are matters of deliberation and learning how to think; they are not matters of mechanical discounting. Preferably use discounting only for short and medium-term economic impacts. - Use MCA scaling and weighing techniques to your benefit, but only in as much as they can be assured of wide consensus. This will often mean not scaling and weighing the highest order criteria. - Unless ‘not-all-that-important impacts’ can be easily aggregated, in the final stages of the evaluation focus only on the most important impacts. 6. Take pride in interpretation of sub-aggregated outcomes. You can: - Calculate various ratios (e.g., cost-effectiveness ratios), - Compare standardized impact scores with other projects (e.g., QualityAdjusted Life Year [QALY]; or GlobalWarmingPotential [GWP] or Naturescores [as in this thesis]) - Analyze outcomes from different stakeholder perspectives (e.g., winners/losers, environmentalists-entrepreneurs). 7. In the final stage of your evaluation, address explicitly the question of whether increased problem understanding stipulates that new project alternatives should be considered or old alternatives should be combined. The second target group consists of decision-makers and stakeholders. To this laymen ‘user’ group, the main points of advice arising out of the MCCBA approach are: 1. If you think that expert evaluators can judge easily whether a project is sustainable or not, that is a wrong assumption. You will have to think together. You may be certain what to do afterwards, but it is likely that the evaluator will not. However, you should all definitively possess greater understanding of the problem situation. 2. You might think that judging sustainability is about looking into the distant future. Perhaps. But be aware; sustainability surely requires that you scrutinize your project from a high international or even global level – next to examining it from your ‘own’ level, certainly. Good courses of action should preferably be good at both levels. 3. It is normal, even essential, that you understand how the evaluation results developed – although you need not understand every single detail. If you do not comprehend the process which has yielded the results, something has gone wrong. 4. You may think that it is best to involve as many stakeholders as possible in an evaluation. However, achieving a consensus on the evaluation outcomes with all stakeholders is illusory, as it is impossible to structure the evaluation by involving everyone. A consensus about easily understood ways to measure the most significant impacts is, however, a realistic goal. To that purpose, the involvement of a small group of stakeholders with widely differing views on a project seems more fruitful.