Road Network Operations
& Intelligent Transport Systems
A guide for practitioners!

You are here

Evaluation Methods

Before beginning to plan an evaluation, the reasons for doing it and the uses which will be made of the results should be clearly defined. Understanding how those using the results will judge the success of failure of a scheme is an important first step in planning an evaluation. It is not sufficient to carry out an evaluation just because it is required in an organisation. An evaluation that is not properly planned may not measure what is considered to be important, and may even fail to measure the changes brought about by the ITS.

Meeting political objectives

There are political aspects to evaluation, given the diversity of users and their needs. ITS can be adapted to serve widely different policy objectives. The key criteria for decision makers are generally that an ITS scheme must be, and must be seen to be:

  • deliverable
  • acceptable
  • affordable
  • measurable

A project evaluation can inform all of these aspects. If ITS can bring about a change which serves the needs of travellers, residents and road network operators within an affordable budget, then ITS will serve the public and decision makers alike. For instance in assessing whether an ITS project is:

  • deliverable - a technical evaluation of the extent to which the technologies have been shown to perform as expected, will inform expectations about the performance of an ITS deployment
  • acceptable - information from users, operators and other stakeholders will show whether, and in what ways, it is acceptable
  • affordable - a cost-benefit analysis can be used to assess affordability
  • measurable – a critical factors analysis will identify the visible results which can be measured – to illustrate how far an ITS investment has provided the desired outcome

Stakeholder interests

The interests of the various stakeholder groups involved in the ITS should be considered before starting an evaluation:

  • role in the scheme
  • how they are likely to be affected
  • contribution to the evaluation process
  • providing monitoring or performance data
  • requirements for evaluation results

Each of these factors should inform the evaluation design. An evaluation which takes them into account and involves stakeholders in the process, will have a much greater chance of producing results which are acceptable to stakeholders. This may mean devoting effort to informing stakeholder groups (including policy-makers) about the scheme – and the reasons for undertaking a thorough independent evaluation, to which their contribution will be key.

At the same time, it is important to remember that one of the key principles in producing a credible evaluation, is that it should be independent. In other words the results are balanced, not biased in favour of the interests of any stakeholder group(s).

To ensure a successful evaluation it is important to agree, at an early stage, clear roles and responsibilities between the various stakeholders involved in evaluation (whether as users of the results or contributors to the study).

Ethical issues

Ethical issues need to be considered when planning how to carry out an evaluation and the methods to use. The evaluation should be designed so that there is no risk to, or other adverse impact on, users and stakeholders. Those carrying out evaluation studies have a responsibility to ensure that participants (such as users or employees) are informed of the purpose of any surveys (or other means of data collection) – and how the data will be used. Processes need to be put in place to ensure anonymity and security of data – including for data that is collected automatically (such as from automatic licence plate recognition techniques or CCTV cameras). Some organisations undertaking evaluations will be required, as part of their quality management, to present their plans and methods to an ethics committee for approval prior to starting the evaluation.

Scale considerations

A variety of techniques exist for measuring impacts. The effort put into an evaluation will depend on the scale, location and objectives of the ITS scheme. A full-scale evaluation, on a scientific basis, is appropriate for innovative ITS where there is little or no published evidence on its costs, benefits and impacts. An ITS application which has been well proven may justify a simpler or smaller scale evaluation. The scale should be agreed by the stakeholders involved at the outset. Considerations include the:

  • anticipated benefits
  • scale of the scheme
  • scheme costs
  • extent of deployment
  • transferability of results to other schemes
  • level of certainty about the outcomes
  • the likely benefits to be obtained from monitoring and evaluation

Over the lifecycle of an ITS application – from concept through initial pilots to larger scale demonstrations and full scale deployment – the extent and coverage of the evaluation needed, reduces. This is because full scale deployment only occurs after a considerable amount of evaluation has been undertaken already on prototypes, pilots and demonstrations – providing a good level of understanding of the ITS application. The European EasyWay programme illustrated this with a graph indicating how the level of evaluation work changes between stages of ITS development.

Level of evaluation required at different stages of application development. Source: Tarry S, Turvey S and Pyne M 2012. EasyWay Project Evaluation Guidelines.  Version 4.0, p 8. EasyWay

EasyWay Programme and Projects (2007-2020)

EasyWay (2007-2020) is a major European Union (EU) programme focusing on the deployment of Intelligent Transport Systems (ITS) on the major road networks across Europe. It involves 30 European countries, bringing together road authorities and road operators – and their major ITS technology partners such as the automotive industry, telecommunications and travel service operators and public transport stakeholders. It addresses EU transport objectives: to improve safety, reduce congestion and negative environmental impacts, and promote the continuity of services at through coordinated deployment of real-time information, traffic management and freight and logistics services.


When planning an evaluation, it is important to consider the following aspects:

  • the importance of socio-economic evaluation (the effects on users, other stakeholders, the economy, the environment) as well as technical evaluation (system performing as expected – for example, it meets the technical specification)
  • the detailed breakdown of impacts as well as the breadth of multiple impacts
  • the need for large scale evaluation of fully implemented schemes as well as small scale evaluation of pilot schemes and tests
  • variances in impacts as well as expected values of impacts
  • both indirect and direct impacts
  • both unintended and intended consequences
  • long term as well as near term effects
  • gradual adaptation to change as well as immediate responses
  • multiple as well as the single most important impacts
  • private sector and public sector concerns
  • interests of third parties as well as main stakeholders
  • distributive as well as aggregate costs and benefits (who gains and who loses)

In preparing an evaluation plan(See Evaluation Plan), it is helpful to draw up a detailed checklist which is specific to the scheme being evaluated. This ensures that the relevant costs, benefits and impacts are identified and assessed. (See Improving Performance)


Several toolkits are available that provide guidance and examples for road network operators and others, supporting activities in different stages of the evaluation cycle – including:

Evaluation of ITS in The Netherlands

The Netherlands assessed and summarised the key principles of the main techniques used in evaluating ITS in 2003 – with reference to ITS implementations on inter-urban routes.

ITS Toolkit – United Kingdom (UK)

The UK’s Department for Transport’s ITS Toolkit is designed to make it easier to specify and procure ITS. It includes guidance on how local authorities can benefit from evaluation results.

An overview of the toolkit is contained in the: Traffic Advisory Unit, Understanding the Costs and Benefits of ITS: A Toolkit Approach, Leaflet ITS 1-06, UK Department for Transport, 2006.

Guidelines for the Evaluation of ITS Projects – Finland

Finland’s guidelines are based on a national framework for project appraisal – which is published in English. They cover both project appraisal (pre- implementation) and project evaluation (post-implementation) – and include checklists of possible impacts, lists of indicators and measurement methods, and checklists for evaluating different aspects of schemes. Four examples are given of how the guidelines have been applied to specific projects.

EasyWay Handbook – Europe

The EasyWay Handbook is aimed at European Road Network Operators and others concerned with evaluating ITS on inter-urban roads in Europe. It discusses the evaluation cycle and includes examples of feedback mechanisms showing how ITS project evaluation is used by decision-makers. See EU ITS Portal


These guidelines are designed for pilots and demonstrations of new transport schemes in Europe. They provide guidance through all stages of the evaluation cycle.

2DECIDE ITS Toolkit – Europe

The 2DECIDE ITS Toolkit is an on-line decision support tool to assist transport organisations, network operators and authorities in selecting and deploying ITS. Users can query the database for ITS solutions relevant to a specific context, problem, goal or ITS service. They receive information back on relevant services with an indication of the impacts which may be expected in a specific context. It includes a database of costs, case studies and evaluation reports. To access the toolkit you must first set up an account (free of charge).

FESTA Handbook (Field Operational Tests Networking and Methodology Promotion) – Europe

This handbook on planning and organising Field Operational Tests (FOTs) of ITS technologies includes guidance on experimental design, data analysis and modelling socio-economic impacts and how to carry out an assessment. Although it is aimed primarily at trials of driver assistance systems, the principles apply equally to other ITS.

FESTA Socio-economic Impact Assessment – Europe

FESTA Socio-economic impact assessment for driver assistance systems (Field Operational Tests Networking and Methodology Promotion) - Europe

This supplements the FESTA Handbook and provides advice on the methodology for socio-economic evaluation of Field Operational Tests (FOTs) of ITS. It covers the assessment framework, stakeholders, scope of assessment, analysis methods, financial analysis and data needs. Although it is aimed primarily at trials of driver assistance systems, the principles apply equally to other ITS.

United States’ Department of Transportation ITS Knowledge Resources

The US Department of Transportation’s ITS Knowledge Resource is a website which provides information on the benefits, costs, deployment levels and lessons learned – from ITS deployments and operations. The information has been assembled from summaries of real life examples starting from the late 1990s. The database supports informed decision making on ITS investments by tracking the effectiveness of systems which have been deployed, over time. Users can search the database, filtering by type of application and country or US state in which it has been evaluated.

The World Bank ITS Toolkit for urban transport

The World Bank provides brief guidance on pre-implementation performance criteria, post-implementation monitoring and evaluation. A series of case studies provide examples of urban ITS implementations. The extent to which they have been evaluated is described. The case studies cover both developing and developed economies. They show that few transport organisations carry out comprehensive evaluation of ITS – which makes it more difficult to obtain support for future schemes.

The World Bank

The World Bank has produced a technical note ‘Independent Evaluation: Principles, Guidelines and Good Practice’. It is aimed at global and regional partnership programmes carrying out independent evaluation. It provides guidance on the principles and purpose of independent evaluation, its conduct and the form of evaluation, developing performance indicators – and key evaluation issues. Although not specifically designed for ITS schemes, the principles apply to any evaluation project. 

IBEC Training Materials

The International Benefit, Evaluation and Cost Working Group IBEC has produced international training materials on international experience in Intelligent Transport Systems in the form of ‘Evaluation 101’ presentations by experts in ITS evaluation. 

Victoria Transport Policy Institute, Canada

The Victoria Transport Policy Institute has published many documents on different aspects of transport evaluation. Topics include: cost and benefit analysis, multi-modal transport evaluation, evaluating the safety impacts of mobility management strategies, evaluation of energy saving and emissions reduction strategies, developing indicators for specific objectives and evaluating research quality. It also provides a critique of a bad evaluation study.


Reference sources

No reference sources found.