Road Network Operations
& Intelligent Transport Systems
A guide for practitioners!
Road Network Operations
& Intelligent Transport Systems
A guide for practitioners!
Before beginning to plan an evaluation, the reasons for doing it and the uses which will be made of the results should be clearly defined. Understanding how those using the results will judge the success of failure of a scheme is an important first step in planning an evaluation. It is not sufficient to carry out an evaluation just because it is required in an organisation. An evaluation that is not properly planned may not measure what is considered to be important, and may even fail to measure the changes brought about by the ITS.
There are political aspects to evaluation, given the diversity of users and their needs. ITS can be adapted to serve widely different policy objectives. The key criteria for decision makers are generally that an ITS scheme must be, and must be seen to be:
A project evaluation can inform all of these aspects. If ITS can bring about a change which serves the needs of travellers, residents and road network operators within an affordable budget, then ITS will serve the public and decision makers alike. For instance in assessing whether an ITS project is:
The interests of the various stakeholder groups involved in the ITS should be considered before starting an evaluation:
Each of these factors should inform the evaluation design. An evaluation which takes them into account and involves stakeholders in the process, will have a much greater chance of producing results which are acceptable to stakeholders. This may mean devoting effort to informing stakeholder groups (including policy-makers) about the scheme – and the reasons for undertaking a thorough independent evaluation, to which their contribution will be key.
At the same time, it is important to remember that one of the key principles in producing a credible evaluation, is that it should be independent. In other words the results are balanced, not biased in favour of the interests of any stakeholder group(s).
To ensure a successful evaluation it is important to agree, at an early stage, clear roles and responsibilities between the various stakeholders involved in evaluation (whether as users of the results or contributors to the study).
Ethical issues need to be considered when planning how to carry out an evaluation and the methods to use. The evaluation should be designed so that there is no risk to, or other adverse impact on, users and stakeholders. Those carrying out evaluation studies have a responsibility to ensure that participants (such as users or employees) are informed of the purpose of any surveys (or other means of data collection) – and how the data will be used. Processes need to be put in place to ensure anonymity and security of data – including for data that is collected automatically (such as from automatic licence plate recognition techniques or CCTV cameras). Some organisations undertaking evaluations will be required, as part of their quality management, to present their plans and methods to an ethics committee for approval prior to starting the evaluation.
A variety of techniques exist for measuring impacts. The effort put into an evaluation will depend on the scale, location and objectives of the ITS scheme. A full-scale evaluation, on a scientific basis, is appropriate for innovative ITS where there is little or no published evidence on its costs, benefits and impacts. An ITS application which has been well proven may justify a simpler or smaller scale evaluation. The scale should be agreed by the stakeholders involved at the outset. Considerations include the:
Over the lifecycle of an ITS application – from concept through initial pilots to larger scale demonstrations and full scale deployment – the extent and coverage of the evaluation needed, reduces. This is because full scale deployment only occurs after a considerable amount of evaluation has been undertaken already on prototypes, pilots and demonstrations – providing a good level of understanding of the ITS application. The European EasyWay programme illustrated this with a graph indicating how the level of evaluation work changes between stages of ITS development.
EasyWay Programme and Projects (2007-2020)
EasyWay (2007-2020) is a major European Union (EU) programme focusing on the deployment of Intelligent Transport Systems (ITS) on the major road networks across Europe. It involves 30 European countries, bringing together road authorities and road operators – and their major ITS technology partners such as the automotive industry, telecommunications and travel service operators and public transport stakeholders. It addresses EU transport objectives: to improve safety, reduce congestion and negative environmental impacts, and promote the continuity of services at through coordinated deployment of real-time information, traffic management and freight and logistics services.
When planning an evaluation, it is important to consider the following aspects:
In preparing an evaluation plan(See Evaluation Plan), it is helpful to draw up a detailed checklist which is specific to the scheme being evaluated. This ensures that the relevant costs, benefits and impacts are identified and assessed. (See Improving Performance)
Several toolkits are available that provide guidance and examples for road network operators and others, supporting activities in different stages of the evaluation cycle – including:
The Netherlands assessed and summarised the key principles of the main techniques used in evaluating ITS in 2003 – with reference to ITS implementations on inter-urban routes.
The UK’s Department for Transport’s ITS Toolkit is designed to make it easier to specify and procure ITS. It includes guidance on how local authorities can benefit from evaluation results.
An overview of the toolkit is contained in the: Traffic Advisory Unit, Understanding the Costs and Benefits of ITS: A Toolkit Approach, Leaflet ITS 1-06, UK Department for Transport, 2006.
Finland’s guidelines are based on a national framework for project appraisal – which is published in English. They cover both project appraisal (pre- implementation) and project evaluation (post-implementation) – and include checklists of possible impacts, lists of indicators and measurement methods, and checklists for evaluating different aspects of schemes. Four examples are given of how the guidelines have been applied to specific projects.
The EasyWay Handbook is aimed at European Road Network Operators and others concerned with evaluating ITS on inter-urban roads in Europe. It discusses the evaluation cycle and includes examples of feedback mechanisms showing how ITS project evaluation is used by decision-makers. See EU ITS Portal
These guidelines are designed for pilots and demonstrations of new transport schemes in Europe. They provide guidance through all stages of the evaluation cycle.
The 2DECIDE ITS Toolkit is an on-line decision support tool to assist transport organisations, network operators and authorities in selecting and deploying ITS. Users can query the database for ITS solutions relevant to a specific context, problem, goal or ITS service. They receive information back on relevant services with an indication of the impacts which may be expected in a specific context. It includes a database of costs, case studies and evaluation reports. To access the toolkit you must first set up an account (free of charge).
This handbook on planning and organising Field Operational Tests (FOTs) of ITS technologies includes guidance on experimental design, data analysis and modelling socio-economic impacts and how to carry out an assessment. Although it is aimed primarily at trials of driver assistance systems, the principles apply equally to other ITS.
FESTA Socio-economic impact assessment for driver assistance systems (Field Operational Tests Networking and Methodology Promotion) - Europe
This supplements the FESTA Handbook and provides advice on the methodology for socio-economic evaluation of Field Operational Tests (FOTs) of ITS. It covers the assessment framework, stakeholders, scope of assessment, analysis methods, financial analysis and data needs. Although it is aimed primarily at trials of driver assistance systems, the principles apply equally to other ITS.
The US Department of Transportation’s ITS Knowledge Resource is a website which provides information on the benefits, costs, deployment levels and lessons learned – from ITS deployments and operations. The information has been assembled from summaries of real life examples starting from the late 1990s. The database supports informed decision making on ITS investments by tracking the effectiveness of systems which have been deployed, over time. Users can search the database, filtering by type of application and country or US state in which it has been evaluated.
The World Bank provides brief guidance on pre-implementation performance criteria, post-implementation monitoring and evaluation. A series of case studies provide examples of urban ITS implementations. The extent to which they have been evaluated is described. The case studies cover both developing and developed economies. They show that few transport organisations carry out comprehensive evaluation of ITS – which makes it more difficult to obtain support for future schemes.
The World Bank has produced a technical note ‘Independent Evaluation: Principles, Guidelines and Good Practice’. It is aimed at global and regional partnership programmes carrying out independent evaluation. It provides guidance on the principles and purpose of independent evaluation, its conduct and the form of evaluation, developing performance indicators – and key evaluation issues. Although not specifically designed for ITS schemes, the principles apply to any evaluation project.
The International Benefit, Evaluation and Cost Working Group IBEC has produced international training materials on international experience in Intelligent Transport Systems in the form of ‘Evaluation 101’ presentations by experts in ITS evaluation.
The Victoria Transport Policy Institute has published many documents on different aspects of transport evaluation. Topics include: cost and benefit analysis, multi-modal transport evaluation, evaluating the safety impacts of mobility management strategies, evaluation of energy saving and emissions reduction strategies, developing indicators for specific objectives and evaluating research quality. It also provides a critique of a bad evaluation study.
Print