What is Evaluation?

The term ‘evaluation of ITS’ is an assessment of the extent to which an ITS scheme has met its objectives. It provides lessons on improving performance in the future. The main issues to consider are:

• why evaluate in the first place – the purpose is to assess the effectiveness of schemes and to provide feed-back into future decisions on investments
• the audience for whom the evaluation is intended, which may determine the focus and extent of the evaluation and the choice of features that are to be monitored
• the evaluation cycle, which is the process of identifying objectives, assessing the extent to which they have been met, and informing future strategies and operational improvements
• evaluation methods – take a systematic approach to planning and conducting evaluations, with different levels of evaluation appropriate for different contexts
• the budget and resources that can be made available, to ensure the statistical reliability and overall credibility of results. In some cases a fixed proportion of scheme costs (say 5%) may be allocated
• how evaluation and monitoring can improve performance – the information collected for monitoring schemes needs to be carefully designed and reported so that it can be used in the evaluation and to improve operational performance and ITS service delivery. It may also be a requirement of a funding programme

The road network operator can play an important role in the evaluation of ITS deployments aimed at supporting Road Network Operations. Positive and negative impacts on network operations should be assessed for any ITS applications that are in the research and development phase or are part of a large-scale field trial. Evaluations of routine deployments of established ITS technologies are also important to establish good practice. For example an evaluation of the use of speed cameras in a completely new context should establish the degree of compliance by road users and whether this has a beneficial effect on traffic efficiency and accidents.

The evaluation is a planned and structured assessment of the impacts of an ITS scheme and the extent to which it has met its objectives. The impacts assessed include the financial costs and negative consequences as well as the benefits. (See ITS Benefits) Evaluation takes place after deployment has been completed, but it is important to plan the evaluation before the deployment takes place and schedule the resources to carry it out. Evaluation is often undertaken by an independent organisation so that the results are seen as a true and unbiased assessment of the scheme. This is particularly so for an ITS scheme that is highly innovative or which has a high public profile, such as the congestion charge scheme in Stockholm.

The significance of evaluation

Monitoring and evaluation are not carried out for their own sake – they are not ends in themselves. ITS projects cover a wide diversity and involve considerable investment of financial and other resources by stakeholder organisations. A formal evaluation is important in order to check that the expected value of an investment has been realised, and to determine who benefits and how those benefits compare with expectations. Evaluation is never simply a matter of “justifying” investment. It provides information which:

• enables lessons to be learned
• cost-effectiveness to be determined
• performance to be improved for the future

Evaluation makes it possible to assess the impacts which the scheme has had on stakeholders (such as travellers and operators) – and on a range of policy objectives such as the environment, safety, sustainability and efficiency. The stakeholder perspective for minority groups who may be disproportionately affected – positively or negatively – will be important, for example people with a disability or vulnerable road users, such as pedestrians and cyclists.

Organisations that fail to undertake full and proper evaluations of their ITS deployments are at a disadvantage when trying to justify proposals for further investments in the future. This applies whether the organisation is a public authority trying to determine the direction of future transport policies, a road operator considering its investment priorities or a commercial organisation that sees new business opportunities. A well-planned and documented evaluation of an existing scheme, helps justify and gain support for the next one.

Types of Evaluation

The type of evaluation to be undertaken is determined by the evaluation objectives. They may include:

• the technical performance of the system
• impact analysis and socio-economic assessment of the ITS deployment including answers to the question “who benefits – or suffers a disbenefit?”
• financial analysis, both for the capital costs and the on-going operation (“whole-life“ costs)
• market analysis for new ITS applications and services
• process analysis (efficiency of the ITS operation itself)
• assessment of the organisational and institutional impacts of an ITS scheme

Role of Road Network Operators

Stakeholders involved in commissioning an evaluation of ITS can include policy makers at national or local level, road or transport authorities, transport operators and users, and the organisations directly involved in Road Network Operations. Depending on the scale and complexity of the evaluation - it may be carried out in-house or by specialist consultants or university researchers.

The involvement of Road Network Operators in the evaluation of ITS for Road Network Operations includes:

• defining the purpose and objectives of the evaluation
• defining the scope of the evaluation plan (See Evaluation Plan)
• defining the indicators to be used in assessing performance against those objectives
• providing (in many cases) data for at least some of the indicators – particularly those relating to the performance of the road network
• reporting the results in a format that is suitable to the primary audience
• using the results to improve road network operations and inform future investment strategy on the network
• and sometimes, commissioning and managing the evaluation in-house

Making the business case for ITS

Road network operators and public road authorities often find it difficult to make a business case for ITS. Evaluation results can support this process. The US DOT’s ePrimer’s Module 12 discusses how to make the business case.

Evaluation of an ITS deployment can help make the business case for investment by:

• demonstrating how ITS is able to address current issues and specific problems
• educating policy makers, decision makers and the public about the costs and benefits of ITS

FURTHER INFORMATION

Public Sources of Evaluation Results

Evaluation results for ITS schemes have been compiled and consolidated in several resources. Some are databases which can be searched for results relating to specific types of scheme or meeting particular objectives. These are invaluable in helping identify expected impacts and potential performance measures. They also demonstrate the advantages of reporting evaluation results using a common framework – to compare studies which may have been carried out in different countries and with different requirements for evaluation and reporting. Sources of ITS evaluation results include:

• US DOT benefits database
• US DOT costs database
• US DOT Lessons learned documents
• EasyWay Evaluation Expert Group 
• 2Decide
• UK DfT ITS Toolkit