The term ‘evaluation of ITS’ is an assessment of the extent to which an ITS scheme has met its objectives. It provides lessons on improving performance in the future. The main issues to consider are:
The road network operator can play an important role in the evaluation of ITS deployments aimed at supporting Road Network Operations. Positive and negative impacts on network operations should be assessed for any ITS applications that are in the research and development phase or are part of a large-scale field trial. Evaluations of routine deployments of established ITS technologies are also important to establish good practice. For example an evaluation of the use of speed cameras in a completely new context should establish the degree of compliance by road users and whether this has a beneficial effect on traffic efficiency and accidents.
The evaluation is a planned and structured assessment of the impacts of an ITS scheme and the extent to which it has met its objectives. The impacts assessed include the financial costs and negative consequences as well as the benefits. (See ITS Benefits) Evaluation takes place after deployment has been completed, but it is important to plan the evaluation before the deployment takes place and schedule the resources to carry it out. Evaluation is often undertaken by an independent organisation so that the results are seen as a true and unbiased assessment of the scheme. This is particularly so for an ITS scheme that is highly innovative or which has a high public profile, such as the congestion charge scheme in Stockholm.
Monitoring and evaluation are not carried out for their own sake – they are not ends in themselves. ITS projects cover a wide diversity and involve considerable investment of financial and other resources by stakeholder organisations. A formal evaluation is important in order to check that the expected value of an investment has been realised, and to determine who benefits and how those benefits compare with expectations. Evaluation is never simply a matter of “justifying” investment. It provides information which:
Evaluation makes it possible to assess the impacts which the scheme has had on stakeholders (such as travellers and operators) – and on a range of policy objectives such as the environment, safety, sustainability and efficiency. The stakeholder perspective for minority groups who may be disproportionately affected – positively or negatively – will be important, for example people with a disability or vulnerable road users, such as pedestrians and cyclists.
Organisations that fail to undertake full and proper evaluations of their ITS deployments are at a disadvantage when trying to justify proposals for further investments in the future. This applies whether the organisation is a public authority trying to determine the direction of future transport policies, a road operator considering its investment priorities or a commercial organisation that sees new business opportunities. A well-planned and documented evaluation of an existing scheme, helps justify and gain support for the next one.
The type of evaluation to be undertaken is determined by the evaluation objectives. They may include:
Stakeholders involved in commissioning an evaluation of ITS can include policy makers at national or local level, road or transport authorities, transport operators and users, and the organisations directly involved in Road Network Operations. Depending on the scale and complexity of the evaluation - it may be carried out in-house or by specialist consultants or university researchers.
The involvement of Road Network Operators in the evaluation of ITS for Road Network Operations includes:
Road network operators and public road authorities often find it difficult to make a business case for ITS. Evaluation results can support this process. The US DOT’s ePrimer’s Module 12 discusses how to make the business case.
Evaluation of an ITS deployment can help make the business case for investment by:
Public Sources of Evaluation Results
Evaluation results for ITS schemes have been compiled and consolidated in several resources. Some are databases which can be searched for results relating to specific types of scheme or meeting particular objectives. These are invaluable in helping identify expected impacts and potential performance measures. They also demonstrate the advantages of reporting evaluation results using a common framework – to compare studies which may have been carried out in different countries and with different requirements for evaluation and reporting. Sources of ITS evaluation results include:
The evaluation cycle uses the principles and values that underpin the development of any community – those of Learning, Evaluation and Planning (LEAP), which are often summarised as: analyse – plan – do – review. The evaluation process is cyclical – positive results can be built on, and less positive results can be analysed to determine what went wrong, what can be done to remedy the situation – and what can be done to improve the results of similar applications in the future. This cycle is illustrated below and shows that evaluation is a key part of the process of implementing ITS. It is not an optional ‘extra’. The diagram shows how monitoring and evaluation can provide both short-term feedback to improve current operations and also results that will feed into the ITS development and investment strategy.
The ITS evaluation cycle
After deployment, post-implementation monitoring and evaluation of an ITS scheme is used to assess whether the system – as installed and operated – is meeting project objectives, delivering the expected performance and matching user requirements (which themselves may change over time). ‘User needs’ are not necessarily those of the driver of a vehicle. They might be the needs of the road authority or network operator to improve system performance and deliver better safety or traffic flow, the needs of the environment, the needs of the wider integrated transport network – or the needs of the communities affected by other people’s travel.
The results of post-implementation analysis should be fed into the evaluation cycle, improving operations and monitoring – and influencing ITS strategies for the future. The feedback provided by the evaluation informs future investments and their design. If the ITS is not performing as expected, feedback can help in understanding how to adjust or adapt the scheme.
Automatic Fare Collection, Turkey
In Izmir, Turkey, the information generated by the automatic fare collection and real time bus information system is being used to re-design the route network, with planned interchanges and improved service quality. It has led to significant increases in passenger numbers. See World Bank Case Study: Izmir, Turkey
A carefully constructed evaluation plan is key to this cyclical process and is an essential part of any programme for developing ITS. (See Evaluation Plan)
Evaluation is significant for any ITS deployment, but especially for a scheme which is innovative or deployed into a new situation. It shows the contribution that the scheme can make to meeting transport policy and everyday operational objectives, and can be used to improve and fine tune ITS operations –providing feedback for subsequent rounds of deployment. Without an adequate evaluation, organisations cannot be certain that they have obtained the properly functioning scheme for which they have paid. Evaluation results can also be used to inform future ITS policy and strategy. (See Improving Performance)
Another use of feedback from ITS evaluation is to improve the process of appraisal prior to deployment of a future scheme. (See Project Appraisal) The results of real life evaluation impacts – which include the costs of building, maintaining and operating ITS schemes and the benefits achieved – can be used in future investment decisions, to make sure that systems selected meet specific user requirements.
Road network operators have an important role throughout the evaluation cycle. They are involved in developing the operating strategies for their networks, identifying ITS project objectives, pre-implementation analysis to inform investment decisions, scheme selection and analysis of user requirements. As scheme ‘owners’ they gather monitoring data during implementations of ITS. Increasingly, ITS includes features that provide the basis for monitoring patterns of traffic demand and the behaviour of road users. (See Network Monitoring) Road network monitoring data is used by operators for pre-implementation analysis and for monitoring traffic, safety, incidents and other performance criteria on the network – after an ITS implementation has been deployed. It provides vital input to the evaluation. Road network operators will often commission independent evaluation and use the results to feed back into future strategies and investment decisions. (See Operational Activities)
Monitoring in conjunction with evaluation is used to learn lessons and improve future performance. It addresses issues such as: does the ITS meet the objectives and are the outcomes as intended? It can identify the range of benefits achieved which can then be quantified with data from internal monitoring. Careful monitoring of costs helps identify the operational and maintenance costs for a specific service – which can be separated from wider organisational costs.
The performance of the relevant elements of the transport system should be monitored to provide the benchmark against which the added value of the ITS scheme is measured. Performance monitoring can be used to improve operation of the ITS, provide data on the impacts and benefits and demonstrate whether the anticipated benefits and impacts have been realised.
The focus of the monitoring activity is likely to vary with the scale and maturity of ITS deployment. For example, for an area where ITS is relatively new and the number of applications is limited – such as cooperative driving based on Vehicle-to-Vehicle (V2V) communications – monitoring is likely to focus in depth on the performance of individual applications. Where deployment is extensive – such as electronic payment or non-stop tolling – the main focus is likely to be on monitoring the overall presence and performance of ITS in the transport system at a strategic level – in addition to the individual applications. These different approaches have different evaluation requirements.
For example, the US Federal Highway Administration (FHWA) has taken a strategic view of progress with ITS deployment through the use of “a few good measures” that are monitored each year. This is illustrated by its “Intelligent Transportation Systems Benefits": 1999 Update”.
The performance criteria defined in the evaluation plan (See Evaluation Plan) should be monitored both before and after implementation. Depending on the ITS implementation, it may be necessary to monitor ‘control’ sites as well as the implementation site.
Performance monitoring covers the services delivered as well as the technologies used. It involves:
Indicators should be selected which are easy to understand and measure, and are clearly linked to the performance requirements and objectives of the scheme. Where possible they should be common indicators, which can be used across a variety of schemes. (See Indicators)
Where possible, the ITS scheme should be designed so that it automatically provides data for performance monitoring. For example:
Monitoring and evaluation are key elements for a learning organisation. An important part of the evaluation cycle is to feedback results from evaluation to inform ITS strategy, ITS operations and the requirement for monitoring. (See Performance Measures)
At a strategic level, the results of monitoring and evaluation can be used to:
Evaluation results are also used as feedback to improve operations and monitoring – for example to optimise performance of the ITS during day-to-day operations. The ITS will often need to be adjusted on the basis of evaluation results in the early stages of operation. These adjustments should increase acceptance of the ITS by travellers operators and other users. Even if evaluation results show the system to be working well, there may be scope to optimise it further. Any adjustments made in the early stages, after implementation, will need to be recorded carefully so that they can be taken into account in the evaluation. The full scale evaluation should be carried out after adjustments have been made and the system has ‘settled down’ – and its effects have stabilised.
Monitoring Automatic Vehicle Location of Buses in South Africa
In South Africa, monitoring of an automatic vehicle location ITS application for buses, identified issues of communication over the mobile network operator’s network – which reduced the performance of the automatic vehicle monitoring system. This was found to be a result of the mobile network operator changing the communications protocols without considering the needs of the public transport operator. An agreement has been reached between the two parties – whereby the mobile network operator:
The monitoring also highlighted potential impacts on the service level agreement – affecting the suppliers of other components of the system – which had not been anticipated when the service was planned. Further information: See World Bank Case Study Johannesburg, South Africa
Monitoring data can be used to assess the extent to which the ITS meets operational requirements, for example, by:
Performance measures can be used to define payments in contracts – to provide incentives for meeting performance targets and to provide the basis for penalty charges if targets are not met. Where payments for completed work depend on meeting performance targets, specialist advice on risk management and performance measurement is suggested.
Performance Measures and Payments in Dublin
In Dublin, the bus company has a Public Service Obligation Contract with the National Transport Authority. Performance measures are linked to payments. The automatic vehicle location and real time information for bus services provides data on performance which is used to resolve operational issues. For example, running times have been reported as a problem by drivers on some routes. The monitoring data can then be used to analyse whether there is real problem which occurs at specific times of day or days of the week, occasional or frequent – so adjustments can be made. Further information: See World Bank Case Study Dublin Ireland
Performance measures can be also used to look towards the future. They can, for instance, define targets for improved performance as part of a process of continuous improvement. More widely, they can be used to provide recommendations for future operations.
Funding programmes may have put in place established processes for monitoring and reporting performance to obtain approval for payment. For example the Asian Development Bank has ‘Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations’.
Monitoring and evaluation data can be used to assess operational effectiveness – identifying whether the ITS is readily incorporated into day-to-day operations and whether there are additional training needs in an organisation. This is one of the aspects addressed in a World Bank Case Study on monitoring and evaluating ITS for automatic bus location, bus scheduling and real time information.See World Bank Case Study Mysore, India
The US DOT’s ePrimer discusses organisational capabilities and refers to work by AASHTO in the US that provides guidance on how to evaluate capabilities and prepare an action plan. This includes capabilities for performance measurement.(See ePrimer Module 12)
Evaluation results also improve the quality of appraisals carried out before investment decisions are taken (See Project Appraisal) – by providing feedback on the performance of ITS options – such as, what did they achieve, what impacts did they have on user demand and the use of other modes. Useful tools are databases and websites which bring together evaluation results from a range of applications in different areas and contexts. This emphasises the value of reporting results and making them widely available.