Road Network Operations
& Intelligent Transport Systems
A guide for practitioners!
Road Network Operations
& Intelligent Transport Systems
A guide for practitioners!
During the lifecycle of any ITS deployment or a field trial – the most appropriate evaluation method will vary:
The Code of Practice for the Design and Evaluation of Advanced Driver Assistance Systems – developed for evaluating prototype is an example of a checklist for concept development and prototypes. See Advanced Driver Assistance Systems
Simulation and modelling are relevant to project appraisal. (See Project Appraisal)
The FESTA handbook provides information on planning and organising Field Operational Tests (or trials) of driver assistance systems. Many of the principles apply equally to other ITS.
Guidelines on evaluation methods for ITS have been produced in several countries and are available in English:
The World Bank provides guidance on monitoring and evaluation methods with information identifying the usefulness of each technique, advantages and disadvantages, costs, skills and time required – and sources of further information. See World Bank Report on Monitoring and Evaluation
The World Bank also provides a guidance resource on how to design influential evaluations. See Influential Evaluations
Many techniques are used in road network evaluation. Several are also suitable at the appraisal stage when planning ITS investments. See Project Appraisal There are a wide variety of approaches:
ITS is often deployed simply because it is obvious.
Where there is a large body of consistent evidence showing that a particular type of investment is value for money, it makes sense to go ahead and implement it, rather than commission further studies. Multi-criteria analysis can be used to double check the gut instinct or help explain the ‘common sense’ to decision makers.
If a road junction is very congested, it is possible to deduce that traffic signals will improve the flow without carrying out a large traffic modelling exercise. Other implementations will provide predictability on costs. The World Bank Case Study on ITS implemented at Dublin Bus is an example of where it was reasonable to rely on common sense. The perceived benefits were reported but not quantified because it was not considered necessary to do so. See World Bank Case Study Dublin, Ireland
The best source will be systems which have consistently proved to deliver value for money. The US DOT and the European 2Decide projects and toolkit provide online databases of ITS costs, benefits and lessons learned from ITS schemes.
Very expensive systems will often need an evaluation, even if it is only a ‘Critical Factors Analysis’. New ITS schemes should always be evaluated because there is no body of evidence to demonstrate value for money. ITS which have different impacts in different circumstances should also be evaluated in the different contexts.
Qualitative methods raise standards and provide benchmarking when planning and evaluating ITS investments.
The methods include focus groups (also known as group discussions), user panels, citizen panels and quality circles. They involve structured discussions between stakeholders which are mediated by an independent facilitator.
These methods provide a source of information for planning investments and reviewing impacts and implications – in a way that takes account of the views of all relevant stakeholders.
Transport consultants, social research companies and market research companies.
Care needs to be taken to ensure proper representation of all relevant stakeholders – and that members of focus groups and panels are able to contribute on an equal basis. These techniques complement quantitative investigations – and are not a substitute for them.
These provide statistical information on road users’ and travellers’ journeys and patterns of movement.
Methods include:
Travel surveys provide an input to traffic models and are a source of travel data for planning ITS services and analysing changes in response to those services.
Transport consultants, social research companies and market research companies.
Poor survey design, weak fieldwork techniques and low response rates can all affect the statistical accuracy and reliability of any survey – and make the results misleading. Quality controls and consistency of survey methods are particularly important for ‘before’ and ‘after’ surveys.
This is the basis for most cost-benefit analysis. It assesses what people will do in response to ITS.
It is a very basic method which looks at the ‘costs’ of journeys before and after the ITS implementation and predicts changes in behaviour on the basis of past responses to changes in costs.
To make rough estimates of response to ITS where the application is fairly common sense, but needs some additional quantitative support. It will be at the heart of any traffic model.
In-house economists or transport consultants.
Generalised cost is a simple concept which is relatively limited in what it describes. Generalised cost models only measure costs which can be calculated in monetary terms (for example standard monetary values of time). Monetary values are not available for some of the policies which ITS can deliver; and several of the policy objectives that ITS does address will show an increase in the generalised cost of travel, rather than a decrease. For instance, the aim of ITS will often be to change the variability in journey times - rather than reduce total travel time, or deliver other policy objectives such as reduced emissions.
Utility models try to simplify the presentation of what people gain from a change. They are closely related to generalised cost models, but are better designed to reflect user behaviour.
Statistical analysis is used to measure the ‘utility’ or ‘good’ that different people receive from a change.
When user behaviour is important to the outcome that the ITS is aiming to achieve – such as when the aim of an ITS scheme is to encourage people to change mode of travel from car to public transport.
Specialist transport consultants and university departments of statistics or economics. This is not a job for amateurs.
A good utility model is a very powerful tool. The form of the model is fundamentally simple but can become complex. Doubts about the ‘truth’ of these models arise because many use stated preference data to derive values – but historic data can be used instead.
Measure whether users have any perception of an impact that benefits them.
‘Before’ and ‘after’ surveys measure perceptions of the factors which the ITS is intended to address.
When the ITS scheme is intended to improve user comfort or community amenity.
Market research companies, social research companies, transport consultancies.
Many surveys use ‘Lickert scales’ to ask whether things are, for example:
Much worse / A little worse/ About the same/ A little better/ Much better’
The value of these is limited. Likert warned that the scale reflects attitudes – not absolute measurements. The surveys can result in unrealistically optimistic scores. It is not easy to design customer satisfaction surveys which yield valuable results. They are probably best used in conjunction with other methods such as measures of changes in behaviour associated with the impact of the scheme.
It measures how behaviour will change in response to ITS. The benefit of ‘stated preference’ is that it does not use past behaviour as a guide. Other transport models are ‘revealed preference’, using past responses to a change – to derive values for responses to future change.
It provides a rigorous assessment of the utility values of different elements of a product or service – based on trade-offs made by a respondent in a highly structured interview.
Where ITS is new, or where the transport policy which the ITS supports, is new.
Transport consultants, some market research companies.
The risks are that respondents will not make the same choices on paper that they will make in practice. The respondent may not see the barriers to using new ITS. The research may not describe the new product’s drawbacks, only its advantages. Some respondents may not be able to predict what they will do – for example in response to some developments in Information and Communications Technologies, people have relocate their homes further away from work, but would they have predicted that in a survey? The concern is that conjoint analysis will derive wrong answers whilst the nature of the answers (a value or number) will be precise – leading to wrong investment decisions.
It can be used to measure changes in factors such as traffic and travel time.
ITS create, collect and transmit data. These data can show the change in the targeted outcomes over time. Baseline ‘before’ data are needed for comparison with internal data obtained after deployment.
Use this in every case where it is technically feasible.
Design the measurement of the results into the ITS.
ITS will only measure quantitative results. Changes in factors such as user confidence cannot be measured, although they may be implied. Public transport patronage cannot usually be measured by ITS unless the application is a ticketing or revenue collection system.
Print