Practical Aspects

There are practical aspects which should be considered before starting an evaluation. In addition to the overall budget for implementing the ITS scheme, a budget should be allocated to evaluation – which will involve work before and after implementation. In some cases this will be considered as part of the implementation. In other cases the evaluation budget will be treated separately. In either case, the evaluation budget represents an investment in planning for future projects. It is important to consider:

• what budget will be needed for the evaluation?
• where will the funds come from and when can they be made available?
• who requires the evaluation and why?
• what effect will any changes in the timing of the deployment programme have on the resources and budget needed for evaluation?

Focus of the evaluation

The evaluation should focus on measuring the key aspect that the scheme is planning to change – not to pick up some general statistics or ‘feel good’ measures that may bear no relation to the effect of the ITS. For example, an evaluation of a real-time journey time information service which just asks users whether the ITS has had an impact on the quality of their journeys will only provide an indication of user satisfaction. Unless traffic flows and journey times are measured on the main route and alternative routes when incidents occur – the evaluation will not be able to determine the extent to which the ITS reduces the impact of incidents by spreading out traffic onto alternative routes. Nor will it be able to identify any detrimental effects to other roads, caused by diverted traffic.

Where possible, the ITS should be programmed to provide data which can be used to monitor its impacts. For example, real-time information at bus stops provides information on bus headways. Storing this data gives a record of intervals between bus services, which, for passengers, is a key aspect of reliability. In the case study on demand responsive transport in Prince William County USA, the service data was used to monitor performance automatically, reducing the cost of data collection and analysis. (See ITS for Demand Responsive Transport Operation, Prince William County USA)

Stakeholder requirements

Specific stakeholder groups may have particular requirements from a scheme – which influences the methods of evaluation, the budget, timing and scale of evaluation. It is important to understand these requirements and constraints at an early stage in planning the evaluation to determine to what extent they are met. The evaluation should be designed so that it is possible to identify how the benefits and costs of the ITS are distributed between the different stakeholders – who gains and who loses.

For example, as well as evaluating the impacts of ITS on the travelling public (the ‘end’ users), evaluation should consider the requirements of the operators and other staff of the organisations involved in delivering the ITS. These impacts can be evaluated by interviewing staff – and in some cases by using data on work patterns and completed tasks.

Who should do the evaluation?

Evaluation is specialist work. Some road network operators, road authorities and other organisations tasked with implementing ITS have limited capabilities within their organisation for monitoring and evaluation of ITS. However evaluation is not always complicated, and does not always have to be carried out by specialists. If there is a need to collect data, this can be done in-house, provided that it is done credibly using an objective approach and sound methods.

For most evaluations it is a good idea to use specialists the first time one is undertaken – so in-house staff can learn how it should be done. Formal training should also be given to enhance the learning experience. It is also worth employing specialists from time to time afterwards so that up-to-date methods can be introduced to in-house staff. In some circumstances it will be important that the evaluation is carried out by an independent organisation – to ensure that the results are credible (for example, in reporting back to funding agencies).

Affordability

Affordability is often an issue. Whichever evaluation method is used, there may be insufficient budget to answer all questions related to the ITS. It then becomes necessary to decide which aspects of the evaluation have the highest priority. A Critical Factors Analysis can be useful at this stage. It looks at the different goals that a scheme is expected to address – and the key indicators needed to measure the extent to which they have been achieved. This is illustrated in the figure below.

Managing evaluation

Evaluation management includes planning and monitoring the work of evaluation to ensure its technical quality, cost-effectiveness and timeliness. Key elements in managing a successful evaluation are:

• forming an evaluation team which is independent (where possible) of the ITS implementation, with a project manager (possibly also a project director), clear objectives and appropriate resources
• reaching agreement with stakeholders and an understanding of what they need from the evaluation process
• developing an Evaluation Management Plan covering resources, timing, coordination with other activities, data management – reviewing and updating it at key points in the project
• establishing a risk register to identify any risks to delivering the evaluation as planned – with measures to minimise or mitigate each risk. This should be reviewed and updated when appropriate
• preparing interim monitoring reports to inform stakeholders of progress and interim results
• defining the format and content of evaluation reports in consultation with stakeholders

Data protection and privacy

Information is a key element of ITS. ITS applications collect, combine, store and communicate data about the transport network, its users – and the stakeholders involved in providing services. In addition to operational data, monitoring and evaluation also requires data collection on individuals and organisations, their views and responses to the ITS. This can raise concerns about privacy. (See Legal and Regulatory Issues)

Those involved in planning, conducting and reporting on evaluation and monitoring activities need to ensure that data protection and privacy issues are considered and addressed – and that any requirements of data protection legislation are met. The US DOT’s ePrimer discusses data protection for ITS at some length and sets out strategies for mitigating privacy issues – which apply to monitoring and evaluation as well as operational data:

• prepare formal data sharing agreements if data is to be shared between organisations
• create a formal policy on data including procedures for disclosure
• collect only personal information that is needed
• use the least sensitive personal information necessary
• set clear and reasonable limits on the time period that data is retained
• if information is required which enables individuals to be identified, this needs to be explained to those taking part in the evaluation. They need to be aware of what is being collected and how it is being used – and individuals should be asked for their consent to their data being used in this way (See ePrimer module 12 )

Specific Issues for Developing Economies

Funding Requirements

External Funding agencies are likely to have specific requirements which must be met by evaluation work, in addition to following best practice in designing and carrying out evaluations. These might relate, for instance to:

• the scale of evaluation – which may be determined by the scale of the deployment (specifying the proportion of the total funding to be devoted to evaluation)
• independence of the evaluation team from the deployment
• governance arrangements for overseeing the evaluation
• requirements for involvement of local representatives within the evaluation team
• requirements for periodic and end-of-project reporting including the format, structure and content of reports
• financial accountability

Learning from other Schemes

Evaluation results from other schemes are helpful when planning an evaluation. They may:

• help identify the likely impacts to be measured – and the indicators to be used
• provide indications of the scale of change expected – and the amount of data that needs to be collected to obtain statistically robust measures of the impacts

Care needs to be taken when considering the methods used and results obtained from elsewhere. The number of case studies of ITS evaluations in developing economies is limited and the temptation will be to look at results from developed economies which are more numerous and readily available – but they may not be appropriate to the context. User reactions and benefit valuation contexts can differ widely between different countries and regions with different social and economic profiles. The key issue to be addressed is whether the planned scheme is similar to the case studies in terms of its context, objectives and user base?