Road Network Operations
& Intelligent Transport Systems
A guide for practitioners!

You are here

Measuring Human Performance

Overall evaluation of ITS products and services is an important activity because the performance of the road user will depend crucially on the usability of the ITS. A benefit of improving the ease and efficiency of ITS technology, is increased user satisfaction. This can provide business advantages, particularly when users have a choice of ITS products or services. Poor usability of ITS, such as a poor user interface, or inadequate and misleading dynamic signage, may have safety implications in the road environment. Good usability will help to manage and predict road user behaviour and so help increase road network performance. For all these reasons, the performance of ITS in terms of its usability needs to be measured. (See Evaluation)

Measurement of Usability

Usability measurement requires assessment of the effectiveness, efficiency and satisfaction with which representative users carry out representative tasks in representative environments. This requires decisions to be made on the following key points:

  • what environment (context of use) is relevant
  • which metrics (measures) will be taken
  • what tools are required – such as stop-watch, questionnaire, driving simulator
  • what method will be used to make the measurements

Steps in performance measurement:

  • define the ITS to be tested (this may be a system in-use or a prototype/pilot)
  • decide what type of user will be considered (first time/infrequent or experienced user)
  • define other aspects of the typical context of use (See Context of Use)
  • decide on the particular functions or aspects of the ITS to be evaluated and the context for the evaluation
  • decide on the usability goals to be tested and their relative importance
  • prepare a plan for measuring a set of metrics (that assess the usability goals)
  • carry out the user tests
  • analyse the test data to derive and interpret the metrics
  • record the results and draw conclusions

The process of measuring human performance when the driver is interacting with ITS (particularly when using information and communication devices) can yield safety benefits – although there are challenges with this form of human performance measurement in this context.

Safety of Drivers’ ITS Interaction

Driver distraction (not focussing on the road ahead and the driving task) is an important issue for road safety. ITS products, such as information and communication devices, can greatly assist the driver (for example by indicating suitable routes) but ITS can also be an additional source of distraction. Distraction can make drivers less aware of other road users such as pedestrians and road workers and less observant of speed limits and traffic lights. (See Road Safety)

Measuring driving performance when interacting with ITS requires specialist equipment and expertise. Measurements made in laboratory settings and driving simulators may not be representative of real driving behaviour. This is because in real driving contexts drivers can choose when to interact (or not) with devices – and can modify their driving style to compensate to some extent for other demands on their attention. On-road measurements have to be designed to be unobtrusive and representative. Field Operational Tests (FOTs) can be designed accordingly and used to investigate both mature and innovative systems. FOTs can involve both professional and ordinary drivers according to the focus of investigation. A “naturalistic” driving study aims to unobtrusively record driver behaviour. Analysis of the drive record is used to identify safety-related events such as distraction, although the interpretation of results can be problematic and controversial.

The weight of scientific evidence points to distraction being an important safety issue. Many governments and Road Operators have sought to restrict drivers’ use of ITS while driving. There are different national and local approaches ranging from guidelines and advice – to bans on specific activities or functions (such as texting or hand-held phone use). 

Deciding on Goals and Metrics

Usability goals for an ITS product or service should be expressed in terms of the usability attribute – such as easy to learn, efficient to use, easy to remember, few errors, subjectively pleasing. Deciding on the relative importance of these goals depends on the ITS and the context but it helps to focus future evaluations on the most important aspects.

Not all performance measurements have to be quantitative but some simple examples of performance metrics that might be of interest to Road Operators are:

  • the time taken by a control room operative to post a road closure on a message sign
  • the number of toll charge queries collected at automated toll booths
  • the ratio between successful and unsuccessful interactions made by users of a bus ticket machine
  • the percentage of drivers seen using a hand-held mobile phone
  • the number of ITS features that an ITS user can remember during a debriefing after a test
  • the frequency of use of the manuals and/or the help system, and the time spent using them by a VMS maintenance operative

It is also important to collect qualitative data. This can help explain the reasons behind a particular performance and may uncover the user's mental processes and beliefs about how the ITS operates (which may be correct or incorrect).

Addressing more strategic performance goals such as “safety” is a wider question in which the ITS has to be considered in the broader transport context. (See Road Safety)

Conducting Performance Testing

Performance testing can be complex. Consult human factors professionals where necessary and:

  • always conduct pilot testing to make sure that the tools and the techniques for data collection work as expected (See Piloting, Feedback and Monitoring)
  • if relevant, testing can be video-recorded so data can be analysed from the recording.

Data Analysis and Conclusions

Simple descriptive statistics (such as average values and spread) may be sufficient to characterise the particular performance metric – but:

  • for more complex analysis, consideration should be given to the extent to which the sample of tested users represents the whole user population
  • if the metric is to be compared against a benchmark, some indication of measurement error or confidence should also be estimated
  • the results from the individual metrics, as well as any qualitative data, should be considered as a whole when deciding if the usability goals have been met
  • data analysis can be complex so consult human factors professionals where necessary


Reference sources

US DOT website on driver distraction:

Bevan, N. and Macleod, M. (1994). Usability measurement in context. Behaviour and Information Technology 13 132-145

Wilson, J. and Corlett, N. (2005). Evaluation of Human Work. CRC Press. ISBN 0-415-26757-9

Sanders, M. and McCormick, E. (1993) Human Factors in Engineering and Design. McGraw-Hill, Inc. ISBN 0-07-054901-X

FOTnet project:

TeleFOT project: