Measuring Human Performance

Driver distraction (not focussing on the road ahead and the driving task) is an important issue for road safety. ITS products, such as information and communication devices, can greatly assist the driver (for example by indicating suitable routes) but ITS can also be an additional source of distraction. Distraction can make drivers less aware of other road users such as pedestrians and road workers and less observant of speed limits and traffic lights. (See Road Safety)

Measuring driving performance when interacting with ITS requires specialist equipment and expertise. Measurements made in laboratory settings and driving simulators may not be representative of real driving behaviour. This is because in real driving contexts drivers can choose when to interact (or not) with devices – and can modify their driving style to compensate to some extent for other demands on their attention. On-road measurements have to be designed to be unobtrusive and representative. Field Operational Tests (FOTs) can be designed accordingly and used to investigate both mature and innovative systems. FOTs can involve both professional and ordinary drivers according to the focus of investigation. A “naturalistic” driving study aims to unobtrusively record driver behaviour. Analysis of the drive record is used to identify safety-related events such as distraction, although the interpretation of results can be problematic and controversial.

The weight of scientific evidence points to distraction being an important safety issue. Many governments and Road Operators have sought to restrict drivers’ use of ITS while driving. There are different national and local approaches ranging from guidelines and advice – to bans on specific activities or functions (such as texting or hand-held phone use). 

Usability goals for an ITS product or service should be expressed in terms of the usability attribute – such as easy to learn, efficient to use, easy to remember, few errors, subjectively pleasing. Deciding on the relative importance of these goals depends on the ITS and the context but it helps to focus future evaluations on the most important aspects.

Not all performance measurements have to be quantitative but some simple examples of performance metrics that might be of interest to Road Operators are:

• the time taken by a control room operative to post a road closure on a message sign
• the number of toll charge queries collected at automated toll booths
• the ratio between successful and unsuccessful interactions made by users of a bus ticket machine
• the percentage of drivers seen using a hand-held mobile phone
• the number of ITS features that an ITS user can remember during a debriefing after a test
• the frequency of use of the manuals and/or the help system, and the time spent using them by a VMS maintenance operative

It is also important to collect qualitative data. This can help explain the reasons behind a particular performance and may uncover the user's mental processes and beliefs about how the ITS operates (which may be correct or incorrect).

Addressing more strategic performance goals such as “safety” is a wider question in which the ITS has to be considered in the broader transport context. (See Road Safety)

Performance testing can be complex. Consult human factors professionals where necessary and:

• always conduct pilot testing to make sure that the tools and the techniques for data collection work as expected (See Piloting, Feedback and Monitoring)
• if relevant, testing can be video-recorded so data can be analysed from the recording.

Simple descriptive statistics (such as average values and spread) may be sufficient to characterise the particular performance metric – but:

• for more complex analysis, consideration should be given to the extent to which the sample of tested users represents the whole user population
• if the metric is to be compared against a benchmark, some indication of measurement error or confidence should also be estimated
• the results from the individual metrics, as well as any qualitative data, should be considered as a whole when deciding if the usability goals have been met
• data analysis can be complex so consult human factors professionals where necessary