Overall evaluation of ITS products and services is an important activity because the performance of the road user will depend crucially on the usability of the ITS. A benefit of improving the ease and efficiency of ITS technology, is increased user satisfaction. This can provide business advantages, particularly when users have a choice of ITS products or services. Poor usability of ITS, such as a poor user interface, or inadequate and misleading dynamic signage, may have safety implications in the road environment. Good usability will help to manage and predict road user behaviour and so help increase road network performance. For all these reasons, the performance of ITS in terms of its usability needs to be measured. (See Evaluation)
Usability measurement requires assessment of the effectiveness, efficiency and satisfaction with which representative users carry out representative tasks in representative environments. This requires decisions to be made on the following key points:
Steps in performance measurement:
The process of measuring human performance when the driver is interacting with ITS (particularly when using information and communication devices) can yield safety benefits – although there are challenges with this form of human performance measurement in this context.
Driver distraction (not focussing on the road ahead and the driving task) is an important issue for road safety. ITS products, such as information and communication devices, can greatly assist the driver (for example by indicating suitable routes) but ITS can also be an additional source of distraction. Distraction can make drivers less aware of other road users such as pedestrians and road workers and less observant of speed limits and traffic lights. (See Road Safety)
Measuring driving performance when interacting with ITS requires specialist equipment and expertise. Measurements made in laboratory settings and driving simulators may not be representative of real driving behaviour. This is because in real driving contexts drivers can choose when to interact (or not) with devices – and can modify their driving style to compensate to some extent for other demands on their attention. On-road measurements have to be designed to be unobtrusive and representative. Field Operational Tests (FOTs) can be designed accordingly and used to investigate both mature and innovative systems. FOTs can involve both professional and ordinary drivers according to the focus of investigation. A “naturalistic” driving study aims to unobtrusively record driver behaviour. Analysis of the drive record is used to identify safety-related events such as distraction, although the interpretation of results can be problematic and controversial.
The weight of scientific evidence points to distraction being an important safety issue. Many governments and Road Operators have sought to restrict drivers’ use of ITS while driving. There are different national and local approaches ranging from guidelines and advice – to bans on specific activities or functions (such as texting or hand-held phone use).
Usability goals for an ITS product or service should be expressed in terms of the usability attribute – such as easy to learn, efficient to use, easy to remember, few errors, subjectively pleasing. Deciding on the relative importance of these goals depends on the ITS and the context but it helps to focus future evaluations on the most important aspects.
Not all performance measurements have to be quantitative but some simple examples of performance metrics that might be of interest to Road Operators are:
It is also important to collect qualitative data. This can help explain the reasons behind a particular performance and may uncover the user's mental processes and beliefs about how the ITS operates (which may be correct or incorrect).
Performance testing can be complex. Consult human factors professionals where necessary and:
Simple descriptive statistics (such as average values and spread) may be sufficient to characterise the particular performance metric – but: