Evaluation of the performance of accelerometer-based gait event detection algorithms in different real-world scenarios using the MAREA gait database

Highlights

• A new database of gait in indoor and outdoor environments using accelerometers.

• Evaluation of the robustness of state-of-the-art gait event detection algorithms.

• Algorithmic performance is significantly decreased in outdoor walking and running.

• Algorithms show better performance for detecting Heel-Strikes compared to Toe-Offs.

• An alternative way of assessing performance based on non-parametric statistical tests.

Abstract

Numerous gait event detection (GED) algorithms have been developed using accelerometers as they allow the possibility of long-term gait analysis in everyday life. However, almost all such existing algorithms have been developed and assessed using data collected in controlled indoor experiments with pre-defined paths and walking speeds. On the contrary, human gait is quite dynamic in the real-world, often involving varying gait speeds, changing surfaces and varying surface inclinations. Though portable wearable systems can be used to conduct experiments directly in the real-world, there is a lack of publicly available gait datasets or studies evaluating the performance of existing GED algorithms in various real-world settings.

This paper presents a new gait database called MAREA (n = 20 healthy subjects) that consists of walking and running in indoor and outdoor environments with accelerometers positioned on waist, wrist and both ankles. The study also evaluates the performance of six state-of-the-art accelerometer-based GED algorithms in different real-world scenarios, using the MAREA gait database. The results reveal that the performance of these algorithms is inconsistent and varies with changing environments and gait speeds. All algorithms demonstrated good performance for the scenario of steady walking in a controlled indoor environment with a combined median F1score of 0.98 for Heel-Strikes and 0.94 for Toe-Offs. However, they exhibited significantly decreased performance when evaluated in other lesser controlled scenarios such as walking and running in an outdoor street, with a combined median F1score of 0.82 for Heel-Strikes and 0.53 for Toe-Offs. Moreover, all GED algorithms displayed better performance for detecting Heel-Strikes as compared to Toe-Offs, when evaluated in different scenarios.

Introduction

The development of gait event detection (GED) algorithms using various sensing modalities has been an active area of research for many years [1]. In the past decade, several GED algorithms have been developed using motion capture systems, present in gait labs, and recent studies have compared and evaluated their performance [2], [3], [4], [5]. Alternatively, inertial sensors are being used as they allow the possibility of long-term monitoring in everyday life and provide spatio-temporal information that can be fused to obtain the entire trajectory of the limb segment [6], [7]. While many GED algorithms have been developed using gyroscopes, others have used accelerometers as they are miniature, inexpensive and low-powered devices [1], [8]. However, as accelerometers suffer heavily from noise due to mechanical vibrations, they require robust algorithms for accurate event detection. Almost all existing accelerometer-based GED algorithms have been developed and assessed using data collected in controlled indoor experiments, that usually involves instructing the subjects to walk in a straight line or a given path at self-selected pace [9], [10] or predefined walking speeds [11], [12], [13]. On the contrary, in the real-world, human gait is quite dynamic in different environments, often involving varying gait speeds, changing walking surfaces and varying surface inclinations, among others. Therefore, it needs to be assessed whether such dynamic and uncontrolled real-world scenarios have an impact on the performance of existing GED methods which have been developed from controlled protocols in laboratory settings. However, as almost all publicly available accelerometer-based gait databases [14], [15] and recent comparative studies [16] also consist of only controlled indoor experiments, there is a lack of gait datasets or any studies that evaluate the performance of existing GED methods in various real-world settings; especially when portable wearable systems can be readily used to conduct experiments directly in humans’ natural environment.

Consequently, a new gait database called MAREA: Movement Analysis in Real-world Environments using Accelerometers, was collected that comprises of various gait activities in different environments, both indoors and outdoors. The objective of this study is two-fold: (1) to introduce the MAREA database which is made publicly available for all readers, and (2) to assess the impact of different real-world scenarios on the performance of state-of-the-art GED algorithms, using the MAREA database. The database is made publicly available at http://islab.hh.se/mediawiki/Gait_database (Table 1).