Abstract

Exoskeletons have decreased physical effort and increased comfort in activities of daily living (ADL) such as walking, squatting, and running. However, this assistance is often activity specific and does not accommodate a wide variety of different activities. To overcome this limitation and increase the scope of exoskeleton application, an automatic human activity recognition (HAR) system is necessary. We developed two deep-learning models for HAR using one-dimensional-convolutional neural network (CNN) and a hybrid model using CNNs and long-short term memory (LSTM). We trained both models using the data collected from a single three-axis accelerometer placed on the chest of ten subjects. We were able to classify five different activities, standing, walking on level ground, walking on an incline, running, and squatting, with an accuracy of 98.1% and 97.8%, respectively. A two subject real-time validation trial was also conducted to validate the real-time applicability of the system. The real-time accuracy was measured at 96.6% and 97.2% for the CNN and the hybrid model, respectively. The high classification accuracy in the test and real-time evaluation suggests that a single sensor could distinguish human activities using machine-learning-based models.

1 Introduction

Recent developments in exoskeletons have enhanced locomotion and increased the quality of life for activities of daily living [16]. Such exoskeletons have shown a substantial reduction in physical effort in walking [79], running [10], squatting [5,11], and lifting applications [12]. Although tested in a lab setting, the activity-specific assistance is poised to improve the activities of daily living of a large population. However, switching between activities and providing appropriate assistance for these activities is still a significant challenge for exoskeleton system controls.

Several studies have developed an automatic assistance-switching method using sensors embedded in the exoskeleton. These sensors identify and provide assistance based on the activity [3,13]. Kim et al. [3] used a hip and thigh mounted inertial measurement unit (IMU) sensor and developed a heuristic classification method to identify the walking and running activity. Since the heuristic method is subject-dependent and requires substantial parameter tuning, this approach is not practical for general-purpose use. Medrano et al. [13] used the ankle angle and torque sensors in an ankle exoskeleton with an extended Kalman filter to accurately identify the ground elevation. These methods, however, are exoskeleton specific, making it challenging to use the same algorithm for different exoskeletons. In addition, during the developmental stage, the algorithms need to be iteratively developed during the process. To address these issues, using additional sensors for activity recognition instead of the sensors embedded in the exoskeleton would be helpful.

Various types of sensors have been used for activity classification: vision-based devices, such as cameras [14], depth-based Kinect sensors [15], or wearable devices such as inertial measurement units [1618]. Hussain et al. [19] has performed a detailed analysis of the various types of sensors used for human activity classification and their different applications. Vision-based and depth-based sensors require constant filtering and stabilization of the image due to the large variation from walking/running movement [20], which increases the computation requirements of the classification system. This disadvantage of the vision-based system makes wearable sensors a better choice for activity classification studies.

For wearable sensors, the position of the sensor on the user has been shown to affect the system's performance significantly [21]. Comparison studies have been done to find the best position for the wearable sensor [22]. Qamar et al. [21] has found that a model's accuracy is higher when the sensor is placed on the user's chest than on the hand and/or ankle. In addition, this position would be independent of most lower limb exoskeletons and would not interfere with their application.

A common approach to wearable sensor-based activity recognition is feature extraction from the data collected by the sensors. Janidarmian et al. [22] compared a model's performance using different features such as mean, standard deviation, pitch, roll angle, etc. Though such methods have progressed well using shallow machine learning techniques [23], they face challenges extracting distinctive features of complex activities such as climbing stairs or walking on an inclined surface without hand-tuning [2426]. A deep learning system could address these issues and develop an activity recognition system with a broad range of activities.

Recently, convolutional neural networks (CNN) have been promising in the field of human activity recognition [27,28]. CNNs are known to capture the relevant feature from the temporal data, and pooling layers can introduce robustness to variance in the measurement [29]. In addition, compared to fully connected layers, CNNs have fewer trainable parameters and hence, are much faster to train [29,30]. Also, CNN is more robust to sensor position and window size of the data than fully connected Neural Networks and traditional machine learning-based approaches like support vector machines (SVM) [31]. In addition to these advantages, CNN has also been shown to have better accuracy per memory usage [32] than other deep-learning architectures.

Recently for human activity recognition (HAR), researchers have also used recurrent neural networks (RNN) due to their efficiency in dealing with time-series data [33]. However, RNNs are plagued by the exploding/vanishing gradient problem which hampers learning model weights. Long short-term memory (LSTM) networks, a form of RNN, can overcome this problem making them more suitable [34]. The forget gate of LSTM controls the amount of long and short-term memory held by the LSTM, thereby enabling them to make unbiased continuous predictions [35]. Britz et al. [36] has shown that LSTMs perform better than other variants of RNNs such as gated recurrent units. Though many variants of LSTMs have been introduced, Greff et al. [37] has shown that these modifications do not significantly improve the standard variant. They also found that the forget gate and the output gate activation influence the performance of an LSTM model. Though LSTMs have shown good performance when dealing with temporal data, they fall behind CNNs due to their computational complexity and memory usage [38,39]. To compensate for these drawbacks, LSTMs have been used in combination with CNNs [40].

Although the current state-of-the-art methods use CNNs and LSTMs, they typically require multiple sensors [41,42]; placed in less practical positions such as ankle [41,42], or hip [43,44]; require many sensors [42,45]; or use longer window size [1.6 s–5.12 s] [42,46], as shown in Table 1. For an exoskeleton application, a model with a low number of sensors or a shorter window size is critical. A large window size could introduce a considerable lag in predictions that could be detrimental to the user. We propose a HAR system with a single sensor and a short time window to address these issues.

In this study, we hypothesized that CNN and LSTM-based models can recognize human activities using a single chest-mounted accelerometer. To address this hypothesis, we developed two models using state-of-the-art deep learning architectures. The first model consists of only CNN layers while the second model is a hybrid of CNN and LSTM layers, as described in Secs. 2.1 and 2.2, respectively. Through a controlled human subject study, we collected 10 subjects' data performing running, walking on level ground, walking on an incline, standing, and squatting activities. We trained and tested both the CNN and hybrid model using the collected data. We then validated the real-time accuracy using a two-subject trial. The data collection process for training and testing is described in Sec. 2.3. Section 3 describes the performance of the two models during the offline and real-time validation. The implications of these results are discussed in Sec. 4. Finally, in Sec. 5, we have provided the concluding remarks and the future scope of this project

2 Methods

2.1 Model Architecture: Convolutional Neural Network.

To develop the CNN model, the architecture followed by Bo Yang et al. [56] was used as a starting point. After preliminary testing, to improve the performance, we increased the number of convolution layers from four to six and increased their kernel size from three to five. Along with these architectural changes, the number of filters in each convolutional layer was also doubled. Padding was used to ensure the input and output tensor lengths remained the same. All convolutional layers use rectified linear unit (ReLU) as the activation. Each layer was then followed by a MaxPool layer with kernel size and stride 2 to reduce the size of the tensor. For every successive convolutional layer, a dropout layer with a dropout rate of 0.3 was added to reduce the tendency of the model to over-fit on the training dataset. Finally, the output of the last convolutional layer is flattened and passed through a fully connected dense layer whose output dimension is equal to the number of classes. For the fully connected output layer, we used the SoftMax layer as the activation, which returns the probability of occurrence of each label for that particular input. This model contains 638,278 trainable parameters. Figure 1(a) illustrates the architecture of the CNN model.

Fig. 1
Representative figure of the human activity recognition system. The data collected from the sensor placed on the chest as shown in the left column of the figure containing the three-axis acceleration is passed to the two Neural Networks, Conv: one-dimensional (1D)-convolutional layer, LSTM: long short term memory layer FC: Fully connected layer: (a) architecture of CNN model: Conv1(64), Conv2(64), Conv3(128), Conv(128), Conv(256), Conv(256), FC(256), Softmax: 1X5 and (b) architecture of hybrid model (CNN+LSTM): Conv1(64), Conv2(128), LSTM(256), Conv3(128), FC(384), Softmax: 1X5. The number in the brackets represents the number of filters/hidden nodes in that layer.
Fig. 1
Representative figure of the human activity recognition system. The data collected from the sensor placed on the chest as shown in the left column of the figure containing the three-axis acceleration is passed to the two Neural Networks, Conv: one-dimensional (1D)-convolutional layer, LSTM: long short term memory layer FC: Fully connected layer: (a) architecture of CNN model: Conv1(64), Conv2(64), Conv3(128), Conv(128), Conv(256), Conv(256), FC(256), Softmax: 1X5 and (b) architecture of hybrid model (CNN+LSTM): Conv1(64), Conv2(128), LSTM(256), Conv3(128), FC(384), Softmax: 1X5. The number in the brackets represents the number of filters/hidden nodes in that layer.
Close modal

2.2 Model Architecture: Hybrid Model (CNN + LSTM).

For this model, we use CNN layers as a feature extractor and LSTM for the time series feature classification. Figure 1(b) illustrates the model architecture of the hybrid model (CNN + LSTM) where two 1D-Convolutional layers precede four LSTM layers. The two convolutional layers, which act as feature extractors, have 64 and 128 filters, respectively. A 1D-MaxPool layer succeeded each convolutional layer with kernel size and stride 3. A Batch Normalization layer was added after every Maxpool layer to decrease overfitting to the training dataset. Each LSTM layer has 256 cells, and the output of the last LSTM layer is fed into another one-dimensional Convolutional layer with 128 filters. After being passed through the third convolutional layer, the tensor is flattened and passed through a dense layer whose output is the number of different classes, in our case, 5. A SoftMax layer is then applied to obtain the probability of occurrence of each class. All convolutional layers use ReLU activation, while the LSTM layers have the Tanh activation function. The convolutional layers have a kernel size of 9 with stride 1. No padding was added for the convolutional layers in the hybrid model.

2.3 Experimental Protocol.

We performed a one-day experiment and collected ten subjects' data to train and test the activity recognition system (10 male, height: 176.6 ± 8.2 cm, mass: 79.2 ± 10.4 kg, age: 24.6 ± 4.0 years). The data collected included five activities that followed the data-collection protocol described in Table 2. During this protocol, each subject walked on level ground, on an incline (+4 deg), stood on a treadmill for 3 min each, and ran on a treadmill for 4 min. Three minutes of sitting rest was provided between each activity. The walking speed during the study was set to 1.25 m/s using a treadmill, for both walking on level ground and incline conditions. The speed during running was set to 2.5 m/s. The detailed protocol followed during squatting activity can be found in Ref. [5]. We used the day1 data of the [5] study which included 180 squats of each subject with the powered, unpowered, and no exoskeleton conditions. Each squat lasted approximately 2 s and was followed by 6 s of standing rest. All the squats were done on the ground.

Table 1

Summary of the current state of the art of CNN and hybrid models for human activity recognition

AuthorsDataset(s) usedPositionActivitiesInput FeaturesWindow sizeAccuracy
Convolutional neural networks
Our modelSelf-collected dataaChest53198.1%
[45]PAMAP2bChest, dominant wrist and ankle12275.1293.75%
[43]Self-collected datacUpper leg63499.28%
[46]PAMAP2Chest, dominant wrist and ankle1235.1291.93%
[41]PAMAP2Chest, dominant wrist and ankle12275.1291.4%
[47]PAMAP2Chest, dominant wrist and ankle12275.1294%
[48]UCI-HARdUpper right leg662.5692.71%
[49]UCI-HARUpper right leg662.5694.8%
[50]Self-collected dataeUpper leg832.5693.8%
Hybrid models: CNN and LSTM
Our modelSelf-collected dataChest53197.8%
[44]UCI-HARUpper right leg662.5696.4%
[51]UCI-HARUpper right leg662.5695.7%
[40]UCI-HARUpper right leg662.5695.8%
[52]WISDMfUpper right leg63497.1%
[42]MHEALTHgChest, dominant wrist and ankle12271.694.86%
[53]PAMAP2Chest, dominant wrist and ankle12275.1293.52%
[54]UCI-HARUpper right leg662.5697.9%
[55]MHEALTHChest, dominant wrist and ankle12271.699.8%
AuthorsDataset(s) usedPositionActivitiesInput FeaturesWindow sizeAccuracy
Convolutional neural networks
Our modelSelf-collected dataaChest53198.1%
[45]PAMAP2bChest, dominant wrist and ankle12275.1293.75%
[43]Self-collected datacUpper leg63499.28%
[46]PAMAP2Chest, dominant wrist and ankle1235.1291.93%
[41]PAMAP2Chest, dominant wrist and ankle12275.1291.4%
[47]PAMAP2Chest, dominant wrist and ankle12275.1294%
[48]UCI-HARdUpper right leg662.5692.71%
[49]UCI-HARUpper right leg662.5694.8%
[50]Self-collected dataeUpper leg832.5693.8%
Hybrid models: CNN and LSTM
Our modelSelf-collected dataChest53197.8%
[44]UCI-HARUpper right leg662.5696.4%
[51]UCI-HARUpper right leg662.5695.7%
[40]UCI-HARUpper right leg662.5695.8%
[52]WISDMfUpper right leg63497.1%
[42]MHEALTHgChest, dominant wrist and ankle12271.694.86%
[53]PAMAP2Chest, dominant wrist and ankle12275.1293.52%
[54]UCI-HARUpper right leg662.5697.9%
[55]MHEALTHChest, dominant wrist and ankle12271.699.8%
a

Activities: Standing, walking on level ground, walking on an incline, running, squatting, sensors: three axis accelerometer.

b

Activities: Standing, walking, running, lying, sitting, cycling, nordic walking, ascending stairs, descending stairs, jumping, vacuum cleaning, ironing, sensors: three axis accelerometer, three axis gyroscope, and three axis magnetometer.

c

Activities: Standing, walking, lying, sitting, cycling, jogging, sensors: three axis accelerometer.

d

Activities: Standing, walking, lying, sitting, ascending stairs, descending stairs, sensors: three axis accelerometer, three axis gyroscope.

e

Activities: Walking, ascending stairs, descending stairs, jumping, falling, step walking, walking quickly, sensors: three axis accelerometer.

f

Activities: Standing, walking, sitting, ascending stairs, descending stairs, jogging, sensors: three axis accelerometer.

g

Activities: Standing, walking, running, squatting, lying, sitting, cycling, ascending stairs, jumping, waist bends forward, frontal elevation of arms, jogging, sensors: three axis accelerometer, three axis gyroscope, and three axis magnetometer.

During all the trials, the subject wore a Polar H10 (Kempele, Finland) chest strap and sensor to measure the accelerometer data. The three-axis accelerometer data were collected at 200 Hz using a custom python-based Bluetooth and Labstreaminglayer (LSL) application. The range of the three-axis accelerometer was −2000 to +2000 cm/s2 . The data from all ten subjects were combined and randomized for training and testing of the algorithms.

To validate the testing model accuracy and generalizability, we did a 2-subject validation trial that followed the validation protocol, as shown in Table 2. Out of the two subjects (Sub1 and Sub2), Sub1 was an experienced participant who participated in the previous data collection while Sub2 was a novice participant. The experimental pipeline remained the same as the data collection pipeline, but alongside the accelerometer data, the model's prediction was also recorded to analyze the accuracy of the prediction. The models' prediction was conducted every 0.5 s. The study protocol was approved by the University of Illinois at Chicago, Institutional review board, and informed consent was obtained for each participant.

2.4 Data Preparation.

Preprocessing of the raw data included converting the data into small windows of fixed length, each of which would act as a separate input to the model. For both models, we selected the window size as 1 s ( i.e., 200 data points) and a window step size of 0.5 s based on the previous testing. As transitioning between different activities takes about 0.3–0.7 s [57,58], a 0.5 s step size should be large enough to detect the transition. A 0.5-second step size would also keep the real-time detection robust. Since the window step size is smaller than the window size, each input sample has a 50% overlap with its predecessor. For the squatting activity, each squat was individually identified and segmented from the standing data.

To remove any influence of subject-specific body posture in the data, the data were normalized by subtracting each subject's mean of the first minute of standing condition data. Alongside this, data was augmented to add more variance to the dataset, avoid overfitting and improve the prediction accuracy of all classes. For the augmentation of data, we used three different techniques, noise injection, scaling, and magnitude warping [5961].

  1. Noise injection: Gaussian white noise with a mean of 0 and a standard deviation within the range of 0–10 cm/s2 was added to the collected data.

  2. Scaling: The absolute value of each sample was multiplied with a Gaussian curve of mean 1 and standard deviation in the range of 0–0.3.

  3. Magnitude warping: Element-wise multiplication of the sample and a third order curve, generated using cubic spline with n number of knots of random magnitude, where n lies between 5 and 12.

The time period of the squats, during data collection, was fixed (1 s ascending state and 1 s descending state) to maintain consistency between subjects and conditions [5]. However, training on such a dataset could lead to bias in the model. To address this potential bias, during the augmentation procedure, we resampled squatting data with varying periods in the range of 1–2 s to make the model more robust and applicable to everyday scenarios.

The final dataset contains 27,895 samples, of which, squatting contains the highest number of samples at 8266 while walking on the level ground contains the least at 3650. Running, standing, and walking on an incline contained 6193, 4583, and 5203 samples, respectively. A 70–30 random split was used on the dataset to create a training and testing dataset. The model parameters were trained only on the training dataset, while the testing dataset was used to validate its performance.

2.5 Training.

Categorical cross-entropy was used as the loss function for both the CNN and hybrid models. The Adam optimizer was used to update the model weights of both models. The two models were trained for 100 epochs, each epoch taking 2 s for the CNN model and 7 s for the hybrid model. Both models were trained using PyTorch and were accelerated by Cuda with an Intel i5 CPU and an Nvidia GeForce GTX 1650 GPU.

To validate the performance of the models, they were trained and tested on two public datasets, UCI and PAMAP2. A window size of 5.12 s was chosen for the PAMAP2 dataset, and 2.56 s was selected for the UCI dataset, similar to the papers shown in Table 1. Both models were trained on the prepartitioned UCI training dataset for 100 epochs and tested on the test dataset. The PAMAP2 dataset was split into the training and testing dataset with a 70–30 ratio and the models were trained for 100 epochs.

Table 2

Protocol table

Data collection protocolValidation protocol
ActivityTimeActivityTime
Standing3 minStanding3 min
Walking on level ground3 minWalking on level ground3 min
Walking on an incline3 minWalking on an incline3 min
Running4 minRunning4 min
Squatting180 repsSquatting10 reps
Data collection protocolValidation protocol
ActivityTimeActivityTime
Standing3 minStanding3 min
Walking on level ground3 minWalking on level ground3 min
Walking on an incline3 minWalking on an incline3 min
Running4 minRunning4 min
Squatting180 repsSquatting10 reps

It details the data collection and the validation protocol used in this study.

3 Results

3.1 Offline Validation.

The CNN model achieved a maximum train accuracy of 99.9% after 100 epochs and a maximum test accuracy of 98.1% after 58 epochs. The hybrid model achieved a maximum train accuracy of 99.9% after 92 epochs and a maximum test accuracy of 97.8% after 89 epochs. The test accuracy of both models is higher than most of the state-of-the-art models, Table 1.

Figure 2(b) shows the training and test loss calculated using the categorical cross-entropy function after each epoch for the two models. The CNN model had the lowest training loss of 0.0029 after 100 epochs and a minimum test loss of 0.067 after 66 epochs. The hybrid model had the lowest train loss of 0.0042 after 95 epochs and a minimum test loss of 0.093 after 29 epochs.

Fig. 2
Accuracy and loss of the CNN and hybrid model during training. (a) Accuracy on the train and test dataset of the two models achieved after each training epoch. The best test accuracy is obtained with the CNN model. (b)Cross Entropy Loss of the train and test dataset for the two models calculated after each training epoch. The CNN model results in a lower loss.
Fig. 2
Accuracy and loss of the CNN and hybrid model during training. (a) Accuracy on the train and test dataset of the two models achieved after each training epoch. The best test accuracy is obtained with the CNN model. (b)Cross Entropy Loss of the train and test dataset for the two models calculated after each training epoch. The CNN model results in a lower loss.
Close modal

Table 3 shows the accuracy and F1-score of each activity for both models. The models with the highest test accuracy during training were used to calculate the F1-score and accuracy. From the table, it can be seen that for both models squatting class returns the highest F1-score at 0.99 and 1, respectively. Both running and squatting have the highest accuracy at 0.99 for both models. It can also be seen that for both models, walking on an incline has the lowest F1-score at 0.95 and 0.94, respectively. Walking on an incline also returns the most insufficient accuracy for the hybrid model at 0.92.

Table 3

Accuracy and F1-score of the two models on each class of the test dataset

CNN modelHybrid model
ActivityAccuracyF1-scoreAccuracyF1-score
Running0.990.990.990.99
Walking on level ground0.950.960.960.95
Walking on an incline0.970.950.920.94
Standing10.9810.99
Squatting0.990.990.990.99
CNN modelHybrid model
ActivityAccuracyF1-scoreAccuracyF1-score
Running0.990.990.990.99
Walking on level ground0.950.960.960.95
Walking on an incline0.970.950.920.94
Standing10.9810.99
Squatting0.990.990.990.99

The two models were trained and tested on the public datasets UCI and PAMAP2 to validate their performance while keeping the parameters from our models the same (e.g., architecture, the number of hidden layers, the learning rate, and the optimizer). The CNN model achieved an accuracy of 0.91 and 0.90 on the two datasets, respectively, and the hybrid model achieved an accuracy of 0.92 and 0.90, respectively.

When tested on a larger window size of 4 s, the performance of the models improved, with the CNN and hybrid models achieving 0.997 and 0.997 test accuracy, respectively. Due to the larger window size, the computation time of training for one epoch increased from 2 s to 7 s for the CNN model and 7 s to 18 s for the hybrid. The inference time also increased from 0.003 s to 0.004 s when using a 4 s window.

3.2 Real-Time Validation.

After offline testing, a two-subject validation trial was conducted where the prediction of both models was collected in real-time.

The CNN model gave an overall accuracy of 0.989 and 0.943 for the two subjects. Running, standing, and squatting returned an accuracy of 1 for both subjects. For Sub1, the lowest accuracy of 0.96 was obtained for walking on level ground while for Sub2, the lowest accuracy was for walking on an incline at 0.72. The hybrid model gave an overall accuracy of 0.996 for Sub1 and 0.948 for Sub2. Similar to the CNN model running, standing and squatting have an accuracy of 1 for both subjects. The lowest accuracy of 0.75 was obtained on Sub2 for walking on an incline. The detailed confusion matrices for the real-time validation on both subjects are shown in Fig. 3.

Fig. 3
Real-time validation results performed on two subjects: (a) and (b) Confusion matrices of the Real-time validation results of Subject 1 and Subject 2 for the CNN model. (c) and (d) Confusion matrices of the real-time validation results of Subject 1 and Subject 2 for the Hybrid model: CNN + LSTM.
Fig. 3
Real-time validation results performed on two subjects: (a) and (b) Confusion matrices of the Real-time validation results of Subject 1 and Subject 2 for the CNN model. (c) and (d) Confusion matrices of the real-time validation results of Subject 1 and Subject 2 for the Hybrid model: CNN + LSTM.
Close modal

4 Discussion

In this study, we developed and compared two Neural Network architectures using CNN and LSTM layers to perform human activity recognition. Both the models were trained on ten subjects' data with test and train accuracy of 98.1% and 99.9% for the CNN model and 97.8% and 99.9% for the hybrid model. Since real-time accuracy is crucial for exoskeleton applications, we validated the real-time performance by conducting a two-subject test. During this test, we found that the accuracy was consistent at 96.6% and 97.2% for the CNN and hybrid models, respectively, indicating that the models can be deployed in real-world applications.

A single chest-mounted accelerometer-based HAR can be versatile for most exoskeletons. Both the CNN and hybrid models used the accelerometer information and could predict the activity with high accuracy, even compared to the multisensor setup [53]. With this high accuracy and sensor position independent of most exoskeleton sensors, the same algorithm and sensor setup could be generalized for different exoskeleton combinations without retraining the model. Future work can include expanding a study to different sensor positions for more practical applications.

The models were trained and tested on ten subject data. Both models achieved a high-test accuracy; this could be due to the similarity between the test and training dataset as it included the same participants. The models' performance could suffer when tested on a completely new subject. To test this aspect, we tested both models in real-time on a new subject (Sub2) whose data was not included in the training dataset. Both models returned a reasonable accuracy of 0.94 and 0.95 for Sub2. Since subject 1 (Sub1), whose data were included in the training set, showed higher performance, future work includes further improving the model robustness to address intersubject variability.

Convolutional neural network model was memory and computationally efficient compared to the hybrid model. The CNN model with 638,021 trainable parameters performed better in terms of accuracy per memory usage when compared to the hybrid model, which had 2,347,525 parameters. The CNN model also takes less time to train, taking 2 s per epoch compared to 7 s for the hybrid model. This could be due to the fact that computations in LSTMs are not parallelized [34] unlike those in CNN.

The performance of the hybrid model was similar to that of the pure CNN model. The CNN model achieved 98.1% and 96.6% accuracy during offline and real-time testing, while the hybrid model gave an accuracy of 97.8% and 97.2%, respectively. LSTMs are generally known to encode time series information [38], while CNNs generally encode spatial information [62]. However, in this study, we saw that adding LSTM layers to a CNN model to classify time series data did not significantly improve the model's performance. This result was different from the finding of Li et al. [63], where adding LSTM layers to a CNN model significantly improved the model's performance. This change in accuracy could be due to our limited sample size or uneven data distribution. Further testing on a more extensive and balanced dataset is needed to compare the performance.

The classification time window influences the accuracy and the computation needs. During the model development, we also tested different time windows of classification based on other papers (Table 1). With a 4 s window size, the accuracy was improved by 2% with the more than doubled training time associated with higher computational costs. In addition, a large window size might not be practical for a short burst activity such as squatting, typically 1–2 s. Considering this computation costs for a small improvement in the performance, we selected the 1 s time window as our approach across activities. However, future applications could consider an activity-specific time window with faster training systems.

One of the main limitations of this study is that transition accuracy is not measured. With a window step size of 0.5 s and a window size of 1 s, we were able to achieve high accuracy of 0.96 in real-time. This high accuracy for both subjects suggests fast transition detection between activities. Rigorous testing, however, is needed to confirm the transition detection speed and accuracy in future work. In particular, for exoskeleton application, improper timing or inaccurate transition could lead to discomfort in the assistance. Future work also includes improving the algorithm using more sensory modalities, such as the gyroscope and the signal derivatives, to improve speed and accuracy.

Additional sensor modalities could help accurately classify walking on level ground and on an incline condition. In most real-world applications, incline walking is a routine activity, making this classification important. In our study, both CNN and hybrid models showed lower classification accuracy between walking on level ground and on an incline for Sub2, as shown in Fig. 3. These misclassifications could be due to intersubject variability in the walking posture and trunk angle across subjects [64]. Another possible reason is the angle of the slope at which walking on incline data was collected (+4 deg), which could be too small to make a noticeable difference in the accelerometer data. Adding a three-dimensional-gyroscope to the chest sensor while keeping the number of sensors constant could significantly improve the performance of the models. This addition could also help distinguish between more activities such as running on level ground and on an incline. When we tested on public datasets using our models, we observed a reasonable accuracy, higher than 0.90, similar to Refs. [65] and [66]. This suggests that using a single sensor makes it possible to obtain comparable accuracy. Future work includes using additional sensor modalities to improve accuracy and speed.

An additional transition state might be required for the practical implementation of this method with an exoskeleton. Since the activity recognition system observes the current state and identifies the activity, using this system in the exoskeleton might lead to a delayed change in control action after transitioning of activity. To address this, adding a transition system [67] between conditions using additional sensor modalities could help smooth the transition of the controllers in the exoskeleton.

Future studies can consider adding data from activities performed in an outdoor environment. For this study, running, walking on level ground, and walking on an incline data was collected on a treadmill. Although most exoskeletons are currently tested on the treadmill, the final application of these systems will be in the outdoor setting. Hence future studies should have both outdoor and indoor experiments to collect data and validate the applicability of the proposed HAR models.

5 Conclusion

A generalizable human activity recognition system is critical to making exoskeletons usable for day-to-day outdoor activities. This paper presents two model architectures: CNN and a hybrid model with CNN and LSTM layers. The two models use the data from a single three-axis accelerometer strapped to the chest, to classify between five activities. The two models achieved a high accuracy of 98.1% and 97.8% during offline testing and 96.6% and 97.2% during real-time validation. The high accuracy suggests that the data from a single accelerometer can be used to classify between activities, and the model's prediction can be passed on to the exoskeleton. Future work could entail adding a three-axis gyroscope to the input data, increasing the number of activities, and integrating this into exoskeleton assistance. Increasing the range of the accelerometer from the current −2000 to +2000 cm/s2 could also help distinguish more activities.

Acknowledgment

We are grateful for the assistance of Michael Jacobson, Inigo Sanz Pena, Sabrina Sullivan, Hyeongkeun Jeong, Gabe Griffin, Atharva Deshpande, and other UIC Rehabilitation Robotics lab members for their support. We also thank Courtney Haynes and Cortney Bradford for their guidance during the project and for their feedback on the paper.

Funding Data

  • US Army (Award No. W911NF2120230; Funder ID: 10.13039/100006754).

Data Availability Statement

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.

References

1.
Ding
,
Y.
,
Kim
,
M.
,
Kuindersma
,
S.
, and
Walsh
,
C. J.
,
2018
, “
Human-in-the-Loop Optimization of Hip Assistance With a Soft Exosuit During Walking
,”
Sci. Rob.
,
3
(
15
).10.1126/scirobotics.aar5438
2.
Zhang
,
J.
,
Fiers
,
P.
,
Witte
,
K. A.
,
Jackson
,
R. W.
,
Poggensee
,
K. L.
,
Atkeson
,
C. G.
, and
Collins
,
S. H.
,
2017
, “
Human-in-the-Loop Optimization of Exoskeleton Assistance During Walking
,”
Science
, 356(6344), pp.
1280
1284
.10.1126/science.aal5054
3.
Kim
,
J.
,
Lee
,
G.
,
Heimgartner
,
R.
,
Revi
,
D. A.
,
Karavas
,
N.
,
Nathanson
,
D.
,
Galiana
,
I.
, et al.,
2019
, “
Reducing the Metabolic Rate of Walking and Running With a Versatile, Portable Exosuit
,”
Science
,
365
(
6454
), pp.
668
672
.10.1126/science.aav7536
4.
Kim
,
M.
,
Jeong
,
H.
,
Kantharaju
,
P.
,
Yoo
,
D.
,
Jacobson
,
M.
,
Shin
,
D.
,
Han
,
C.
, and
Patton
,
J. L.
,
2022
, “
Visual Guidance Can Help With the Use of a Robotic Exoskeleton During Human Walking
,”
Sci. Rep.
, 12(1), pp.
1
10
.10.1038/s41598-022-07736-w
5.
Kantharaju
,
P.
,
Jeong
,
H.
,
Ramadurai
,
S.
,
Jacobson
,
M.
,
Jeong
,
H.
, and
Kim
,
M.
,
2022
, “
Reducing Squat Physical Effort Using Personalized Assistance From an Ankle Exoskeleton
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
30
, pp.
1786
1795
.10.1109/TNSRE.2022.3186692
6.
Housley
,
S. N.
,
Wu
,
D.
,
Richards
,
K.
,
Belagaje
,
S.
,
Ghovanloo
,
M.
, and
Butler
,
A. J.
,
2017
, “
Improving Upper Extremity Function and Quality of Life With a Tongue Driven Exoskeleton: A Pilot Study Quantifying Stroke Rehabilitation
,”
Stroke Res. Treat.
, 2017(19), p.
3603860
.10.1155/2017/3603860
7.
Bryan
,
G. M.
,
Franks
,
P. W.
,
Song
,
S.
,
Voloshina
,
A. S.
,
Reyes
,
R.
,
ODonovan
,
M. P.
,
Gregorczyk
,
K. N.
, and
Collins
,
S. H.
, “
Optimized Hip–Knee–Ankle Exoskeleton Assistance at a Range of Walking Speeds
,”
J. Neuroeng. Rehabil.
, 18(1), pp.
1
12
.10.1186/s12984-021-00943-y
8.
Zhou
,
L.
,
Chen
,
W.
,
Chen
,
W.
,
Bai
,
S.
,
Zhang
,
J.
, and
Wang
,
J.
,
2020
, “
Design of a Passive Lower Limb Exoskeleton for Walking Assistance With Gravity Compensation
,”
Mech. Mach. Theory.
, 150(
2020
), Article No. 103840.10.1016/j.mechmachtheory.2020.103840
9.
Jacobson
,
M.
,
Kantharaju
,
P.
,
Jeong
,
H.
,
Ryu
,
J.-K.
,
Park
,
J.-J.
,
Chung
,
H.-J.
, and
Kim
,
M.
,
2022
, “
Foot Contact Forces Can Be Used to Personalize a Wearable Robot During Human Walking
,”
Sci. Rep.
, 12(1), pp.
1
12
.10.1038/s41598-022-14776-9
10.
Elliott
,
G.
,
Marecki
,
A.
, and
Herr
,
H.
,
2014
, “
Design of a Clutch-Spring Knee Exoskeleton for Running
,”
ASME J. Med. Dev.
,
8
(
3
), p.
031002
.10.1115/1.4027841
11.
Sado
,
F.
,
Yap
,
H. J.
,
Ghazilla
,
R. A. R.
, and
Ahmad
,
N.
,
2019
, “
Design and Control of a Wearable Lower-Body Exoskeleton for Squatting and Walking Assistance in Manual Handling Works
,”
Mechatronics
,
63
, p.
102272
.10.1016/j.mechatronics.2019.102272
12.
Yong
,
X.
,
Yan
,
Z.
,
Wang
,
C.
,
Wang
,
C.
,
Li
,
N.
, and
Wu
,
X.
, 2019. “
Ergonomic Mechanical Design and Assessment of a Waist Assist Exoskeleton for Reducing Lumbar Loads During Lifting Task
,”
Micromachines
, 10(7), p.
463
.10.3390/mi10070463
13.
Medrano
,
R. L.
,
Thomas
,
G. C.
,
Keais
,
C. G.
,
Rouse
,
E. J.
, and
Gregg
,
R. D.
,
2022
, “
Real-Time Gait Phase and Task Estimation for Controlling a Powered Ankle Exoskeleton on Extremely Uneven Terrain
,”
arXiv preprint arXiv:2205.00155
.10.48550/arXiv.2205.00155
14.
Feichtenhofer
,
C.
,
Fan
,
H.
,
Malik
,
J.
, and
He
,
K.
,
2019
, “
SlowFast Networks for Video Recognition
,”
Proceedings of the IEEE/CVF International Conference on Computer Vision
, Seoul, Korea, Oct. 27–28, pp.
6202
6211
.https://openaccess.thecvf.com/content_ICCV_2019/papers/Feichtenhofer_SlowFast_Networks_for_Video_Recognition_ICCV_2019_paper.pdf
15.
Park
,
S. U.
,
Park
,
J. H.
,
Al-Masni
,
M. A.
,
Al-Antari
,
M. A.
,
Uddin
,
M. Z.
, and
Kim
,
T. S.
,
2016
, “
A Depth Camera-Based Human Activity Recognition Via Deep Learning Recurrent Neural Network for Health and Social Care Services
,”
Procedia Comput. Sci.
,
100
, pp.
78
84
.10.1016/j.procs.2016.09.126
16.
Kim
,
Y. W.
,
Joa
,
K. L.
,
Jeong
,
H. Y.
, and
Lee
,
S.
,
2021
, “
Wearable IMU-Based Human Activity Recognition Algorithm for Clinical Balance Assessment Using 1D-CNN and GRU Ensemble Model
,”
Sensors
, 21(22), p.
7628
.10.3390/s21227628
17.
Bai
,
L.
,
Yao
,
L.
,
Wang
,
X.
,
Kanhere
,
S. S.
, and
Xiao
,
Y.
,
2020
, “
Prototype Similarity Learning for Activity Recognition
,”
Advances in Knowledge Discovery and Data Mining12084
, pp.
649
661
.10.1007/978-3-030-47426-3_50
18.
Sanz-Pena
,
I.
,
Blanco
,
J.
, and
Kim
,
J. H.
,
2021
, “
Computer Interface for Real-Time Gait Biofeedback Using a Wearable Integrated Sensor System for Data Acquisition
,”
IEEE Trans. Human-Mach. Syst.
, 51(5), pp.
484
493
.10.1109/THMS.2021.3090738
19.
Hussain
,
Z.
,
Sheng
,
Q. Z.
, and
Zhang
,
W. E.
,
2020
, “
A Review and Categorization of Techniques on Device-Free Human Activity Recognition
,”
J. Network Comput. Appl.
,
167
, p.
102738
.10.1016/j.jnca.2020.102738
20.
D'Sa
,
A. G.
, and
Prasad
,
B.
,
2019
, “
A Survey on Vision Based Activity Recognition, Its Applications and Challenges
,” 2019 Second International Conference on Advanced Computational and Communication Paradigms (
ICACCP
),
IEEE
, Gangtok, India, Feb. 25–28, pp.
1
8
.10.1109/ICACCP.2019.8882896
21.
Qamar
,
N.
,
Siddiqui
,
N.
,
Ehatisham-Ul-Haq
,
M.
,
Azam
,
M. A.
, and
Naeem
,
U.
,
2020
, “
An Approach Towards Position-Independent Human Activity Recognition Model Based on Wearable Accelerometer Sensor
,”
Procedia Comput. Sci.
,
177
, pp.
196
203
.10.1016/j.procs.2020.10.028
22.
Janidarmian
,
M.
,
Fekr
,
A. R.
,
Radecka
,
K.
, and
Zilic
,
Z.
,
2017
, “
A Comprehensive Analysis on Wearable Acceleration Sensors in Human Activity Recognition
,”
Sensors
, 17(3), p.
529
.10.3390/s17030529
23.
Lara
,
O. D.
, and
Labrador
,
M. A.
,
2013
, “
A Survey on Human Activity Recognition Using Wearable Sensors
,”
IEEE Commun. Surv. Tutorials
,
15
(
3
), pp.
1192
1209
.10.1109/SURV.2012.110112.00192
24.
Wang
,
J.
,
Chen
,
Y.
,
Hao
,
S.
,
Peng
,
X.
, and
Hu
,
L.
,
2017
, “
Deep Learning for Sensor-Based Activity Recognition: A Survey
,”
Pattern Recognit. Lett.
, 119(7), pp.
3
11
.10.1016/j.patrec.2018.02.010
25.
Ramamurthy
,
S. R.
,
Roy
,
N.
, and
Ramamurthy
,
C. S. R.
,
2018
, “
Recent Trends in Machine Learning for Human Activity Recognition - A Survey
,”
Wiley Interdiscip. Rev.: Data Min. Knowl. Discov.
, 2018(8), p. e1254.https://mdsoar.org/bitstream/handle/11603/8781/Royrecent-trends-machine%20%281%29.pdf?sequence=1&isAllowed=y
26.
Nweke
,
H. F.
,
Teh
,
Y. W.
,
Al-Garadi
,
M. A.
, and
Alo
,
U. R.
,
2018
, “
Deep Learning Algorithms for Human Activity Recognition Using Mobile and Wearable Sensor Networks: State of the Art and Research Challenges
,”
Expert Syst. Appl.
, 105(9), pp.
233
261
.10.1016/j.eswa.2018.03.056
27.
Lee
,
S. M.
,
Yoon
,
S. M.
, and
Cho
,
H.
,
2017
, “
Human Activity Recognition From Accelerometer Data Using Convolutional Neural Network
,”
2017 IEEE International Conference on Big Data and Smart Computing, BigComp 2017
, Jeju, Korea,
Feb. 13–16, pp. 131–134
.10.1109/BIGCOMP.2017.7881728
28.
Cho
,
H.
, and
Yoon
,
S. M.
,
2018
, “
Divide and ConquerBased 1D CNN Human Activity Recognition Using Test Data Sharpening
,”
Sensors
, 18(4), p.
1055
.10.3390/s18041055
29.
Horn
,
Z. C.
,
Auret
,
L.
,
McCoy
,
J. T.
,
Aldrich
,
C.
, and
Herbst
,
B. M.
,
2017
, “
Performance of Convolutional Neural Networks for Feature Extraction in Froth Flotation Sensing
,”
IFAC-PapersOnLine
,
50
(
2
), pp.
13
18
.10.1016/j.ifacol.2017.12.003
30.
Cruciani
,
F.
,
Vafeiadis
,
A.
,
Nugent
,
C.
,
Cleland
,
I.
,
Mccullagh
,
P.
,
Votis
,
K.
,
Giakoumis
,
D.
,
Tzovaras
,
D.
,
Chen
,
L.
, and
Hamzaoui
,
R.
,
2020
, “
Feature Learning for Human Activity Recognition Using Convolutional Neural Networks a Case Study for Inertial Measurement Unit and Audio Data
,”
CCF Trans. Pervasive Comput. Interact.
,
2
(
1
), pp.
18
32
.10.1007/s42486-020-00026-2
31.
Chung
,
S.
,
Lim
,
J.
,
Noh
,
K. J.
,
Kim
,
G.
, and
Jeong
,
H.
,
2019
, “
Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning
,”
Sensors
, 19(7), p.
1716
.10.3390/s19071716
32.
Sansano
,
E.
,
Montoliu
,
R.
, and
Belmonte Fernández
,
Ó.
,
2020
, “
A Study of Deep Neural Networks for Human Activity Recognition
,”
Comput. Intell.
,
36
(
3
), pp.
1113
1139
.10.1111/coin.12318
33.
Madan
,
R.
, and
Mangipudi
,
P. S.
,
2018
, “
Predicting Computer Network Traffic: A Time Series Forecasting Approach Using Dwt, Arima and Rnn
,” 2018 Eleventh International Conference on Contemporary Computing (
IC3
), Noida, India, Aug. 2–4, pp.
1
5
.10.1109/IC3.2018.8530608
34.
Hochreiter
,
S.
, and
Schmidhuber
,
J.
,
1997
, “
Long Short-Term Memory
,”
Neural Computation
,
9
(
8
), pp.
1735
1780
.10.1162/neco.1997.9.8.1735
35.
Lyu
,
Q.
, and
Zhu
,
J.
,
2014
, “
Revisit Long Short-Term Memory: An Optimization Perspective
,” Deep Learning and Representation Learning Workshop (
NIPS 2014
), Montreal, QC, Canada, Dec. 8–13, pp.
1
9
.https://ml.cs.tsinghua.edu.cn/~jun/pub/lstmparallel.pdf
36.
Britz
,
D.
,
Goldie
,
A.
,
Luong
,
M.
, and
Le
,
Q. V.
,
2017
, “
Massive Exploration of Neural Machine Translation Architectures
,”
CoRR.
, abs/1703.03906.10.48550/arXiv.1703.03906
37.
Greff
,
K.
,
Srivastava
,
R. K.
,
Koutnik
,
J.
,
Steunebrink
,
B. R.
, and
Schmidhuber
,
J.
,
2017
, “
Lstm: A Search Space Odyssey
,”
IEEE Trans. Neural Networks Learn. Syst.
,
28
(
10
), pp.
2222
2232
.10.1109/TNNLS.2016.2582924
38.
Lindemann
,
B.
,
Müller
,
T.
,
Vietz
,
H.
,
Jazdi
,
N.
, and
Weyrich
,
M.
,
2021
, “
A Survey on Long Short-Term Memory Networks for Time Series Prediction
,”
Procedia CIRP
,
99
, pp.
650
655
.10.1016/j.procir.2021.03.088
39.
Gopali
,
S.
,
Abri
,
F.
,
Siami-Namini
,
S.
, and
Namin
,
A. S.
,
2021
, “
A Comparison of Tcn and Lstm Models in Detecting Anomalies in Time Series Data
,” 2021 IEEE International Conference on Big Data (
Big Data
), Orlando, FL, Dec. 15–18, pp.
2415
2420
.10.1109/BigData52589.2021.9671488
40.
Xia
,
K.
,
Huang
,
J.
, and
Wang
,
H.
,
2020
, “
LSTM-CNN Architecture for Human Activity Recognition
,”
IEEE Access
, 8, pp.
56855
56866
.10.1109/ACCESS.2020.2982225
41.
Tang
,
Y.
,
Teng
,
Q.
,
Zhang
,
L.
,
Min
,
F.
, and
He
,
J.
,
2021
, “
Layer-Wise Training Convolutional Neural Networks With Smaller Filters for Human Activity Recognition Using Wearable Sensors
,”
IEEE Sens. J.
, 21(1), pp.
581
592
.10.1109/JSEN.2020.3015521
42.
Singh
,
S. P.
,
Lay-Ekuakille
,
A.
,
Gangwar
,
D.
,
Sharma
,
M. K.
, and
Gupta
,
S.
,
2020
, “
Deep ConvLSTM With Self-Attention for Human Activity Decoding Using Wearables
,”
IEEE Sens. J.
, 21(
6
), pp.
8575
8582
.10.1109/JSEN.2020.3045135
43.
Hysenllari
,
E.
,
Ottenbacher
,
J.
, and
McLennan
,
D.
,
2022
, “
Validation of Human Activity Recognition Using Aconvolutional Neural Network on Accelerometer and Gyroscope Data
,”
German J. Exercise Sport Res.
,
52
(
2
), pp.
248
252
.10.1007/s12662-022-00817-y
44.
Challa
,
S. K.
,
Kumar
,
A.
, and
Semwal
,
V. B.
,
2021
, “
A Multibranch CNN-BiLSTM Model for Human Activity Recognition Using Wearable Sensor Data
,”
Visual Comput.
, 1(8), pp.
1
15
.10.1007/s00371-021-02283-3
45.
Tang
,
Y.
,
Zhang
,
L.
,
Min
,
F.
, and
He
,
J.
,
2022
, “
Multi-Scale Deep Feature Learning for Human Activity Recognition Using Wearable Sensors
,”
IEEE Trans. Ind. Electron.
, 70(2), pp.
2106
2116
.10.1109/TIE.2022.3161812
46.
Huang
,
W.
,
Zhang
,
L.
,
Gao
,
W.
,
Min
,
F.
, and
He
,
J.
,
2021
, “
Shallow Convolutional Neural Networks for Human Activity Recognition Using Wearable Sensors
,”
IEEE Trans. Instrum. Meas.
,
70
, pp.
1
11
.10.1109/TIM.2021.3091990
47.
Cheng
,
X.
,
Zhang
,
L.
,
Tang
,
Y.
,
Liu
,
Y.
,
Wu
,
H.
, and
He
,
J.
,
2020
, “
Real-Time Human Activity Recognition Using Conditionally Parametrized Convolutions on Mobile and Wearable Devices
,”
IEEE Sens. J.
, 22(6), pp.
5889
5901
.10.1109/JSEN.2022.3149337
48.
Wan
,
S.
,
Qi
,
L.
,
Xu
,
X.
,
Tong
,
C.
, and
Gu
,
Z.
,
2020
, “
Deep Learning Models for Real-Time Human Activity Recognition With Smartphones
,”
Mobile Networks Appl.
,
25
(
2
), pp.
743
755
.10.1007/s11036-019-01445-x
49.
Ronao
,
C. A.
, and
Cho
,
S. B.
,
2016
, “
Human Activity Recognition With Smartphone Sensors Using Deep Learning Neural Networks
,”
Expert Syst. Appl.
,
59
, pp.
235
244
.10.1016/j.eswa.2016.04.032
50.
Chen
,
Y.
, and
Xue
,
Y.
,
2016
, “
A Deep Learning Approach to Human Activity Recognition Based on Single Accelerometer
,”
Proceedings - 2015 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2015
, Hong Kong, China, Oct. 9–12, pp.
1488
1492
.10.1109/SMC.2015.263
51.
Mutegeki
,
R.
, and
Han
,
D. S.
,
2020
, “
A CNN-LSTM Approach to Human Activity Recognition
,”
2020 International Conference on Artificial Intelligence in Information and Communication, ICAIIC 2020
, Fukuoka, Japan, Feb. 19–21.10.1109/ICAIIC48513.2020.9065078
52.
Mukherjee
,
D.
,
Mondal
,
R.
,
Singh
,
P. K.
,
Sarkar
,
R.
, and
Bhattacharjee
,
D.
,
2020
, “
EnsemConvNet: A Deep Learning Approach for Human Activity Recognition Using Smartphone Sensors for Healthcare Applications
,”
Multimedia Tools Appl.
,
79
(
41–42
), pp.
31663
31690
.10.1007/s11042-020-09537-7
53.
Lv
,
T.
,
Wang
,
X.
,
Jin
,
L.
,
Xiao
,
Y.
, and
Song
,
M.
,
2020
, “
Margin-Based Deep Learning Networks for Human Activity Recognition
,”
Sensors
, 20(7), p.
1871
.10.3390/s20071871
54.
Su
,
T.
,
Sun
,
H.
,
Ma
,
C.
,
Jiang
,
L.
, and
Xu
,
T.
,
2019
, “
HDL: Hierarchical Deep Learning Model Based Human Activity Recognition Using Smartphone Sensors
,”
Proceedings of the International Joint Conference on Neural Networks
, Vol.
7
, Budapest, Hungary, July 14–19, pp.
1
8
.10.1109/IJCNN.2019.8851889
55.
Gumaei
,
A.
,
Hassan
,
M. M.
,
Alelaiwi
,
A.
, and
Alsalman
,
H.
,
2019
, “
A Hybrid Deep Learning Model for Human Activity Recognition Using Multimodal Body Sensing Data
,”
IEEE Access
,
7
, pp.
99152
99160
.10.1109/ACCESS.2019.2927134
56.
Bo Yang
,
J.
,
Nhut Nguyen
,
M.
,
Phyo San
,
P.
,
Li Li
,
X.
, and
Krishnaswamy
,
S.
,
2015
, “
Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition
,”
IJCAI
, AAAI Press, Buenos Aires, Argentina, pp.
3995
4001
.https://www.ijcai.org/Proceedings/15/Papers/561.pdf
57.
Hreljac
,
A.
,
Imamura
,
R. T.
,
Escamilla
,
R. F.
, and
Edwards
,
W. B.
,
2007
, “
When Does a Gait Transition Occur During Human Locomotion?
,”
J. Sports Sci. Med.
,
6
(
3
), pp.
36
43
.https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3778697/
58.
Jones
,
G. D.
,
James
,
D. C.
,
Thacker
,
M.
, and
Green
,
D. A.
,
2018
, “
Parameters That Remain Consistent Independent of Pausing Before Gait-Initiation During Normal Rise-to-Walk Behaviour Delineated by Sit-Towalk and Sit-to-Stand-and-Walk
,”
PLoS One
,
13
(
10
), p.
e0205346
.10.1371/journal.pone.0205346
59.
Iwana
,
B. K.
, and
Uchida
,
S.
,
2021
, “
An Empirical Survey of Data Augmentation for Time Series Classification With Neural Networks
,”
PLoS ONE
, 16(7), pp.
1
32
.10.1371/journal.pone.0254841
60.
Wen
,
Q.
,
Sun
,
L.
,
Yang
,
F.
,
Song
,
X.
,
Gao
,
J.
,
Wang
,
X.
, and
Xu
,
H.
,
2020
, “
Time Series Data Augmentation for Deep Learning: A Survey
,”
IJCAI
, pp.
4653
4660
.10.24963/ijcai.2021/631
61.
Eyobu
,
O. S.
, and
Han
,
D. S.
,
2018
, “
Feature Representation and Data Augmentation for Human Activity Classification Based on Wearable IMU Sensor Data Using a Deep LSTM Neural Network
,”
Sensors
, 18(9), p.
2892
.10.3390/s18092892
62.
Girshick
,
R.
,
Donahue
,
J.
,
Darrell
,
T.
, and
Malik
,
J.
,
2014
, “
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, June 23–28, pp.
580
587
.
63.
Li
,
F.
,
Shirahama
,
K.
,
Nisar
,
M. A.
,
Koping
,
L
¨., and
Grzegorzek
,
M.
,
2018
, “
Comparison of Feature Learning Methods for Human Activity Recognition Using Wearable Sensors
,”
Sensors
, 18(2), p.
679
.10.3390/s18020679
64.
Chan
,
S. C.
,
Chia
,
J. W.
, and
Tan
,
Y. Q.
,
2018
, “
Kinetic and Kinematic Impacts During Level Walking, Uphill Walking, Level Running and Uphill Running
,”
2018 IEEE 16th Student Conference on Research and Development, SCOReD
, Selangor, Malaysia, Nov. 26–28, pp.
1
4
.10.1109/SCORED.2018.8710809
65.
Anguita
,
D.
,
Ghio
,
A.
,
Oneto
,
L.
,
Parra
,
X.
, and
ReyesOrtiz
,
J. L.
,
2012
, “
Human Activity Recognition on Smartphones Using a Multiclass Hardware-Friendly Support Vector Machine
,”
Ambient Assisted Living andHome Care: 4th International Workshop
, IWAAL2012, Vitoria-Gasteiz, Spain, Dec. 3–5, pp. 216–223.10.1007/978-3-642-35395-6_30
66.
Reiss
,
A.
, and
Stricker
,
D.
,
2012
, “
Introducing a New Benchmarked Dataset for Activity Monitoring
,”
Proceedings of International Symposium on Wearable Computers, ISWC,
Newcastle, UK, June 18–22, pp.
108
109
.10.1109/ISWC.2012.13
67.
Reyes-Ortiz
,
J.-L.
,
Oneto
,
L.
,
Sama
,
A.
,
Parra
,
X.
, and
Anguita
,
D.
,
2016
, “
Transition-Aware Human Activity Recognition Using Smartphones
,”
Neurocomputing
, 171, pp.
754
767
.10.1016/j.neucom.2015.07.085