Abstract

Behavior of animals living in the wild is often studied using visual observations made by trained experts. However, these observations tend to be used to classify behavior during discrete time periods and become more difficult when used to monitor multiple individuals for days or weeks. In this work, we present automatic tools to enable efficient behavior and dynamic state estimation/classification from data collected with animal borne bio-logging tags, without the need for statistical feature engineering. A combined framework of an long short-term memory (LSTM) network and a hidden Markov model (HMM) was developed to exploit sequential temporal information in raw motion data at two levels: within and between windows. Taking a moving window data segmentation approach, LSTM estimates the dynamic state corresponding to each window by parsing the contiguous raw data points within the window. HMM then links all of the individual window estimations and further improves the overall estimation. A case study with bottlenose dolphins was conducted to demonstrate the approach. The combined LSTM–HMM method achieved a 6% improvement over conventional methods such as K-nearest neighbor (KNN) and support vector machine (SVM), pushing the accuracy above 90%. In addition to performance improvements, the proposed method requires a similar amount of training data to traditional machine learning methods, making the method easily adaptable to new tasks.

References

1.
LeCun
,
Y.
,
Bengio
,
Y.
, and
Hinton
,
G.
,
2015
, “
Deep Learning
,”
Nature
,
512
(
7553
), pp.
436
444
. 10.1038/nature14539
2.
Arifoglu
,
D.
, and
Bouchachia
,
A.
,
2017
, “
Activity Recognition and Abnormal Behaviour Detection With Recurrent Neural Networks
,”
Procedia Comput. Sci.
,
110
, pp.
86
93
. 10.1016/j.procs.2017.06.121
3.
Ching
,
T.
,
Himmelstein
,
D. S.
,
Beaulieu-Jones
,
B. K.
,
Kalinin
,
A. A.
,
Do
,
B. T.
,
Way
,
G. P.
, and
Ferrero
,
E.
,
2018
, “
Opportunities and Obstacles for Deep Learning in Biology and Medicine
,”
J. R. Soc. Interface
,
15
(
141
), p.
20170387
. 10.1098/rsif.2017.0387
4.
Manning
,
A.
, and
Dawkins
,
M. S.
,
1998
,
An Introduction to Animal Behaviour
,
Cambridge University Press
,
Cambridge
.
5.
Martin
,
P.
, and
Bateson
,
P.
,
2007
,
Measuring Behaviour: An Introductory Guide
, 3rd ed.,
Cambridge University Press
,
Cambridge
.
6.
Preisler
,
H. K.
,
Ager
,
A. A.
,
Johnson
,
B. K.
, and
Kie
,
J. G.
,
2004
, “
Modeling Animal Movements Using Stochastic Differential Equations
,”
Environmetrics
,
15
(
7
), pp.
643
657
. 10.1002/env.636
7.
Zhang
,
D.
,
van der Hoop
,
J. M.
,
Petrov
,
V.
,
Rocho-Levine
,
J.
,
Moore
,
M. J.
, and
Shorter
,
K. A.
,
2020
, “
Simulated and Experimental Estimates of Hydrodynamic Drag From Bio-Logging Tags
,”
Mar. Mammal Sci.
,
36
(
1
), pp.
136
157
. 10.1111/mms.12627
8.
Gabaldon
,
J.
,
Zhang
,
D.
,
Barton
,
K.
,
Johnson-Roberson
,
M.
, and
Shorter
,
K. A.
,
2017
, “
A Framework for Enhanced Localization of Marine Mammals Using Auto-Detected Video and Wearable Sensor Data Fusion
,”
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Vancouver, BC, Canada
,
Sept. 24
, pp.
2505
2510
.
9.
Zhang
,
D.
,
Shorter
,
K. A.
,
Rocho-Levine
,
J.
,
van der Hoop
,
J.
,
Moore
,
M.
, and
Barton
,
K.
,
2019
, “
Localization and Tracking of Uncontrollable Underwater Agents: Particle Filter Based Fusion of On-Body Imus and Stationary Cameras
,”
International Conference on Robotics and Automation (ICRA)
,
Montreal, QC, Canada, Canada
,
May 20
, pp.
6575
6581
.
10.
Grünewälder
,
S.
,
Broekhuis
,
F.
,
Macdonald
,
D. W.
,
Wilson
,
A. M.
,
Mcnutt
,
J. W.
,
Shawe-Taylor
,
J.
, and
Hailes
,
S.
,
2012
, “
Movement Activity Based Classification of Animal Behaviour With an Application to Data From Cheetah (Acinonyx jubatus)
,”
PLoS ONE
,
7
(
11
), p.
e49120
. 10.1371/journal.pone.0049120
11.
Vieira
,
A.
,
2015
, “
Predicting Online User Behaviour Using Deep Learning Algorithms
,”
arXiv preprint
. arXiv:1511.06247
12.
Hammerla
,
N. Y.
,
Halloran
,
S.
, and
Plotz
,
T.
,
2016
, “
Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
,”
Twenty-Fifth International Joint Conference on Artificial Intelligence
,
New York, NY
,
July 9
, pp.
1533
1540
.
13.
Gers
,
F. A.
,
Schmidhuber
,
J.
, and
Cummins
,
F. A.
,
2000
, “
Learning to Forget: Continual Prediction With LSTM
,”
Neural Comput.
,
12
(
10
), pp.
2451
2471
. 10.1162/089976600300015015
14.
Saleh
,
K.
,
Hossny
,
M.
, and
Nahavandi
,
S.
,
2017
, “
Driving Behavior Classification Based on Sensor Data Fusion Using LSTM Recurrent Neural Networks
,”
2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC)
,
Yokohama, Japan
,
Oct. 16
, pp.
1
16
.
15.
Ghahramani
,
Z.
,
2001
, “
An Introduction to Hidden Markov Models and Bayesian Networks
,”
Int. J. Pattern Recognit. Artif. Netw.
,
15
(
1
), pp.
9
42
. 10.1142/S0218001401000836
16.
Rabiner
,
L. R.
,
1989
, “
A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition
,”
Proc. IEEE
,
77
(
2
), pp.
257
286
. 10.1109/5.18626
17.
Viterbi
,
A. J.
,
1967
, “
Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm
,”
IEEE Trans. Inf. Theory
,
13
(
2
), pp.
260
269
. 10.1109/TIT.1967.1054010
18.
Zhang
,
D.
,
Shorter
,
K. A.
,
Rocho-Levine
,
J.
,
van der Hoop
,
J.
,
Moore
,
M.
, and
Barton
,
K.
,
2018
, “
Behavior Inference From Bio-Logging Sensors: A Systematic Approach for Feature Generation, Selection and State Classification
,”
Proceedings of the 2018 ASME Dynamic Systems and Control Conference (DSCC)
,
Atlanta, GA, Sept
. 10.1115/DSCC2018-9213
You do not currently have access to this content.