Representation Learning for Deviation Detection

Title	Representation Learning for Deviation Detection
Summary	Optimise Echo State Networks for time series forecasting and reconstruction. Propose methods, e.g. objective functions, to train Echo State Networks for deviation detection
Keywords	Representation learning, Deviation detection, Echo State Network, Optimization, Differential evolution
TimeFrame	Fall 2019
References	Bengio, Yoshua, Aaron Courville, and Pascal Vincent. "Representation learning: A review and new perspectives." IEEE transactions on pattern analysis and machine intelligence 35.8 (2013): 1798-1828. Jaeger, Herbert. "Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the" echo state network" approach. GMD-Forschungszentrum Informationstechnik, 2002. Jaeger, Herbert, et al. "Optimization and applications of echo state networks with leaky-integrator neurons." Neural networks 20.3 (2007): 335-352. Lukoševičius, Mantas. "A practical guide to applying echo state networks." Neural networks: Tricks of the trade. Springer Berlin Heidelberg, 2012. 659-686. Wang, Lin, Zhigang Wang, and Shan Liu. "An effective multivariate time series classification approach using echo state network and adaptive differential evolution algorithm." Expert Systems with Applications 43 (2016): 237-249. Li, Decai, Min Han, and Jun Wang. "Chaotic time series prediction based on a novel robust echo state network." IEEE Transactions on Neural Networks and Learning Systems 23.5 (2012): 787-799. Krause13, André Frank, et al. "Evolutionary Optimization of Echo State Networks: multiple motor pattern learning." (2010). Marco Rigamonti et al., "Echo State Network for the Remaining Useful Life Prediction of a Turbofan Engine." Third European Conference of the Prognostics and Health Management Society 2016, Bilbao, Spain, 5-8 July, 2016. PHM Society, 2016. Chen, Huanhuan, Peter Tiňo, and Xin Yao. "Cognitive fault diagnosis in Tennessee Eastman Process using learning in the model space." Computers & Chemical Engineering 67 (2014): 33-42. Quevedo, Joseba, et al. "Combining learning in model space fault diagnosis with data validation/reconstruction: Application to the Barcelona water network." Engineering Applications of Artificial Intelligence 30 (2014): 18-29. Fan, Yuantao, et al. "Predicting Air Compressor Failures with Echo State Networks." Third European Conference of the Prognostics and Health Management Society 2016, Bilbao, Spain, 5-8 July, 2016. PHM Society, 2016.
Prerequisites	Artificial Intelligence and Learning Systems courses; good knowledge of machine learning and neural networks; programming skills for implementing machine learning algorithms
Author
Supervisor	Sławomir Nowaczyk, Yuantao Fan
Level	Master
Status	Open

Echo State Network (ESN) is a special class of recurrent neural networks that can be used for supervised learning, e.g. time series forecasting, pattern recognition and deviation detection. The main idea of ESN is to drive a large random, fixed neural network that induce input signal into each neurons. The nonlinear responses of all neurons are combined by a trainable linear combination to a desired output. The network captures temporal features of time series data. Compared to other recurrent neural networks, it can be trained much faster. ESN are characterized by a number of parameters (e.g. network architecture, spectral radius, type of neurons and input scaling etc.). In order to achieve desired performance, these parameters need to be optimized and, preferably, in an efficient way.

There are many applications [0,1,2] ESN can be applied to. ESN can be used for general learning tasks, e.g. time series forecasting, or as representation of the data. In [3,4] ESN is employed to classify time series for fault detection. Similarly, in [6], air pressure signals are encoded into ESNs and anomalies are detected using an unsupervised deviation detection method, called Consensus Self-Organizing Models (COSMO) method. One key feature of COSMO method is the ability to capture and encode characteristics of the signals by using different representations. Generic representation that can be adapted to different type of signals and captures various characteristics is preferred for anomaly detection. Paper [6] demonstrated that ESNs are promising for this purpose. How to optimize the parameters of ESNs for deviation detection can be further investigated.

Objectives:

1. Propose a framework to optimize Echo State Networks for forecasting and reconstructing time series.

2. Use ESNs as self-organizing data representation (can be trained without external supervision) for unsupervised deviation detection. Determine what are the criteria for ESNs to learn interesting characteristics of the time series for deviation detection. Note that, for example, minimizing the reconstruction error does not necessary to be the objective function when train ESNs, but rather how ESNs can be trained to generalize various properties that characterize the observed system.

3. Perform deviation detection (e.g. COSMO method) on both synthetic data and data from real world application based on Echo State Networks and other data representations. Compare and analyze the performance.