Transformers and Time Series Forecasting

Abstract

Computation and Quantitative methods regularly transform the fields they are employed in. By harnessing the exponentially increasing power of silicon, engineers are able to approach and model problems that would be beyond the wildest imaginations of mathematicians only a few generations ago. Modern hardware and software enable us to make ever more precise predictions of all different types of phenomenon. Of focus in this paper, will be the prediction of price processes. The prediction of price processes are an area of great research for a variety of reasons – chief among them is the value such prediction algorithms can offer to investors for the purposes of making ever more educated investment decisions. By utilizing technology, it may become possible to offer efficient, accurate forecasts of the future movement of price processes. Researchers have always been interested in finding new, ever more accurate and efficient means of calculating and forecasting processes in prices and time series in general. Countless different computational methods have been developed both at the academic and private levels. The challenge lies in determining which languages and which algorithms would make themselves most amenable to our modeling goals, in our case, time series forecasting. I hope to evaluate the accuracy of Transformers in time series forecasting. Transformer neural networks are recently developed generative non recursive modeling algorithms which specialize in the prediction of future elements within sequences. They are utilized traditionally for the purposes of natural language translation. We will repurpose it to forecast time series and compare its accuracy to predictions generated by Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN). There are numerous benefits to utilizing the Transformer architecture over LSTM RNN. The two chief differences between the Transformer Architecture and the LSTM architecture are in the elimination of recurrence, thus decreasing complexity, and the enabling of parallelization, thus improving efficiency in computation. This may translate into faster overall computation. Given that Transformers have actually achieved greater accuracy than LSTM architectures in certain language translation tasks (Green & Radev 2017), it would worthwhile to also compare their accuracy in time series forecasting as well.