PERTANIKA JOURNAL OF SCIENCE AND TECHNOLOGY

A year ago, one thousand USD invested in Bitcoin (BTC) alone would have appreciated to three thousand five hundred USD. Deep reinforcement learning (DRL) recent outstanding performance has opened up the possibilities to predict price fluctuations in changing markets and determine effective trading points, making a significant contribution to the finance sector. Several DRL methods have been tested in the trading domain. However, this research proposes implementing the proximal policy optimisation (PPO) algorithm, which has not been integrated into an automated trading system (ATS). Furthermore, behavioural biases in human decision-making often cloud one’s judgement to perform emotionally. ATS may alleviate these problems by identifying and using the best potential strategy for maximising profit over time. Motivated by the factors mentioned, this research aims to develop a stable, accurate, and robust automated trading system that implements a deep neural network and reinforcement learning to predict price movements to maximise investment returns by performing optimal trading points. Experiments and evaluations illustrated that this research model has outperformed the baseline buy and hold method and exceeded models of other similar works.

• Anthony, M., Bartlett, P. L., & Bartlett, P. L. (1999). Neural network learning: Theoretical Foundations. Cambridge University Press.

• Azulay, A., & Weiss, Y. (2018). Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning, 20(184), 1-25. https://doi.org/10.48550/arXiv.1805.12177

• Benesty, J., Chen, J., Huang, Y., & Cohen, I. (2009). On the importance of the Pearson correlation coefficient in noise reduction. IEEE Transactions on Audio, Speech, and Language Processing, 16(4), 757-765. https://doi.org/10.1109/tasl.2008.919072

• Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford University Press.

• Cartwright, T. J. (1991). Planning and chaos theory. Journal of the American Planning Association, 57(1), 44-56. https://doi.org/10.1080/01944369108975471

• Chollet, F. (2017). Deep learning with Python. Simon and Schuster.

• Cowpertwait, P. S. P., & Metcalfe, A. V. (2009) Time series data. In Introductory time series with R (pp. 1-25). Springer. https://doi.org/10.1007/978-0-387-88698-5_1

• Dempster, M. A. H., & Romahi, Y. S. (2002). Intraday FX trading: An evolutionary reinforcement learning approach. In H. Yin, N. Allinson, R. Freeman, J. Keane & S. Hubbard (Eds.), Intelligent Data Engineering and Automated Learning - IDEAL 2002 (pp. 347-358). Springer. https://doi.org/10.1007/3-540-45675-9_52

• Dempster, M. A., & Leemans, V. (2006). An automated FX trading system using adaptive reinforcement learning. Expert Systems with Applications, 30(3), 543-552. https://doi.org/10.1016/j.eswa.2005.10.012

• Fang, F., Ventre, C., Basios, M., Kong, H., Kanthan, L., Li, L., Martinez-Regoband, D., & Wu, F. (2022). Cryptocurrency trading: A comprehensive survey. Financial Innovation, 8(13). https://doi.org/10.1186/s40854-021-00321-6

• Ganesh, S., Vadori, N., Xu, M., Zheng, H., Reddy, P., & Veloso, M. (2019). Reinforcement learning for market making in a multi-agent dealer market. arXiv Preprint. https://doi.org/10.48550/arXiv.1911.05892

• Graves, A. (2012). Long short-term memory. In Supervised sequence labelling with recurrent neural networks (pp. 37-45). Springer. https://doi.org/10.1007/978-3-642-24797-2_4

• Grondman, I., Busoniu, L., Lopes, G. A., & Babuska, R. (2012). A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(6), 1291-1307. https://doi.org/10.1109/tsmcc.2012.2218595

• Gronwald, M. (2014). The economics of bitcoins - Market characteristics and price jumps. (Working Paper No. 5121). https://www.cesifo.org/DocDL/cesifo1_wp5121.pdf

• Haferkorn, M., & Diaz, J. M. Q. (2014). Seasonality and interconnectivity within cryptocurrencies - An analysis on the basis of bitcoin, litecoin and namecoin. In A. Lugmayr (Ed). International Workshop on Enterprise Applications and Services in the Finance Industry (pp. 106-120). Springer. https://doi.org/10.1007/978-3-319-28151-3_8

• Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735

• Hossin, M., & Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International Journal of Data Mining & Knowledge Management Process, 5(2). https://doi.org/10.5121/ijdkp.2015.5201

• Huang, B., Huan, Y., Xu, L. D., Zheng, L., & Zou, Z. (2019). Automated trading systems statistical and machine learning methods and hardware implementation: A survey. Enterprise Information Systems, 13(1), 132-144. https://doi.org/10.1080/17517575.2018.1493145

• Huang, C. Y. (2018). Financial trading as a game: A deep reinforcement learning approach. arXiv Preprint. https://doi.org/10.48550/arXiv.1807.02787

• Hyndman, R. J., & Athanasopoulos, G. (2018). Forecasting: Principles and practice. OTexts.

• Jeong, G., & Kim, H. Y. (2019). Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning. Expert Systems with Applications, 117, 125-138. https://doi.org/10.1016/j.eswa.2018.09.036

• Kolm, P. N., & Ritter, G. (2019). Modern perspectives on reinforcement learning in finance. Journal of Machine Learning in Finance, 1(1).

• Konda, V. R., & Tsitsiklis, J. N. (1999). Actor-critic algorithms. In S. Solla, T. Leen & K. Müller (Eds.), NIPS’99: Proceedings of the 12th International Conference on Neural Information Processing Systems (pp. 1008-1014). MIT Press.

• Kotsiantis, S. B., Kanellopoulos, D., & Pintelas, P. E. (2006). Data preprocessing for supervised leaning. International Journal of Computer and Information Engineering, 1(12), 4104-4109. https://doi.org/10.5281/zenodo.1082415

• Li, Y. (2017). Deep reinforcement learning: An overview. arXiv Preprint. https://doi.org/10.48550/arXiv.1701.07274

• Liao, S., Wang, J., Yu, R., Sato, K., & Cheng, Z. (2017). CNN for situations understanding based on sentiment analysis of twitter data. Procedia Computer Science, 111, 376-381. https://doi.org/10.1016/j.procs.2017.06.037

• Liu, X. Y., Yang, H., Chen, Q., Zhang, R., Yang, L., Xiao, B., & Wang, C. (2020). FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance. arXiv Preprint. https://doi.org/10.48550/arXiv.2011.09607

• Livieris, I. E., Pintelas, E., & Pintelas, P. (2020). A CNN-LSTM model for gold price time-series forecasting. Neural Computing and Applications, 32, 17351-17360. https://doi.org/10.1007/s00521-020-04867-x

• Lu, W., Li, J., Li, Y., Sun, A., & Wang, J. (2020). A CNN-LSTM-based model to forecast stock prices. Artificial Intelligence for Smart System Simulation, 2020, Article 6622927 https://doi.org/10.1155/2020/6622927

• Lucarelli, G., & Borrotti, M. (2019). A deep reinforcement learning approach for automated cryptocurrency trading. In J. MacIntyre, I. Maglogiannis, L. Iliadis & E. Pimenidis (Eds.), Artificial Intelligence Applications and Innovations (pp. 247-258). Springer. https://doi.org/10.1007/978-3-030-19823-7_20

• Moody, J., Saffell, M., Andrew, W. L., Abu-Mostafa, Y. S., LeBaraon, B., & Weigend, A. S. (1999). Minimizing downside risk via stochastic dynamic programming. Computational Finance, 403-415.

• Neely, C. J., Rapach, D. E., Tu, J., & Zhou, G. (2014). Forecasting the equity risk premium: The role of technical indicators. Management Science, 60(7), 1772-1791. https://doi.org/10.1287/mnsc.2013.1838

• Pan, S. J., & Yang, Q. (2009). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345-1359. https://doi.org/10.1109/tkde.2009.191

• Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386-408. https://doi.org/10.1037/h0042519

• Sattarov, O., Muminov, A., Lee, C. W., Kang, H. K., Oh, R., Ahn, J., Oh, H. J., & Jeon, H. S. (2020). Recommending cryptocurrency trading points with deep reinforcement learning approach. Applied Sciences, 10(4), Article 1506. https://doi.org/10.3390/app10041506

• Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-dimensional continuous control using generalized advantage estimation. arXiv Preprint. https://doi.org/10.48550/arXiv.1506.02438

• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv Preprint. https://doi.org/10.48550/arXiv.1707.06347

• Shrestha, A., & Mahmood, A. (2019). Review of deep learning algorithms and architectures. IEEE Access, 7, 53040-53065. https://doi.org/10.1109/access.2019.2912200

• Sola, J., & Sevilla, J. (1997). Importance of input data normalization for the application of neural networks to complex industrial problems. IEEE Transactions on Nuclear Science, 44(3), 1464-1468. https://doi.org/10.1109/23.589532

• Tucnik, P. (2010). Optimization of automated trading system’s interaction with market environment. In P. Forbrig & H. Günther (Eds.), Perspectives in Business Informatics Research (pp. 55-61). Springer. https://doi.org/10.1007/978-3-642-16101-8_5

• Van Otterlo, M., & Wiering, M. (2012). Reinforcement learning and Markov decision processes. In M. Wiering & M. Van Otterlo (Eds.), Reinforcement Learning. Adaptation, Learning, and Optimization (pp. 3-42). Springer. https://doi.org/10.1007/978-3-642-27645-3_1

• Vrigazova, B. (2021). The proportion for splitting data into training and test set for the bootstrap in classification problems. Business Systems Research Journal, 12(1) 228-242. https://doi.org/10.2478/bsrj-2021-0015

• Wu, C. H., Lu, C. C., Ma, Y. F., & Lu, R. S. (2018). A new forecasting framework for bitcoin price with LSTM. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW) (pp. 168-175). IEEE Publishing. https://doi.org/10.1109/icdmw.2018.00032

• Xiong, Z., Liu, X. Y., Zhong, S., Yang, H., & Walid, A. (2018). Practical deep reinforcement learning approach for stock trading. arXiv Preprint. https://doi.org/10.48550/arXiv.1811.07522

• Yang, H., Liu, X. Y., Zhong, S., & Walid, A. (2020, October 15-16). Deep reinforcement learning for automated stock trading: An ensemble strategy. In Proceedings of the First ACM International Conference on AI in Finance (pp. 1-8). ACM Publishing. https://doi.org/10.1145/3383455.3422540

• Zhang, W., Yang, Z., Shen, J., Liu, M., Huang, Y., Zhang, X., Tang, R., & Li, Z. (2021). Learning to build high-fidelity and robust environment models. In N. Oliver, F. Pérez-Cruz, S. Kramer, J. Read & J. A. Lozano (Eds.), Machine Learning and Knowledge Discovery in Databases (pp. 104-121). Springer. https://doi:10.1007/978-3-030-86486-6_7

• Zhang, Z., Zhang, Y., & Li, Z. (2018). Removing the feature correlation effect of multiplicative noise. arXiv Preprint. https://doi.org/10.48550/arXiv.1809.07023