IDEAS home Printed from https://ideas.repec.org/p/zbw/iwqwdp/122018.html
   My bibliography  Save this paper

Reinforcement learning in financial markets - a survey

Author

Listed:
  • Fischer, Thomas G.

Abstract

The advent of reinforcement learning (RL) in financial markets is driven by several advantages inherent to this field of artificial intelligence. In particular, RL allows to combine the "prediction" and the "portfolio construction" task in one integrated step, thereby closely aligning the machine learning problem with the objectives of the investor. At the same time, important constraints, such as transaction costs, market liquidity, and the investor's degree of risk-aversion, can be conveniently taken into account. Over the past two decades, and albeit most attention still being devoted to supervised learning methods, the RL research community has made considerable advances in the finance domain. The present paper draws insights from almost 50 publications, and categorizes them into three main approaches, i.e., critic-only approach, actor-only approach, and actor-critic approach. Within each of these categories, the respective contributions are summarized and reviewed along the representation of the state, the applied reward function, and the action space of the agent. This cross-sectional perspective allows us to identify recurring design decisions as well as potential levers to improve the agent's performance. Finally, the individual strengths and weaknesses of each approach are discussed, and directions for future research are pointed out.

Suggested Citation

  • Fischer, Thomas G., 2018. "Reinforcement learning in financial markets - a survey," FAU Discussion Papers in Economics 12/2018, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    • Handle: RePEc:zbw:iwqwdp:122018
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/183139/1/1032172355.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Krauss, Christopher & Do, Xuan Anh & Huck, Nicolas, 2017. "Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500," European Journal of Operational Research, Elsevier, vol. 259(2), pages 689-702.
    2. Sarlan, Haldun, 2001. "Cyclical aspects of business cycle turning points," International Journal of Forecasting, Elsevier, vol. 17(3), pages 369-382.
    3. John Moody & Lizhong Wu, "undated". "Optimization of Trading Systems and Portfolios," Computing in Economics and Finance 1997 55, Society for Computational Economics.
    4. Bekiros, Stelios D., 2010. "Heterogeneous trading strategies with adaptive fuzzy Actor-Critic reinforcement learning: A behavioral approach," Journal of Economic Dynamics and Control, Elsevier, vol. 34(6), pages 1153-1170, June.
    5. David O. Lucca & Emanuel Moench, 2015. "The Pre-FOMC Announcement Drift," Journal of Finance, American Finance Association, vol. 70(1), pages 329-371, February.
    6. Nicholas T. Chan and Christian Shelton, 2001. "An Adaptive Electronic Market-Maker," Computing in Economics and Finance 2001 146, Society for Computational Economics.
    7. Christopher Krauss & Anh Do & Nicolas Huck, 2017. "Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500," Post-Print hal-01768895, HAL.
    8. French, Kenneth R., 1980. "Stock returns and the weekend effect," Journal of Financial Economics, Elsevier, vol. 8(1), pages 55-69, March.
    9. Zhengyao Jiang & Dixing Xu & Jinjun Liang, 2017. "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem," Papers 1706.10059, arXiv.org, revised Jul 2017.
    10. Marco Corazza & Francesco Bertoluzzo, 2014. "Q-Learning-based financial trading systems with applications," Working Papers 2014:15, Department of Economics, University of Venice "Ca' Foscari".
    11. Bertsimas, Dimitris & Lo, Andrew W., 1998. "Optimal control of execution costs," Journal of Financial Markets, Elsevier, vol. 1(1), pages 1-50, April.
    12. Francesco Bertoluzzo & Marco Corazza, 2012. "Reinforcement Learning for automatic financial trading: Introduction and some applications," Working Papers 2012:33, Department of Economics, University of Venice "Ca' Foscari", revised 2012.
    13. Xufre Casqueiro, Patricia & Rodrigues, Antonio J.L., 2006. "Neuro-dynamic trading methods," European Journal of Operational Research, Elsevier, vol. 175(3), pages 1400-1412, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Weiguang Han & Boyi Zhang & Qianqian Xie & Min Peng & Yanzhao Lai & Jimin Huang, 2023. "Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning," Papers 2301.10724, arXiv.org, revised Feb 2023.
    2. Charl Maree & Christian W. Omlin, 2022. "Balancing Profit, Risk, and Sustainability for Portfolio Management," Papers 2207.02134, arXiv.org.
    3. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Feb 2023.
    4. Xiao-Yang Liu & Hongyang Yang & Jiechao Gao & Christina Dan Wang, 2021. "FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance," Papers 2111.09395, arXiv.org.
    5. Tidor-Vlad Pricope, 2021. "Deep Reinforcement Learning in Quantitative Algorithmic Trading: A Review," Papers 2106.00123, arXiv.org.
    6. Jonas Hanetho, 2023. "Commodities Trading through Deep Policy Gradient Methods," Papers 2309.00630, arXiv.org.
    7. Maximilian Wehrmann & Nico Zengeler & Uwe Handmann, 2021. "Observation Time Effects in Reinforcement Learning on Contracts for Difference," JRFM, MDPI, vol. 14(2), pages 1-15, January.
    8. Schnaubelt, Matthias, 2022. "Deep reinforcement learning for the optimal placement of cryptocurrency limit orders," European Journal of Operational Research, Elsevier, vol. 296(3), pages 993-1006.
    9. Xiao-Yang Liu & Hongyang Yang & Qian Chen & Runjia Zhang & Liuqing Yang & Bowen Xiao & Christina Dan Wang, 2020. "FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance," Papers 2011.09607, arXiv.org, revised Mar 2022.
    10. Adrian Millea, 2021. "Deep Reinforcement Learning for Trading—A Critical Survey," Data, MDPI, vol. 6(11), pages 1-25, November.
    11. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    12. Jiwon Kim & Moon-Ju Kang & KangHun Lee & HyungJun Moon & Bo-Kwan Jeon, 2023. "Deep Reinforcement Learning for Asset Allocation: Reward Clipping," Papers 2301.05300, arXiv.org.
    13. MohammadAmin Fazli & Mahdi Lashkari & Hamed Taherkhani & Jafar Habibi, 2022. "A Novel Experts Advice Aggregation Framework Using Deep Reinforcement Learning for Portfolio Management," Papers 2212.14477, arXiv.org.
    14. Eric Benhamou & David Saltiel & Sandrine Ungari & Abhishek Mukhopadhyay, 2020. "Bridging the gap between Markowitz planning and deep reinforcement learning," Papers 2010.09108, arXiv.org.
    15. Zihao Zhang & Stefan Zohren & Stephen Roberts, 2019. "Deep Reinforcement Learning for Trading," Papers 1911.10107, arXiv.org.
    16. Longbing Cao, 2021. "AI in Finance: Challenges, Techniques and Opportunities," Papers 2107.09051, arXiv.org.
    17. Federico Cornalba & Constantin Disselkamp & Davide Scassola & Christopher Helf, 2022. "Multi-Objective reward generalization: Improving performance of Deep Reinforcement Learning for applications in single-asset trading," Papers 2203.04579, arXiv.org, revised Feb 2023.
    18. Jingyuan Wang & Yang Zhang & Ke Tang & Junjie Wu & Zhang Xiong, 2019. "AlphaStock: A Buying-Winners-and-Selling-Losers Investment Strategy using Interpretable Deep Reinforcement Attention Networks," Papers 1908.02646, arXiv.org.
    19. Weiguang Han & Jimin Huang & Qianqian Xie & Boyi Zhang & Yanzhao Lai & Min Peng, 2023. "Mastering Pair Trading with Risk-Aware Recurrent Reinforcement Learning," Papers 2304.00364, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    2. Ha, Youngmin & Zhang, Hai, 2020. "Algorithmic trading for online portfolio selection under limited market liquidity," European Journal of Operational Research, Elsevier, vol. 286(3), pages 1033-1051.
    3. Hyungjun Park & Min Kyu Sim & Dong Gu Choi, 2019. "An intelligent financial portfolio trading strategy using deep Q-learning," Papers 1907.03665, arXiv.org, revised Nov 2019.
    4. Ahmet Murat Ozbayoglu & Mehmet Ugur Gudelek & Omer Berat Sezer, 2020. "Deep Learning for Financial Applications : A Survey," Papers 2002.05786, arXiv.org.
    5. Schnaubelt, Matthias & Fischer, Thomas G. & Krauss, Christopher, 2020. "Separating the signal from the noise – Financial machine learning for Twitter," Journal of Economic Dynamics and Control, Elsevier, vol. 114(C).
    6. Weiguang Han & Boyi Zhang & Qianqian Xie & Min Peng & Yanzhao Lai & Jimin Huang, 2023. "Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning," Papers 2301.10724, arXiv.org, revised Feb 2023.
    7. Baoqiang Zhan & Shu Zhang & Helen S. Du & Xiaoguang Yang, 2022. "Exploring Statistical Arbitrage Opportunities Using Machine Learning Strategy," Computational Economics, Springer;Society for Computational Economics, vol. 60(3), pages 861-882, October.
    8. Kamaladdin Fataliyev & Aneesh Chivukula & Mukesh Prasad & Wei Liu, 2021. "Stock Market Analysis with Text Data: A Review," Papers 2106.12985, arXiv.org, revised Jul 2021.
    9. Steven L. Heston & Robert A. Korajczyk & Ronnie Sadka, 2010. "Intraday Patterns in the Cross‐section of Stock Returns," Journal of Finance, American Finance Association, vol. 65(4), pages 1369-1407, August.
    10. Amir Mosavi & Pedram Ghamisi & Yaser Faghan & Puhong Duan, 2020. "Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics," Papers 2004.01509, arXiv.org.
    11. Kentaro Imajo & Kentaro Minami & Katsuya Ito & Kei Nakagawa, 2020. "Deep Portfolio Optimization via Distributional Prediction of Residual Factors," Papers 2012.07245, arXiv.org.
    12. Alexandre Carbonneau & Fr'ed'eric Godin, 2021. "Deep equal risk pricing of financial derivatives with non-translation invariant risk measures," Papers 2107.11340, arXiv.org.
    13. Knoll, Julian & Stübinger, Johannes & Grottke, Michael, 2017. "Exploiting social media with higher-order Factorization Machines: Statistical arbitrage on high-frequency data of the S&P 500," FAU Discussion Papers in Economics 13/2017, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    14. Moews, Ben & Ibikunle, Gbenga, 2020. "Predictive intraday correlations in stable and volatile market environments: Evidence from deep learning," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 547(C).
    15. Cheema, Arbab K. & Eshraghi, Arman & Wang, Qingwei, 2023. "Macroeconomic news and price synchronicity," Journal of Empirical Finance, Elsevier, vol. 73(C), pages 390-412.
    16. Guillaume Coqueret & Tony Guida, 2020. "Training trees on tails with applications to portfolio choice," Post-Print hal-04144665, HAL.
    17. Zhou, Hao & Kalev, Petko S., 2019. "Algorithmic and high frequency trading in Asia-Pacific, now and the future," Pacific-Basin Finance Journal, Elsevier, vol. 53(C), pages 186-207.
    18. Fischer, Thomas & Krauss, Christopher, 2017. "Deep learning with long short-term memory networks for financial market predictions," FAU Discussion Papers in Economics 11/2017, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    19. Mercadier, Mathieu & Lardy, Jean-Pierre, 2019. "Credit spread approximation and improvement using random forest regression," European Journal of Operational Research, Elsevier, vol. 277(1), pages 351-365.
    20. Amirhosein Mosavi & Yaser Faghan & Pedram Ghamisi & Puhong Duan & Sina Faizollahzadeh Ardabili & Ely Salwana & Shahab S. Band, 2020. "Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics," Mathematics, MDPI, vol. 8(10), pages 1-42, September.

    More about this item

    Keywords

    financial markets; reinforcement learning; survey; trading systems; machine learning;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:iwqwdp:122018. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/vierlde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.