Skip to main content
Log in

Discovering novel memory cell designs for sentiment analysis on tweets

Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

Designing a Recurrent Neural Network to extract sentiment from tweets is a very hard task. When using memory cells in their design, the task becomes even harder due to the large number of design alternatives and the costly process of finding a performant design. In this paper we propose an original evolutionary algorithm to address the hard challenge of discovering novel Recurrent Neural Network memory cell designs for sentiment analysis on tweets. We used three different tasks to discover and evaluate the designs. We conducted experiments and the results show that the best obtained designs surpass the baselines—which are the most popular cells, LSTM and GRU. During the discovery process we evaluated roughly 17,000 cell designs. The selected winning candidate outperformed the others for the overall sentiment analysis problem, hence showing generality. We made the winner selection by using the cumulated accuracies on all three considered tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. L. Araujo, Genetic programming for natural language processing. Genet. Program. Evol. Mach. 21(1), 11–32 (2020)

    Article  Google Scholar 

  2. T. Bäck, H.P. Schwefel, An overview of evolutionary algorithms for parameter optimization. Evol. Comput. 1(1), 1–23 (1993)

    Article  Google Scholar 

  3. B. Baker, O. Gupta, R. Raskar, N. Naik, Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705.10823 (2017)

  4. J. Bayer, D. Wierstra, J. Togelius, J. Schmidhuber, Evolving memory cell structures for sequence learning. in International Conference on Artificial Neural Networks (Springer, 2009), pp. 755–764

  5. M. Becker, M. Staniek, V. Nastase, A. Palmer, A. Frank, Classifying semantic clause types: modeling context and genre characteristics with recurrent neural networks and attention. in Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017) (2017), pp. 230–240

  6. R. Breuer, R. Kimmel, A deep learning perspective on the origin of facial expressions. arXiv preprint arXiv:1705.01842 (2017)

  7. M.Y. Chen, H.S. Chiang, A.K. Sangaiah, T.C. Hsieh, Recurrent neural network with attention mechanism for language model. in Neural Computing and Applications (2019), pp. 1–9

  8. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)

  9. T. Chugh, K. Sindhya, J. Hakanen, K. Miettinen, A survey on handling computationally expensive multiobjective optimization problems with evolutionary algorithms. Soft Comput. 23(9), 3137–3166 (2019)

    Article  Google Scholar 

  10. A.M. Ciobanu, L.P. Dinu, Automatic identification and production of related words for historical linguistics. Comput. Linguist. 667–704 (2012)

  11. K. Deb, Multi-objective Optimization Using Evolutionary Algorithms, vol. 16 (Wiley, Hoboken, 2001)

    MATH  Google Scholar 

  12. S. Ding, H. Li, C. Su, J. Yu, F. Jin, Evolutionary artificial neural networks: a review. Artif. Intell. Rev. 39(3), 251–260 (2013)

    Article  Google Scholar 

  13. P. Doetsch, M. Kozielski, H. Ney, Fast and robust training of recurrent neural networks for offline handwriting recognition. in 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 279–284. (IEEE, 2014)

  14. C.A. Duchanoy, M.A. Moreno-Armendáriz, L. Urbina, C.A. Cruz-Villar, H. Calvo, J.d.J. Rubio, A novel recurrent neural network soft sensor via a differential evolution training algorithm for the tire contact patch. Neurocomputing 235, 71–82 (2017)

  15. F.A. Gers, J. Schmidhuber, Recurrent nets that time and count. in Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, vol. 3 (IEEE, 2000), pp. 189–194

  16. F. Gers, J. Schmidhuber, F. Cummins, Learning to forget: continual prediction with LSTM. Neural Comput. 12, 2451–2471 (2000). https://doi.org/10.1162/089976600300015015

    Article  Google Scholar 

  17. F. Gomez, R. Miikkulainen, Incremental evolution of complex general behavior. Adapt. Behav. 5(3–4), 317–342 (1997)

    Article  Google Scholar 

  18. K. Greff, R.K. Srivastava, J. Koutník, B.R. Steunebrink, J. Schmidhuber, LSTM: A search space odyssey. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2222–2232 (2016)

    Article  MathSciNet  Google Scholar 

  19. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 770–778

  20. S. Hochreiter, J. Schmidhuber, Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  21. T.J. Hsieh, H.F. Hsiao, W.C. Yeh, Forecasting stock markets using wavelet transforms and recurrent neural networks: An integrated system based on artificial bee colony algorithm. Appl. Soft Comput. 11(2), 2510–2525 (2011)

    Article  Google Scholar 

  22. E. Kouloumpis, T. Wilson, J. Moore, Twitter sentiment analysis: the good the bad and the omg! in Fifth International AAAI Conference on Weblogs and Social Media (2011)

  23. T. Lei, Y. Zhang, S.I. Wang, H. Dai, Y. Artzi, Simple recurrent units for highly parallelizable recurrence. arXiv preprint arXiv:1709.02755 (2017)

  24. Z.C. Lipton, J. Berkowitz, C. Elkan, A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.00019 (2015)

  25. D.C. Mallqui, R.A. Fernandes, Predicting the direction, maximum, minimum and closing prices of daily bitcoin exchange rate using machine learning techniques. Appl. Soft Comput. 75, 596–606 (2019)

    Article  Google Scholar 

  26. S.M. Mohammad, F. Bravo-Marquez, Emotion intensities in tweets. arXiv preprint arXiv:1708.03696 (2017)

  27. C.I. Muntean, G.A. Morar, D. Moldovan, Exploring the meaning behind twitter hashtags through clustering. in International Conference on Business Information Systems. (Springer, 2012), pp. 231–242

  28. I. Naji, Twitter Sentiment Analysis Training Corpus (Dataset). http://thinknook.com/twitter-sentiment-analysis-training-corpus-dataset-2012-09-22/ (2012). Accessed 21 July 2020

  29. S. Otte, M. Liwicki, A. Zell, Dynamic cortex memory: enhancing recurrent neural networks for gradient-based sequence learning. in International Conference on Artificial Neural Networks (Springer, 2014), pp. 1–8

  30. A. Pak, P. Paroubek, Twitter as a corpus for sentiment analysis and opinion mining. LREc 10, 1320–1326 (2010)

    Google Scholar 

  31. A. Rawal, R. Miikkulainen, From nodes to networks: Evolving recurrent neural networks. arXiv preprint arXiv:1803.04439 (2018)

  32. H. Sak, A. Senior, F. Beaufays, Long short-term memory recurrent neural network architectures for large scale acoustic modeling. in: Fifteenth Annual Conference of the International Speech Communication Association (2014)

  33. R. Sprugnoli, S. Tonelli, Novel event detection and classification for historical texts. Comput. Linguist. 45(2), 229–265 (2019)

    Article  Google Scholar 

  34. G. Stilo, P. Velardi, Hashtag sense clustering based on temporal similarity. Comput. Linguist. 43(1), 181–200 (2017)

    Article  Google Scholar 

  35. D. Stojanovski, G. Strezoski, G. Madjarov, I. Dimitrovski, Finki at semeval-2016 task 4: Deep learning architecture for twitter sentiment analysis. in Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) (2016), pp. 149–154

  36. M. Suganuma, S. Shirakawa, T. Nagao, A genetic programming approach to designing convolutional neural network architectures. in Proceedings of the Genetic and Evolutionary Computation Conference (2017), pp. 497–504

  37. I. Sutskever, O. Vinyals, Q. Le, Sequence to sequence learning with neural networks. Advances in NIPS (2014)

  38. J. Wehrmann, W. Becker, H.E. Cagnini, R.C. Barros, A character-based convolutional neural network for language-agnostic twitter sentiment analysis. in 2017 International Joint Conference on Neural Networks (IJCNN). (IEEE, 2017), pp. 2384–2391

  39. P.J. Werbos, Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990)

    Article  Google Scholar 

  40. S. Xingjian, Z. Chen, H. Wang, D.Y. Yeung, W.K. Wong, W.c. Woo: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. in Advances in Neural Information Processing Systems (2015), pp. 802–810

  41. X. Yao, Evolving artificial neural networks. Proc. IEEE 87(9), 1423–1447 (1999)

    Article  Google Scholar 

  42. E. Zitzler, Evolutionary Algorithms for Multiobjective Optimization: Methods and Applications, vol. 63. (Citeseer, 1999)

  43. B. Zoph, Q.V. Le, Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Răzvan Liviu Nistor.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The handling Associate Editor "Sebastian Risi" is a Neural Systems editor for this paper.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nistor, S.C., Moca, M. & Nistor, R.L. Discovering novel memory cell designs for sentiment analysis on tweets. Genet Program Evolvable Mach 22, 147–187 (2021). https://doi.org/10.1007/s10710-020-09395-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-020-09395-0

Keywords

Navigation