Skip to main content
SpringerLink
Log in

Evolutionary Computation Meets Stream Processing

  • Conference paper
  • First Online:
Applications of Evolutionary Computation (EvoApplications 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14634))

  • 134 Accesses

Abstract

Evolutionary computation (EC) has a great potential of exploiting parallelization, a feature often underemphasized when describing evolutionary algorithms (EAs). In this paper, we show that the paradigm of stream processing (SP) can be used to express EAs in a way that allows the immediate exploitation of parallel and distributed computing, not at the expense of the agnosticity of the EAs with respect to the application domain. We introduce the first formal framework for EC based on SP and describe several building blocks tailored to EC. Then, we experimentally validate our framework and show that (a) it can be used to express common EAs, (b) it scales when deployed on real-world stream processing engines (SPEs), and (c) it facilitates the design of EA modifications which would require a larger effort with traditional implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  1. Alba, E., Luque, G., Nesmachnow, S.: Parallel metaheuristics: recent advances and new trends. Int. Trans. Oper. Res. 20(1), 1–48 (2013)

    Article  Google Scholar 

  2. Bartoli, A., Manzoni, L., Medvet, E.: Commentary on “Jaws 30’’, by W.B. Langdon. Genet. Program Evolvable Mach. 24, 23 (2023). https://doi.org/10.1007/s10710-023-09471-1

    Article  Google Scholar 

  3. Carbone, P., Fragkoulis, M., Kalavri, V., Katsifodimos, A.: Beyond analytics: the evolution of stream processing systems. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pp. 2651–2658 (2020)

    Google Scholar 

  4. Cardellini, V., Lo Presti, F., Nardelli, M., Russo, G.R.: Runtime adaptation of data stream processing systems: the state of the art. ACM Comput. Surv. 54(11s), 1–36 (2022)

    Article  Google Scholar 

  5. De Lorenzo, A., Bartoli, A., Castelli, M., Medvet, E., Xue, B.: Genetic programming in the twenty-first century: a bibliometric and content-based analysis from both sides of the fence. Genet. Program Evolvable Mach. 21, 181–204 (2020)

    Article  Google Scholar 

  6. Duvignau, R., Gulisano, V., Papatriantafilou, M., Savic, V.: Streaming piecewise linear approximation for efficient data management in edge computing. In: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing (2019)

    Google Scholar 

  7. Flink: Apache Flink (2023). https://flink.apache.org. Accessed 27 Jan 2023

  8. FlinkML: Apache Flink ML Documentation (2023). https://nightlies.apache.org/flink/flink-ml-docs-stable/. Accessed 14 Nov 2023

  9. Fortin, F.A., De Rainville, F.M., Gardner, M.A.G., Parizeau, M., Gagné, C.: DEAP: evolutionary algorithms made easy. J. Mach. Learn. Res. 13(1), 2171–2175 (2012)

    MathSciNet  Google Scholar 

  10. Frasca, F., Gulisano, V., Mencagli, G., Palyvos-Giannas, D., Torquati, M.: Accelerating stream processing queries with congestion-aware scheduling and real-time Linux threads. In: Proceedings of the 20th ACM International Conference on Computing Frontiers, pp. 144–153 (2023)

    Google Scholar 

  11. Gulisano, V., Jimenez-Peris, R., Patino-Martinez, M., Soriente, C., Valduriez, P.: StreamCloud: an elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23(12), 2351–2365 (2012)

    Article  Google Scholar 

  12. Gulisano, V., Palyvos-Giannas, D., Havers, B., Papatriantafilou, M.: The role of event-time order in data streaming analysis. In: Proceedings of the 14th ACM International Conference on Distributed and Event-Based Systems, DEBS 2020, pp. 214–217. Association for Computing Machinery, New York (2020). ISBN 9781450380287. https://doi.org/10.1145/3401025.3404088

  13. Gulisano, V., Papadopoulos, A.V., Nikolakopoulos, Y., Papatriantafilou, M., Tsigas, P.: Performance modeling of stream joins. In: Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems, pp. 191–202 (2017)

    Google Scholar 

  14. Harada, T., Alba, E.: Parallel genetic algorithms: a useful survey. ACM Comput. Surv. 53(4), 1–39 (2020)

    Article  Google Scholar 

  15. Havers, B., Duvignau, R., Najdataei, H., Gulisano, V., Koppisetty, A.C., Papatriantafilou, M.: DRIVEN: a framework for efficient data retrieval and clustering in vehicular networks. In: 2019 IEEE 35th International Conference on Data Engineering (ICDE), pp. 1850–1861. IEEE (2019)

    Google Scholar 

  16. Hummer, W., Satzger, B., Dustdar, S.: Elastic stream processing in the cloud. Wiley Interdisc. Rev. Data Min. Knowl. Disc. 3(5), 333–345 (2013)

    Article  Google Scholar 

  17. Isah, H., Abughofa, T., Mahfuz, S., Ajerla, D., Zulkernine, F., Khan, S.: A survey of distributed data stream processing frameworks. IEEE Access 7, 154300–154316 (2019)

    Article  Google Scholar 

  18. La Cava, W., et al.: Contemporary symbolic regression methods and their relative performance. arXiv preprint arXiv:2107.14351 (2021)

  19. Maitre, O., Baumes, L.A., Lachiche, N., Corma, A., Collet, P.: Coarse grain parallelization of evolutionary algorithms on GPGPU cards with EASEA. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 1403–1410 (2009)

    Google Scholar 

  20. Medvet, E., Bartoli, A., Carminati, B., Ferrari, E.: Evolutionary inference of attribute-based access control policies. In: Gaspar-Cunha, A., Henggeler Antunes, C., Coello, C.C. (eds.) EMO 2015. LNCS, vol. 9018, pp. 351–365. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15934-8_24

    Chapter  Google Scholar 

  21. Medvet, E., Nadizar, G., Manzoni, L.: JGEA: a modular java framework for experimenting with evolutionary computation. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 2009–2018 (2022)

    Google Scholar 

  22. Najdataei, H., Gulisano, V., Tsigas, P., Papatriantafilou, M.: pi-Lisco: parallel and incremental stream-based point-cloud clustering. In: Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, pp. 460–469 (2022)

    Google Scholar 

  23. Najdataei, H., Nikolakopoulos, Y., Gulisano, V., Papatriantafilou, M.: Continuous and parallel LiDAR point-cloud clustering. In: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), pp. 671–684. IEEE (2018)

    Google Scholar 

  24. Palyvos-Giannas, D., Havers, B., Papatriantafilou, M., Gulisano, V.: Ananke: a streaming framework for live forward provenance. Proc. VLDB Endow. 14(3), 391–403 (2020)

    Article  Google Scholar 

  25. Palyvos-Giannas, D., Mencagli, G., Papatriantafilou, M., Gulisano, V.: Lachesis: a middleware for customizing OS scheduling of stream processing queries. In: Proceedings of the 22nd International Middleware Conference, pp. 365–378 (2021)

    Google Scholar 

  26. Palyvos-Giannas, D., Tzompanaki, K., Papatriantafilou, M., Gulisano, V.: Erebus: explaining the outputs of data streaming queries. In: Very Large Data Base, vol. 16, pp. 230–242 (2023)

    Google Scholar 

  27. Pigozzi, F., Medvet, E.: Evolving modularity in soft robots through an embodied and self-organizing neural controller. Artif. Life 28(3), 322–347 (2022)

    Article  Google Scholar 

  28. Pospichal, P., Jaros, J., Schwarz, J.: Parallel genetic algorithm on the CUDA architecture. In: Di Chio, C., et al. (eds.) EvoApplications 2010. LNCS, vol. 6024, pp. 442–451. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12239-2_46

    Chapter  Google Scholar 

  29. Rathore, M.M., Son, H., Ahmad, A., Paul, A., Jeon, G.: Real-time big data stream processing using GPU with spark over Hadoop ecosystem. Int. J. Parallel Program. 46(3), 630–646 (2017). https://doi.org/10.1007/s10766-017-0513-2

    Article  Google Scholar 

  30. Röger, H., Mayer, R.: A comprehensive survey on parallelization and elasticity in stream processing. ACM Compu. Surv. (CSUR) 52(2), 1–37 (2019)

    Article  Google Scholar 

  31. Rovito, L., De Lorenzo, A., Manzoni, L.: Evolution of Walsh Transforms with genetic programming. In: Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pp. 2386–2389 (2023)

    Google Scholar 

  32. Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1(5), 206–215 (2019)

    Article  Google Scholar 

  33. Russo, G.R., Cardellini, V., Presti, F.L.: Reinforcement learning based policies for elastic stream processing on heterogeneous resources. In: Proceedings of the 13th ACM International Conference on Distributed and Event-Based Systems, pp. 31–42 (2019)

    Google Scholar 

  34. Stephens, R.: A survey of stream processing. Acta Informatica 34, 491–541 (1997)

    Article  MathSciNet  Google Scholar 

  35. Virgolin, M., Alderliesten, T., Bosman, P.A.: Linear scaling with and within semantic backpropagation-based genetic programming for symbolic regression. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1084–1092 (2019)

    Google Scholar 

  36. White, D.R., et al.: Better GP benchmarks: community survey results and proposals. Genet. Program Evolvable Mach. 14, 3–29 (2013)

    Article  Google Scholar 

Download references

Acknowledgements

This study was carried out within the PNRR research activities of the consortium iNEST (Interconnected North-Est Innovation Ecosystem) funded by the European Union Next-GenerationEU (Piano Nazionale di Ripresa e Resilienza (PNRR) - Missione 4 Componente 2, Investimento 1.5 - D.D. 1058 23/06/2022, ECS_00000043), and by the Marie Skłodowska-Curie Doctoral Network project RELAX-DN, funded by the European Union under Horizon Europe 2021–2027 Framework Programme Grant Agreement number 101072456.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden

    Vincenzo Gulisano

  2. Department of Engineering and Architecture, University of Trieste, Trieste, Italy

    Eric Medvet

Authors
  1. Vincenzo Gulisano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Medvet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eric Medvet .

Editor information

Editors and Affiliations

  1. University of York, York, UK

    Stephen Smith

  2. University of Coimbra, Coimbra, Portugal

    João Correia

  3. University of Málaga, Málaga, Spain

    Christian Cintrano

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gulisano, V., Medvet, E. (2024). Evolutionary Computation Meets Stream Processing. In: Smith, S., Correia, J., Cintrano, C. (eds) Applications of Evolutionary Computation. EvoApplications 2024. Lecture Notes in Computer Science, vol 14634. Springer, Cham. https://doi.org/10.1007/978-3-031-56852-7_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56852-7_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56851-0

  • Online ISBN: 978-3-031-56852-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation