Skip to main content
SpringerLink
Log in

Program Synthesis in a Continuous Space Using Grammars and Variational Autoencoders

  • Conference paper
  • First Online:
Parallel Problem Solving from Nature – PPSN XVI (PPSN 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12270))

Included in the following conference series:

Abstract

An important but elusive goal of computer scientists is the automatic creation of computer programs given only input and output examples. We present a novel approach to program synthesis based on the combination of grammars, generative neural models, and evolutionary algorithms. Programs are described by sequences of productions sampled from a Backus-Naur form grammar. A sequence-to-sequence Variational Autoencoder (VAE) is trained to embed randomly sampled programs in a continuous space – the VAE’s encoder maps a sequence of productions (a program) to a point z in the latent space, and the VAE’s decoder reconstructs the program given z. After the VAE has converged, we can engage the decoder as a generative model that maps locations in the latent space to executable programs. Hence, an Evolutionary Algorithm can be employed to search for a vector z (and its corresponding program) that solves the synthesis task. Experiments on the program synthesis benchmark suite suggest that the proposed approach is competitive with tree-based GP and PushGP. Crucially, code can be synthesised in any programming language.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Only \((e_2,e_5)\) are displayed for clarity, but in practice \((e_0, e_2, e_5, e_8)\) would be used.

References

  1. Balog, M., Gaunt, A.L., Brockschmidt, M., Nowozin, S., Tarlow, D.: Deepcoder: learning to write programs. In: Proceedings International Conference on Learning Representations 2017. OpenReviews.net (2017)

    Google Scholar 

  2. Orzechowski, P., Magiera, F., Moore, J.H.: Benchmarking manifold learning methods on a large collection of datasets. In: Hu, T., Lourenço, N., Medvet, E., Divina, F. (eds.) EuroGP 2020. LNCS, vol. 12101, pp. 135–150. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-44094-7_9

    Chapter  Google Scholar 

  3. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)

  4. Chollet, F.: The measure of intelligence. arXiv preprint arXiv:1911.01547 (2019)

  5. De Jong, K.A.: Evolutionary Computation: A Unified Approach. MIT Press, Cambridge (2006)

    MATH  Google Scholar 

  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  7. Fenton, M., McDermott, J., Fagan, D., Forstenlechner, S., Hemberg, E., O’Neill, M.: PonyGE2: grammatical evolution in python. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1194–1201 (2017)

    Google Scholar 

  8. Forstenlechner, S.: Program Synthesis with Grammars and Semantics in Genetic Programming. PhD Thesis pp. 162–175 (2019)

    Google Scholar 

  9. Forstenlechner, S., Fagan, D., Nicolau, M., O’Neill, M.: A Grammar Design Pattern for Arbitrary Program Synthesis Problems in Genetic Programming. In: McDermott, J., Castelli, M., Sekanina, L., Haasdijk, E., García-Sánchez, P. (eds.) EuroGP 2017. LNCS, vol. 10196, pp. 262–277. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-55696-3_17

  10. Forstenlechner, S., Fagan, D., Nicolau, M., O’Neill, M.: Extending Program Synthesis Grammars for Grammar-Guided Genetic Programming. In: Auger, A., Fonseca, C.M., Lourenço, N., Machado, P., Paquete, L., Whitley, D. (eds.) PPSN 2018. LNCS, vol. 11101, pp. 197–208. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99253-2_16

  11. Friedberg, R.M.: A learning machine: part i. IBM J. Res. Dev. 2(1), 2–13 (1958)

    Article  Google Scholar 

  12. Friedberg, R.M., Dunham, B., North, J.H.: A learning machine: part ii. IBM J. Res. Dev. 3(3), 282–287 (1959)

    Article  Google Scholar 

  13. Fujiki, C., Dickinson, J.: Using the genetic algorithm to generate LISP source code to solve the prisoner’s dilemma. In: Proceedings of the 2nd International Conference on Genetic Algorithms, Cambridge, MA, USA, July 1987. pp. 236–240 (1987)

    Google Scholar 

  14. Gaunt, A.L., et al.: TerpreT: A probabilistic programming language for program induction. CoRR abs/1608.04428 (2016)

    Google Scholar 

  15. Gómez-Bombarelli, R.: Automatic chemical design using a data-driven continuous representation of molecules. ACS central science 4(2), 268–276 (2018)

    Article  Google Scholar 

  16. Gulwani, S.: Automating string processing in spreadsheets using input-output examples. SIGPLAN Notices 46(1), 317–330 (2011)

    Article  Google Scholar 

  17. Helmuth, T., McPhee, N.F., Pantridge, E., Spector, L.: Improving generalization of evolved programs through automatic simplification. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 937–944 (2017)

    Google Scholar 

  18. Helmuth, T., Spector, L.: General program synthesis benchmark suite. In: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 1039–1046 (2015)

    Google Scholar 

  19. Helmuth, T., Spector, L., Matheson, J.: Solving uncompromising problems with lexicase selection. IEEE T. Evolut. Comput. 19(5), 630–643 (2014)

    Article  Google Scholar 

  20. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  Google Scholar 

  21. Holland, J.H.: Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology. Control and Artificial Intelligence. MIT Press, Cambridge (1975)

    MATH  Google Scholar 

  22. Katayama, S.: Recent Improvements of magichaskeller. In: Schmid, U., Kitzelmann, E., Plasmeijer, R. (eds.) AAIP 2009. LNCS, vol. 5812, pp. 174–193. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-11931-6_9

    Chapter  Google Scholar 

  23. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  24. Koza, J.R.: Human-competitive results produced by genetic programming. Genet. Program. Evol. Mach. 11(3–4), 251–284 (2010)

    Article  Google Scholar 

  25. Koza, J.R., Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection, vol. 1. MIT press, Cambridge (1992)

    Google Scholar 

  26. Krawiec, K., O’Reilly, U.M.: Behavioral programming: a broader and more detailed take on semantic GP. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 935–942 (2014)

    Google Scholar 

  27. Krawiec, K., Swan, J.: Pattern-guided genetic programming. In: Proceedings of the 15th Annual Conference On Genetic And Evolutionary Computation, pp. 949–956 (2013)

    Google Scholar 

  28. Kusner, M.J., Paige, B., Hernández-Lobato, J.M.: Grammar variational autoencoder. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. pp. 1945–1954. JMLR. org (2017)

    Google Scholar 

  29. Mckay, R.I., Hoai, N.X., Whigham, P.A., Shan, Y., O’Neill, M.: Grammar-based genetic programming: a survey. Genet. Program. Evol. Mach. 11(3–4), 365–396 (2010)

    Article  Google Scholar 

  30. Muggleton, S.: Inductive logic programming: issues, results and the challenge of learning language in logic. Artif. Intell. 114(1–2), 283–296 (1999)

    Article  Google Scholar 

  31. Nordin, P., Francone, F., Banzhaf, W.: Explicitly defined introns and destructive crossover in genetic programming. Adv. Genetic Program. 2, 111–134 (1995)

    Google Scholar 

  32. O’Neill, M., Fagan, D.: The Elephant in the Room: Towards the Application of Genetic Programming to Automatic Programming. In: Banzhaf, W., Spector, L., Sheneman, L. (eds.) Genetic Programming Theory and Practice XVI. GEC, pp. 179–192. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-04735-1_9

    Chapter  Google Scholar 

  33. O’Neill, M., Ryan, C.: Grammatical Evolution: Evolutionary Automatic Programming in a Arbitrary Language (2003)

    Google Scholar 

  34. O’Neill, M., Spector, L.: Automatic programming: The open issue? Genetic Programming and Evolvable Machines pp. 1–12 (2019)

    Google Scholar 

  35. Poli, R., Langdon, W.B., McPhee, N.F., Koza, J.R.: A Field Guide to Genetic Programming. Lulu.com (2008)

    Google Scholar 

  36. Rich, C., Waters, R.C.: Automatic programming: Myths and prospects. Computer 21(8), 40–51 (1988)

    Article  Google Scholar 

  37. Samuel, A.L.: Some studies in machine learning using the game of checkers. IBM J. Res. Dev. 3(3), 210–229 (1959)

    Article  MathSciNet  Google Scholar 

  38. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)

    Article  Google Scholar 

  39. Spector, L., Klein, J., Keijzer, M.: The Push3 execution stack and the evolution of control. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, pp. 1689–1696 (2005)

    Google Scholar 

  40. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

  41. Zhao, S., Song, J., Ermon, S.: InfoVAE: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262 (2017)

Download references

Acknowledgements

This research is based upon works supported by the Science Foundation Ireland under grant 13/IA/1850.

Author information

Authors and Affiliations

  1. Natural Computing Research and Applications Group, UCD, Dublin, Ireland

    David Lynch & Michael O’Neill

  2. School of Computer Science, National University of Ireland, Galway, Ireland

    James McDermott

Authors
  1. David Lynch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James McDermott
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael O’Neill
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to David Lynch , James McDermott or Michael O’Neill .

Editor information

Editors and Affiliations

  1. Leiden University, Leiden, The Netherlands

    Thomas Bäck

  2. Leiden University, Leiden, The Netherlands

    Mike Preuss

  3. Leiden University, Leiden, The Netherlands

    André Deutz

  4. Sorbonne University, Paris, France

    Hao Wang

  5. Sorbonne University, Paris, France

    Carola Doerr

  6. Leiden University, Leiden, The Netherlands

    Michael Emmerich

  7. University of Münster, Münster, Germany

    Heike Trautmann

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lynch, D., McDermott, J., O’Neill, M. (2020). Program Synthesis in a Continuous Space Using Grammars and Variational Autoencoders. In: Bäck, T., et al. Parallel Problem Solving from Nature – PPSN XVI. PPSN 2020. Lecture Notes in Computer Science(), vol 12270. Springer, Cham. https://doi.org/10.1007/978-3-030-58115-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58115-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58114-5

  • Online ISBN: 978-3-030-58115-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation