Skip to main content
SpringerLink
Log in

Scaling Up Inductive Logic Programming: An Evolutionary Wrapper Approach

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Inductive logic programming (ILP) algorithms are classification algorithms that construct classifiers represented as logic programs. ILP algorithms have a number of attractive features, notably the ability to make use of declarative background (user-supplied) knowledge. However, ILP algorithms deal poorly with large data sets (>104 examples) and their widespread use of the greedy set-covering algorithm renders them susceptible to local maxima in the space of logic programs.

This paper presents a novel approach to address these problems based on combining the local search properties of an inductive logic programming algorithm with the global search properties of an evolutionary algorithm. The proposed algorithm may be viewed as an evolutionary wrapper around a population of ILP algorithms.

The evolutionary wrapper approach is evaluated on two domains. The chess-endgame (KRK) problem is an artificial domain that is a widely used benchmark in inductive logic programming, and Part-of-Speech Tagging is a real-world problem from the field of Natural Language Processing. In the latter domain, data originates from excerpts of the Wall Street Journal. Results indicate that significant improvements in predictive accuracy can be achieved over a conventional ILP approach when data is plentiful and noisy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. T.G. Dietterich, “Machine learning research: Four current directions,” AI Magazine, vol. 18, No. 4, pp. 97-136, 1997.

    Google Scholar 

  2. T. Bäck. Evolutionary Algorithms in Theory and Practice. Oxford University Press, 1996.

  3. K.A. DeJong and W.M. Spears, “Using genetic algorithms to solve NP-complete problems,” in International Conference on Genetic Algorithms, 1989, pp. 124-132.

  4. D.E. Goldberg, “Genetic and evolutionary algorithms come of age,” Communications of the ACM, vol. 37, pp. 113-119, 1994.

    Google Scholar 

  5. K.A. DeJong, W.M. Spears, and D.F. Gordon, “Using genetic algorithms for concept learning,” Machine Learning, vol. 13, pp. 161-188, 1993.

    Google Scholar 

  6. A. Giordana and L. Saitta, “Regal: An integrated system for learning relations using genetic algorithms,” in Proceedings of 2nd InternationalWorkshop on Multistrategy Learning, Morgan Kaufmann, pp. 234-249, 1993.

  7. A.C. Schultz and J.J. Grefenstette, “Improving tactical plans with genetic algorithms,” in Proceedings of the 2nd International IEEE Conference on Tools for Artificial Intelligence, number IEEE Cat. No. 90CH2915-7, Herndon,VA, 6-9Nov. 1990. IEEE Computer Society Press: Los Alamitos, CA, pp. 328-334.

    Google Scholar 

  8. M.L. Wong and K.S. Leung, “Inductive logic programming using genetic algorithms,” in Advances in Artificial Intelligence-Theory and Application II, edited by J.W. Brahan and G.E. Lasker, pp. 119-124, 1994.

  9. J.R. Koza, Genetic Programming: On the Programming of Computers by Natural Selection, MIT Press: Cambridge, MA, 1992.

    Google Scholar 

  10. N.J. Radcliffe and P.D. Surry, “Cooperation through hierarchical competition in genetic data mining,” Technical Report TR9409, University of Edinburgh, Parallel Computing Centre, Edinburgh, 1994.

    Google Scholar 

  11. D.J. Montana, “Strongly typed genetic programming,” BBN Technical Report #7866, Bolt Beranek and Newman, Inc., 10 Moulton Street, Cambridge, MA 02138, 1994.

  12. T.D. Haynes, D.A. Schoenefeld, and R.L. Wainwright, “Type inheritance in strongly typed genetic programming,” in Advances in Genetic Programming 2, edited by P.J. Angeline and K.E. Kinnear, Jr., MIT Press: Cambridge, MA, Chap. 18, pp. 359-376, 1996.

    Google Scholar 

  13. M.L. Wong and K.S. Leung, “An induction system that learns programs in different programming languages using genetic programming and logic grammars,” in Seventh International Conference on Tools with Artificial Intelligence, Herndon, Virginia, Nov. 1995, The Chinese University of Hong Kong, IEEE Computer Society Press, pp. 380-387.

  14. M.L. Wong and K.S. Leung, “Combining genetic programming and inductive logic programming using logic grammars,” in Proceedings of the IEEE International Conference on Evolutionary Computation, Perth, Western Australia, Nov. 1995, IEEE Computer Society Press, pp. 733-736.

  15. M.L. Wong and K.S. Leung, “Applying logic grammars to induce sub-functions in genetic programming,” in Proceedings of the IEEE International Conference on Evolutionary Computation, Perth, Western Australia, Nov. 1995, IEEE Computer Society Press, pp. 737-740.

    Google Scholar 

  16. M.L. Wong and K.S. Leung, “Evolving recursive functions for the even-parity problem using genetic programming,” in Advances in Genetic Programming 2, edited by P.J. Angeline and K.E. Kinnear Jr., MIT Press, Cambridge, MA, 1996.

    Google Scholar 

  17. P.A. Whigham, “Grammatically-based genetic programming,” in Proc. Workshop on Genetic Programming: From Theory to Real-World Applications, edited by J. Rosca, Morgan Kaufmann, pp. 33-41, 1995.

  18. P.A. Whigham, “Search bias, language bias, and genetic programming,” in Genetic Programming 1996: Proceedings of the First Conference, Stanford University, CA, 28-31 July 1996, edited by John R. Koza, David E. Goldberg, David B. Fogel, and Rick L. Riolo, MIT Press.

  19. F. Gruau, “On using syntactic constraints with genetic programming,” in Advances in Genetic Programming III, edited by P.J. Angeline and K.E. Kinnear, MIT Press: Cambridge, MA, Chap. 19, pp. 377-394, 1996.

    Google Scholar 

  20. P.S. Ngan, M.L. Wong, W. Lam, K.S. Leung, and J.C.Y. Cheng, “Medical data mining using evolutionary computation,” Artificial Intelligence in Medicine, vol. 16, pp. 73-96, 1999.

    Google Scholar 

  21. P.A. Whigham and P.F. Crapper, “Time series modelling using genetic programming: An application to rainfall-runoff models,” in Advances in Genetic Programming, edited by L. Spector, W.B. Langdon, U.-M. O'Reilly, and P.J. Angeline, MIT Press: Cambridge, MA, Chap. 5, pp. 89-104. 1999.

    Google Scholar 

  22. K.A. DeJong and W.M. Spears, “Learning concept classification rules using genetic algorithms,” in Proceedings of the Twelfth International Joint Conference on Artificial Intelligence, 1991, pp. 651-656.

  23. C.Z. Janikow, “A knowledge-intensive genetic algorithm for supervised learning,” Machine Learning, vol. 13, pp. 189-228, 1993.

    Google Scholar 

  24. J.J. Grefenstette, “A system for learning control strategies with genetic algorithms,” George Mason University, pp. 183-190. Naval Research Laboratory, 1989

  25. J.J. Grefenstette, “Learning decision strategies with genetic algorithms,” in Proceedings of the International Workshop on Analogical and Inductive Inference, Lecture Notes in Artificial Intelligence 642, pp. 35-50, 1992.

  26. J.J. Grefenstette, “The evolution of strategies for multiagent environments,” Adaptive Behaviour, vol. 1, no. 1, pp. 65-90, 1992.

    Google Scholar 

  27. J.J. Grefenstette, “Lamarckian learning in multiagent environments,” in Proceedings of 4th International Conference of Scaling Up Inductive Logic Programming 197 Genetic Algorithms, Morgan Kaufmann, pp. 303-310, 1992.

  28. S. Augier, G. Venturini, and Y. Kodratoff, “Learning first order logic rules with a genetic algorithm,” in Proceedings of the First International Conference on Knowledge Discovery and Data Mining, 1995, pp. 21-26.

  29. A. Giordana, L. Saitta, and F. Zini, “Learning disjunctive concepts by means of genetic algorithms,” in Proceedings of the 11th International Conference on Machine Learning, 1994, pp. 96-104.

  30. F. Neri, “First order logic concept learning by means of a distributed genetic algorithm,” Ph.D. dissertation, Università of Torino, Dipartimento di Informatica, 1997.

  31. L. Davis, Handbook of Genetic Algorithms, Van Nostrand Reinhold: New York, 1991.

    Google Scholar 

  32. W.E. Hart and R.K. Belew, “Optimization with genetic algorithm hybrids that use local search,” in Adaptive Individuals in Evolving Populations: Models and Algorithms, edited by R.K. Belew and M. Mitchell, SFI Studies in the Sciences of Complexity, vol. 26, Chap. 27, pp. 483-496, 1996.

  33. D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley: Reading, MA, 1989.

    Google Scholar 

  34. S. Muggleton, “Inductive logic programming,” New Generation Computing, vol. 8, no. 4, pp. 295-318, 1991.

    Google Scholar 

  35. S. Muggleton, “Inductive logic programming: Derivations, successes and shortcomings,” SIGART Bulletin, vol. 5, no. 1, pp. 5-11, 1994.

    Google Scholar 

  36. S. Muggleton, “Inverse entailment and progol,” New Generation Computing, vol. 13, 1995.

  37. U. Pompe, M. Kovačič, and I. Kononenko, “SFOIL: Stochastic approach to inductive logic programming,” in Proceedings of the 2nd Electrotechnical and Computer Science Conference, Portoroz, Slovenija, p. 31, 1993.

  38. M. Kovačič, “Stochastic inductive logic programming,” Ph.D. Thesis, University of Ljubljana, Slovenia, 1994.

    Google Scholar 

  39. L. Breiman, “Bagging predictors,” Technical Report No. 421, University of California, Berkeley, Department of Statistics, 1994.

  40. S.H. Muggleton, M. Bain, J. Hayes-Michie, and D. Michie, “An experimental comparison of human and machine learning formalisms,” in Proc. Sixth International Workshop on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1989. pp. 113-118.

    Google Scholar 

  41. J. Cussens, “Part-of-speech tagging using progol,” in Inductive Logic Programming: Proceedings of the 7th InternationalWorkshop (ILP-97), 1997, volume 1297 of Lecture Notes in Artificial Intelligence, Springer pp. 93-108.

  42. M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz, “Building a large annotated corpus of English: the Penn treebank,” Computational Linguistics, vol. 19, 1993.

  43. J. Cussens, “Part-of-speech disambiguation using ILP,” Technical Report PRG-TR-25-96, Oxford University Computing laboratory, 1996.

  44. J.R. Quinlan, “Discovering rules from large collections of examples: A case study,” in Expert Systems in the Micro-electronic Age, edited by D. Michie, Edinburgh University Press, Edinburgh, pp. 168-201, 1979.

    Google Scholar 

  45. J.W. Lloyd, Foundations of Logic Programming, Springer-Verlag: Berlin, 1984.

    Google Scholar 

  46. T.G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Computation, vol. 10, no. 7, pp. 1895-1924, 1998.

    Google Scholar 

  47. D. Fogel and C. deSilva, eds. Proceedings of the IEEE International Conference on Evolutionary Computation, Perth,Western Australia, Nov. 1995, IEEE Computer Society Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Auckland, Auckland, New Zealand

    Philip G.K. Reiser & Patricia J. Riddle

Authors
  1. Philip G.K. Reiser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patricia J. Riddle
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Reiser, P.G., Riddle, P.J. Scaling Up Inductive Logic Programming: An Evolutionary Wrapper Approach. Applied Intelligence 15, 181–197 (2001). https://doi.org/10.1023/A:1011239013893

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1011239013893

Search

Navigation