Skip to main content
SpringerLink
Log in

Multi-class pattern classification using single, multi-dimensional feature-space feature extraction evolved by multi-objective genetic programming and its application to network intrusion detection

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

In this paper we investigate using multi-objective genetic programming to evolve a feature extraction stage for multiple-class classifiers. We find mappings which transform the input space into a new, multi-dimensional decision space to increase the discrimination between all classes; the number of dimensions of this decision space is optimized as part of the evolutionary process. A simple and fast multi-class classifier is then implemented in this multi-dimensional decision space. Mapping to a single decision space has significant computational advantages compared to k-class-to-2-class decompositions; a key design requirement in this work has been the ability to incorporate changing priors and/or costs associated with mislabeling without retraining. We have employed multi-objective optimization in a Pareto framework incorporating solution complexity as an independent objective to be minimized in addition to the main objective of the misclassification error. We thus give preference to simpler solutions which tend to generalize well on unseen data, in accordance with Occam’s Razor. We obtain classification results on a series of benchmark problems which are essentially identical to previous, more complex decomposition approaches. Our solutions are much simpler and computationally attractive as well as able to readily incorporate changing priors/costs. In addition, we have also applied our approach to the KDD-99 intrusion detection dataset and obtained results which are highly competitive with the KDD-99 Cup winner but with a significantly simpler classification framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Usage of the terms “feature extraction”, “selection” and “construction” varies in the literature. In this paper we follow the terminology of [1] and consider feature selection as the selection of a subset of the raw pattern attributes, and feature extraction as the (possibly non-linear) combination of that selected subset of features to form new, more discriminatory features in the space in the which the decision is made.

  2. Although finding a true optimum can only be guaranteed by exhaustive evaluation of all the possible combinations, like other stochastic techniques, GP typically locates near-optimal solutions which usually represent practical solutions to problems.

  3. Downloadable from http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.

References

  1. I. Guyon, A. Elisseeff, An introduction to feature extraction. In Feature Extraction, Foundations and Applications, eds. by I. Guyon, S. Gunn, M. Nikravesh, L. Zadeh. (Springer, Heidelberg, 2006)

    Google Scholar 

  2. Y. Zhang, P.I. Rockett, Domain-independent feature extraction for multi-classification using multi-objective genetic programming. Pattern Anal. Appl. 13(3), 273–288 (2010)

    Article  MathSciNet  Google Scholar 

  3. A. Bailey, Class-dependent Features and Multicategory Classification. PhD thesis, Department of Electronics and Computer Science, University of Southampton, Southampton, UK, 2001

  4. A. Kattan, A. Agapitos, R. Poli, Unsupervised problem decomposition using genetic programming. In 13th European Conference (EuroGP 2010), eds. by A. Esparcia-Alcźar, A. Ekárt, S. Silva, S. Dignum, A. Uyar (Springer, Berlin/Heidelberg, Istanbul, 2010), pp. 122–133 (In LNCS vol. 6021)

  5. A. McIntyre, M. Heywood et al., MOGE: GP classification problem decomposition using multi-objective optimization. In 8th Annual Conference on Genetic and Evolutionary Computation (GECCO2006), vol. 1, eds. by M. Keijzer. (ACM Press, Seattle, WA, 2006), pp. 863–870

    Chapter  Google Scholar 

  6. A.R. McIntyre, M.I. Heywood, Pareto cooperative-competitive genetic programming: a classification benchmarking study. In Genetic Programming Theory and Practice VI, eds. by R. Riolo, T. Soule, B. Worzel. (Springer, Ann Arbor, MI, 2008), pp. 43–60.

    Google Scholar 

  7. T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. (Springer, Berlin, 2009)

    MATH  Google Scholar 

  8. R. Poli, W.B. Langdon, N.F. McPhee, A Field Guide to Genetic Programming (2008). Published via http://lulu.com and freely available at http://www.gp-field-guide.org.uk

  9. Y. Zhang, P.I. Rockett, Feature extraction using multi-objective genetic programming. In: Y. Jin (eds) Multi-Objective Machine Learning, (Springer, Heidelberg, 2006)

    Google Scholar 

  10. R. Kohavi, D. Sommerfield, Feature subset selection using the wrapper method: overfitting and dynamic search space topology. In 1st International Conference on Knowledge Discovery and Data Mining (KDD-95). (AAAI Press, 1995), pp. 192–197

  11. C.A. Coello, van D. Veldhuizen, G.B. Lamont, Evolutionary Algorithms for Solving Multi-Objective Problems. (Kluwer, Norwell, MA, 2002)

    MATH  Google Scholar 

  12. K. Deb, Multi-objective Optimization Using Evolutionary Algorithms. (Wiley, New York, 2001)

    MATH  Google Scholar 

  13. Y. Jin (eds), Multi-objective Machine Learning. (Springer, Berlin, 2006)

    MATH  Google Scholar 

  14. Y. Jin, B. Sendhoff, Pareto-based multi-objective machine learning: an overview and case studies. IEEE Trans. Syst. Man Cyber. Part C 38(3), 397–415 (2008)

    Article  Google Scholar 

  15. Y. Zhang, P.I. Rockett, A generic optimising feature extraction method using multiobjective genetic programming. Appl. Soft Comput. 11(1), 1087–1097 (2011)

    Article  Google Scholar 

  16. Y. Zhang, M. Zhang, A Multiple Output Program Tree Structure in Genetic Programming. Technical Report CS-TR-04/14 (Department of Computer Science, School of Mathematical and Computing Sciences, Victoria University, Wellington, New Zealand, 2004)

  17. K. Krawiec, Genetic programming-based construction of features for machine learning and knowledge discovery tasks. Gene. Prog. Evol. Mach. 3(4), 329–343 (2002)

    Article  MATH  Google Scholar 

  18. C. Estébanez, R. Aler, J.M. Valls, A method based on genetic programming for improving the quality of datasets in classification problems. Int. J. Comput. Sci. Appl. 4(1), 69–80 (2007)

    Google Scholar 

  19. M. Oltean, L. Dioşan, An autonomous GP-based system for regression and classification problems. Appl. Soft Comput. 9(1), 49–60 (2009)

    Article  Google Scholar 

  20. J.R. Sherrah, Automatic Feature Extraction for Pattern Recognition. PhD Thesis, University of Adelaide, Adelaide, Australia, 1998

  21. J.R. Sherrah, R.E. Bogner, A. Bouzerdoum, The evolutionary pre-processor: automatic feature extraction for supervised classification using genetic programming. In 2nd International Conference on Genetic Programming. Morgan Kaufmann 1997), pp. 304–312

  22. M.G. Smith, L. Bull, Genetic programming with a genetic algorithm for feature construction and selection. Gene. Prog. Evol. Mach. 6(3), 265–281 (2005)

    Article  Google Scholar 

  23. H. Vafaie, K. de Jong, Genetic algorithms as a tool for restructuring feature space representations. In 7th International Conference on Tools with Artificial Intelligence. (IEEE Computer Society, 1995), pp. 8–11

  24. Y. Zhang, P.I. Rockett, Evolving optimal feature extraction using multi-objective genetic programming: a methodology and preliminary study on edge detection. In Genetic and Evolutionary Computation Conference (GECCO 2005). (ACM Press, 2005), pp. 795–802

  25. Y. Zhang, P.I. Rockett, A generic multi-dimensional feature extraction method using multiobjective genetic programming. Evol. Comput. 17(1), 89–115 (2009)

    Article  Google Scholar 

  26. C. Fonseca, P.J. Fleming, Multiobjective optimization and multiple constraint handling with evolutionary algorithms—part I: a unified formulation. IEEE Trans. Syst. Man Cyber. Part A Syst. Hum. 28(1), 26–37 (1998)

    Article  Google Scholar 

  27. K. Chellapilla, Evolving computer programs without subtree crossover. IEEE Trans. Evol. Comput. 1(3), 209–216 (1997)

    Article  Google Scholar 

  28. M.F. Brameier, W. Banzhaf, Linear Genetic Programming. (Springer, New York, 2007)

    MATH  Google Scholar 

  29. K. Badran, P.I. Rockett, The influence of mutation on population dynamics in multiobjective genetic programming. Gene. Prog. Evol. Mach. 11(1), 5–31 (2010)

    Article  Google Scholar 

  30. T. Ito, H. Iba, S. Sato, Depth-dependent crossover for genetic programming. In IEEE World Congress on Computational Intelligence, (IEEE Press, 1998), pp. 775–780

  31. R. Kumar, P.I. Rockett, Improved sampling of the Pareto-front in multiobjective genetic optimizations by steady-state evolution: a Pareto converging genetic algorithm. Evol. Comput. 10(3), 283–314 (2002)

    Article  Google Scholar 

  32. Y. Zhang, P.I. Rockett, Comparison of evolutionary strategies for multi-objective genetic programming. In IEEE Systems, Man and Cybernetics Society Conference on Advances in Cybernetic Systems (AICS2006) (2006)

  33. E. Burke, S. Gustafson, G. Kendall, A survey and analysis of diversity measures in genetic programming. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2002). (Morgan Kaufmann, 2002), pp. 716–723

  34. A. Ekárt, S.Z. Németh, Selection based on the Pareto nondomination criterion for controlling code growth in genetic programming. Gene. Prog. Evol. Mach. 2(1), 61–73 (2001)

    Article  MATH  Google Scholar 

  35. R.O. Duda, P.E. Hart, D.G. Stork, Pattern Recognition, 2nd edn. (Wiley, New York, 2001)

    Google Scholar 

  36. G.H. Golub, van C.F. Loan, Matrix Computations, 3rd edn. (Johns Hopkins University Press, Maryland, 1996)

    MATH  Google Scholar 

  37. M.T. Heath, Scientific Computing: An Introductory Survey, 2nd edn. (McGraw-Hill, New York, 2005)

    Google Scholar 

  38. A. Asuncion, D.J. Newman, UCI machine learning repository. http://www.ics.uci.edu/~mlearn/MLRepository.html (University of California, Irvine, School of Information and Computer Sciences, 2007)

  39. T.G. Dietterich, Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput. 10(7), 1895–1923 (1998)

    Article  Google Scholar 

  40. C. Nadeau, Y. Bengio, Inference for the generalization error. Mach. Learn. 52(3), 239–281 (2003)

    Article  MATH  Google Scholar 

  41. D.W. Aha, D. Kibler, M.K. Albert, Instance-based learning algorithms. Mach. Learn. 6(1), 37–66 (1991)

    Google Scholar 

  42. R.P. Lippmann, D.J. Fried, I. Graf, J.W. Haines, K.R. Kendall, D. McClung, D. Weber, S.E. Webster, D. Wyschogrod, R.K. Cunningham, M.A. Zissman, Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. In DARPA Information Survivability Conference and Exposition, vol. 2. (IEEE, 2000), pp. 12–26.

  43. K. Kendall, A Database of Computer Attacks for the Evaluation of Intrusion Detection Systems. Masters Thesis, Massachusetts Institute of Technology, 1998.

  44. R. Coolen, H. Luiijf, Intrusion detection: generics and state-of-the-art. Technical report RTO TR-049 / IST-008, The Research and Technology Organisation (RTO) of NATO (2002)

  45. K. Labib, Computer security and intrusion detection. ACM Crossroads 11(1), 2 (2004)

    Article  Google Scholar 

  46. W. Lee, S. Stolfo, Framework for constructing features and models for intrusion detection systems. ACM Trans. Info. Syst. Secur. (TISSEC) 3(4), 227–261 (2000)

    Article  Google Scholar 

  47. C. Elkan, Results of the KDD’99 classifier learning. SIGKDD Explor. 1(2), 63–64 (2000)

    Article  Google Scholar 

  48. B. Pfahringer, Winning the KDD99 classification cup: bagged boosting. SIGKDD Explor. 1(2), 65–66 (2000)

    Article  Google Scholar 

  49. I. Levin, KDD-99 classifier learning contest Llsoft’s results overview. SIGKDD Explor. 1(2), 67–75 (2000)

    Article  Google Scholar 

  50. V. Miheev, A. Vopilov, I. Shabalin, The MP13 approach to the KDD’99 classifier learning contest. SIGKDD Explor. 1(2), 76–77 (2000)

    Article  Google Scholar 

  51. E. Eskin, A. Arnold, M. Prerau , L. Portnoy, S. Stolfo, A geometric framework for unsupervised anomaly detection: detecting intrusions in unlabeled data. In Applications of Data Mining in Computer Security, eds. by D. Barbara, S. Jajodia. (Kluwer, 2002)

  52. H.G. Kayacik, A.N. Zincir-Heywood, M.I. Heywood, On the capability of an SOM based intrusion detection system. In International Joint Conference on Neural Networks. (IEEE, 2003), pp. 1808–1813

  53. R. Agarwal, M.V. Joshi, PNrule: a new framework for learning classifier models in data mining (a case-study in network intrusion detection). In 1st SIAM International Conference on Data Mining, ed. by G.R, K. V. (Chicago, IL, 2000), pp. 1–17

  54. D. Song, M.I. Heywood, A.N. Zincir-Heywood, Training genetic programming on half a million patterns: an example from anomaly detection. IEEE Trans. Evol. Comput. 9(3), 225–239 (2005)

    Article  Google Scholar 

  55. K.M. Faraoun, A. Boukelif, Genetic programming approach for multi-category pattern classification applied to network intrusions detection. Int. J. Comput. Intell. Appl. 6(1), 77–99 (2006)

    Article  Google Scholar 

  56. K. Badran, P. Rockett, Integrating categorical variables with multiobjective genetic programming for classifier construction. In 11th European Conference on Genetic Programming (EuroGP2008). (Springer, 2008), pp. 301–311

  57. M. Sabhnani, G. Serpen, Application of machine learning algorithms to KDD intrusion detection dataset within misuse detection context. In International Conference on Machine Learning, Models, Technologies and Applications. (MLMTA, 2003), pp. 209–215

  58. M. Sabhnani, G. Serpen, Why machine learning algorithms fail in misuse detection on KDD intrusion detection data set. Intell. Data Anal. 8(4), 403–415 (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Vision and Information Engineering Research Group, Department of Electronic and Electrical Engineering, The University of Sheffield, Mappin Street, Sheffield, S1 3D, UK

    Khaled Badran & Peter Rockett

Authors
  1. Khaled Badran
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Rockett
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter Rockett.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Badran, K., Rockett, P. Multi-class pattern classification using single, multi-dimensional feature-space feature extraction evolved by multi-objective genetic programming and its application to network intrusion detection. Genet Program Evolvable Mach 13, 33–63 (2012). https://doi.org/10.1007/s10710-011-9143-4

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-011-9143-4

Keywords

Search

Navigation