A Comparison of Machine Learning Methods in a High-Dimensional Classification Problem

• Authors: Marijana Zekić-Sušac , Sanja Pfeifer , Nataša Šarlija
• Languages of publication: EN

Abstracts

EN

Background: Large-dimensional data modelling often relies on variable reduction methods in the pre-processing and in the post-processing stage. However, such a reduction usually provides less information and yields a lower accuracy of the model. Objectives: The aim of this paper is to assess the high-dimensional classification problem of recognizing entrepreneurial intentions of students by machine learning methods. Methods/Approach: Four methods were tested: artificial neural networks, CART classification trees, support vector machines, and k-nearest neighbour on the same dataset in order to compare their efficiency in the sense of classification accuracy. The performance of each method was compared on ten subsamples in a 10-fold cross-validation procedure in order to assess computing sensitivity and specificity of each model. Results: The artificial neural network model based on multilayer perceptron yielded a higher classification rate than the models produced by other methods. The pairwise t-test showed a statistical significance between the artificial neural network and the k-nearest neighbour model, while the difference among other methods was not statistically significant. Conclusions: Tested machine learning methods are able to learn fast and achieve high classification accuracy. However, further advancement can be assured by testing a few additional methodological refinements in machine learning methods.

Keywords

EN

machine learning; support vector machines; artificial neural networks; CART classification trees; k-nearest neighbour; large-dimensional data; crossvalidation

• Publisher: Sciendo
• Journal: Business Systems Research Journal
• Year: 2014
• Volume: 5
• Issue: 3
• Pages: 82-96

Dates

published

2014-09-01

received

2013-12-15

accepted

2014-05-18

online

2014-09-25

Contributors

author

Marijana Zekić-Sušac

• University of Josip Juraj Strossmayer in Osijek, Faculty of Economics, Croatia

author

Sanja Pfeifer

• University of Josip Juraj Strossmayer in Osijek, Faculty of Economics, Croatia

author

Nataša Šarlija

• University of Josip Juraj Strossmayer in Osijek, Faculty of Economics, Croatia

References

• 1. Apte, C., Weiss, S.(1997), “Data Mining with Decision Trees and Decision Rules”, Future Generation Computer Systems , Vol. 13, No.2, pp. 197-210.
• 2. Behzad, M., Asghar, K., Eazi, M., Palhang, M. (2009), „Generalization performance of support vector machines and neural networks in runoff modeling“, Expert Systems with Applications, Vol. 36, No.4, pp. 7624-7629.
• 3. Bensic, M., Sarlija, N., Zekic-Susac, M. (2005), „Modeling Small Business Credit Scoring Using Logistic Regression, Neural Networks, and Decision Trees“, Intelligent Systems in Accounting, Finance and Management, Vol. 13, No. 3, pp. 133-150.
• 4. Bishop, C. (1995), Neural Networks for Pattern Recognition, University Press, Oxford, UK.
• 5. Bolivar-Cime, A., Marron, J.S. (2013), “Comparison of binary discrimination methods for high dimension low sample size data”, Journal of Multivariate Analysis, Vol. 115, No. 1, pp. 108-121
• 6. Brown, D.E., Corruble, V., Pittard, C.L. (1993). “A comparison of decision tree classifiers with backpropagation neural networks for multimodal classification problems”, Pattern Recognition, Vol. 26, No. 6, pp. 953-961.[Crossref]
• 7. Carr, J.C., Sequeira, J.M. (2007), “Prior family business exposure as intergenerational entrepreneurial intent: A theory of planned behavior approach”, Journal of Business Research, Vol. 60, No.10, pp.1090-1098.[Crossref]
• 8. Dai, Y-H. (2002), “Convergence properties of the BFGS algorithm“, SIAM Journal of Optimization, Vol. 13, No. 3, pp. 693-701.[Crossref]
• 9. Farmer, S.M., X. Yao., Kung-Mcintyre, K. (2011), “The behavioral impact of entrepreneur identity aspiration and prior entrepreneurial experience”, Entrepreneurship Theory and Practice, Vo. 35, No. 2, pp. 245-273.[Crossref][WoS]
• 10. Haykin, S. (1999), Neural Networks: A Comprehensive Foundation, Prentice Hall International, Inc., New Jersey, USA.
• 11. Kolvereid, L., Isaksen, E. (2006), “New business start-up and subsequent entry into self-employment”, Journal of Business Venturing, Vol. 21, No.6, pp. 866-885.[Crossref]
• 12. Krueger, N.F. Jr. (2000), “The Cognitive Infrastructure of Opportunity Emergence”, Entrepreneurship: Theory and Practice, Vol. 24, No. 3, pp. 5-23.
• 13. Krueger, N.F. JR., Reilly, M.D., Carsrud, A.L. (2000), “Competing Models of Entrepreneurial Intentions”, Journal of Business Venturing, Vol. 15, No.5, pp. 411-432.[WoS][Crossref]
• 14. Kuzey, C., Uyar, A., Delen, D. (2014), “The impact of multinationality on firm value: A comparative analysis of machine learning techniques”, Decision Support Systems, Vol. 59, No. 1, pp. 127-142.
• 15. Lee, S.(2010), “Using data envelopment analysis and decision trees for efficiency analysis and recommendation of B2C controls”, Decision Support Systems, Vol. 49, No.4, pp. 486-497.
• 16. Lin, W.B. (2006), „A comparative study on the trends of entrepreneurial behaviors of enterprises in different strategies: Application of the social cognition theory“, Expert Systems with Applications, Vol. 31, No.2, pp. 207-220.
• 17. Liu, G., Yi, Z., Yang, S. (2007), „A hierarchical intrusion detection model based on the PCA neural networks“, Neurocomputing, Vol. 70, No.7, pp. 1561-1568.[WoS][Crossref]
• 18. Masters, T. (1995), Advanced Algorithms for Neural Networks, A C++ Sourcebook, John Wiley & Sons, Inc., New York, USA.
• 19. McGee, J., Peterson, M., Mueller, S., Sequeira, J.M. (2009), “Entrepreneurial selfefficacy: Refining the measure and examining its relationship to attitudes toward venturing and nascent entrepreneurship”, Entrepreneurship Theory and Practice, Vol. 33, No.4, pp. 965-988.[Crossref]
• 20. Min, J.H., Lee, Y.-C. (2005), „Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters“, Expert Systems with Applications, Vol. 28, No. 4, pp. 603-614.
• 21. Nga, J.K.H., Shamuganathan, G. (2010), “The influence of personality traits and demographic factors on social entrepreneurship start up intentions”, Journal of Business Ethics, Vol. 95, No.2, pp. 259-282.[WoS]
• 22. Paliwal, M., Kumar, U.A. (2009), “Neural networks and statistical techniques: A review of applications”, Expert Systems with Applications, Vol. 36, No.1, pp. 2-17.
• 23. Questier, F., Put, R., Coomans, D., Walczak, B., Vander Heyden, Y. (2005), “The use of CART and multivariate regression trees for supervised and unsupervised feature selection”, Chemometrics and Intelligent Laboratory Systems, Vol. 76, No.1, pp. 45-54.
• 24. Shao, Y., Lunetta, R.S. (2012), “Comparison of support vector machine, neural network, and CART algorithms for the land-cover classification using limited training data points”, ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 70, June 2012, pp. 78-87.[Crossref]
• 25. Shin, H.J., Eom, D.-H., Kim, S.S. (2005), “One-class support vector machines - an application in machine fault detection and classification“, Computers & Industrial Engineering, Vol. 48, No.2, pp. 395-408.[Crossref]
• 26. Simon, D. and Boring, J.R.(1990), “Sensitivity, Specificity, and Predictive Value”, in Walker, H.K., Hall, W.D., Hurst, J.W. (Eds.), Clinical Methods: The History, Physical, and Laboratory Examinations, Butterworths, Boston, pp. 49-54.
• 27. Smith, T.W. (2009), Altruism and Empathy in America: Trends and Correlates, National Opinion Research Center, University of Chicago, Chicago.
• 28. St. John, C.H., Balakrishnan, N., Fiet, J.O. (2000), „Modeling the relationship between corporate strategy and wealth creation using neural networks“, Computers & Operations Research, Vol. 27, No. 11, pp. 1077-1092.
• 29. Talukder, A., Casasent, D. (2001), “A closed-form neural network for discriminatory feature extraction from high-dimensional data”, Neural Networks, Vol. 14, No. 9, pp. 1201-1218.[Crossref]
• 30. Thompson, E.R. (2009), “Individual entrepreneurial intent: Construct clarification and development of an internationally reliable metric”, Entrepreneurship Theory and Practice, Vol. 33, No. 3, pp. 669-694.[Crossref][WoS]
• 31. Triandis, H.C., Gelfand, M.J. (1998), “Converging Measurement of Horizontal and Vertical Individualism and Collectivism”, Journal of Personality and Social Psychology, Vol. 74, No.1, pp.118-128.[Crossref]
• 32. Witten, I.H., Frank, E. (2000), Data Mining: Practical Machine Learning Tools and Techniques with Java Implementation, Morgan Kaufman Publishers, San Francisco.
• 33. Yeh, C.C, et al. (2010), „A hybrid approach of DEA, rough set and support vector machines for business failure prediction”, Expert Systems with Applications, Vol. 37, No.2, pp. 1535-1541.
• 34. Yu, H., Yang, J., Han, J. (2003),„Classifying Large Data Sets Using SVMs with Hierarchical Clustering“, in Proceedings of the ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM New York, NY, USA, pp. 306-315.
• 35. Zanaty, E.A. (2012), “Support Vector Machines (SVMs) versus Multilayer Perception (MLP) in data classification”, Egyptian Informatics Journal, Vol. 13, No. 3, pp. 177-183.
• 36. Zekic-Susac, M., Pfeifer, S., Djurdjevic, I. (2010), „Classification of entrepreneurial intentions by neural networks, decision trees and support vector machines“, Croatian Operational Research Review, Vol. 1, No.1, pp. 62-71.
• 37. Zekic-Susac, M., Šarlija, N., Pfeifer, S. (2012), „Combining PCA analysis and neural networks in modelling entrepreneurial intentions of students“, Croatian Operational Research Review, Vol. 4, No.1, pp. 306-317.

Identifiers

DOI

10.2478/bsrj-2014-0021