Comparison of Particle Swarm Optimization Algorithms in Hyperparameter Optimization Problem of Multi Layered Perceptron

Kenta Shiomi; Tetsuya Sato; Eisuke Kita

doi:10.24423/cames.2025.1730

Kenta Shiomi Graduate School of Informatics, Nagoya University
Tetsuya Sato Graduate School of Informatics, Nagoya University
Eisuke Kita Graduate School of Informatics, Nagoya University

DOI: http://dx.doi.org/10.24423/cames.2025.1730

Abstract

This paper describes the application of particle swarm optimization (PSO) for the hyperparameter optimization problem of multi-layered perceptron (MLP) model. Several PSO algorithms are presented by many researchers; basic PSO, PSO with inertia weight (PSO-w), PSO with constriction factor (PSO-cf), local PSO-w, local PSO-cf, union of local and global PSOs (UPSO), PSO with second global best particle (SG-PSO), and PSO with second local best particle (SP-PSO). The wine dataset is taken as a numerical example and hyperparameters of MLP the model are determined by the above-mentioned PSO algorithms. The sets of hyperparameters determined by these PSO algorithms are compared with the results of the traditional algorithms for hyperparameter optimization such as random search, tree-structured Parzen estimator (TPE), and covariance matrix adaptation evolution strategy (CMA-ES).

Numerical results indicate that PSO-cf is the best-performing and local PSO-w is the second best among the PSO algorithms. The sets of hyperparameters determined by the PSO algorithms were relatively similar. An important finding from the numerical results is that PSO algorithms could find better hyperparameters than random search, TPE, and CMA-ES. This demonstrates that PSO is suitable for the hyperparameter optimization problem in MLP models.

Keywords

multi layered perceptron (MLP), hyperparameter optimization, particle swarm optimization (PSO), wine dataset,

References

1. I. Goodfellow et al., Generative adversarial networks, Communications of the ACM, 63(11): 139–144, 2020, https://doi.org/10.1145/3422622.
2. V. Mnih et al., Human-level control through deep reinforcement learning, Nature, 518: 529–533, 2015, https://doi.org/10.1038/nature14236.
3. M. Ferrari Dacrema, P. Cremonesi, D. Jannach, Are we really making much progress? A worrying analysis of recent neural recommendation approaches, [in:] Proceedings of the 13th ACM Conference on Recommender Systems (RecSys ’19), Association for Computing Machinery, New York, NY, USA, pp. 101–109, 2019, https://doi.org/10.1145/3298689.3347058.
4. J. Bergstra, R. Bardenet, Y. Bengio, B. K´egl, Algorithms for hyper-parameter optimization, [in:] Advances in Neural Information Processing Systems 24 (NIPS 2011), Vol. 24, pp. 2546–2554, 2011.
5. J. Snoek, H. Larochelle, R.P. Adams, Practical Bayesian optimization of machine learning algorithms, [in:] Advances in Neural Information Processing Systems 25 (NIPS 2012), Vol. 25, pp. 2951–2959, 2012.
6. M. Feurer, F. Hutter, Hyperparameter Optimization, [in:] F. Hutter, L. Kotthoff, J. Vanschoren [Eds.], Automated Machine Learning: Methods, Systems, Challenges, Chapter 1, pp. 3–33, Springer, Cham, 2019, https://doi.org/10.1007/978-3-030-05318-5 1.
7. R.Z. Cabada, H.R. Rangel, M.L.B. Estrada, H.M.C. Lopez, Hyperparameter optimization in CNN for learning-centered emotion recognition for intelligent tutoring systems, Soft Computing, 24(10): 7593–7602, 2020, https://doi.org/10.1007/s00500-019-04387-4.
8. P. Singh, S. Chaudhury, B.K. Panigrahi, Hybrid MPSO-CNN: Multi-level particle swarm optimized hyperparameters of convolutional neural network, Swarm and Evolutionary Computation, 63: 100863, 2021, https://doi.org/10.1016/j.swevo.2021.100863.
9. N.M. Aszemi, P.D.D. Dominic, Hyperparameter optimization in convolutional neural network using genetic algorithms, International Journal of Advanced Computer Science and Applications, 10(6): 269–278, 2019, https://doi.org/10.14569/IJACSA.2019.0100638.
10. P. Ribalta Lorenzo, J. Nalepa, M. Kawulok, L. Sanchez Ramos, J. Ranilla, Particle swarm optimization for hyper-parameter selection in deep neural networks, [in:] Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’17), Association for Computing Machinery, pp. 481–488, 2017, https://doi.org/10.1145/3071178.3071208.
11. P. Ribalta Lorenzo, J. Nalepa, L. Sanchez Ramos, J. Ranilla, Hyper-parameter selection in deep neural networks using parallel particle swarm optimization, [in:] Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO ’17), Association for Computing Machinery, pp. 1864–1871, 2017, https://doi.org/10.1145/3067695.3084211.
12. Y. Wang, H. Zhang, G. Zhang, cPSO-CNN: An efficient PSO-based algorithm for finetuning hyper-parameters of convolutional neural networks, Swarm and Evolutionary Computation, 49: 114–123, 2019, https://doi.org/10.1016/j.swevo.2019.06.002.
13. D. Sarkar, T. Khan, F. Ahmed Talukdar, Hyperparameters optimization of neural network using improved particle swarm optimization for modeling of electromagnetic inverse problems, International Journal of Microwave and Wireless Technologies, 14(10): 1326–1337, 2022, https://doi.org/10.1017/S1759078721001690.
14. J. Kennedy, R.C. Eberhart, Particle swarm optimization, [in:] Proceedings of ICNN’95 – International Conference on Neural Networks, Vol. 4, pp. 1942–1948, 1995, https://doi.org/10.1109/ICNN.1995.488968.
15. Y. Shi, R. Eberhart, A modified particle swarm optimizer, [in:] 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence, Anchorage, AK, USA, pp. 69–73, 1998, https://doi.org/10.1109/ICEC.1998.699146.
16. A. Maleki, M. Ameri, F. Keynia, Scrutiny of multifarious particle swarm optimization for finding the optimal size of a PV/wind/battery hybrid system, Renewable Energy, 80: 552–563, 2015, https://doi.org/10.1016/j.renene.2015.02.045.
17. Y. Sun, Z.Wang, B.J. van Wyk, Local and global search based PSO algorithm, [in:] Y. Tan, Y. Shi, H. Mo [Eds.], Advances in Swarm Intelligence. ICSI 2013. Lecture Notes in Computer Science, Vol. 7928, pp. 129–136, Springer, Berlin, Heidelberg, 2013, https://doi.org/10.1007/978-3-642-38703-6 15.
18. K.E. Parsopoulos, M.N. Vrahatis, UPSO: A unified particle swarm optimization scheme, [in:] International Conference of Computational Methods in Sciences and Engineering (ICCMSE 2004), pp. 868–873, CRC Press, 2019.
19. Y.-B. Shin, E. Kita, Search performance improvement of particle swarm optimization by second best particle information, Applied Mathematics and Computation, 246: 346–354, 2014, https://doi.org/10.1016/j.amc.2014.08.013.
20. S. Watanabe, F. Hutter, c-TPE: Tree-structured Parzen estimator with inequality constraints for expensive hyperparameter optimization, [in:] E. Elkind [Ed.], Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 4371–4379, International Joint Conferences on Artificial Intelligence Organization, 2023, https://doi.org/10.48550/arXiv.2211.14411.
21. Y. Mei, H. Wang, Covariance matrix adaptation evolution strategy assisted by principal component analysis, arXiv, 2021, https://doi.org/10.48550/arXiv.2105.03687.
22. I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, The MIT Press, 2016.