Summary
This chapter discusses an approach to the problem of algorithm selection, which exploits the performance metadata of algorithms (workflows) on prior tasks to generate recommendations for a given target dataset. The recommendations are in the form of rankings of candidate algorithms. The methodology involves two phases. In the first one, rankings of algorithms/workflows are elaborated on the basis of historical performance data on different datasets. These are subsequently aggregated into a single ranking (e.g. average ranking). In the second phase, the average ranking is used to schedule tests on the target dataset with the objective of identifying the best performing algorithm. This approach requires that an appropriate evaluation measure, such as accuracy, is set beforehand. In this chapter we also describe a method that builds this ranking based on a combination of accuracy and runtime, yielding good anytime performance. While this approach is rather simple, it can still provide good recommendations to the user. Although the examples in this chapter are from the classification domain, this approach can be applied to other tasks besides algorithm selection, namely hyperparameter optimization (HPO), as well as the combined algorithm selection and hyperparameter optimization (CASH) problem. As this approach works with discrete data, continuous hyperparameters need to be discretized first.
Chapter PDF
Similar content being viewed by others
References
Abdulrahman, S., Brazdil, P., van Rijn, J. N., and Vanschoren, J. (2018). Speeding up algorithm selection using average ranking and active testing by introducing runtime. Machine Learning, 107(1):79–108.
Abdulrahman, S., Brazdil, P., Zainon, W., and Alhassan, A. (2019). Simplifying the algorithm selection using reduction of rankings of classification algorithms. In ICSCA ’19, Proceedings of the 2019 8th Int. Conf. on Software and Computer Applications, Malaysia, pages 140–148. ACM, New York.
Asuncion, A. and Newman, D. (2007). UCI machine learning repository.
Brazdil, P., Gama, J., and Henery, B. (1994). Characterizing the applicability of classification algorithms using meta-level learning. In Bergadano, F. and De Raedt, L., editors, Proceedings of the European Conference on Machine Learning (ECML94), pages 83–102. Springer-Verlag.
Brazdil, P., Soares, C., and da Costa, J. P. (2003). Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3):251–277.
Brazdil, P., Soares, C., and Pereira, R. (2001). Reducing rankings of classifiers by eliminating redundant cases. In Brazdil, P. and Jorge, A., editors, Proceedings of the 10th Portuguese Conference on Artificial Intelligence (EPIA2001). Springer.
Cachada, M. (2017). Ranking classification algorithms on past performance. Master’s thesis, Faculty of Economics, University of Porto.
Cachada, M., Abdulrahman, S., and Brazdil, P. (2017). Combining feature and algorithm hyperparameter selection using some metalearning methods. In Proc. of Workshop AutoML 2017, CEUR Proceedings Vol-1998, pages 75–87.
Charnes, A., Cooper, W., and Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6):429–444.
Cook, W. D., Golany, B., Penn, M., and Raviv, T. (2007). Creating a consensus ranking of proposals from reviewers’ partial ordinal rankings. Computers & Operations Research, 34(4):954–965.
Cook, W. D., Kress, M., and Seiford, L. W. (1996). A general framework for distance based consensus in ordinal ranking models. European Journal of Operational Research, 96(2):392–397.
Gama, J. and Brazdil, P. (1995). Characterization of classification algorithms. In Pinto-Ferreira, C. and Mamede, N. J., editors, Progress in Artificial Intelligence, Proceedings of the Seventh Portuguese Conference on Artificial Intelligence, pages 189–200. Springer-Verlag.
Hall, M. (1999). Correlation-based feature selection for machine learning. PhD thesis, University of Waikato.
Hand, D., Mannila, H., and Smyth, P. (2001). Principles of Data Mining. MIT Press.
Kalousis, A. (2002). Algorithm Selection via Meta-Learning. PhD thesis, University of Geneva, Department of Computer Science.
Kalousis, A. and Theoharis, T. (1999). NOEMON: Design, implementation and performance results of an intelligent assistant for classifier selection. Intelligent Data Analysis, 3(5):319–337.
Keller, J., Paterson, I., and Berrer, H. (2000). An integrated concept for multi-criteria ranking of data-mining algorithms. In Keller, J. and Giraud-Carrier, C., editors, Proceedings of the ECML Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pages 73–85.
Leite, R. and Brazdil, P. (2010). Active testing strategy to predict the best classification algorithm via sampling and metalearning. In Proceedings of the 19th European Conference on Artificial Intelligence (ECAI), pages 309–314.
Lin, S. (2010). Rank aggregation methods. WIREs Computational Statistics, 2:555–570.
Mitchell, T. M. (1997). Machine Learning. McGraw-Hill.
Nakhaeizadeh, G. and Schnabl, A. (1997). Development of multi-criteria metrics for evaluation of data mining algorithms. In Proceedings of the Fourth International Conference on Knowledge Discovery in Databases & Data Mining, pages 37–42. AAAI Press.
Nakhaeizadeh, G. and Schnabl, A. (1998). Towards the personalization of algorithms evaluation in data mining. In Agrawal, R. and Stolorz, P., editors, Proceedings of the Third International Conference on Knowledge Discovery & Data Mining, pages 289–293. AAAI Press.
Neave, H. R. and Worthington, P. L. (1992). Distribution-Free Tests. Routledge.
Pavan, M. and Todeschini, R. (2004). New indices for analysing partial ranking diagrams. Analytica Chimica Acta, 515(1):167–181.
Pfahringer, B., Bensusan, H., and Giraud-Carrier, C. (2000). Meta-learning by landmarking various learning algorithms. In Langley, P., editor, Proceedings of the 17th International Conference on Machine Learning, ICML’00, pages 743–750.
Pfisterer, F., van Rijn, J. N., Probst, P., Müller, A., and Bischl, B. (2018). Learning multiple defaults for machine learning algorithms. arXiv preprint arXiv:1811.09409.
Pihur, V., Datta, S., and Datta, S. (2009). RankAggreg, an R package for weighted rank aggregation. BMC Bioinformatics, 10(1):62.
Soares, C. and Brazdil, P. (2000). Zoomed ranking: Selection of classification algorithms based on relevant performance information. In Zighed, D. A., Komorowski, J., and Zytkow, J., editors, Proceedings of the Fourth European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD 2000), pages 126–135. Springer.
Todorovski, L. and Džeroski, S. (1999). Experiments in meta-level learning with ILP. In Rauch, J. and Zytkow, J., editors, Proceedings of the Third European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD99), pages 98–106. Springer.
van Rijn, J. N., Abdulrahman, S., Brazdil, P., and Vanschoren, J. (2015). Fast algorithm selection using learning curves. In International Symposium on Intelligent Data Analysis XIV, pages 298–309.
Wistuba, M., Schilling, N., and Schmidt-Thieme, L. (2015). Sequential model-free hyperparameter tuning. In 2015 IEEE International Conference on Data Mining, pages 1033–1038.
Author information
Authors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2022 The Author(s)
About this chapter
Cite this chapter
Brazdil, P., van Rijn, J.N., Soares, C., Vanschoren, J. (2022). Metalearning Approaches for Algorithm Selection I (Exploiting Rankings). In: Metalearning. Cognitive Technologies. Springer, Cham. https://doi.org/10.1007/978-3-030-67024-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-67024-5_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67023-8
Online ISBN: 978-3-030-67024-5
eBook Packages: Computer ScienceComputer Science (R0)