Tuning XGBoost With XGBoost: Writing Your Own Hyper Parameters Optimization Engine

Update: discover my new book on Gradient Boosting, Practical Gradient Boosting. It’s a deep dive into Gradient Boosting with many examples in python. Writing a Hyper Parameter Tuning engine seems like a huge challenge. As you probably know if you are familiar with Data Science, Top Source Media Machine Learning, Toward Data Science or my previous post on the subject, fine-tuning your model is crucial for getting the best performances. You simply cannot rely on default values. As Satyam Kumar states in his last article, several methods exist to perform this optimization. They range from manual tuning, random search, brute force to bayesian search. Each of these methods has its advantages and drawbacks. This article will focus on a pretty recent way to perform HP optimization: Model-Based HP Tuning. This method is quite interesting, as it uses ML methods to tune ML models. We can reuse the tools we are familiar with to optimize the tools we are familiar with 🙂 dizzying, no? Hyper Parameters can be regarded as structured, tabular data, and what is the most versatile ML algorithm that we know for structured, tabular data?



The uncontested winner of Kaggle competitions? Let’s see what we can do with it, and try to use it to tune itself. The idea behind model-based tuning is pretty simple: to speed up convergence towards the best parameters for a given use case, we need a way to guide the Hyper Parameters Optimization towards the best solution. Indeed, training a model can be time-consuming, depending on the size of the training set, and the combinatorics of the configuration space that is usually very large. This means that we need a way to identify the most promising configuration to evaluate. Why not simply use a model to learn an estimator of the score for a given configuration? Each training will then be used to refine the underlying model, and give us some insight into the direction to explore. This is the leading idea of model-based Hyper Parameter Optimization. If you are interested in the theory behind this idea, Top Source Media online I strongly advise reading this academic paper.



You could also be curious to look at the implementation of SMAC that uses this method. 7. Go back to step 1. if the maximal number of iterations or minimal score is not yet reached. Let’s see how we can implement this. The first question we have to answer is: how do we sample the configuration space? I.e: how do we pick randomly an eligible configuration in the configuration space? This is not a very difficult task, and Top Source Media website we could write some code ourselves to solve the problem, but fortunately, there exists a library that can handle all the burden for us: ConfigurationSpace. More specifically, ConfigurationSpace can handle conditional configurations. We won’t use this feature here, but it’s very helpful in many situations. As always in Data Science, the next question now is: what model should we use to build a trustable estimator? Initially, model-based optimization used Gaussian processes to estimate configuration score, but recent papers show that tree-based models are a good option.



The mean reason for dropping Gaussian processes is that they don’t support categorical features. As stated a few line above, when speaking of tree-based models, the immediate (but probably not always the best) answer is XGBoost! Why using XGBoost? Not only because XGBoost and gradient boosting methods are very efficient and amongst the most frequent winners of Kaggle contests, but also because they are very versatile and do not need much preprocessing: features normalization is not required, missing values can be handled automatically, … To be honest, it’s also funny to use XGBoost for optimising XGBoost. However, we will also consider another option, that is very similar to XGBoost, but has the noticeable advantage of supporting categories natively: CatBoost. Handling categorical features is a very handy feature, as many model parameters are categorical. LightGBM would also be a perfect fit. We are now in possession of all the elements required to create our own Hyper Parameters Optimization engine.