Hyperparameter Tuning with Scikit-Optimize and Hyperopt in Python

By Viraj Nayak

This packet contains an overview of different hyperparameter tuning methods in the Scikit-Optimize and Hyperopt libraries.

This project contains an overview of different Sequential model-based optimization methods in the Scikit-Optimize and Hyperopt libraries.

Hyperparameters are parameters that are defined before training and are not directly learned by the learning algorithm. Hyperparameters constrain the algorithm by controlling how flexible the model is.

Hyperparameter Tuning is the method of choosing the best combination of hyperparameters for a given dataset.

Different types of hyperparameter tuning methods are:

1. Manual Search: Essentially, we input the hyperparameters based on our judgment and update them based on model performance. The process repeats till desired performance is achieved.
2. Grid Search: We define a hyperparameter search space, and models are trained on all the points randomly sampled in that space.
3. Random Search:  We define a hyperparameter search space, and models are trained on randomly sampled points in that space.
4. Bayesian Optimization: Bayesian optimization is a sequential strategy for global optimization of black-box functions, which does not assume any functional forms.

Manual search is an example of informed search as we choose parameters based on the previous set of hyperparameters and model performance. Whereas, Grid and Random Search are examples of uninformed search where the next set of hyperparameters are chosen at random from the search space.

Sequential Model-Based Optimization (SMBO):

Sequential search techniques pick a few hyperparameter settings, evaluate their quality, then decide where to sample next. In other words, SMBO looks back at the result of the last runs to focus future searches on areas that look more promising.

The optimization process has three parts:

1. Objective function: The function that we want to minimize.
2. Search/Hyperparameter Space
3. Hyperparameter optimization function: constructs surrogate function and chooses next set to evaluate. 

SMBO methods are a formalization of Bayesian optimization that update the probability model sequentially. Every evaluation of the objective function with a set of values updates the model with the idea that eventually the model will come to represent the true objective function.

Scikit-Optimize:

Scikit-Optimize is a simple and efficient library to minimize expensive and noisy black-box functions. It implements several methods for sequential model-based optimization. 

The scikit-optimize models discussed in the project are:

1. Randomized Search
2. Bayesian Search with Gaussian Process
3. Bayesian Search with Random Forests
4. Bayesian Search with Gradient Boosting Machine (GBM)
5. BayeSearchCV: Scikit-learn hyperparameter search wrapper

Installing Scikit-Optimize: 

1. Via pip: pip install scikit-optimize
2. Via conda: conda install -c conda-forge scikit-optimize

Hyperopt:

The hyperopt models discussed in the project are:

1. Randomized Search
2. Anneal
3. Tree of Parzen Estimators (TPE)

Installing Hyperopt: 

1. pip install hyperopt