By Viraj Nayak
This packet contains an overview of different hyperparameter tuning methods in the Scikit-Optimize and Hyperopt libraries.
This project contains an overview of different Sequential model-based optimization methods in the Scikit-Optimize and Hyperopt libraries.
Hyperparameters are parameters that are defined before training and are not directly learned by the learning algorithm. Hyperparameters constrain the algorithm by controlling how flexible the model is.
Hyperparameter Tuning is the method of choosing the best combination of hyperparameters for a given dataset.
Different types of hyperparameter tuning methods are:
Manual search is an example of informed search as we choose parameters based on the previous set of hyperparameters and model performance. Whereas, Grid and Random Search are examples of uninformed search where the next set of hyperparameters are chosen at random from the search space.
Sequential search techniques pick a few hyperparameter settings, evaluate their quality, then decide where to sample next. In other words, SMBO looks back at the result of the last runs to focus future searches on areas that look more promising.
The optimization process has three parts:
SMBO methods are a formalization of Bayesian optimization that update the probability model sequentially. Every evaluation of the objective function with a set of values updates the model with the idea that eventually the model will come to represent the true objective function.
Scikit-Optimize is a simple and efficient library to minimize expensive and noisy black-box functions. It implements several methods for sequential model-based optimization.
The scikit-optimize models discussed in the project are:
The hyperopt models discussed in the project are: