Hyperparameter Tuning

One of the most important aspects of machine learning is hyperparameter tuning. Many machine learning models have a number of hyperparameters that control aspects of the model. These hyperparameters typically cannot be learned directly by the same learning algorithm used for the rest of learning and have to be set in an alternate fashion. The dc.hyper module contains utilities for hyperparameter tuning.

DeepChem’s hyperparameter optimzation algorithms are simple and run in single-threaded fashion. They are not intended to be production grade hyperparameter utilities, but rather useful first tools as you start exploring your parameter space. As the needs of your application grow, we recommend swapping to a more heavy duty hyperparameter optimization library.

Hyperparameter Optimization API

class deepchem.hyper.HyperparamOpt(model_builder)[source]

Abstract superclass for hyperparameter search classes.

This class is an abstract base class for hyperparameter search classes in DeepChem. Hyperparameter search is performed on dc.models.Model classes. Each hyperparameter object accepts a dc.models.Model class upon construct. When the hyperparam_search class is invoked, this class is used to construct many different concrete models which are trained on the specified training set and evaluated on a given validation set.

Different subclasses of HyperparamOpt differ in the choice of strategy for searching the hyperparameter evaluation space. This class itself is an abstract superclass and should never be directly instantiated.

__delattr__

Implement delattr(self, name).

__dir__()

Default dir() implementation.

__eq__

Return self==value.

__format__()

Default object formatter.

__ge__

Return self>=value.

__getattribute__

Return getattr(self, name).

__gt__

Return self>value.

__hash__

Return hash(self).

__init__(model_builder)[source]

Initialize Hyperparameter Optimizer.

Note this is an abstract constructor which should only be used by subclasses.

Parameters:model_builder (constructor function.) – This parameter must be constructor function which returns an object which is an instance of dc.models.Model. This function must accept two arguments, model_params of type dict and model_dir, a string specifying a path to a model directory. See the example.
__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__le__

Return self<=value.

__lt__

Return self<value.

__ne__

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__

Return repr(self).

__setattr__

Implement setattr(self, name, value).

__sizeof__()

Size of object in memory, in bytes.

__str__

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

Conduct Hyperparameter search.

This method defines the common API shared by all hyperparameter optimization subclasses. Different classes will implement different search methods but they must all follow this common API.

Parameters:
  • params_dict (dict) – Dictionary mapping strings to values. Note that the precise semantics of params_dict will change depending on the optimizer that you’re using. Depending on the type of hyperparameter optimization, these values can be ints/floats/strings/lists/etc. Read the documentation for the concrete hyperparameter optimization subclass you’re using to learn more about what’s expected.
  • train_dataset (dc.data.Dataset) – dataset used for training
  • valid_dataset (dc.data.Dataset) – dataset used for validation(optimization on valid scores)
  • output_transformers (list[dc.trans.Transformer]) – Transformers for evaluation. This argument is needed since train_dataset and valid_dataset may have been transformed for learning and need the transform to be inverted before the metric can be evaluated on a model.
  • use_max (bool, optional) – If True, return the model with the highest score. Else return model with the minimum score.
  • logdir (str, optional) – The directory in which to store created models. If not set, will use a temporary directory.
Returns:

  • (best_model, best_hyperparams, all_scores) where best_model is
  • an instance of dc.models.Models, best_hyperparams is a
  • dictionary of parameters, and all_scores is a dictionary mapping
  • string representations of hyperparameter sets to validation
  • scores.

Grid Hyperparameter Optimization

This is the simplest form of hyperparameter optimization that simply involves iterating over a fixed grid of possible values for hyperaparameters.

class deepchem.hyper.GridHyperparamOpt(model_builder)[source]

Provides simple grid hyperparameter search capabilities.

This class performs a grid hyperparameter search over the specified hyperparameter space. This implementation is simple and simply does a direct iteration over all possible hyperparameters and doesn’t use parallelization to speed up the search.

Example

This example shows the type of constructor function expected.

>>> import sklearn
>>> import deepchem as dc
>>> optimizer = dc.hyper.GridHyperparamOpt(lambda **p: dc.models.GraphConvModel(**p))

Here’s a more sophisticated example that shows how to optimize only some parameters of a model. In this case, we have some parameters we want to optimize, and others which we don’t. To handle this type of search, we create a model_builder which hard codes some arguments (in this case, n_tasks and n_features which are properties of a dataset and not hyperparameters to search over.)

>>> def model_builder(**model_params):
...   n_layers = model_params['layers']
...   layer_width = model_params['width']
...   dropout = model_params['dropout']
...   return dc.models.MultitaskClassifier(
...     n_tasks=5,
...     n_features=100,
...     layer_sizes=[layer_width]*n_layers,
...     dropouts=dropout
...   )
>>> optimizer = dc.hyper.GridHyperparamOpt(model_builder)
__delattr__

Implement delattr(self, name).

__dir__()

Default dir() implementation.

__eq__

Return self==value.

__format__()

Default object formatter.

__ge__

Return self>=value.

__getattribute__

Return getattr(self, name).

__gt__

Return self>value.

__hash__

Return hash(self).

__init__(model_builder)[source]

Initialize Hyperparameter Optimizer.

Note this is an abstract constructor which should only be used by subclasses.

Parameters:model_builder (constructor function.) – This parameter must be constructor function which returns an object which is an instance of dc.models.Model. This function must accept two arguments, model_params of type dict and model_dir, a string specifying a path to a model directory. See the example.
__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__le__

Return self<=value.

__lt__

Return self<value.

__ne__

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__

Return repr(self).

__setattr__

Implement setattr(self, name, value).

__sizeof__()

Size of object in memory, in bytes.

__str__

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

Perform hyperparams search according to params_dict.

Each key to hyperparams_dict is a model_param. The values should be a list of potential values for that hyperparam.

Parameters:
  • params_dict (Dict[str, list]) – Maps hyperparameter names (strings) to lists of possible parameter values.
  • train_dataset (dc.data.Dataset) – dataset used for training
  • valid_dataset (dc.data.Dataset) – dataset used for validation(optimization on valid scores)
  • output_transformers (list[dc.trans.Transformer]) – transformers for evaluation
  • metric (dc.metrics.Metric) – metric used for evaluation
  • use_max (bool, optional) – If True, return the model with the highest score. Else return model with the minimum score.
  • logdir (str, optional) – The directory in which to store created models. If not set, will use a temporary directory.
Returns:

  • (best_model, best_hyperparams, all_scores) where best_model is
  • an instance of dc.model.Models, best_hyperparams is a
  • dictionary of parameters, and all_scores is a dictionary mapping
  • string representations of hyperparameter sets to validation
  • scores.

Gaussian Process Hyperparameter Optimization

class deepchem.hyper.GaussianProcessHyperparamOpt(model_builder)[source]

Gaussian Process Global Optimization(GPGO)

This class uses Gaussian Process optimization to select hyperparameters. Underneath the hood it uses pyGPGO to optimize models. If you don’t have pyGPGO installed, you won’t be able to use this class.

Note that params_dict has a different semantics than for GridHyperparamOpt. param_dict[hp] must be an int/float and is used as the center of a search range.

Example

This example shows the type of constructor function expected.

>>> import sklearn
>>> import deepchem as dc
>>> optimizer = dc.hyper.GaussianProcessHyperparamOpt(lambda **p: dc.models.GraphConvModel(n_tasks=1, **p))

Here’s a more sophisticated example that shows how to optimize only some parameters of a model. In this case, we have some parameters we want to optimize, and others which we don’t. To handle this type of search, we create a model_builder which hard codes some arguments (in this case, n_tasks and n_features which are properties of a dataset and not hyperparameters to search over.)

>>> def model_builder(**model_params):
...   n_layers = model_params['layers']
...   layer_width = model_params['width']
...   dropout = model_params['dropout']
...   return dc.models.MultitaskClassifier(
...     n_tasks=5,
...     n_features=100,
...     layer_sizes=[layer_width]*n_layers,
...     dropouts=dropout
...   )
>>> optimizer = dc.hyper.GaussianProcessHyperparamOpt(model_builder)
__delattr__

Implement delattr(self, name).

__dir__()

Default dir() implementation.

__eq__

Return self==value.

__format__()

Default object formatter.

__ge__

Return self>=value.

__getattribute__

Return getattr(self, name).

__gt__

Return self>value.

__hash__

Return hash(self).

__init__(model_builder)[source]

Initialize Hyperparameter Optimizer.

Note this is an abstract constructor which should only be used by subclasses.

Parameters:model_builder (constructor function.) – This parameter must be constructor function which returns an object which is an instance of dc.models.Model. This function must accept two arguments, model_params of type dict and model_dir, a string specifying a path to a model directory. See the example.
__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__le__

Return self<=value.

__lt__

Return self<value.

__ne__

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__

Return repr(self).

__setattr__

Implement setattr(self, name, value).

__sizeof__()

Size of object in memory, in bytes.

__str__

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

Perform hyperparameter search using a gaussian process.

Parameters:
  • params_dict (dict) – Maps hyperparameter names (strings) to possible parameter values. The semantics of this list are different than for GridHyperparamOpt. params_dict[hp] must map to an int/float, which is used as the center of a search with radius search_range since pyGPGO can only optimize numerical hyperparameters.
  • train_dataset (dc.data.Dataset) – dataset used for training
  • valid_dataset (dc.data.Dataset) – dataset used for validation(optimization on valid scores)
  • transformers (list[dc.trans.Transformer]) – transformers for evaluation
  • metric (dc.metrics.Metric) – metric used for evaluation
  • use_max (bool, (default True)) – Specifies whether to maximize or minimize metric. maximization(True) or minimization(False)
  • logdir (str, optional) – The directory in which to store created models. If not set, will use a temporary directory.
  • max_iter (int, (default 20)) – number of optimization trials
  • search_range (int(float)/dict (default 4)) –

    The search_range specifies the range of parameter values to search for. If search_range is an int/float, it is used as the global search range for parameters. This creates a search problem on the following space:

    optimization on [initial value / search_range,
    initial value * search_range]

    If search_range is a dict, it must contain the same keys as for params_dict. In this case, search_range specifies a per-parameter search range. This is useful in case some parameters have a larger natural range than others. For a given hyperparameter hp this would create the following search range:

    optimization on hp on [initial value[hp] / search_range[hp],
    initial value[hp] * search_range[hp]]
  • logfile (str) – Name of logfile to write results to. If specified, this is must be a valid file. If not specified, results of hyperparameter search will be written to logdir/.txt.
Returns:

  • (best_model, best_hyperparams, all_scores) where best_model is
  • an instance of dc.model.Models, best_hyperparams is a
  • dictionary of parameters, and all_scores is a dictionary mapping
  • string representations of hyperparameter sets to validation
  • scores.