Keras Tuner for Hyperparameter Optimization

Keras Tuner is a scalable and user-friendly framework designed to automate the hyperparameter optimization process for deep learning models built using Keras and TensorFlow. Hyperparameter tuning is important for optimizing model performance, but manual tuning is often tedious, time-consuming and sub-optimal. Keras Tuner addresses these challenges by providing built-in search algorithms and tools to efficiently search for the best hyperparameters within a defined space, enabling improved model accuracy and generalization.It is an open-source library that simplifies hyperparameter search by allowing users to define a search space for their model hyperparameters and then automatically exploring different configurations to find the optimal combination.

Key Features of Keras Tuner

Easy integration with Keras and TensorFlow: Designed specifically for Keras models, it seamlessly fits into existing workflows.

Multiple search algorithms: Includes popular and efficient algorithms such as:

• Random Search: Samples hyperparameters randomly.
• Bayesian Optimization: Uses probabilistic models to choose promising hyperparameters based on previous trials.
• Hyperband: An optimized bandit algorithm that allocates resources efficiently by early-stopping poorly performing trials.

Define-by-run syntax: Allows dynamic and flexible definition of the hyperparameter search space in a single function.

Extensible design: Researchers can extend the tuner to implement custom algorithms.

Support for Scikit-Learn models: Extends usability beyond deep learning models.

How Keras Tuner Works

At its core, Keras Tuner requires the user to provide a model-building function, often called a hypermodel builder or simply a model builder. This function accepts a HyperParameters object (hp) that specifies the hyperparameter search space and builds the model accordingly.

Inside this function, you define ranges or choices for hyperparameters such as number of layers, units per layer, learning rates, dropout rates, activation functions, etc., using methods like:

• hp.Int() for integer ranges (e.g., number of filters, units)
• hp.Float() for floating values (e.g., learning rate)
• hp.Choice() for discrete choices (e.g., activation functions)
• hp.Boolean() for binary options (e.g., whether to use dropout)

Once the hypermodel is defined, you instantiate a Tuner object corresponding to the desired algorithm (RandomSearch, BayesianOptimization or Hyperband). You specify the objective metric to optimize (like validation loss or accuracy), maximum trials and other configurations.

Call the tuner’s .search() method, passing training data and optionally validation data and callbacks. The tuner performs multiple trials, training models with different hyperparameters, evaluates them and keeps track of the best performers.

After search concludes, you can retrieve the best hyperparameters and build the optimized model to train or evaluate further.

Implementation

Step 1 : Import libraries and define the model building function

• This function defines the model, using hyperparameters units and learning_rate that Keras Tuner will try different values for.

import keras
import keras_tuner

def build_model(hp):
    model = keras.Sequential()
    model.add(keras.layers.Dense(
        units=hp.Int('units', min_value=8, max_value=128, step=8),
        activation='relu'))
    model.add(keras.layers.Dense(1, activation='linear'))
    model.compile(
        optimizer=keras.optimizers.Adam(
            hp.Float('learning_rate', 1e-4, 1e-2, sampling='log')),
        loss='mse',
        metrics=['mae'])
    return model

Step 2 : Instantiate the RandomSearch tuner

This creates a tuner that will try up to 10 different hyperparameter combinations.

• It looks at the validation loss (val_loss) to select the best hyperparameters.
• Results will be stored in my_dir/keras_tuner_demo.

tuner = keras_tuner.RandomSearch(
    build_model,
    objective='val_loss',
    max_trials=10,
    executions_per_trial=1,
    directory='my_dir',
    project_name='keras_tuner_demo')

Step 3 : Prepare your training data

• Make sure your x_train and y_train variables are defined with your training inputs and targets as NumPy arrays or compatible structures.

Step 4 :Run the search for the best hyperparameters

• This starts training models with different hyperparameters for 10 epochs each.
• 20% of your training data is used as validation to compute val_loss.

tuner.search(x_train, y_train, epochs=10, validation_split=0.2)

Output

Step 5 :Retrieve the best hyperparameter

• After the search completes, this code gets the best hyperparameter set found.

best_hp = tuner.get_best_hyperparameters(num_trials=1)[0]
print(f"Best units: {best_hp.get('units')}")
print(f"Best learning rate: {best_hp.get('learning_rate')}")

Output

Best units: 32

Best learning rate: 0.00046073971733178836

Step 6 : Build and train the optimized model with these hyperparameters:

• Use the best hyperparameters to build a new model.
• Train it for 20 epochs on your full training data.

model = tuner.hypermodel.build(best_hp)
model.fit(x_train, y_train, epochs=20, validation_split=0.2)

Output

Google Colab link : Keras Tuner for Hyperparameter Optimisation

Search Algorithms in Keras Tuner

• Random Search: Samples hyperparameters randomly from the space; simple but potentially inefficient.
• Bayesian Optimization: Uses past evaluation results to guide the search toward promising regions. It uses probabilistic models (like Gaussian processes) to choose next parameters, aiming for efficiency.
• Hyperband: Uses early stopping and adaptive resource allocation to speed up the search by pruning bad trials early, effectively balancing exploration and exploitation.

Advantages of Using Keras Tuner

• Efficiency: Automates the repetitive trial-and-error hyperparameter tuning process.
• Scalability: Can handle complex models and search spaces, with support for running multiple trials in parallel.
• Flexibility: Can customize search space; extendable to advanced use cases.
• Integration: Works well with other TensorFlow ecosystem tools like callbacks and TensorBoard for monitoring.

Common Use Cases

• Optimizing deep neural network architecture: number of layers, units per layer.
• Finding best regularization parameters: dropout rates, batch normalization usage.
• Tuning training parameters: learning rates, optimizers, batch size.
• Model selection and architecture search for convolutional, recurrent or transformer models.
• Extending hyperparameter tuning to scikit-learn models (to some extent).

Tips for Effective Hyperparameter Tuning with Keras Tuner

• Define a meaningful and reasonable search space to reduce needless trials.
• Use EarlyStopping callback to prevent overfitting and to reduce training time during search.
• Adjust max_trials and executions_per_trial based on available computational budget.
• Consider Hyperband for large search spaces and limited time due to its resource efficiency.
• Monitor tuning progress using TensorBoard by specifying a directory for logs.

Limitations of Keras Tuner for hyperparameter tuning

• Limited to Keras/TensorFlow ecosystem: Primarily designed for Keras models; less support or integration for other frameworks (though there is partial support for Scikit-learn).
• Search space definition complexity: Users must manually define hyperparameter search spaces using the HyperParameters API, which can be error-prone or require deep understanding.
• Computationally expensive: Hyperparameter tuning involves training multiple models, which can be time-consuming and resource-heavy especially for large models/datasets.
• Basic algorithmic options: While it offers Random Search, Bayesian Optimization and Hyperband, more advanced or customized optimization algorithms are not built-in.
• Scalability constraints: Keras Tuner by itself does not natively support distributed or multi-node tuning; scaling large hyperparameter searches requires manual orchestration or external tools.