documentation/tutorials/automatic_tuning_colab.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z2adOh0OtVRg"
      },
      "source": [
        "##### Copyright 2023 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "cDKTycmGtZ84"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-60qQWZytc5P"
      },
      "source": [
        "# Automated hyper-parameter tuning\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/decision_forests/tutorials/automatic_tuning_colab\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/decision-forests/blob/main/documentation/tutorials/automatic_tuning_colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/decision-forests/blob/main/documentation/tutorials/automatic_tuning_colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/decision-forests/documentation/tutorials/automatic_tuning_colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6v3R0N_TxCru"
      },
      "source": [
        "Welcome to the **Automated hyper-parameter tuning** tutorial. In this colab, you\n",
        "will learn how to improve your models using automated hyper-parameter tuning\n",
        "with TensorFlow Decision Forests.\n",
        "\n",
        "More precicely we will:\n",
        "\n",
        "1.  Train a model **without hyper-parameter tuning**. This model will be used to\n",
        "    measure the quality improvement of hyper-parameter tuning.\n",
        "1.  Train a model **with hyper-parameter tuning** using **TF-DF's tuner**. The\n",
        "    hyper-parameters to optimize will be **defined manually**.\n",
        "1.  Train another model **with hyper-parameter tuning** using **TF-DF's tuner**.\n",
        "    But this time, the hyper-parameters to optimize will be **set\n",
        "    automatically**. **This is the recommanded first approach to try when using\n",
        "    hyper-parameter tuning**.\n",
        "1.  Finally, we will train a model **with hyper-parameter tuning** using\n",
        "    **Keras's tuner**.\n",
        "\n",
        "## Introduction\n",
        "\n",
        "A learning algorithm trains a machine learning model on a training dataset. The\n",
        "parameters of a learning algorithm\u0026ndash;called \"hyper-parameters\"\u0026ndash;control\n",
        "how the model is trained and impact its quality. Therefore, finding the best\n",
        "hyper-parameters is an important stage of modeling.\n",
        "\n",
        "Some hyper-parameters are simple to configure. For example, increasing the\n",
        "number of trees (`num_trees`) in a random forest increases the quality of the\n",
        "model until a plateau. Therefore, setting the largest value compatible with the\n",
        "serving constraints (more trees means a larger model) is a valid rule of thumb.\n",
        "However, other hyper-parameters have a more complex interaction with the model\n",
        "and cannot be chosen with such a simple rule. For example, increasing the\n",
        "maximum tree depth (`max_depth`) of a gradient boosted tree model can both\n",
        "increase or decrease the quality of the model. Furthermore, hyper-parameters can\n",
        "interact between each others, and the optimal value of a hyper-parameter cannot\n",
        "be found in isolation.\n",
        "\n",
        "There are three main approaches to select the hyper-parameter values:\n",
        "\n",
        "1.  **The default approach**: Learning algorithms come with default values.\n",
        "    While not ideal in all cases, those values produce reasonable results in\n",
        "    most situations. This approach is recommended as the first approach to use\n",
        "    in any modeling.\n",
        "    [This page](https://ydf.readthedocs.io/en/latest/hyper_parameters.html)\n",
        "    lists the default values of TF Decision Forests.\n",
        "\n",
        "2.  **The template hyper-parameter approach**: In addition to the default\n",
        "    values, TF Decision Forests also exposes the hyper-parameter templates.\n",
        "    Those are benchmark-tuned hyper-parameter values with excellent performance\n",
        "    but high training cost (e.g. `hyperparameter_template=\"benchmark_rank1\"`).\n",
        "\n",
        "3.  **The manual tuning approach:** You can manually test different\n",
        "    hyper-parameter values and select the one that performs best.\n",
        "    [This guide](https://ydf.readthedocs.io/en/latest/improve_model.html)\n",
        "    give some advice.\n",
        "\n",
        "4.  **The automated tuning approach:** A tuning algorithm can be used to find\n",
        "    automatically the best hyper-parameter values. This approach gives often the\n",
        "    best results and does not require expertise. The main downside of this\n",
        "    approach is the time it takes for large datasets.\n",
        "\n",
        "In this colab, we shows the **default** and **automated tuning** approaches with\n",
        "the TensorFlow Decision Forests library.\n",
        "\n",
        "## Hyper-parameter tuning algorithms\n",
        "\n",
        "Automated tuning algorithms work by generating and evaluating a large number of\n",
        "hyper-parameter values. Each of those iterations is called a \"trial\". The\n",
        "evaluation of a trial is expensive as it requires to train a new model each\n",
        "time. At the end of the tuning, the hyper-parameter with the best evaluation is\n",
        "used.\n",
        "\n",
        "Tuning algorithm are configured as follow:\n",
        "\n",
        "**The search space**\n",
        "\n",
        "The search space is the list of hyper-parameters to optimize and the values they\n",
        "can take. For example, the maximum depth of a tree could be optimized for values\n",
        "in between 1 and 32. Exploring more hyper-parameters and more possible values\n",
        "often leads to better models but also takes more time. The hyper-parameters\n",
        "hyper-parameters are listed in\n",
        "the\n",
        "[documentation](https://ydf.readthedocs.io/en/latest/hyper_parameters.html).\n",
        "\n",
        "When the possible value of one hyper-parameter depends on the value of another\n",
        "hyper-parameter, the search space is said to be conditional.\n",
        "\n",
        "**The number of trials**\n",
        "\n",
        "The number of trials defines how many models will be trained and evaluated.\n",
        "Larger number of trials generally leads to better models, but takes more time.\n",
        "\n",
        "**The optimizer**\n",
        "\n",
        "The optimizer selects the next hyper-parameter to evaluate the past trial\n",
        "evaluations. The simplest and often reasonable optimizer is the one that selects\n",
        "the hyper-parameter at random.\n",
        "\n",
        "**The objective / trial score**\n",
        "\n",
        "The objective is the metric optimized by the tuner. Often, this metric is a\n",
        "measure of quality (e.g. accuracy, log loss) of the model evaluated on a\n",
        "validation dataset.\n",
        "\n",
        "*Train-valid-test*\n",
        "\n",
        "The validation dataset should be different from the training datasets: If the\n",
        "training and validation datasets are the same, the selected hyper-parameters\n",
        "will be irrelevant. The validation dataset should also be different from the\n",
        "testing dataset (also called holdout dataset): Because hyper-parameter tuning is\n",
        "a form of training, if the testing and validation datasets are the same, you are\n",
        "effectively training on the test dataset. In this case, you might overfit on\n",
        "your test dataset without a way to measure it.\n",
        "\n",
        "*Cross-validation*\n",
        "\n",
        "In the case of a small dataset, for example a dataset with less than 100k\n",
        "examples, hyper-parameter tuning can be coupled with\n",
        "[cross-validation](https://en.wikipedia.org/wiki/Cross-validation_\\(statistics\\)):\n",
        "Instead of being evaluated from a single training-test round, the\n",
        "objective/trial score is evaluated as the average of the metric over multiple\n",
        "cross-validation rounds.\n",
        "\n",
        "Similarly as to the train-valid-and-test datasets, the cross-validation used to\n",
        "evaluate the objective/score during hyper-parameter tuning should be different\n",
        "from the cross-validation used to evaluate the quality of the model.\n",
        "\n",
        "*Out-of-bag evaluation*\n",
        "\n",
        "Some models, like Random Forests, can be evaluated on the training datasets\n",
        "using the \"out-of-bag evaluation\" method. While not as accurate as\n",
        "cross-validation, the \"out-of-bag evaluation\" is much faster than\n",
        "cross-validation and does not require a separate validation datasets.\n",
        "\n",
        "*In tensorflow decision forests*\n",
        "\n",
        "In TF-DF, the model\n",
        "[\"self\" evaluation](https://www.tensorflow.org/decision_forests/tutorials/beginner_colab#model_self_evaluation)\n",
        "is always a fair way to evaluate a model. For example, an out-of-bag evaluation\n",
        "is used for Random Forest models while a validation dataset is used for Gradient\n",
        "Boosted models.\n",
        "\n",
        "## Hyper-parameter tuning with TF Decision Forests\n",
        "\n",
        "TF-DF supports automatic hyper-parameter tuning with minimal configuration. In\n",
        "the next example, we will train and compare two models: One trained with default\n",
        "hyper-parameters, and one trained with hyper-parameter tuning.\n",
        "\n",
        "**Note:** Hyper-parameters tuning can take a long time in the case of large\n",
        "datasets. In this case, it is recommended to use TF-DF with distributed training\n",
        "to drastically speed-up the hyper-parameter tuning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ehdDZCgWti0H"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OeWh16r3sipp"
      },
      "outputs": [],
      "source": [
        "# Install TensorFlow Dececision Forests\n",
        "!pip install tensorflow_decision_forests -U -qq"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BQRnwsJDtm9v"
      },
      "source": [
        "Install [Wurlitzer](https://pypi.org/project/wurlitzer/). Wurlitzer is required to show the detailed training logs in colabs (with `verbose=2`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lh0cM23rtlwm"
      },
      "outputs": [],
      "source": [
        "!pip install wurlitzer -U -qq"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MMtytrNvtpLk"
      },
      "source": [
        "Import the necessary libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "usa_5pEGtq2o"
      },
      "outputs": [],
      "source": [
        "import tensorflow_decision_forests as tfdf\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1Uwb9Ov0RnF"
      },
      "source": [
        "The hidden code cell limits the output height in colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "SzxVn31c0S5w"
      },
      "outputs": [],
      "source": [
        "#@title Define \"set_cell_height\".\n",
        "\n",
        "from IPython.core.magic import register_line_magic\n",
        "from IPython.display import Javascript\n",
        "from IPython.display import display\n",
        "\n",
        "# Some of the model training logs can cover the full\n",
        "# screen if not compressed to a smaller viewport.\n",
        "# This magic allows setting a max height for a cell.\n",
        "@register_line_magic\n",
        "def set_cell_height(size):\n",
        "  display(\n",
        "      Javascript(\"google.colab.output.setIframeHeight(0, true, {maxHeight: \" +\n",
        "                 str(size) + \"})\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uz1E3qbZt96o"
      },
      "source": [
        "## Training a model **without** Automated hyper-parameter tuning\n",
        "\n",
        "We will train a model on the [Adult](https://archive.ics.uci.edu/ml/datasets/adult) dataset available on the [UCI](https://archive.ics.uci.edu/ml/datasets/adult). Let's download the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WZZEvxJsIJwF"
      },
      "outputs": [],
      "source": [
        "# Download a copy of the adult dataset.\n",
        "!wget -q https://raw.githubusercontent.com/google/yggdrasil-decision-forests/main/yggdrasil_decision_forests/test_data/dataset/adult_train.csv -O /tmp/adult_train.csv\n",
        "!wget -q https://raw.githubusercontent.com/google/yggdrasil-decision-forests/main/yggdrasil_decision_forests/test_data/dataset/adult_test.csv -O /tmp/adult_test.csv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j9iyC3MaRz3k"
      },
      "source": [
        "Split the dataset into a training and a testing dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q428LI7zRUSp"
      },
      "outputs": [],
      "source": [
        "# Load the dataset in memory\n",
        "train_df = pd.read_csv(\"/tmp/adult_train.csv\")\n",
        "test_df = pd.read_csv(\"/tmp/adult_test.csv\")\n",
        "\n",
        "# , and convert it into a TensorFlow dataset.\n",
        "train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_df, label=\"income\")\n",
        "test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(test_df, label=\"income\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6H3r4wehxOrX"
      },
      "source": [
        "First, we train and evaluate the quality of a [Gradient Boosted Trees](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/GradientBoostedTreesModel) model trained with the default hyper-parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8NzRvyQHxZsH"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "# Train a model with default hyper-parameters\n",
        "model = tfdf.keras.GradientBoostedTreesModel()\n",
        "model.fit(train_ds)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VsLHJVgcyT9J"
      },
      "outputs": [],
      "source": [
        "# Evaluate the model\n",
        "model.compile([\"accuracy\"])\n",
        "test_accuracy = model.evaluate(test_ds, return_dict=True, verbose=0)[\"accuracy\"]\n",
        "print(f\"Test accuracy without hyper-parameter tuning: {test_accuracy:.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dm_lJxjkyatR"
      },
      "source": [
        "The default hyper-parameters of the model are available with the `learner_params` function. The definition of those parameters is available in the [documentation](https://ydf.readthedocs.io/en/latest/hyper_parameters.html)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r2UsHw2dyKfZ"
      },
      "outputs": [],
      "source": [
        "print(\"Default hyper-parameters of the model:\\n\", model.learner_params)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eGSv9fBRuKUH"
      },
      "source": [
        "## Training a model **with** automated hyper-parameter tuning and **manual** definition of the hyper-parameters\n",
        "\n",
        "Hyper-parameter tuning is enabled by specifying the `tuner` constructor argument of the model. The tuner object contains all the configuration of the tuner (search space, optimizer, trial and objective).\n",
        "\n",
        "**Note:** *In the next section, you will see how to automtically configure hyper-parameters values. However, setting the hyper-parameters manually as shown here is still valuable to understand.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xf2M0eVGzcwd"
      },
      "outputs": [],
      "source": [
        "# Configure the tuner.\n",
        "\n",
        "# Create a Random Search tuner with 50 trials.\n",
        "tuner = tfdf.tuner.RandomSearch(num_trials=50)\n",
        "\n",
        "# Define the search space.\n",
        "#\n",
        "# Adding more parameters generaly improve the quality of the model, but make\n",
        "# the tuning last longer.\n",
        "\n",
        "tuner.choice(\"min_examples\", [2, 5, 7, 10])\n",
        "tuner.choice(\"categorical_algorithm\", [\"CART\", \"RANDOM\"])\n",
        "\n",
        "# Some hyper-parameters are only valid for specific values of other\n",
        "# hyper-parameters. For example, the \"max_depth\" parameter is mostly useful when\n",
        "# \"growing_strategy=LOCAL\" while \"max_num_nodes\" is better suited when\n",
        "# \"growing_strategy=BEST_FIRST_GLOBAL\".\n",
        "\n",
        "local_search_space = tuner.choice(\"growing_strategy\", [\"LOCAL\"])\n",
        "local_search_space.choice(\"max_depth\", [3, 4, 5, 6, 8])\n",
        "\n",
        "# merge=True indicates that the parameter (here \"growing_strategy\") is already\n",
        "# defined, and that new values are added to it.\n",
        "global_search_space = tuner.choice(\"growing_strategy\", [\"BEST_FIRST_GLOBAL\"], merge=True)\n",
        "global_search_space.choice(\"max_num_nodes\", [16, 32, 64, 128, 256])\n",
        "\n",
        "tuner.choice(\"use_hessian_gain\", [True, False])\n",
        "tuner.choice(\"shrinkage\", [0.02, 0.05, 0.10, 0.15])\n",
        "tuner.choice(\"num_candidate_attributes_ratio\", [0.2, 0.5, 0.9, 1.0])\n",
        "\n",
        "# Uncomment some (or all) of the following hyper-parameters to increase the\n",
        "# quality of the search. The number of trial should be increased accordingly.\n",
        "\n",
        "# tuner.choice(\"split_axis\", [\"AXIS_ALIGNED\"])\n",
        "# oblique_space = tuner.choice(\"split_axis\", [\"SPARSE_OBLIQUE\"], merge=True)\n",
        "# oblique_space.choice(\"sparse_oblique_normalization\",\n",
        "#                      [\"NONE\", \"STANDARD_DEVIATION\", \"MIN_MAX\"])\n",
        "# oblique_space.choice(\"sparse_oblique_weights\", [\"BINARY\", \"CONTINUOUS\"])\n",
        "# oblique_space.choice(\"sparse_oblique_num_projections_exponent\", [1.0, 1.5])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f708dxUwFqRk"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "%set_cell_height 300\n",
        "\n",
        "# Tune the model. Notice the `tuner=tuner`.\n",
        "tuned_model = tfdf.keras.GradientBoostedTreesModel(tuner=tuner)\n",
        "tuned_model.fit(train_ds, verbose=2)\n",
        "\n",
        "# The `num_threads` model constructor argument (not specified in the example\n",
        "# above) controls how many trials are run in parallel (one per thread). If\n",
        "# `num_threads` is not specified (like in the example above), one thread is\n",
        "# allocated for each available CPU core.\n",
        "#\n",
        "# If the training is interrupted (for example, by pressing on the \"stop\" button\n",
        "# on the top-left of the colab cell), the best model so-far will be returned.\n",
        "\n",
        "# In the training logs, you can see lines such as `[10/50] Score: -0.45 / -0.40\n",
        "# HParams: ...`. This indicates that 10 of the 50 trials have been completed.\n",
        "# And that the last trial returned a score of \"-0.45\" and that the best trial so\n",
        "# far has a score of \"-0.40\". In this example, the model is optimized by\n",
        "# logloss. Since scores are maximized and log loss should be minimized, the\n",
        "# score is effectively minus the log loss."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mMfJS2FWz6tt"
      },
      "outputs": [],
      "source": [
        "# Evaluate the model\n",
        "tuned_model.compile([\"accuracy\"])\n",
        "tuned_test_accuracy = tuned_model.evaluate(test_ds, return_dict=True, verbose=0)[\"accuracy\"]\n",
        "print(f\"Test accuracy with the TF-DF hyper-parameter tuner: {tuned_test_accuracy:.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FiQFjV_ab2YE"
      },
      "source": [
        "The hyper-parameters and objective scores of the trials are available in the model inspector. The `score` value is always maximized. In this example, the score is the negative log loss on the validation dataset (selected automatically)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFn3AEykrlQ2"
      },
      "outputs": [],
      "source": [
        "# Display the tuning logs.\n",
        "tuning_logs = tuned_model.make_inspector().tuning_logs()\n",
        "tuning_logs.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JXLXTBv9cOiG"
      },
      "source": [
        " The single rows with `best=True` is the one used in the final model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "En6HzmTYEaRl"
      },
      "outputs": [],
      "source": [
        "# Best hyper-parameters.\n",
        "tuning_logs[tuning_logs.best].iloc[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DA-fcmgBcU__"
      },
      "source": [
        "**Note:** Parameters with NaN value are conditional parameters that are not set.\n",
        "\n",
        "Next, we plot the evaluation of the best score during the tuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VVrJZ5qerzTF"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(10, 5))\n",
        "plt.plot(tuning_logs[\"score\"], label=\"current trial\")\n",
        "plt.plot(tuning_logs[\"score\"].cummax(), label=\"best trial\")\n",
        "plt.xlabel(\"Tuning step\")\n",
        "plt.ylabel(\"Tuning score\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CLQggOs1Yr_y"
      },
      "source": [
        "## Training a model **with** automated hyper-parameter tuning and **automatic** definition of the hyper-parameters (*recommended approach*)\n",
        "\n",
        "As before, hyper-parameter tuning is enabled by specifying the `tuner` constructor argument of the model. Set `use_predefined_hps=True` to automatically configure the search space for the hyper-parameters.\n",
        "\n",
        "**Note:** The automatic hyper-parameter configuration explores some powerful but slow to train hyper-parameters. For example, oblique splits (commented/disabled in the previous section; see `SPARSE_OBLIQUE`) are tested. This mean the tuning will be slower, but hopefully give a result of significantly higher quality."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8nfzv5siYrJ7"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "%set_cell_height 300\n",
        "\n",
        "# Create a Random Search tuner with 50 trials and automatic hp configuration.\n",
        "tuner = tfdf.tuner.RandomSearch(num_trials=50, use_predefined_hps=True)\n",
        "\n",
        "# Define and train the model.\n",
        "tuned_model = tfdf.keras.GradientBoostedTreesModel(tuner=tuner)\n",
        "tuned_model.fit(train_ds, verbose=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U0g8e3inf8Ef"
      },
      "outputs": [],
      "source": [
        "# Evaluate the model\n",
        "tuned_model.compile([\"accuracy\"])\n",
        "tuned_test_accuracy = tuned_model.evaluate(test_ds, return_dict=True, verbose=0)[\"accuracy\"]\n",
        "print(f\"Test accuracy with the TF-DF hyper-parameter tuner: {tuned_test_accuracy:.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PgdSseACZD4x"
      },
      "source": [
        "Same as before, display the tuning logs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GFGQvXxZZAiU"
      },
      "outputs": [],
      "source": [
        "# Display the tuning logs.\n",
        "tuning_logs = tuned_model.make_inspector().tuning_logs()\n",
        "tuning_logs.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "80k6nTULZKEb"
      },
      "source": [
        "Same as before, shows the best hyper-parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n5HN4_n9ZA1l"
      },
      "outputs": [],
      "source": [
        "# Best hyper-parameters.\n",
        "tuning_logs[tuning_logs.best].iloc[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P21MkKxgZQjm"
      },
      "source": [
        "Finally, plots the evolution of the quality of the model during tuning:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1DPZjHrKZVNO"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(10, 5))\n",
        "plt.plot(tuning_logs[\"score\"], label=\"current trial\")\n",
        "plt.plot(tuning_logs[\"score\"].cummax(), label=\"best trial\")\n",
        "plt.xlabel(\"Tuning step\")\n",
        "plt.ylabel(\"Tuning score\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N6U8DwBddCT8"
      },
      "source": [
        "## Training a model with Keras Tuner *(Alternative approach)* \n",
        "\n",
        "TensorFlow Decision Forests is based on the Keras framework, and it is compatible with the [Keras tuner](https://keras.io/keras_tuner/#quick-introduction).\n",
        "\n",
        "Currently, the **TF-DF Tuner** and the **Keras Tuner** are complementary.\n",
        "\n",
        "*TF-DF Tuner*\n",
        "\n",
        "- Automatic configuration of the objective.\n",
        "- Automatic extraction of validation dataset (if needed).\n",
        "- Support model self evaluation (e.g. out-of-bag evaluation).\n",
        "- Distributed hyper-parameter tuning.\n",
        "- Shared dataset access in between the trials: The tensorflow dataset is read only once, speeding-up tuning significantly on small datasets.\n",
        "\n",
        "*Keras Tuner*\n",
        "\n",
        "- Support tuning of the pre-processing parameters.\n",
        "- Support hyper-band optimizer.\n",
        "- Support custom objectives.\n",
        "\n",
        "Let's tune a TF-DF model using the Keras tuner."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rjKRFlGSBJ-O"
      },
      "outputs": [],
      "source": [
        "# Install the Keras tuner\n",
        "!pip install keras-tuner -U -qq\n",
        "import keras_tuner as kt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8crUJKg_CDFG"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "\n",
        "def build_model(hp):\n",
        "  \"\"\"Creates a model.\"\"\"\n",
        "\n",
        "  model = tfdf.keras.GradientBoostedTreesModel(\n",
        "      min_examples=hp.Choice(\"min_examples\", [2, 5, 7, 10]),\n",
        "      categorical_algorithm=hp.Choice(\"categorical_algorithm\", [\"CART\", \"RANDOM\"]),\n",
        "      max_depth=hp.Choice(\"max_depth\", [4, 5, 6, 7]),\n",
        "      # The keras tuner convert automaticall boolean parameters to integers.\n",
        "      use_hessian_gain=bool(hp.Choice(\"use_hessian_gain\", [True, False])),\n",
        "      shrinkage=hp.Choice(\"shrinkage\", [0.02, 0.05, 0.10, 0.15]),\n",
        "      num_candidate_attributes_ratio=hp.Choice(\"num_candidate_attributes_ratio\", [0.2, 0.5, 0.9, 1.0]),\n",
        "  )\n",
        "\n",
        "  # Optimize the model accuracy as computed on the validation dataset.\n",
        "  model.compile(metrics=[\"accuracy\"])\n",
        "  return model\n",
        "\n",
        "keras_tuner = kt.RandomSearch(\n",
        "    build_model,\n",
        "    objective=\"val_accuracy\",\n",
        "    max_trials=50,\n",
        "    overwrite=True,\n",
        "    directory=\"/tmp/keras_tuning\")\n",
        "\n",
        "# Important: The tuning should not be done on the test dataset.\n",
        "\n",
        "# Extract a validation dataset from the training dataset. The new training\n",
        "# dataset is called the \"sub-training-dataset\".\n",
        "\n",
        "def split_dataset(dataset, test_ratio=0.30):\n",
        "  \"\"\"Splits a panda dataframe in two.\"\"\"\n",
        "  test_indices = np.random.rand(len(dataset)) \u003c test_ratio\n",
        "  return dataset[~test_indices], dataset[test_indices]\n",
        "\n",
        "sub_train_df, sub_valid_df = split_dataset(train_df)\n",
        "sub_train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(sub_train_df, label=\"income\")\n",
        "sub_valid_ds = tfdf.keras.pd_dataframe_to_tf_dataset(sub_valid_df, label=\"income\")\n",
        "\n",
        "# Tune the model\n",
        "keras_tuner.search(sub_train_ds, validation_data=sub_valid_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xmh00lVlm3ko"
      },
      "source": [
        "The best hyper-parameter are available with `get_best_hyperparameters`:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9TVvu3YUm8va"
      },
      "outputs": [],
      "source": [
        "# Tune the model\n",
        "best_hyper_parameters = keras_tuner.get_best_hyperparameters()[0].values\n",
        "print(\"Best hyper-parameters:\", keras_tuner.get_best_hyperparameters()[0].values)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4gHVX50tokH2"
      },
      "source": [
        "The model should be re-trained with the best hyper-parameters:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yO0cDwO2opGt"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 300\n",
        "# Train the model\n",
        "# The keras tuner convert automaticall boolean parameters to integers.\n",
        "best_hyper_parameters[\"use_hessian_gain\"] = bool(best_hyper_parameters[\"use_hessian_gain\"])\n",
        "best_model = tfdf.keras.GradientBoostedTreesModel(**best_hyper_parameters)\n",
        "best_model.fit(train_ds, verbose=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0yDSeZ_ho3rw"
      },
      "source": [
        "We can then evaluate the tuned model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AhI0gHMMo7-O"
      },
      "outputs": [],
      "source": [
        "# Evaluate the model\n",
        "best_model.compile([\"accuracy\"])\n",
        "tuned_test_accuracy = best_model.evaluate(test_ds, return_dict=True, verbose=0)[\"accuracy\"]\n",
        "print(f\"Test accuracy with the Keras Tuner: {tuned_test_accuracy:.4f}\")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Automated hyper-parameter tuning",
      "provenance": [
        {
          "file_id": "1iIftewut1alGR8OZntBKNK5g1xKl2KV2",
          "timestamp": 1652168623502
        }
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}