Pushing the docs to dev/ for branch: main, commit 071293d7ded1f7f9bc6c5466d014a000560e167c

Author: sklearn-ci

dev/_downloads/07fcc19ba03226cd3d83d4e40ec44385/auto_examples_python.zip

@@ -11,7 +11,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## FixedThresholdClassifier: Setting the decision threshold of a binary classifier\nAll binary classifiers of scikit-learn use a fixed decision threshold of 0.5 to\nconvert probability estimates (i.e. output of `predict_proba`) into class\npredictions. However, 0.5 is almost never the desired threshold for a given problem.\n:class:`~model_selection.FixedThresholdClassifier` allows to wrap any binary\nclassifier and set a custom decision threshold.\n\n"
+        "## FixedThresholdClassifier: Setting the decision threshold of a binary classifier\nAll binary classifiers of scikit-learn use a fixed decision threshold of 0.5\nto convert probability estimates (i.e. output of `predict_proba`) into class\npredictions. However, 0.5 is almost never the desired threshold for a given\nproblem. :class:`~model_selection.FixedThresholdClassifier` allows wrapping any\nbinary classifier and setting a custom decision threshold.\n\n"
       ]
     },
     {
@@ -22,7 +22,7 @@
       },
       "outputs": [],
       "source": [
-        "from sklearn.datasets import make_classification\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import confusion_matrix\n\nX, y = make_classification(n_samples=1_000, weights=[0.9, 0.1], random_state=0)\nclassifier = LogisticRegression(random_state=0).fit(X, y)\n\nprint(\"confusion matrix:\\n\", confusion_matrix(y, classifier.predict(X)))"
+        "from sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import ConfusionMatrixDisplay\n\n\nX, y = make_classification(n_samples=10_000, weights=[0.9, 0.1], random_state=0)\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n\nclassifier_05 = LogisticRegression(C=1e6, random_state=0).fit(X_train, y_train)\n_ = ConfusionMatrixDisplay.from_estimator(classifier_05, X_test, y_test)"
       ]
     },
     {
@@ -40,14 +40,14 @@
       },
       "outputs": [],
       "source": [
-        "from sklearn.model_selection import FixedThresholdClassifier\n\nwrapped_classifier = FixedThresholdClassifier(classifier, threshold=0.1).fit(X, y)\n\nprint(\"confusion matrix:\\n\", confusion_matrix(y, wrapped_classifier.predict(X)))"
+        "from sklearn.model_selection import FixedThresholdClassifier\n\nclassifier_01 = FixedThresholdClassifier(classifier_05, threshold=0.1)\nclassifier_01.fit(X_train, y_train)\n_ = ConfusionMatrixDisplay.from_estimator(classifier_01, X_test, y_test)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## TunedThresholdClassifierCV: Tuning the decision threshold of a binary classifier\nThe decision threshold of a binary classifier can be tuned to optimize a given\nmetric, using :class:`~model_selection.TunedThresholdClassifierCV`.\n\n"
+        "## TunedThresholdClassifierCV: Tuning the decision threshold of a binary classifier\nThe decision threshold of a binary classifier can be tuned to optimize a\ngiven metric, using :class:`~model_selection.TunedThresholdClassifierCV`.\n\nIt is particularly useful to find the best decision threshold when the model\nis meant to be deployed in a specific application context where we can assign\ndifferent gains or costs for true positives, true negatives, false positives,\nand false negatives.\n\nLet's illustrate this by considering an arbitrary case where:\n\n- each true positive gains 1 unit of profit, e.g. euro, year of life in good\n  health, etc.;\n- true negatives gain or cost nothing;\n- each false negative costs 2;\n- each false positive costs 0.1.\n\nOur metric quantifies the average profit per sample, which is defined by the\nfollowing Python function:\n\n"
       ]
     },
     {
@@ -58,14 +58,14 @@
       },
       "outputs": [],
       "source": [
-        "from sklearn.metrics import balanced_accuracy_score\n\n# Due to the class imbalance, the balanced accuracy is not optimal for the default\n# threshold. The classifier tends to over predict the majority class.\nprint(f\"balanced accuracy: {balanced_accuracy_score(y, classifier.predict(X)):.2f}\")"
+        "from sklearn.metrics import confusion_matrix\n\n\ndef custom_score(y_observed, y_pred):\n    tn, fp, fn, tp = confusion_matrix(y_observed, y_pred, normalize=\"all\").ravel()\n    return tp - 2 * fn - 0.1 * fp\n\n\nprint(\"Untuned decision threshold: 0.5\")\nprint(f\"Custom score: {custom_score(y_test, classifier_05.predict(X_test)):.2f}\")"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Tuning the threshold to optimize the balanced accuracy gives a smaller threshold\nthat allows more samples to be classified as the positive class.\n\n"
+        "It is interesting to observe that the average gain per prediction is negative\nwhich means that this decision system is making a loss on average.\n\nTuning the threshold to optimize this custom metric gives a smaller threshold\nthat allows more samples to be classified as the positive class. As a result,\nthe average gain per prediction improves.\n\n"
       ]
     },
     {
@@ -76,21 +76,21 @@
       },
       "outputs": [],
       "source": [
-        "from sklearn.model_selection import TunedThresholdClassifierCV\n\ntuned_classifier = TunedThresholdClassifierCV(\n    classifier, cv=5, scoring=\"balanced_accuracy\"\n).fit(X, y)\n\nprint(f\"new threshold: {tuned_classifier.best_threshold_:.4f}\")\nprint(\n    f\"balanced accuracy: {balanced_accuracy_score(y, tuned_classifier.predict(X)):.2f}\"\n)"
+        "from sklearn.model_selection import TunedThresholdClassifierCV\nfrom sklearn.metrics import make_scorer\n\ncustom_scorer = make_scorer(\n    custom_score, response_method=\"predict\", greater_is_better=True\n)\ntuned_classifier = TunedThresholdClassifierCV(\n    classifier_05, cv=5, scoring=custom_scorer\n).fit(X, y)\n\nprint(f\"Tuned decision threshold: {tuned_classifier.best_threshold_:.3f}\")\nprint(f\"Custom score: {custom_score(y_test, tuned_classifier.predict(X_test)):.2f}\")"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        ":class:`~model_selection.TunedThresholdClassifierCV` also benefits from the\nmetadata routing support (`Metadata Routing User Guide<metadata_routing>`)\nallowing to optimze complex business metrics, detailed\nin `Post-tuning the decision threshold for cost-sensitive learning\n<sphx_glr_auto_examples_model_selection_plot_cost_sensitive_learning.py>`.\n\n"
+        "We observe that tuning the decision threshold can turn a machine\nlearning-based system that makes a loss on average into a beneficial one.\n\nIn practice, defining a meaningful application-specific metric might involve\nmaking those costs for bad predictions and gains for good predictions depend on\nauxiliary metadata specific to each individual data point such as the amount\nof a transaction in a fraud detection system.\n\nTo achieve this, :class:`~model_selection.TunedThresholdClassifierCV`\nleverages metadata routing support (`Metadata Routing User\nGuide<metadata_routing>`) allowing to optimize complex business metrics as\ndetailed in `Post-tuning the decision threshold for cost-sensitive\nlearning\n<sphx_glr_auto_examples_model_selection_plot_cost_sensitive_learning.py>`.\n\n"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Performance improvements in PCA\n:class:`~decomposition.PCA` has a new solver, \"covariance_eigh\", which is faster\nand more memory efficient than the other solvers for datasets with a large number\nof samples and a small number of features.\n\n"
+        "## Performance improvements in PCA\n:class:`~decomposition.PCA` has a new solver, `\"covariance_eigh\"`, which is\nup to an order of magnitude faster and more memory efficient than the other\nsolvers for datasets with many data points and few features.\n\n"
       ]
     },
     {
@@ -101,14 +101,14 @@
       },
       "outputs": [],
       "source": [
-        "from sklearn.datasets import make_low_rank_matrix\nfrom sklearn.decomposition import PCA\n\nX = make_low_rank_matrix(\n    n_samples=10_000, n_features=100, tail_strength=0.1, random_state=0\n)\n\npca = PCA(n_components=10).fit(X)\n\nprint(f\"explained variance: {pca.explained_variance_ratio_.sum():.2f}\")"
+        "from sklearn.datasets import make_low_rank_matrix\nfrom sklearn.decomposition import PCA\n\nX = make_low_rank_matrix(\n    n_samples=10_000, n_features=100, tail_strength=0.1, random_state=0\n)\n\npca = PCA(n_components=10, svd_solver=\"covariance_eigh\").fit(X)\nprint(f\"Explained variance: {pca.explained_variance_ratio_.sum():.2f}\")"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The \"full\" solver has also been improved to use less memory and allows to\ntransform faster. The \"auto\" option for the solver takes advantage of the\nnew solver and is now able to select an appropriate solver for sparse\ndatasets.\n\n"
+        "The new solver also accepts sparse input data:\n\n"
       ]
     },
     {
@@ -119,7 +119,14 @@
       },
       "outputs": [],
       "source": [
-        "from scipy.sparse import random\n\nX = random(10000, 100, format=\"csr\", random_state=0)\n\npca = PCA(n_components=10, svd_solver=\"auto\").fit(X)"
+        "from scipy.sparse import random\n\nX = random(10_000, 100, format=\"csr\", random_state=0)\n\npca = PCA(n_components=10, svd_solver=\"covariance_eigh\").fit(X)\nprint(f\"Explained variance: {pca.explained_variance_ratio_.sum():.2f}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The `\"full\"` solver has also been improved to use less memory and allows\nfaster transformation. The default `svd_solver=\"auto\"`` option takes\nadvantage of the new solver and is now able to select an appropriate solver\nfor sparse datasets.\n\nSimilarly to most other PCA solvers, the new `\"covariance_eigh\"` solver can leverage\nGPU computation if the input data is passed as a PyTorch or CuPy array by\nenabling the experimental support for `Array API <array_api>`.\n\n"
       ]
     },
     {

dev/_downloads/6f1e7a639e0699d6164445b55e6c116d/auto_examples_jupyter.zip

@@ -24,89 +24,136 @@
 # %%
 # FixedThresholdClassifier: Setting the decision threshold of a binary classifier
 # -------------------------------------------------------------------------------
-# All binary classifiers of scikit-learn use a fixed decision threshold of 0.5 to
-# convert probability estimates (i.e. output of `predict_proba`) into class
-# predictions. However, 0.5 is almost never the desired threshold for a given problem.
-# :class:`~model_selection.FixedThresholdClassifier` allows to wrap any binary
-# classifier and set a custom decision threshold.
+# All binary classifiers of scikit-learn use a fixed decision threshold of 0.5
+# to convert probability estimates (i.e. output of `predict_proba`) into class
+# predictions. However, 0.5 is almost never the desired threshold for a given
+# problem. :class:`~model_selection.FixedThresholdClassifier` allows wrapping any
+# binary classifier and setting a custom decision threshold.
 from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import confusion_matrix
+from sklearn.metrics import ConfusionMatrixDisplay
+
 
-X, y = make_classification(n_samples=1_000, weights=[0.9, 0.1], random_state=0)
-classifier = LogisticRegression(random_state=0).fit(X, y)
+X, y = make_classification(n_samples=10_000, weights=[0.9, 0.1], random_state=0)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
 
-print("confusion matrix:\n", confusion_matrix(y, classifier.predict(X)))
+classifier_05 = LogisticRegression(C=1e6, random_state=0).fit(X_train, y_train)
+_ = ConfusionMatrixDisplay.from_estimator(classifier_05, X_test, y_test)
 
 # %%
 # Lowering the threshold, i.e. allowing more samples to be classified as the positive
 # class, increases the number of true positives at the cost of more false positives
 # (as is well known from the concavity of the ROC curve).
 from sklearn.model_selection import FixedThresholdClassifier
 
-wrapped_classifier = FixedThresholdClassifier(classifier, threshold=0.1).fit(X, y)
-
-print("confusion matrix:\n", confusion_matrix(y, wrapped_classifier.predict(X)))
+classifier_01 = FixedThresholdClassifier(classifier_05, threshold=0.1)
+classifier_01.fit(X_train, y_train)
+_ = ConfusionMatrixDisplay.from_estimator(classifier_01, X_test, y_test)
 
 # %%
 # TunedThresholdClassifierCV: Tuning the decision threshold of a binary classifier
 # --------------------------------------------------------------------------------
-# The decision threshold of a binary classifier can be tuned to optimize a given
-# metric, using :class:`~model_selection.TunedThresholdClassifierCV`.
-from sklearn.metrics import balanced_accuracy_score
+# The decision threshold of a binary classifier can be tuned to optimize a
+# given metric, using :class:`~model_selection.TunedThresholdClassifierCV`.
+#
+# It is particularly useful to find the best decision threshold when the model
+# is meant to be deployed in a specific application context where we can assign
+# different gains or costs for true positives, true negatives, false positives,
+# and false negatives.
+#
+# Let's illustrate this by considering an arbitrary case where:
+#
+# - each true positive gains 1 unit of profit, e.g. euro, year of life in good
+#   health, etc.;
+# - true negatives gain or cost nothing;
+# - each false negative costs 2;
+# - each false positive costs 0.1.
+#
+# Our metric quantifies the average profit per sample, which is defined by the
+# following Python function:
+from sklearn.metrics import confusion_matrix
+
+
+def custom_score(y_observed, y_pred):
+    tn, fp, fn, tp = confusion_matrix(y_observed, y_pred, normalize="all").ravel()
+    return tp - 2 * fn - 0.1 * fp
 
-# Due to the class imbalance, the balanced accuracy is not optimal for the default
-# threshold. The classifier tends to over predict the majority class.
-print(f"balanced accuracy: {balanced_accuracy_score(y, classifier.predict(X)):.2f}")
+
+print("Untuned decision threshold: 0.5")
+print(f"Custom score: {custom_score(y_test, classifier_05.predict(X_test)):.2f}")
 
 # %%
-# Tuning the threshold to optimize the balanced accuracy gives a smaller threshold
-# that allows more samples to be classified as the positive class.
+# It is interesting to observe that the average gain per prediction is negative
+# which means that this decision system is making a loss on average.
+#
+# Tuning the threshold to optimize this custom metric gives a smaller threshold
+# that allows more samples to be classified as the positive class. As a result,
+# the average gain per prediction improves.
 from sklearn.model_selection import TunedThresholdClassifierCV
+from sklearn.metrics import make_scorer
 
+custom_scorer = make_scorer(
+    custom_score, response_method="predict", greater_is_better=True
+)
 tuned_classifier = TunedThresholdClassifierCV(
-    classifier, cv=5, scoring="balanced_accuracy"
+    classifier_05, cv=5, scoring=custom_scorer
 ).fit(X, y)
 
-print(f"new threshold: {tuned_classifier.best_threshold_:.4f}")
-print(
-    f"balanced accuracy: {balanced_accuracy_score(y, tuned_classifier.predict(X)):.2f}"
-)
+print(f"Tuned decision threshold: {tuned_classifier.best_threshold_:.3f}")
+print(f"Custom score: {custom_score(y_test, tuned_classifier.predict(X_test)):.2f}")
 
 # %%
-# :class:`~model_selection.TunedThresholdClassifierCV` also benefits from the
-# metadata routing support (:ref:`Metadata Routing User Guide<metadata_routing>`)
-# allowing to optimze complex business metrics, detailed
-# in :ref:`Post-tuning the decision threshold for cost-sensitive learning
+# We observe that tuning the decision threshold can turn a machine
+# learning-based system that makes a loss on average into a beneficial one.
+#
+# In practice, defining a meaningful application-specific metric might involve
+# making those costs for bad predictions and gains for good predictions depend on
+# auxiliary metadata specific to each individual data point such as the amount
+# of a transaction in a fraud detection system.
+#
+# To achieve this, :class:`~model_selection.TunedThresholdClassifierCV`
+# leverages metadata routing support (:ref:`Metadata Routing User
+# Guide<metadata_routing>`) allowing to optimize complex business metrics as
+# detailed in :ref:`Post-tuning the decision threshold for cost-sensitive
+# learning
 # <sphx_glr_auto_examples_model_selection_plot_cost_sensitive_learning.py>`.
 
 # %%
 # Performance improvements in PCA
 # -------------------------------
-# :class:`~decomposition.PCA` has a new solver, "covariance_eigh", which is faster
-# and more memory efficient than the other solvers for datasets with a large number
-# of samples and a small number of features.
+# :class:`~decomposition.PCA` has a new solver, `"covariance_eigh"`, which is
+# up to an order of magnitude faster and more memory efficient than the other
+# solvers for datasets with many data points and few features.
 from sklearn.datasets import make_low_rank_matrix
 from sklearn.decomposition import PCA
 
 X = make_low_rank_matrix(
     n_samples=10_000, n_features=100, tail_strength=0.1, random_state=0
 )
 
-pca = PCA(n_components=10).fit(X)
+pca = PCA(n_components=10, svd_solver="covariance_eigh").fit(X)
+print(f"Explained variance: {pca.explained_variance_ratio_.sum():.2f}")
 
-print(f"explained variance: {pca.explained_variance_ratio_.sum():.2f}")
 
 # %%
-# The "full" solver has also been improved to use less memory and allows to
-# transform faster. The "auto" option for the solver takes advantage of the
-# new solver and is now able to select an appropriate solver for sparse
-# datasets.
+# The new solver also accepts sparse input data:
 from scipy.sparse import random
 
-X = random(10000, 100, format="csr", random_state=0)
+X = random(10_000, 100, format="csr", random_state=0)
 
-pca = PCA(n_components=10, svd_solver="auto").fit(X)
+pca = PCA(n_components=10, svd_solver="covariance_eigh").fit(X)
+print(f"Explained variance: {pca.explained_variance_ratio_.sum():.2f}")
+
+# %%
+# The `"full"` solver has also been improved to use less memory and allows
+# faster transformation. The default `svd_solver="auto"`` option takes
+# advantage of the new solver and is now able to select an appropriate solver
+# for sparse datasets.
+#
+# Similarly to most other PCA solvers, the new `"covariance_eigh"` solver can leverage
+# GPU computation if the input data is passed as a PyTorch or CuPy array by
+# enabling the experimental support for :ref:`Array API <array_api>`.
 
 # %%
 # ColumnTransformer is subscriptable