Partial derivatives in Machine Learning

In machine learning, models have many parameters like weights and biases. The loss function shows how far predictions are from the correct output. To improve the model, we need to know how each parameter affects the loss. Partial derivatives help us update each parameter individually, which is the core idea behind gradient descent. By measuring how the loss changes with respect to each parameter, we adjust them to reduce error.

• Partial Derivatives: A function of many variables is said to have a partial derivative if it is only related to one of the variables, holding the rest constant. For a function f(x_1,x_2,....,x_n) the partial derivative with respect to xi is denoted as ∂x/∂f.
• Gradient: A function's gradient is a vector that, at a given moment in time, indicates the direction of the function's maximum rate of growth. When optimizing a cost function in machine learning, the gradient often indicates the direction of the sharpest rise or decline.
• Gradient Descent: It is an optimization process that moves repeatedly in the direction of the steepest descent, which is indicated by the gradient's negative, in order to minimize a function.

To understand machine learning partial derivatives, let us examine a basic linear regression model:

f(x) = wx+b

Where w is the weight, b is the bias and x is the input variable.

Partial Derivatives: In order to optimise the model, we must calculate the partial derivatives of the cost function J(w,b) with respect to the parameters w and b.

\frac{\partial J}{\partial w} = \frac{1}{m} \sum_{i=1}^{m} (wx_i + b - y_i) \cdot x_i ,where (x_i, y_i) denotes the input-output pairings and m is the number of training samples.

Gradient Descent Update Rule: We use the gradient descent technique repeatedly to update the parameters:

\omega = \omega - \alpha .\frac{\partial J}{\partial w}

b = b - \alpha .\frac{\partial J}{\partial b}

Where:

• w and b are the model parameters (weight and bias).
• \alpha is the learning rate, which controls how big each step is during optimization.
• \frac{\partial J}{\partial w} and \frac{\partial J}{\partial b} are the partial derivatives of the cost (loss) function J(w,b) with respect to w and b. These tell us how much the loss changes when we tweak each parameter a little bit, all else being equal.

Implementation Of Partial derivatives in Machine Learning

• Initialization: Initialize the weight parameter w and bias parameter b to 0 and set the learning rate α and number of epochs.
• Prediction: Compute the model's predictions for house prices using the current values of w and b.
• Gradient Calculation: Compute the gradients of the mean squared error cost function with respect to w and b using partial differentiation.
• Parameter Update: Update the parameters w and b using the computed gradients and the learning rate α.
• Iteration: Repeat the prediction, gradient calculation and parameter update steps for a specified number of epochs to optimize the parameters for the linear model.

import numpy as np

X = np.array([1, 2, 3, 4, 5])
y = np.array([100, 200, 300, 400, 500])

w = 0
b = 0
learning_rate = 0.01
epochs = 100

for epoch in range(epochs):
    predictions = w * X + b

    dw = (1 / len(X)) * np.sum((predictions - y) * X)
    db = (1 / len(X)) * np.sum(predictions - y)

    w -= learning_rate * dw
    b -= learning_rate * db

print("Optimal parameters: \n w =", w, "\n b =", b)

Output:

Optimal parameters:
 w = 93.98340961256555   
 b = 21.720572459273797

Partial Derivative Examples

Example: Find the partial derivatives of the function: f(x,y) = x²y +3xy²

Solution:

1. Partial derivative with respect to x

Treat y as a constant, f(x,y) = x²y +3xy²

Differentiate: \frac{\partial f}{\partial x} = 2xy + 3y^2

2. Partial derivative with respect to y

Treat x as a constant, f(x,y) = x²y+ 3xy²

Differentiate: \frac{\partial f}{\partial y} = x^2 + 6xy

Final Answers:

\frac{\partial f}{\partial x} = 2xy + 3y^2

\frac{\partial f}{\partial y} = x^2 + 6xy