Feedforward Neural Network

Feedforward Neural Network (FNN) is an artificial neural network in which information flows in one direction, from the input layer through any hidden layers to the output layer, without forming cycles or feedback connections. FNNs are commonly used for classification and regression tasks on fixed-size input data.

For example:

In a credit scoring system, banks use an FNN which analyze users financial profiles such as income, credit history and spending habits to determine their creditworthiness.

Each piece of information flows through the network’s layers where various calculations are made to produce a final score.

Architecture

1. Input Layer: The input layer consists of neurons that receive the input data. Each neuron in the input layer represents a feature of the input data.
2. Hidden Layers: One or more hidden layers are placed between the input and output layers. These layers are responsible for learning the complex patterns in the data. Each neuron in a hidden layer applies a weighted sum of inputs followed by a non-linear activation function.
3. Output Layer: The output layer provides the final output of the network. The number of neurons in this layer corresponds to the number of classes in a classification problem or the number of outputs in a regression problem.

Each connection between neurons in these layers has an associated weight that is adjusted during the training process to minimize the error in predictions.

Working

1. InputData: The input features are provided to the input layer.

For example: X = [x₁, x₂, x₃]

Each value represents a feature of the input data.

2. Calculate Weighted Sum: Each neuron calculates a weighted sum of its inputs along with a bias: z = w₁x₁ + w₂x₂ + w₃x₃ + b

• where: w1, w2, w3 are the weights and b is the bias.

3. Apply Activation Function: The weighted sum is passed through an activation function: a = f(z)

Activation functions introduce non-linearity into the network, allowing it to learn complex relationships in the data. Common activation functions include:

Common activation functions include:

• Sigmoid: σ(x)=11+e−x
• Tanh: tanh(x)=ex−e−xex+e−x
• ReLU: ReLU(x)=max(0,x)

4. Forward Propagation: The output of one layer is passed as input to the next layer.

For a simple network: X -> H₁ -> H₂ -> Y

The final layer produces the model's prediction.

5. Calculate Loss: During training, the predicted output is compared with the actual output using a loss function.

For example: Mean Squared Error (MSE) can be used for regression:

MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2

where:

• y_{i}is the actual value.
• \hat{y}_i is the predicted value.
• n is the number of observations.

6. Backpropagation: The network calculates how much each weight contributed to the prediction error. The gradients of the loss are propagated backward through the network using the chain rule of calculus.

7. Update Weights: An optimization algorithm such as Gradient Descent uses these gradients to update the weights:

w_{\mathrm{new}} = w_{\mathrm{old}} - \eta\frac{\partial L}{\partial w}

where:

• η is the learning rate.
• L is the loss function.
• w is a model weight.

The forward propagation, loss calculation, backpropagation and weight update steps are repeated for multiple batches and epochs until the model learns the patterns in the training data.

8. Evaluate the Model: After training, the model is evaluated on unseen test data to measure how well it performs on new inputs.

Common evaluation metrics include:

• Accuracy: Proportion of correctly classified instances out of the total instances.
• Precision: Proportion of correctly predicted positive instances among all predicted positive instances.
• Recall: Proportion of correctly predicted positive instances among all actual positive instances.
• F1 Score: Harmonic mean of precision and recall.
• Confusion Matrix: Shows the numbers of true positives, true negatives, false positives and false negatives.

Note: During training, a Feedforward Neural Network performs forward propagation followed by backpropagation and weight updates. During prediction, only forward propagation is required.

Example

Suppose we want to predict whether a student will pass an examination based on:

• Study hours
• Attendance

The input is: X = [ x₁ , x₂ ]

Suppose a hidden neuron has weights w₁ , w₂ and bias b. It calculates the weighted sum: z = w₁x₁ + w₂x₂ + b

The ReLU activation is then applied: a = max(0, z)

• The resulting activation is passed to the output layer.

For binary classification, the output layer can use the sigmoid function: \hat{y} = \frac{1}{1 + e^{-z}}

• If the output is 0.85, the model predicts a high probability that the student will pass.

Implementation

This example uses TensorFlow and Keras to build a Feedforward Neural Network for classifying handwritten digits from the MNIST dataset.

Step 1: Load the MNIST Dataset

Load the training and test images along with their corresponding digit labels.

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import SparseCategoricalCrossentropy
from tensorflow.keras.metrics import SparseCategoricalAccuracy

mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()

Step 2: Normalize the Data

Scale the pixel values from 0-255 to approximately 0-1 to make the model training more efficient.

x_train, x_test = x_train / 255.0, x_test / 255.0

Step 3: Build the Model

Create a Sequential model with:

• A Flatten layer to convert each 28 x 28 image into a 1D array.
• A Dense layer with 128 neurons and ReLU activation.
• A Dense output layer with 10 neurons and softmax activation, representing the ten digit classes (0-9).

model = Sequential([
    Flatten(input_shape=(28, 28)),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')
])

Step 4: Compile the Model

Configure the model with:

• Adam optimizer
• Sparse Categorical Crossentropy loss function
• Sparse Categorical Accuracy as the evaluation metric

model.compile(optimizer=Adam(),
              loss=SparseCategoricalCrossentropy(),
              metrics=[SparseCategoricalAccuracy()])

Step 5: Train the Model

Train the network on the MNIST training data for 5 epochs.

model.fit(x_train, y_train, epochs=5)

Step 6: Evaluate the Model

Evaluate the trained model on the test dataset and print its classification accuracy.

test_loss, test_acc = model.evaluate(x_test, y_test)
print(f'\nTest accuracy: {test_acc}')

Output:

Test accuracy: 0.9765999913215637

You can download the complete source code from here.

Applications

• Classification: Spam detection, credit-risk classification and medical diagnosis.
• Regression: Predicting prices, demand or other continuous values.
• Pattern recognition: Classifying fixed-size feature representations.
• Recommendation and scoring: Producing scores from structured input features.

Limitations

• Limited handling of sequential dependencies: Standard FNNs do not maintain a hidden state or recurrence for sequence data.
• Fixed-size input: Conventional FNNs generally require inputs with a consistent dimensionality.
• Parameter growth: Large fully connected layers can require many parameters as the input and hidden-layer sizes increase.
• May be unsuitable for structured data: Specialized architectures such as CNNs or sequence/attention-based models may be better suited to spatial or sequential relationships.