Hidden Layer Perceptron in TensorFlow

In this article, we will learn about hidden layer perceptron. A hidden layer perceptron is nothing but a hi-fi terminology for a neural network with one or more hidden layers. The purpose which is being served by these hidden layers is that they help to learn complex and non-linear functions for a task.

The above image is the simplest representation of the hidden layer perceptron with a single hidden layer. Here we can see that the input for the final layer is the neurons of the hidden layers. So, in a hidden layer perceptron network input for the current layer is the output of the previous layer.

We will try to understand how one can implement a Hidden layer perceptron network using TensorFlow. Also, the data used for this purpose is the famous Facial Recognition dataset.

Importing Libraries and Dataset

• Pandas – This library helps to load the data frame in a 2D array format and has multiple functions to perform analysis tasks in one go.
• Numpy – Numpy arrays are very fast and can perform large computations in a very short time.
• Matplotlib – This library is used to draw visualizations.
• Sklearn – This module contains multiple libraries having pre-implemented functions to perform tasks from data preprocessing to model development and evaluation.
• OpenCV – This is an open-source library mainly focused on image processing and handling.
• Tensorflow – This is an open-source library that is used for Machine Learning and Artificial intelligence and provides a range of functions to achieve complex functionalities with single lines of code.

import numpy as np
import pandas as pd
import seaborn as sb
import matplotlib.pyplot as plt

import cv2
from glob import glob
import tensorflow as tf
from tensorflow import keras
from keras import layers

from tqdm.notebook import tqdm, trange
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, StandardScaler

import warnings
warnings.filterwarnings('ignore')

Now let's create a data frame of the image path and the classes from which they belong. Creating a data frame helps us to analyze the distribution of the data across various classes.

images = glob('images/train/*/*.jpg')
len(images)

Output:

28821

df = pd.DataFrame({'image_path': images})
df.head()

Output:

df['label'] = df['image_path'].str.split('/', expand=True)[2]
df.head()

Output:

df.groupby('label').count().plot.bar()
plt.show()

Output:

Data Visualization

Here we can certainly say that this dataset is not balanced but in this article, our main motive is to learn what a hidden layer perceptron is and how can we use it.

plt.subplots(figsize=(10, 6))
emotions = df['label'].unique()
for i, emotion in enumerate(emotions):
    plt.subplot(2, 4, i+1)
    x = df[df['label'] == emotion].image_path
    path = x.values[5]
    img = cv2.imread(path)
    plt.imshow(img)
    plt.title(emotion)
plt.show()

Output:

X, Y = [], []
for path in tqdm(images):
    img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
    X.append(img.flatten())
    Y.append(path.split('/')[2])

le = LabelEncoder()
Y = le.fit_transform(Y)

Now let's convert the image list as a NumPy array and convert the labels as one-hot encoded vectors from the 7 classes.

X = np.asarray(X)
Y = pd.get_dummies(Y).values

X.shape, Y.shape

Output:

((28821, 2304), (28821, 7))

Now to evaluate the performance of the model as the training goes on we need to split the whole data into training as well as the training data.

X_train, X_val,\
    Y_train, Y_val = train_test_split(X, Y,
                                      test_size=0.05,
                                      random_state=10)
X_train.shape, X_val.shape

Output:

((27379, 2304), (1442, 2304))

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_val = scaler.transform(X_val)

Model Architecture

We will implement a Sequential model which will contain the following parts:

• Then we will have two fully connected layers followed by the output of the flattened layer.
• We have included some BatchNormalization layers to enable stable and fast training and a Dropout layer before the final layer to avoid any possibility of overfitting.
• The final layer is the output layer which outputs soft probabilities for the seven classes. 

Now we will be implementing a neural network with two hidden layers with 256 neurons each. These hidden layers are nothing but hidden layer perceptrons.

model = keras.Sequential([
    layers.Dense(256, activation='relu', input_shape=[2304]),
    layers.BatchNormalization(),
    layers.Dense(256, activation='relu'),
    layers.Dropout(0.3),
    layers.BatchNormalization(),
    layers.Dense(7, activation='softmax')
])

model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['AUC']
)

While compiling a model we provide these three essential parameters:

• optimizer – This is the method that helps to optimize the cost function by using gradient descent.
• loss – The loss function by which we monitor whether the model is improving with training or not.
• metrics – This helps to evaluate the model by predicting the training and the validation data.

Let's print the summary of our hidden layer perceptron model to understand the number of parameters present.

model.summary()

Output:

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 256)               590080    
                                                                 
 batch_normalization (BatchN  (None, 256)              1024      
 ormalization)                                                   
                                                                 
 dense_1 (Dense)             (None, 256)               65792     
                                                                 
 dropout (Dropout)           (None, 256)               0         
                                                                 
 batch_normalization_1 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 dense_2 (Dense)             (None, 7)                 1799      
                                                                 
=================================================================
Total params: 659,719
Trainable params: 658,695
Non-trainable params: 1,024
_________________________________________________________________

Model Training and Evaluation

Now we are ready to train our model.

model.fit(X_train, Y_train,
          epochs=5,
          batch_size=64,
          verbose=1,
          validation_data=(X_val, Y_val))

Output:

Epoch 1/5
428/428 [==============================] - 5s 8ms/step - loss: 1.8563 - auc: 0.6886 - val_loss: 1.6245 - val_auc: 0.7530
Epoch 2/5
428/428 [==============================] - 3s 7ms/step - loss: 1.6319 - auc: 0.7554 - val_loss: 1.5624 - val_auc: 0.7769
Epoch 3/5
428/428 [==============================] - 4s 8ms/step - loss: 1.5399 - auc: 0.7845 - val_loss: 1.5510 - val_auc: 0.7814
Epoch 4/5
428/428 [==============================] - 5s 11ms/step - loss: 1.4883 - auc: 0.7999 - val_loss: 1.5106 - val_auc: 0.7929
Epoch 5/5
428/428 [==============================] - 3s 8ms/step - loss: 1.4408 - auc: 0.8146 - val_loss: 1.4992 - val_auc: 0.7971

By using this neural network with two hidden layers we have achieved a 0.8 AUC-ROC score which implies that the predictions made will be around 80% accurate.

results = model.evaluate(X_val, Y_val, verbose=0)
print('Validation loss :', results[0])
print('Validation Accuracy :', results[1])

Output:

Validation loss : 1.4992401599884033
Validation Accuracy : 0.7971429824829102