Deep Learning Models with TensorFlow and Keras: A Guide to Sequential, Functional, CNN, RNN and Autoencoder Models with Example Code

TensorFlow is a powerful and popular deep learning library, and Keras is a high-level API built on top of TensorFlow that makes it easier to build and train deep learning models.

Here are some common types of deep learning models that you can build using the tf.keras API:

1. Sequential model: A sequential model is a linear stack of layers, where you can easily add, remove or modify layers. This is the most commonly used model in Keras.
2. Functional API model: A functional API model allows you to define complex models that have multiple inputs, multiple outputs, shared layers, and more. This type of model is useful when you need more flexibility and control over the model architecture.
3. Recurrent Neural Network (RNN): RNNs are a type of neural network that can process sequences of input data. They are commonly used in natural language processing, speech recognition, and time series prediction tasks.
4. Convolutional Neural Network (CNN): CNNs are a type of neural network that are designed for image processing tasks. They work by using convolutional layers that can identify features in an image, such as edges, corners, and textures.
5. Autoencoder: An autoencoder is a type of neural network that can learn to reconstruct its input data. It is often used for unsupervised learning, where you train the model on a dataset without labels.
6. Generative Adversarial Network (GAN): GANs are a type of neural network that can generate new data that is similar to the input data. They consist of two neural networks that are trained together: a generator network and a discriminator network.
7. Transformer: Transformers are a type of neural network that are commonly used in natural language processing tasks, such as language translation and sentiment analysis. They work by processing sequences of input data and are designed to handle long-term dependencies.

These are just a few of the many types of models that you can build using the tf.keras API. It’s important to choose the right model for your task and to experiment with different architectures to find the one that works best for your data.

Code Example with tf.keras API

Sequential model

Here’s an example of how to create a Sequential model using the tf.keras API:

import tensorflow as tf
print("TensorFlow version:", tf.__version__)

from keras.datasets import mnist
from keras.utils import to_categorical

# Use Keras MNIST dataset
mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# One-hot encode the labels
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

# Define the layers for the model
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(10, activation='softmax')
])

# Define a loss function for training
loss_fn = tf.keras.losses.categorical_crossentropy

# Compile the model
model.compile(optimizer='adam',
              loss=loss_fn,
              metrics=['accuracy'])

# Train the model
model.fit(x_train, y_train, epochs=10)

# Evaluate the model
test_loss, test_acc = model.evaluate(x_test, y_test)
print('Test accuracy:', test_acc)

In this example, we’re creating a simple Sequential model with two layers: a Dense layer with 128 units and a ReLU activation function, and a Dense layer with 10 units and a softmax activation function. We’re compiling the model with the Adam optimizer, categorical crossentropy loss, and accuracy metric. Then, we’re training the model on our training data for 10 epochs and evaluating it on our test data.

Functional API model

Here’s an example of how to create a Functional API model using the tf.keras API:

import tensorflow as tf
print("TensorFlow version:", tf.__version__)

# Use Keras IMDb dataset
from keras.datasets import imdb

# Load the dataset
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=10000)

# Preprocess the data
x_train_title = tf.keras.preprocessing.sequence.pad_sequences(x_train, maxlen=32)
x_test_title = tf.keras.preprocessing.sequence.pad_sequences(x_test, maxlen=32)
x_train_desc = tf.keras.preprocessing.sequence.pad_sequences(x_train, maxlen=32)
x_test_desc = tf.keras.preprocessing.sequence.pad_sequences(x_test, maxlen=32)
y_train = tf.keras.utils.to_categorical(y_train, num_classes=10)
y_test = tf.keras.utils.to_categorical(y_test, num_classes=10)

# Define the inputs for the model
input1 = tf.keras.Input(shape=(32,))
input2 = tf.keras.Input(shape=(32,))

# Define the layers for the model
dense1 = tf.keras.layers.Dense(64, activation='relu')
dense2 = tf.keras.layers.Dense(64, activation='relu')
concat = tf.keras.layers.Concatenate()
output = tf.keras.layers.Dense(10, activation='softmax')

# Connect the layers to create the model
x1 = dense1(input1)
x2 = dense2(input2)
merged = concat([x1, x2])
predictions = output(merged)
model = tf.keras.Model(inputs=[input1, input2], outputs=predictions)

# Compile the model
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit([x_train_title, x_train_desc], y_train, epochs=10, validation_data=([x_test_title, x_test_desc], y_test))

# Evaluate the model
test_loss, test_acc = model.evaluate([x_test_title, x_test_desc], y_test)
print('Test accuracy:', test_acc)

In this example, we’re using the IMDb movie review dataset, which consists of 50,000 movie reviews that are labeled as positive or negative. We’re pre-processing the data by padding the sequences to a fixed length of 32 and converting the labels to categorical format.

We’re defining a model with two input layers, one for the words in the title of the review and the other for the words in the body of the review. The model consists of two dense layers with ReLU activation functions, followed by a concatenation layer and a softmax output layer.

We’re compiling the model with the Adam optimizer, categorical cross-entropy loss, and accuracy metric. Then, we’re training the model on our training data for 10 epochs, with a validation split on the test data.

Recurrent Neural Network (RNN)

Here’s an example of how to create a simple Recurrent Neural Network (RNN) using the tf.keras API:

import tensorflow as tf
print("TensorFlow version:", tf.__version__)

# Use Keras IMDb dataset
from keras.datasets import imdb

# Load the dataset
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=10000)

# Preprocess the data
x_train = tf.keras.preprocessing.sequence.pad_sequences(x_train, maxlen=100)
x_test = tf.keras.preprocessing.sequence.pad_sequences(x_test, maxlen=100)

# Define the model architecture
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(10000, 32),
    tf.keras.layers.SimpleRNN(32),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# Compile the model
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(x_train, y_train, epochs=10, batch_size=128, validation_split=0.2)

# Evaluate the model
test_loss, test_acc = model.evaluate(x_test, y_test)
print('Test accuracy:', test_acc)

In this example, we’re using the IMDB movie review dataset, which consists of 25,000 movie reviews that are labeled as positive or negative. We’re pre-processing the data by padding the sequences to a fixed length of 100.

We’re defining a simple RNN model with an embedding layer that converts the input integers to dense vectors of fixed size, a SimpleRNN layer with 32 units, and a Dense layer with a sigmoid activation function for binary classification.

We’re compiling the model with the RMSprop optimizer, binary cross-entropy loss, and accuracy metric. Then, we’re training the model on our training data for 10 epochs, with a batch size of 128 and a validation split of 0.2.

Convolutional Neural Network (CNN)

Here’s an example of how to create a Convolutional Neural Network (CNN) using the tf.keras API:

import tensorflow as tf
print("TensorFlow version:", tf.__version__)

from keras.datasets import cifar10

# Load the dataset
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

# Preprocess the data
x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0
y_train = tf.keras.utils.to_categorical(y_train)
y_test = tf.keras.utils.to_categorical(y_test)

# Define the model architecture
model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(x_train, y_train, epochs=10, batch_size=64, validation_data=(x_test, y_test))

# Evaluate the model
test_loss, test_acc = model.evaluate(x_test, y_test)
print('Test accuracy:', test_acc)

In this example, we’re using the CIFAR-10 dataset, which consists of 60,000 32×32 color images in 10 classes. We’re pre-processing the data by scaling the pixel values to the range of [0, 1] and one-hot encoding the labels.

We’re defining a CNN model with three convolutional layers with increasing filter sizes, max pooling layers to reduce the spatial dimensionality, a flatten layer to convert the output of the convolutional layers into a 1D vector, and two fully connected layers for classification.

We’re compiling the model with the Adam optimizer, categorical cross-entropy loss, and accuracy metric. Then, we’re training the model on our training data for 10 epochs, with a batch size of 64 and validating on our test data.

Autoencoder

Here’s an example of how to create an Autoencoder using the tf.keras API:

import tensorflow as tf
print("TensorFlow version:", tf.__version__)

import numpy as np

import matplotlib.pyplot as plt
from keras.layers import Input, Dense
from keras.models import Model

# Create a custom dataset
data = np.random.random((1000, 100))

# Define the size of the encoding layer
encoding_dim = 32

# Define the input layer
input_layer = Input(shape=(data.shape[1],))

# Define the encoding layer
encoded = Dense(encoding_dim, activation='relu')(input_layer)

# Define the decoding layer
decoded = Dense(data.shape[1], activation='sigmoid')(encoded)

# Create the autoencoder
autoencoder = Model(input_layer, decoded)

# Compile the autoencoder
autoencoder.compile(optimizer='adam', loss='mse')

# Train the autoencoder
autoencoder.fit(data, data,
                epochs=50,
                batch_size=32,
                shuffle=True,
                validation_split=0.2)

# Get the encoder and decoder separately
encoder = Model(input_layer, encoded)
decoder_input = Input(shape=(encoding_dim,))
decoder_layer = autoencoder.layers[-1]
decoder = Model(decoder_input, decoder_layer(decoder_input))

# Generate some encoded and decoded data
encoded_data = encoder.predict(data)
decoded_data = decoder.predict(encoded_data)



# Plot the encoded and decoded data
n = 10  # number of data points to plot
plt.figure(figsize=(20, 4))
for i in range(n):
    # Original data
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(data[i].reshape(10, 10))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # Encoded data
    ax = plt.subplot(2, n, i + n + 1)
    plt.imshow(encoded_data[i].reshape(4, 8))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()

In this example, we’re using the MNIST dataset, which consists of 60,000 28×28 grayscale images of the digits 0-9. We’re pre-processing the data by scaling the pixel values to the range of [0, 1].

We’re defining an Autoencoder model with an input layer, a Dense layer with 32 units in the encoding layer, and a Dense layer with 784 units in the decoding layer.

We’re then compiling the Autoencoder model with the Adam optimizer and binary cross-entropy loss. Then, we’re training the model on our training data for 50 epochs with a batch size of 256 and validating on our test data.

We’re also defining an Encoder model and a Decoder model, which will allow us to get the encoded representation of the test data and reconstruct it from that representation.

Finally, we’re using the Encoder and Decoder models to get the encoded representation of the test data and reconstruct it from that representation. We’re then plotting the original and reconstructed images side by side for visual comparison.

Further readings

Here are some useful resources for further reading on deep learning models with TensorFlow and Keras:

1. TensorFlow documentation: https://www.tensorflow.org/api_docs
2. Keras documentation: https://keras.io/api/
3. Deep Learning with Python by François Chollet: https://www.manning.com/books/deep-learning-with-python
4. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition by Aurélien Géron: https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/
5. TensorFlow Tutorials: https://www.tensorflow.org/tutorials
6. Keras Tutorials: https://keras.io/examples/
7. Deep Learning Specialization by Andrew Ng: https://www.coursera.org/specializations/deep-learning
8. Dive into Deep Learning by Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola: https://d2l.ai/
9. The Hundred-Page Machine Learning Book by Andriy Burkov: https://www.amazon.com/Hundred-Page-Machine-Learning-Book/dp/199957950X
10. Hands-On Deep Learning for Images with TensorFlow by Will Ballard: https://www.packtpub.com/product/hands-on-deep-learning-for-images-with-tensorflow/9781801075303