CUSTOMER ANALYTICS USING MULTIHEAD ATTENTION (Part 1)

Multihead Transformer Architecture

Key Components:

Embedding Layer: Converts input tokens into dense vectors of fixed size.

Positional Encoding: Adds information about the positions of the tokens in the sequence, essential for capturing the order of words.

Multi-Head Attention: Allows the model to focus on different parts of the input sequence simultaneously, capturing various aspects of the input.

Feed-Forward Network: Applies a series of transformations to the attended representations.

Layer Normalization: Normalizes the inputs across the features.

Global Max Pooling: Reduces the output of the Transformer block to a fixed size.

Output Layers: Separate dense layers for sentiment analysis and behavior prediction.

The architecture of our Transformer model is illustrated in the image below:

Here:

𝑝𝑜𝑠 is the position.

𝑖 is the dimension.

𝑑model is the dimensionality of the embeddings.

The sine and cosine functions are used to generate unique patterns for each position, ensuring that each position has a unique encoding. The use of different frequencies allows the model to capture information at varying levels of granularity.

Code Snippet:

class PositionalEncoding(tf.keras.layers.Layer):

    def __init__(self, position, d_model):

        super(PositionalEncoding, self).__init__()

        self.pos_encoding = self.positional_encoding(position, d_model)

    def get_config(self):

        config = super(PositionalEncoding, self).get_config()

        config.update({"position": self.position, "d_model": self.d_model})

        return config

    def positional_encoding(self, position, d_model):

        angle_rads = self.get_angles(np.arange(position)[:, np.newaxis], np.arange(d_model)[np.newaxis, :], d_model)

        angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])

        angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])

        pos_encoding = angle_rads[np.newaxis, ...]

        return tf.cast(pos_encoding, dtype=tf.float32)

    def get_angles(self, pos, i, d_model):

        angle_rates = 1 / np.power(10000, (2 * (i // 2)) / np.float32(d_model))

        return pos * angle_rates

    def call(self, inputs):

        return inputs + self.pos_encoding[:, :tf.shape(inputs)[1], :]

x = PositionalEncoding(max_length, embedding_dim)(x)

Detailed Code Explanation

Class Initialization:

The PositionalEncoding class inherits from tf.keras.layers.Layer.

The __init__ method initializes the positional encoding matrix by calling the positional_encoding function with the given position and d_model.

Positional Encoding Calculation:

The positional_encoding function generates the positional encoding matrix using sine and cosine functions.

np.arange(position)[:, np.newaxis] creates a column vector of positions.

np.arange(d_model)[np.newaxis, :] creates a row vector of dimensions.

The get_angles function calculates the angles for the sine and cosine functions.

The sine function is applied to even indices: angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2]).

The cosine function is applied to odd indices: angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2]).

The resulting matrix pos_encoding has shape (1, position, d_model).

Adding Positional Encoding to Input:

The call method adds the positional encoding to the input embeddings.

The positional encoding matrix is broadcasted to match the shape of the input embeddings and added element-wise.

In the next part, we will discuss how we will build the transformer model.