A Brief Overview of Recurrent Neural Networks (RNN)

The most important component of RNN is the Hidden state, which remembers specific information about a sequence

The Architecture of a Traditional RNN

RNN architectures

1. One To One: There is only one pair here. A one-to-one architecture is used in traditional neural networks.

2. One To Many: A single input in a one-to-many network might result in numerous outputs. One too many networks are used in the production of music, for example.

3. Many To One:  In this scenario, a single output is produced by combining many inputs from distinct time steps. Sentiment analysis and emotion identification use such networks, in which the class label is determined by a sequence of words.

4. Many To Many: For many to many, there are numerous options. Two inputs yield three outputs. Machine translation systems, such as English to French or vice versa translation systems, use many to many networks.

   

   The input layer x receives and processes the neural network’s input before passing it on to the middle layer

   Multiple hidden layers can be found in the middle layer h, each with its own activation functions, weights, and biases. You can utilize a recurrent neural network if the various parameters of different hidden layers are not impacted by the preceding layer, i.e. There is no memory in the neural network

   The different activation functions, weights, and biases will be standardized by the Recurrent Neural Network, ensuring that each hidden layer has the same characteristics. Rather than constructing numerous hidden layers, it will create only one and loop over it as many times as necessary

   Common Activation Functions

   

   • Sigmoid: The formula $g(z) = \frac{1}{(1 + e^{-z})}$ is used to express this.

   • Tanh: The formula $g(z) = \frac{e^{z} – e^{-z}}{(e^z + e^{-z}}$ is used to express this.

   • Relu: The formula $g(z) = max(0 , z)$ is used to express this.

     

   A feed-forward neural network has only one route of information flow: from the input layer to the output layer, passing through the hidden layers. The data flows across the network in a straight route, never going through the same node twice

   Feed-forward neural networks are poor predictions of what will happen next because they have no memory of the information they receive. Because it simply analyses the current input, a feed-forward network has no idea of temporal order. Apart from its training, it has no memory of what transpired in the past

   The information is in an RNN cycle via a loop. Before making a judgment, it evaluates the current input as well as what it has learned from past inputs. A recurrent neural network, on the other hand, may recall due to internal memory. It produces output, copies it, and then returns it to the network

   

   The output of the neural network is used to calculate and collect the errors once it has trained on a time set and given you an output. The network is then rolled back up, and weights are recalculated and adjusted to account for the faults.

   

   With regard to its inputs, a gradient is a partial derivative. If you’re not sure what that implies, consider this: a gradient quantifies how much the output of a function varies when the inputs are changed slightly.

   A function’s slope is also known as its gradient. The steeper the slope, the faster a model can learn, the higher the gradient. The model, on the other hand, will stop learning if the slope is zero. A gradient is used to measure the change in all weights in relation to the change in error

   • Exploding Gradients: Exploding gradients occur when the algorithm gives the weights an absurdly high priority for no apparent reason. Fortunately, truncating or squashing the gradients is a simple solution to this problem.
   • Vanishing Gradients: Vanishing gradients occur when the gradient values are too small, causing the model to stop learning or take far too long. This was a big issue in the 1990s, and it was far more difficult to address than the exploding gradients. Fortunately, Sepp Hochreiter and Juergen Schmidhuber’s LSTM concept solved the problem.

   Recurrent Neural Network	Deep Neural Network
   Weights are same across all the layers number of a Recurrent Neural Network	Weights are different for each layer of the network
   Recurrent Neural Networks are used when the data is sequential and the number of inputs is not predefined.	A Simple Deep Neural network does not have any special method for sequential data also here the the number of inputs is fixed
   The Numbers of parameter in the RNN are higher than in simple DNN	The Numbers of Parameter are lower than RNN
   Exploding and vanishing gradients is the  the major drawback of RNN	These problems also occur in DNN but these are not the major problem with DNN