Four Common Types of Neural Network Layers (and When to use Them)
Neural networks (NN) are the backbone of many of today's machine learning (ML) models, loosely mimicking the neurons of the human brain to recognize patterns from input data. As a result, numerous types of neural network topologies have been designed over the years, built using different types of neural network layers.
You can easily build a model with different types of neural network topologies in PerceptiLabs, by dragging and dropping different neural network layer components into the workspace, configuring their hyperparameters, and then creating connections between those components. And because of our visual approach to ML modeling in the PerceptiLabs tool, you also gain transparency into your neural networks during design and training.
PerceptiLabs supports the four most common types of neural network layers: Fully connected, Convolution, Deconvolution, and Recurrent.
Fully Connected Layer
Fully connected layers connect every neuron in one layer to every neuron in the next layer. Fully connected layers are found in all different types of neural networks ranging from standard neural networks to convolutional neural networks (CNN).
Fully connected layers can become computationally expensive as their input grows, resulting in a combinatorial explosion of vector operations to be performed, and potentially poor scalability. As such, they are commonly used for specific purposes within neural networks such as classifying image data.
Use Cases
- Experimentation or learning ML using fully connected neural networks.
- In CNNs to classify images for computer vision.
Usage in PerceptiLabs
Look for this component in PerceptiLabs and drag it onto your workspace:
Configurable Hyperparameters
- Number of neurons
- Activation function
- Dropout
Convolution Layer
A Convolution Layer is an important type of layer in a CNN. Its most common use is for detecting features in images, in which it uses a filter to scan an image, a few pixels at a time, and outputs a feature map that classifies each feature found.
The filter (sometimes called kernel) is a set of n-dimensional weights that are multiplied against the input, where the filter's dimensions match that of the input (e.g., two dimensions when dealing with 2D images). The filter describes the probabilities that a given pattern of pixels represents a feature. Thus the number of filter weights (i.e., size of the filter) are smaller than the input, and the multiplication performed by the layer's convolution process is performed on "patches" of the image that match the filter size.
Multiplication is systematically repeated from left to right and top to bottom over the entire image to detect features. The number of pixels by which the filter moves for the next iteration is called the stride. Padding may be added around the input image to ensure that the filter always fits within the total bounds of the image for a given stride.
Use Cases
- Analyzing imagery for image recognition and classification.
Usage in PerceptiLabs
Look for this component in PerceptiLabs and drag it onto your workspace:
Configurable Hyperparameters
- Dimensionality
- Patch size
- Stride
- Number of feature maps to generate
- Padding strategy
- Activate function
- Dropout
- Pooling
Deconvolution Layer
A Deconvolution Layer is a transposed convolution process that effectively upsamples data to a higher resolution. This can include image data and/or feature maps generated from a convolution layer, or other types of data. For image data, the upsampling resolution output by deconvolution may be the same as the original input image, or may be different.
Common Use Cases
- Upsampling images
Usage in PerceptiLabs
Look for this component in PerceptiLabs and drag it onto your workspace:
Configurable Hyperparameters
- Dimensionality
- Stride
- Number of feature maps to generate
- Padding strategy
- Activate function
- Dropout
Recurrent Layer
A Recurrent Layer includes a "looping" capability such that its input consists of both the data to analyze as well as the output from a previous calculation performed by that layer. Recurrent layers form the basis of recurrent neural networks (RNNs), effectively providing them with memory (i.e., maintain a state across iterations), while their recursive nature makes RNNs useful for cases involving sequential data like natural language and time series. They're also useful for mapping inputs to outputs of different types and dimensions.
Common Use Cases
- Classifying sentiments into positive and negative.
- Generating text descriptions of what an image contains.
- Translating paragraphs of text to another language.
Usage in PerceptiLabs
Look for this component in PerceptiLabs and drag it onto your workspace:
Configurable Hyperparameters
- Dimensionality
- Type of recurrent neural network (LSTM, GRU, or standard RNN layer)
- Return sequence
- Dropout
Image Recognition Example
Now that we've looked at the common neural network layers, let's look at a simple example of how a neural network model can be built in PerceptiLabs using different types of layers.
The image recognition example project included with PerceptiLabs solves the classical problem of identifying the digits 0 through 9 from 28x28 pixel grayscale images of handwritten digits. This example project has the following topology that includes Convolutional Layer and Fully Connected Layer components:
Figure 1: Topology of the image recognition sample project included in PerceptiLabs.
The image data is initially in the form of a one-dimensional array of 784 pixels, so a Reshape component converts the data into a two dimensional array of 28x28 values, which are then input into the Convolutional layer for processing.
Figure 2: The Convolutional Layer component connected to the Reshape Layer component. Note how PerceptiLabs shows the resulting dimension of a given layer above its icon.
The Convolutional Layer component's hyperparameters define the expected input dimensions, patch size, stride, number of feature maps to produce, and the padding strategy to employ.
The output of the Convolution Layer component consists of eight, 14x14 feature maps which are input to a fully connected layer, configured with 10 neurons:
Figure 3: The addition of a Fully Connected Layer component to calculate the final predictions for a given image's digit representation.
Each node in each feature map is connected to each neuron in the fully connected layer. These 10 neurons contain the probabilities that a given image represents each digit from 0 through 9 based on the features detected.
As Figure 3 shows, this part of the project's topology defines both the input data and its shape, as well as a basic neural network consisting of two layers. The output from the neural network can then be sent to a supervised learning layer (that's the green icon with the sprocket shown in Figure 1 above) to train the neural network.
And while this is a simple example, you can start to see how easy it is to build models with more complex neural networks using PerceptiLabs.
Conclusion
Neural networks are the current state-of-the-art when it comes to ML and there are many topologies and layer types to choose from. However, their inner workings can often be a black box which is where PerceptiLabs visual approach can be beneficial. With PerceptiLabs, ML practitioners can better understand their neural network models from both the visualization of how the neural network's layers are connected and through their explicit hyperparameter fields.
Interested in learning more? Be sure to check out the neural networks section of our Machine Learning Handbook, and be sure to post your questions or comments on our Slack channels.