Hello, and welcome! In this video, we’ll be covering the basic concepts and the motivation behind autoencoders, a type of neural network used in unsupervised machine learning. Let’s say that you want to extract the feeling or emotion of a person in a photograph. The photograph that you see here is a small image that’s just 256 by 256 pixels… But, this means that there are over 65 thousand pixels in play in defining the dimension of the input! As we increase the dimensionality, the time to deal with data increases exponentially, in order to train and fit the raw data into a neural network that can detect the emotion. So, we need a way to extract the most important features of a face, and represent each image with those features which are of lower dimensions. An autoencoder works well for this type of problem. An autoencoder is a type of unsupervised learning algorithm that will find patterns in a dataset by detecting key features. It is a type of neural net that analyzes all of the images in your dataset and extracts some useful features automatically in such a way that it can distinguish images using those features. Generally speaking, autoencoders excel in tasks that involve feature learning or extraction, data compression, and learning generative models of data and dimensionality reduction. Let me talk more about Dimension Reduction. High-dimensional data is a significant problem for machine learning tasks. In fact, data scientists commonly refer to it as the “CURSE OF DIMENSIONALITY”. Many common problems that we want to target have a large number of dimensions. Even that 256 x 256 pixel image that we saw earlier, would have over 65 thousand dimensions; or in other words, one dimension for each pixel. A high-resolution image from a smart phone would be even larger. According to a study, the time to fit a model is, at best, the function you see here, which is a function of the number of points, dimensionality, and parameters. So, as we increase the dimensionality, our time to fit our model will increase exponentially. So, what happens when we increase or reduce the dimension of data? Well, if we have a huge number of dimensions, our data will start to get sparse, which results in an over-allocation of memory and slow training time. We run into additional problems when we try to reduce the dimensions. If we have a small number of dimensions, our data could overlap, resulting in a loss of data characteristics. You can see how that looks in one dimension and three dimensions. Overlap and sparsity make it difficult to determine the underlying patterns. However, with the proper number of dimensions, the patterns become much clearer. It’s important to know that an Autoencoder is not the only dimension reduction method in Machine Leaning. Principal Component Analysis (or PCA) has been around for a long time, and is a classic algorithm for dimensionality reduction. Take a look at the data separation outputs for the MNIST dataset of handwritten digits. These charts show the image data after reducing their dimensions to 2 dimensions. The left output is from PCA. The output on the right is from an autoencoder. As you can see, it’s easier to discern the data with the autoencoder’s output. Here is another comparison between PCA and an autoencoder, now applied to news stories. The separability of the autoencoder is far better than the PCA, in this case. Since separability is important for applying clustering algorithms, this difference in quality is significant. By now, you should understand that an autoencoder can extract key image features, improve training times of other networks, and improve the separability of reduced datasets when compared to other methods. For these reasons, the autoencoder was a breakthrough in the unsupervised learning research field. Thanks for watching this video.
