Welcome to Understanding Learned Relationships using the embedding projector. In this activity, you'll use a tool called the embedding projector to understand what models learn about data. Creating a model involves making use of standard algorithms and lots of data. Before deploying a model, it's important to evaluate that model in order to ensure that it's performant and not discriminatory. There are many ways of doing so, some of which we introduce in the lecture. For example, you can look at the accuracy of the model or examine a confusion matrix. In some business use cases, it may also be important to understand what the models learned. To understand why it makes the predictions that it does. One way of doing this is to look at how the model groups the data. To see what data points it thinks are similar and what data points it thinks are dissimilar. We can sometimes explain the model's predictions by explaining what the model has learned in terms of the relationship between different input values. TensorBoard's embedding projector is a tool designed for this purpose. In this activity we'll examine how a model trained to work with text groups the meanings of words. These groupings will reveal things about the relationships between those words. The model has learned about these relationships solely by looking at documents and seeing which groups of words tend to occur together. This is the embedding projector. The embedding projector has four panels. The data panel, on the top left, where you can choose the data set to examine. The projections panel, on the bottom left, which can be used among other things to define how to orient the visualization. The inspector panel, on the top right, where you can search for particular points. In this case words, and see a list of nearest neighbors. The bookmarks panel, on the bottom right, displays saved sets of parameters for the tool. And the visualization panel in the center where the data is displayed, which in this case is a blue rotating cloud. When you load the page you should see a slowly rotating cloud of blue points in the visualization panel. Mouse over a point. In this case what you see is that it corresponds to the word cellular and that there's a count attached to it. Which in this case is the number of times that the model saw cellular in its data set. You can also see in the visualization pane, the number of words that have been highlighted. These are the nearest neighbors of cellular, and then they're also displayed conveniently on the right hand-side here. And they shouldn't surprise you. There are things like telephones and telephone, and mobile and phones. Displayed next to them is a numerical value corresponding to the cosine of the two vectors, which is basically a proxy for how similar they are. The visualization panel also supports zooming and panning. So you can use your mouse to zoom in and out and explore different regions. However, there's a lot of points here. And it can be challenging to simply look at individual neighborhoods and try and get a sense. So we're going to come up with a more targeted way of exploring. In the data panel, change the data set being visualized to Word2Vec All. Each of the data points here comes from a model trained on Wikipedia articles. One of the most basic sanity checks for understanding what a model has learned is to do a neighborhood analysis. A neighborhood analysis involves looking at the nearest neighbors for a set of predefined points and seeing if they make intuitive sense. In the inspector panel, type the word mouse. The interface will offer suggestions for matching words. In this case mousetrap and mouseketeer and others. Click on mouse to select it. Note how, as before, we also get a list of nearest points in space, which include words that are similar to mouse. After you've clicked on the word mouse, click on the gray button labeled, Isolate 101 points. This will restrict the set of points being visualized at the point representing mouse, as well as its 100 nearest neighbors. In this case you should see words related to mouse, but perhaps some that may surprise you. So examine the set of nearest neighbors and see if you can find any patterns. I find it helpful to rotate the space and to zoom and pan a little bit. In this case, it seems like we have one region over here that corresponds to characters and animals. Notice Walt Disney, mickey, mighty, rat and dogs. And then [COUGH] at the top, we have another region that seems related to hardware. In the case of words with more than one meaning, a neighborhood that reflects this fact may be totally acceptable. But sometimes models learn things that even after we notice them can be cause for concern. The embedding projector also allows you to reorient the visualization in order to do more sophisticated tests for these cases. And we will use this to demonstrate how a model has learned that male words are more related to math, while female words are more related to art. First, click on Show All Data again, and then in the inspector panel, paste this string. And it's okay that it doesn't find any matches. And then select the period star button. Notice how it now matches things. Then click on the gray box labeled Isolate 32 points. So what I've done for you is I've configured the searcher to find 32 words related to gender, math and art. Then in the projections panel, select the tab labeled Custom. In the text box labeled Up, paste the following. And then select the period star, similarly paste the different string in the other box. And what we see now is that our visualizations become reoriented. The top we have male, at the bottom we have female. And all the words are somehow projected into this line. And unfortunately we also see is that many of the words that are toward the female end have to do with art. So for instance you see art and dance and sculpture. Now this isn't universally true, there are also some that have to do with math, like numbers. On the other side of the spectrum we also see that a lot of the math words are clustered near the male side. That includes equations and computation and algebra, and calculus and geometry and math. Now, in this case, we considered a model absent from the context in which it was used. But imagine if these data points were used to predict how good someone was at their job, and that job was related to art or math. The model might mistakenly believe that their gender affected how good a candidate they were. In summary, today we looked at the embedding projector, a tool that's freely available as part of the TensorFlow software package. And we used it to understand what models learn about data without being told. Specifically looking through the lens at which points are similar to which other points.
