Here are a few well established methods that you can use to generate word embeddings. Word2vec by Google, which initially popularized the use of machine learning, to generate word embeddings. Word2vec, uses a shallow neural network to learn word embeddings. It proposes two model architectures, continuous bag of words, which is a simple but efficient approach that you will implement this week. The objective of the model is to learn to predict a missing word given the surrounding words. Continuous skip-gram, also known as the skip-gram with negative sampling, which does the reverse of the continuous bag of words method. The model learns to predict the word surrounding a given input word. Global vectors or glove by Stanford, which involves factorizing the logarithm of the corpuses word co-occurrence matrix, which is similar to the counter matrix you've used before. fastText by Facebook, which is based on the skip-gram model and takes into account the structure of words by representing words as an n-gram of characters. This enables the model to support previously unseen words, known as outer vocabulary words, by inferring their embedding from the sequence of characters they are made of and the corresponding sequences that it was initially trained on. For example, it would create similar embeddings for kitty and kitten, even if it had never seen the word kitty before. As kitty and kitten are made of similar sequences of characters. Another benefit of fastText, is that word embedding vectors can be averaged together to make vector representations of phrases and sentences. Next up, some more sophisticated modeling approaches. These methods use advanced deep neural network architectures, to refine the representation of the words meaning according to their contexts. In the previous models a given word always has the same embedding. In these more advanced models, the words have different embeddings depending on their context. This adds supports for polysemy or words with similar meanings. Such as the word plants, which can refer to an organism like a flower, a factory or which can be an adverb, with yet more different meanings. A few examples of advanced models that generate word embeddings are, BERT or bidirectional encoder representations from transformers by Google, ELMO for embeddings from language models, by the Allen Institute for AI or GPT-2 or generative pre-training 2 by Open AI. If you want to use these advanced methods, you can find off the shelf pre-trained models on the Internet. You can fine tune these models using your own corpus to generate high-quality, domain, specific word embeddings. To recap, you now have some tools in your toolbox for creating word embeddings, including some pretty sophisticated new models. That's nice. Up next, I will introduce the continuous bag of words model, that you'll be using for this week's assignment.