{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre style=\"float: right\">version 1.0.1</pre>\n",
    "# FNLP: Lab Session 2: Smoothing and Authorship Identification\n",
    "\n",
    "## Aim\n",
    "\n",
    "The aims of this lab session are to:\n",
    "- explore Laplace, Lidstone and backoff smoothing methods for language models\n",
    "- use of language models in authorship identification. \n",
    "\n",
    "Successful completion of this lab will help you solidify your understanding of smoothing (important not just for LMs but all over NLP), cross-entropy (important also for assignment 1), and one type of text classification (authorship identification). By the end of this lab session, you should be able to:\n",
    "- Compute smoothed bigram probabilities by hand for simple smoothing methods.\n",
    "- Train an ``NgramModel``  with smoothing for unseen n-grams.\n",
    "- Make use of language models to identify the author of a text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "For this lab, we will continue to use ``nltk`` and ``nltk_models`` package from the previous lab. Moreover, we will only work with Gutenberg corpus. Execute the bottom cell to import all libraries for this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import gutenberg\n",
    "\n",
    "# The NgramModel from NLTK version 2 has been removed from NLTK 3.\n",
    "# So we're using a ported version from a local directory.\n",
    "try:\n",
    "    from nltk_model import *  # See the README inside the nltk_model folder for more information\n",
    "except ImportError:\n",
    "    from .nltk_model import *  # Compatibility depending on how this script was run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Smoothing\n",
    "\n",
    "In the final exercise of Lab 1, you were asked to calculate the probability of a word given its context, using a bigram language model with no smoothing. For the first two word-context pairs, these bigrams had been seen in the data used to train the language model. For the third word-context pair, the bigram had not been seen in the training data, which led to an estimated probability of 0.0.\n",
    "\n",
    "Zero probabilities for unseen n-grams cause problems. Suppose for example you take a bigram language model and use it to score an automatically generated sentence of 10 tokens (say the output of a machine translation system). If one of the bigrams in that sentence is unseen, the probability of the sentence will be zero.\n",
    "\n",
    "Smoothing is a method of assigning probabilities to unseen n-grams. As language models are typically trained using large amounts of data, any n-gram not seen in the training data is probably unlikely to be seen in other (test) data. A good smoothing method is, therefore, one that assigns a fairly small probability to unseen n-grams.\n",
    "\n",
    "We’ll explore two different smoothing methods: Laplace (add-one) and Lidstone (add-alpha), and we will also consider the effects of backoff."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Maximum-Likelihood estimation\n",
    "\n",
    "Before implementing any smoothing, you should make sure you understand how to implement maximum likelihood estimation. In last week’s lab, we used NLTK to do this for us by training a bigram language model with an MLE estimator. We could then use the language model to find the MLE probability of any word given its context. Here, you’ll do the same thing but without using NLTK, just to make sure you understand how. We will also compare the smoothed probabilities you compute later to these MLE probabilities.\n",
    "\n",
    "### Exercise 0\n",
    "The function below extracts all the words from the specified document in Gutenberg corpus and then computes a list of bigram tuples by pairing up each word in the corpus with the following word. Using the resulting lists of unigrams and bigrams, complete the code block so it returns the MLE probability of a word given a single word of context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myMLE(doc_name, word,context):\n",
    "    \"\"\"\n",
    "    :type doc_name: str\n",
    "    :param doc_name: name of the document to use for estimation\n",
    "    :type word: str\n",
    "    :param word: The input word\n",
    "    :type context: str\n",
    "    :param context: The preceding word\n",
    "    :rtype: float\n",
    "    :return: The MLE probability of word given context\n",
    "    \"\"\"\n",
    "    # Preprocessing all words to be lowercased\n",
    "    words = [w.lower() for w in gutenberg.words(doc_name)]\n",
    "    # list of bigrams as tuples (doesn't include begin/end of corpus: but basically this is fine)\n",
    "    bigrams = list(zip(words[:-1], words[1:])) \n",
    "    # Compute probability of word given context\n",
    "    prob = 0\n",
    "\n",
    "    return prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your estimates using Jane Austen’s “Sense and Sensibility” from Gutenberg Corpus by computing the probabilities:\n",
    "    \n",
    "1. $ P_{MLE}(“end”|“the”) $\n",
    "2. $ P_{MLE}(“the”|“end”) $\n",
    "\n",
    "Make sure your answers match the MLE probability estimates from Exercise 5 of Lab 1, where we used NLTK to compute these estimates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_name = 'austen-sense.txt'\n",
    "print(\"MLE probability of 'end' given 'the': {:.5f}\".format(myMLE(doc_name, 'end', 'the')))\n",
    "print(\"MLE probability of 'the' given 'end': {:.5f}\".format(myMLE(doc_name, 'the', 'end')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Laplace (add-1)\n",
    "\n",
    "Laplace smoothing adds a value of 1 to the sample count for each “bin” (possible observation, in this case, each possible bigram), and then takes the maximum likelihood estimate of the resulting frequency distribution.\n",
    "\n",
    "### Exercise 1\n",
    "\n",
    "Assume that the size of the vocabulary is just the number of different words observed in the training data (that is, we will not deal with unseen words). Complete the function ``myLaplace`` to compute Laplace smoothed probabilities, again without using NLTK. Hint: if you have trouble, study the equations and example in Lectures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myLaplace(doc_name, word, context):\n",
    "    \"\"\"\n",
    "    :type doc_name: str\n",
    "    :param doc_name: name of the document to use for estimation\n",
    "    :type word: str\n",
    "    :param word: The input word\n",
    "    :type context: str\n",
    "    :param context: The preceding word\n",
    "    :rtype: float\n",
    "    :return: The Laplace-smoothed probability of word given context\n",
    "    \"\"\"\n",
    "    # Preprocessing all words to be lowercased\n",
    "    words = [w.lower() for w in gutenberg.words(doc_name)]\n",
    "    # list of bigrams as tuples (doesn't include begin/end of corpus: but basically this is fine)\n",
    "    bigrams = list(zip(words[:-1], words[1:]))\n",
    "    # Estimate the size of the vocabluary\n",
    "    V = 0\n",
    "    # Compute probability of word given context\n",
    "    prob = 0\n",
    "\n",
    "    return prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now test your code and look at the estimates for:\n",
    "\n",
    "1. $P_{+1} (“end”|“the”)$\n",
    "2. $P_{+1} (“the”|“end”)$\n",
    "\n",
    "using Jane Austen’s “Sense and Sensibility” as training data. How do these probabilities differ from the MLE estimates performed previously?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_name = 'austen-sense.txt'\n",
    "print(\"LAPLACE probability of 'end' given 'the': {:.5f}\".format(myLaplace(doc_name, 'end', 'the')))\n",
    "print(\"LAPLACE probability of 'the' given 'end': {:.5f}\".format(myLaplace(doc_name, 'the', 'end')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lidstone (add-alpha)\n",
    "\n",
    "In practice, Laplace smoothing assigns too much mass to unseen n-grams. The Lidstone method works in a similar way, but instead of adding 1, it adds a value between 0 and 1 to the sample count for each bin (in class we called this value alpha, NLTK calls it gamma).\n",
    "\n",
    "### Exercise 2\n",
    "\n",
    "Complete function ``myLidstone`` to compute Lidstone smoothed probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myLidstone(doc_name, word, context, alpha=1.0):\n",
    "    \"\"\"\n",
    "    :type doc_name: str\n",
    "    :param doc_name: name of the document to use for estimation\n",
    "    :type word: str\n",
    "    :param word: The input word\n",
    "    :type context: str\n",
    "    :param context: The preceding word\n",
    "    :type alpha: float \n",
    "    :param alpha: smoothing constant\n",
    "    :rtype: float\n",
    "    :return: The Lidstone-smoothed probability of word given context\n",
    "    \"\"\"\n",
    "    # Preprocessing all words to be lowercased\n",
    "    words = [w.lower() for w in gutenberg.words(doc_name)]\n",
    "    # list of bigrams as tuples (doesn't include begin/end of corpus: but basically this is fine)\n",
    "    bigrams = list(zip(words[:-1], words[1:]))\n",
    "    # Estimate the size of the vocabluary\n",
    "    V = 0\n",
    "    # Compute probability of word given context\n",
    "    prob = 0\n",
    "\n",
    "    return prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "test code again using Jane Austen Novel. Look at the probability estimates that are computed for the same bigrams as before using various values of alpha.\n",
    "\n",
    "What do you notice about using `alpha = 0` and `alpha = 1`? (Compare to the probabilities computed by the previous methods.) What about when `alpha = 0.01`? Are the estimated probabilities more similar to MLE or Laplace smoothing in this case?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_name = 'austen-sense.txt'\n",
    "print(\"alpha=0\")\n",
    "print(\"LIDSTONE probability of 'end' given 'the': {:.5f}\".format(myLidstone(doc_name, 'end', 'the', 0)))\n",
    "print(\"LIDSTONE probability of 'the' given 'end': {:.5f}\".format(myLidstone(doc_name, 'the', 'end', 0)))\n",
    "print(\"alpha=1\")\n",
    "print(\"LIDSTONE probability of 'end' given 'the': {:.5f}\".format(myLidstone(doc_name, 'end', 'the', 1)))\n",
    "print(\"LIDSTONE probability of 'the' given 'end': {:.5f}\".format(myLidstone(doc_name, 'the', 'end', 1)))\n",
    "print(\"alpha=0.1\")\n",
    "print(\"LIDSTONE probability of 'end' given 'the': {:.5f}\".format(myLidstone(doc_name, 'end', 'the', 0.1)))\n",
    "print(\"LIDSTONE probability of 'the' given 'end': {:.5f}\".format(myLidstone(doc_name, 'the', 'end', 0.1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backoff\n",
    "\n",
    "Now we will look at the effects of incorporating backoff in addition to some of these simple smoothing methods. In a bigram language model with backoff, the probability of an unseen bigram is computed by “backing off”: that is, if a word has never been seen in a particular context, then we compute its probability by using one fewer context words. Backing off from a bigram model (one word of context) therefore means we’d get estimates based on unigram frequencies (no context).\n",
    "\n",
    "The mathematical details of backoff are a bit complex to ensure all the probabilities sum to 1. You needn’t understand all the details of backoff but you should understand these basic principles:\n",
    "\n",
    "- Bigram probabilities for seen bigrams will be slightly lower than MLE to allocate some probability mass to unseen bigrams.\n",
    "- The unigram probabilities inside the backoff (i.e. the ones we use if we didn’t see the bigram) are similar in their relatives sizes to the unigram probabilities we would get if we just estimated a unigram model directly.\n",
    "\n",
    "That is, a word with high corpus frequency will have a higher unigram backoff probability than a word with a low corpus frequency. Look back at the initialization method for NgramModel earlier in the lab. If you pass in MLEProbDist as the estimator (which we did in the last lab), then no backoff is used. However, with any other estimator (i.e., smoothing), the NgramModel does use backoff.\n",
    "\n",
    "### Exercise 3\n",
    "\n",
    "Complete the function ``myLaplaceBackoff`` to estimate the Laplace Language model with backoff for the given document of Gutenberg corpus, using ``NgramModel``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myLaplaceBackoff(doc_name, word, context):\n",
    "    \"\"\"\n",
    "    :type doc_name: str\n",
    "    :param doc_name: name of the document to use for estimation\n",
    "    :type word: str\n",
    "    :param word: The input word\n",
    "    :type context: str\n",
    "    :param context: The preceding word\n",
    "    :rtype: float\n",
    "    :return: The Laplace-smoothed probability of word given context\n",
    "    \"\"\"\n",
    "    words = [w.lower() for w in gutenberg.words(doc_name)]\n",
    "    est = lambda fdist,bins: nltk.probability.LaplaceProbDist(fdist,bins+1)\n",
    "    # Train a bigram language model using a LAPLACE estimator AND BACKOFF\n",
    "    lm = NgramModel(<order>,<word_list>,estimator=<estimator>)\n",
    "    # Compute probability of word given context (note lm requires a list context)\n",
    "    prob = 0\n",
    "\n",
    "    return prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your function again with Jane Austen novel by explore how diffrent values of the interpolation constant effect the probability estimate. How different are the estimated probabilities, compared to previously implemented ones?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_name = 'austen-sense.txt'\n",
    "print(\"LAPLACE(backoff) probability of 'end' given 'the': {:.5f}\".format(myLaplaceBackoff(doc_name, 'end', 'the')))\n",
    "print(\"LAPLACE(backoff) probability of 'the' given 'end': {:.5f}\".format(myLaplaceBackoff(doc_name, 'the', 'end')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Authorship Identification\n",
    "\n",
    "## Cross-entropy\n",
    "\n",
    "In language modelling, a model is trained on a set of data (i.e. the training data). The cross-entropy of this model may then be measured on a test set (i.e. another set of data that is different from the training data) to assess how accurate the model is in predicting the test data.\n",
    "\n",
    "Another way to look at this is: if we used the trained model to generate new sentences by sampling words from its probability distribution, how similar would those new sentences be to the sentences in the test data? This interpretation allows us to use cross-entropy for authorship detection.\n",
    "\n",
    "`NgramModel` contains the following cross-entropy method:\n",
    "```python\n",
    "def entropy(self, text, pad_left=False, pad_right=False,\n",
    "    verbose=False, perItem=False):\n",
    "    \"\"\"\n",
    "    Calculate the approximate cross-entropy of the n-gram model for a\n",
    "    given evaluation text.\n",
    "    This is the average log probability of each item in the text.\n",
    "    :param text: items to use for evaluation\n",
    "    :type text: iterable(str)\n",
    "    :param pad_left: whether to pad the left of each text with an (n-1)-gram\\\n",
    "    of <s> markers\n",
    "    :type pad_left: bool\n",
    "    :param pad_right: whether to pad the right of each sentence with an </s>\\\n",
    "    marker\n",
    "    :type pad_right: bool\n",
    "    :param perItem: normalise for length if True\n",
    "    :type perItem: bool\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4\n",
    "\n",
    "We can use cross-entropy in authorship detection. For example, suppose we have a language model trained on Jane Austen’s “Sense and Sensibility” (training data) plus the texts for two other novels (test data), one by Jane Austen and one by another author, but we don’t know which is which. We can work out the cross-entropy of the model on each of the texts and from the scores, determine which of the two test texts was more likely written by Jane Austen. For testing use :\n",
    "\n",
    "- text a: ``austen-emma.txt`` (Jane Austen’s “Emma”)\n",
    "- text b: ``chesterton-ball.txt`` (G.K. Chesterton’s “The Ball and Cross”)\n",
    "\n",
    "and complete functions bellow in which you will:\n",
    "\n",
    "- Evaluate a trigram language model with a Lidstone probability distribution. \n",
    "- Compute total document cross-entropy for each text\n",
    "- Compute per word cross-entropy for each text\n",
    "\n",
    "Note:  The “f.B()+1” argument (already provided for you in the code) means that we lump together all the unseen n-grams as a single “unknown” token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def estimateLM(doc_name):\n",
    "    \"\"\"\n",
    "    type doc_name: string\n",
    "    param doc_name: name of the document in Gutenberg corpus\n",
    "    rtype: NgramModel\n",
    "    return: Lidstone smoothed language model with backoff\n",
    "    \"\"\"\n",
    "    # Construct a list of lowercase words from the document (training data for lm)\n",
    "    doc_words = [w.lower() for w in gutenberg.words(doc_name)]\n",
    "    # a Lidstone probability distribution with +0.01 added to the sample count for each bin\n",
    "    est = lambda fdist,bins:nltk.LidstoneProbDist(fdist,0.01,fdist.B()+1)\n",
    "    # Train a trigram language model with backoff using doc_words and    \n",
    "    lm = NgramModel(<order>,<word_list>,estimator=<estimator>)\n",
    "    # Return the language model\n",
    "    return lm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def document_xent(lm, doc_name):\n",
    "    \"\"\"\n",
    "    Use a language model to compute the total word-level cross-entropy of a document\n",
    "    \n",
    "    :type lm: NgramModel\n",
    "    :param lm: a language model\n",
    "    :type doc_name: str\n",
    "    :param doc_name: A gutenberg document name\n",
    "    :rtype: float\n",
    "    :return: The total entropy of the named document per the model\n",
    "    \"\"\"\n",
    "    # Construct a list of lowercase words from the document (test document)\n",
    "    doc_words = [w.lower() for w in gutenberg.words(doc_name)]\n",
    "    # Compute the total cross entropy of the text in doc_name\n",
    "    xent = 0\n",
    "    \n",
    "    return xent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perword_xent(lm, doc_name):\n",
    "    \"\"\"\n",
    "    Use a language model to compute the total average (per-word) word-level cross-entropy of a document\n",
    "    \n",
    "    :type lm: NgramModel\n",
    "    :param lm: a language model\n",
    "    :type doc_name: str\n",
    "    :param doc_name: A gutenberg document name\n",
    "    :rtype: float\n",
    "    :return: The total entropy of the named document per the model\n",
    "    \"\"\"\n",
    "    # Construct a list of lowercase words from the document (test document)\n",
    "    doc_words = [w.lower() for w in gutenberg.words(doc_name)]\n",
    "    # Compute the total cross entropy of the text in doc_name\n",
    "    xent = 0\n",
    "    \n",
    "    return xent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_doc = 'austen-sense.txt'\n",
    "test_a = 'austen-emma.txt'\n",
    "test_b = 'chesterton-ball.txt'\n",
    "lm = estimateLM(train_doc)\n",
    "\n",
    "print('Document {}:'.format(test_a))\n",
    "print('document xent: {} perword xent {}'.format(document_xent(lm, test_a), perword_xent(lm, test_a)))\n",
    "\n",
    "print('Document {}:'.format(test_b))\n",
    "print('document xent: {} perword xent {}'.format(document_xent(lm, test_b), perword_xent(lm, test_b)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##  Going further\n",
    "\n",
    "###  Padding\n",
    "\n",
    "Redo exercise 4 setting `pad_left` and `pad_right` to `True` both when initialising\n",
    "the n-gram model and when computing entropy. What difference does this\n",
    "make?\n",
    "\n",
    "### Sentences\n",
    "\n",
    "Using one enormous string of words as the training and test data is less than optimal, as it trains/tests across sentence boundaries.  Look back at the argument description for the `train` argument to `NgramModel` and see that it will actually train on an input which is a list of list of words, that is, a list of *sentences*, padding each sentence appropriately.  Redo exercise 4 training and testing on the sentences in the specified documents.\n",
    "\n",
    "### Case\n",
    "\n",
    "If we're training on sentences, maybe we shouldn't be down-casing?  Give it a try."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}