{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FNLP: Lab Session 3\n",
    "\n",
    "# Hidden Markov Models - Construction and Use\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the packages used for this lab\n",
    "\n",
    "import nltk\n",
    "\n",
    "# import brown corpus\n",
    "from nltk.corpus import brown\n",
    "\n",
    "# module for training a Hidden Markov Model and tagging sequences\n",
    "from nltk.tag.hmm import HiddenMarkovModelTagger\n",
    "\n",
    "# module for computing a Conditional Frequency Distribution\n",
    "from nltk.probability import ConditionalFreqDist\n",
    "\n",
    "# module for computing a Conditional Probability Distribution\n",
    "from nltk.probability import ConditionalProbDist\n",
    "\n",
    "# module for computing a probability distribution with the Maximum Likelihood Estimate\n",
    "from nltk.probability import MLEProbDist\n",
    "\n",
    "# pretty printing\n",
    "import pprint\n",
    "pp = pprint.PrettyPrinter(indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Corpora tagged with Part-of-Speech information\n",
    "\n",
    "NLTK provides corpora annotated with Part-of-Speech (POS) information and\n",
    "some tools to access this information. The Penn Treebank tagset is commonly\n",
    "used for annotating English sentences. We can inspect this tagset in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.help.upenn_tagset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Brown corpus provided with NLTK is also tagged with POS information,\n",
    "although the tagset is slightly different than the Penn Treebank tagset. Information about the Brown corpus tagset can be found here:\n",
    "http://www.scs.leeds.ac.uk/ccalas/tagsets/brown.html\n",
    "\n",
    "We can retrieve the tagged sentences in the Brown corpus by calling the `tagged_sents()`\n",
    "function and looking at an annotated sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagged_sentences = brown.tagged_sents(categories='news')\n",
    "print('Sentence tagged with Penn Treebank POS labels:')\n",
    "pp.pprint(tagged_sentences[29])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes it is useful to use a coarser label set in order to avoid data sparsity\n",
    "or to allow a mapping between the POS labels for different languages. The Universal tagset was designed to be applicable for all languages:\n",
    "\n",
    "https://github.com/slavpetrov/universal-pos-tags\n",
    "\n",
    "There are mappings between the POS tagset of several languages and the Universal tagset. We can access the Universal tags for the Brown corpus sentences\n",
    "by changing the tagset argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagged_sentences_universal = brown.tagged_sents(categories='news', tagset='universal')\n",
    "print('Sentence tagged with Universal POS:')\n",
    "pp.pprint(tagged_sentences_universal[29])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This initial universal tagset was later expanded as part of the Universal Dependencies project. The resulting tagset is called UPOS and you can find more information in the link below. This tagset is not yet supported by NLTK. However, it is important that you know about it since it is the most used multi-lingual tagset nowadays.\n",
    "\n",
    "https://universaldependencies.org/u/pos/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1: Exploring Brown corpus\n",
    "\n",
    "In this exercise we will explore the Brown corpus, specifically its frequency distribution over POS tags.\n",
    "The Brown corpus is divided in topical categories called 'genres'. Let's see what genres we have in the corpus.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.pprint(brown.categories())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You task in this exercise is to implement a function that computes the Frequency Distribution over a Brown genre category and a tagset scheme.\n",
    "The template of the function is given below. It takes two parameters: one is the genre category and the other is the tagset name.\n",
    "Your job is to do the following:\n",
    "\n",
    "1. Convert the list of (word,tag) pairs to a list of tags\n",
    "2. Use the list of tags to compute a frequency distribution over the tags. Use NLTK's `FreqDist()`\n",
    "3. Compute the total number of tags in the Frequency Distribution\n",
    "4. Return the total number of tags and the top 10 most frequent tags\n",
    "\n",
    "You are given the code to retrieve the list of (word, tag) tuples from the brown corpus corresponding to the given category and tagset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def explore_tagset_frequency_distro(genre, tagset):\n",
    "    \"\"\"Compute a Frequency distribution of the POS tags in a genre of the tagged Brown corpus\n",
    "    \n",
    "    :param genre: A Brown corpus genre\n",
    "    :type genre: str or iterable(str) or None\n",
    "    :param tagset: A Brown tagset name\n",
    "    :type tagset: str or None (defaults to 'brown')\n",
    "    :return: number of tag types, top 10 tags\n",
    "    :rtype: tuple(int,list(tuple(str,int))\"\"\"\n",
    "\n",
    "    # get the tagged words from the corpus\n",
    "    tagged_words = brown.tagged_words(categories=genre, tagset=tagset)\n",
    "\n",
    "    # TODO: convert tagged_words to a list of tags\n",
    "    # tags =\n",
    "\n",
    "    # TODO: using the above list compute a Frequency Distribution\n",
    "    # hint: use nltk.FreqDist()\n",
    "    # tagsFDist =\n",
    "\n",
    "    # TODO: retrieve the number of tag types in the tagset\n",
    "    # hint: help(nltk.FreqDist)\n",
    "    number_of_tags = 0\n",
    "\n",
    "    # TODO: retrieve the top 10 most frequent tags and their counts\n",
    "    top_tags = []\n",
    "\n",
    "    return number_of_tags, top_tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your code with this function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_ex1():\n",
    "    print('Tag FreqDist for news with Penn Treebank tagset:')\n",
    "    pp.pprint(explore_tagset_frequency_distro('news', None))\n",
    "\n",
    "    print('Tag FreqDist for science_fiction with Penn Treebank tagset:')\n",
    "    pp.pprint(explore_tagset_frequency_distro('science_fiction', None))\n",
    "\n",
    "    # Do the same thing for a different tagset: Universal\n",
    "\n",
    "    print('Tag FreqDist for news with Universal tagset:')\n",
    "    pp.pprint(explore_tagset_frequency_distro('news', 'universal'))\n",
    "\n",
    "    print('Tag FreqDist for science_fiction with Universal tagset:')\n",
    "    pp.pprint(explore_tagset_frequency_distro('science_fiction', 'universal'))\n",
    "\n",
    "test_ex1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the top tags for different genre and tagsets. Observe differences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training and Evaluating an HMM Tagger\n",
    "\n",
    "NLTK provides a module for training a Hidden Markov Model for sequence tagging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(nltk.tag.hmm.HiddenMarkovModelTagger)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can train the HMM for POS tagging given a labelled dataset. At the begging of this lab we learned how to access the labelled sentences of the Brown corpus.\n",
    "We will use this dataset to study the effect of the size of the training corpus on\n",
    "the accuracy of the tagger.\n",
    "\n",
    "### Exercise 2\n",
    "\n",
    "In this exercise we will train a HMM tagger on a training set and evaluate it\n",
    "on a test set. The template of the function that you have to implement takes\n",
    "two parameters: a sentence to be tagged and the size of the training corpus in\n",
    "number of sentences. You are given the code that creates the training and test\n",
    "datasets from the tagged sentences in the Brown corpus.\n",
    "\n",
    "1. Train a Hidden Markov Model tagger on the training dataset. Refer to `help(nltk.tag.hmm.HiddenMarkovModelTagger.train)` if necessary.\n",
    "2. Use the trained model to tag the sentence\n",
    "3. Use the trained model to evaluate the tagger on the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_trainer(sentence, size):\n",
    "    \"\"\" Create an HMM tagger from the Brown news corpus, and test it by tagging a sample sentence\n",
    "    \n",
    "    :param sentence: An untagged sentence as an example\n",
    "    :type sentence: list(str)\n",
    "    :param size: Number of sentences to train on (be sure to leave room for the test data)\n",
    "    :type size: int\n",
    "    :return: The tagger, the sample sentence with tags, entropy of model wrt 100 test sentences\n",
    "    :rtype: tuple(nltk.tag.hmm.HiddenMarkovModelTagger, list(tuple(str,str)), float)\"\"\"\n",
    "    tagged_sentences = brown.tagged_sents(categories='news')\n",
    "\n",
    "    # set up the training data\n",
    "    train_data = tagged_sentences[-size:]\n",
    "\n",
    "    # set up the test data\n",
    "    test_data = tagged_sentences[:100]\n",
    "    \n",
    "    # Hint: use help on HiddenMarkovModelTagger to find out how to train, tag and evaluate an HMM tagger\n",
    "\n",
    "    # TODO: train a HiddenMarkovModelTagger, using the train() method\n",
    "    tagger = None\n",
    "\n",
    "    # TODO: using the hmm tagger tag the sentence\n",
    "    hmm_tagged_sentence = []\n",
    "\n",
    "    # TODO: using the hmm tagger, evaluate accuracy score on the test data\n",
    "    acc = 0.0\n",
    "\n",
    "    return tagger, hmm_tagged_sentence, acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your implementation by running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_my_trainer():\n",
    "    tagged_sentences = brown.tagged_sents(categories='news')\n",
    "    words = [tp[0] for tp in tagged_sentences[42]]\n",
    "    tagger, hmm_tagged_sentence, acc = my_trainer(words, 500)\n",
    "    print('Training nltk.HiddenMarkovModelTagge with 500 sentences...')\n",
    "    print('\\tSentence tagged with model:')\n",
    "    pp.pprint(hmm_tagged_sentence)\n",
    "    print('\\tAccuracy score on the test set: %.4f%%' % (100.0*acc))\n",
    "    print()\n",
    "\n",
    "    tagger, hmm_tagged_sentence, acc = my_trainer(words, 3000)\n",
    "    print('Training nltk.HiddenMarkovModelTagge with 3000 sentences...')\n",
    "    print('\\tSentence tagged with model:')\n",
    "    pp.pprint(hmm_tagged_sentence)\n",
    "    print('\\tAccuracy score on the test set: %.4f%%' % (100.0*acc))\n",
    "\n",
    "test_my_trainer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the tagged sentence and the accuracy of the tagger. How does the size of the training set affect the accuracy?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computing the Transition and Emission Probabilities\n",
    "\n",
    "In the previous exercise we learned how to train and evaluate an HMM tagger.\n",
    "We have used the HMM tagger as a black box and have seen how the training\n",
    "data affects the accuracy of the tagger. In order to get a better understanding\n",
    "of the HMM we will look at the two components of this model:\n",
    "    \n",
    "* The transition model\n",
    "* The emission model\n",
    "\n",
    "The transition model estimates $P (tag_{i+1} |tag_i )$, the probability of a POS tag\n",
    "at position $i+1$ given the previous tag (at position $i$). The emission model\n",
    "estimates $P (word|tag)$, the probability of the observed word given a tag.\n",
    "\n",
    "Given the above definitions, we will need to learn a Conditional Probability\n",
    "Distribution for each of the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(nltk.probability.ConditionalProbDist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3: Emission Model\n",
    "\n",
    "In this exercise we will estimate the emission model. In order to compute the\n",
    "Conditional Probability Distribution of $P (word|tag)$ we first have to compute\n",
    "the Conditional Frequency Distribution of a word given a tag.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(nltk.probability.ConditionalFreqDist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The constructor of the ConditionalFreqDist class takes as input a list of tuples,\n",
    "each tuple consisting of a condition and an observation. For the emission model,\n",
    "the conditions are tags and the observations are the words. The template of the\n",
    "function that you have to implement takes as argument the list of tagged words\n",
    "from the Brown corpus.\n",
    "\n",
    "1. Build the dataset to be passed to the `ConditionalFreqDist()` constructor. Words should be lowercased. Each item of data should be a tuple of tag (a condition) and word (an observation).\n",
    "2. Compute the Conditional Frequency Distribution of words given tags.\n",
    "3. Return the top 10 most frequent words given the tag NN.\n",
    "4. Compute the Conditional Probability Distribution for the above Conditional Frequency Distribution. Use the `MLEProbDist` estimator when calling the ConditionalProbDist constructor.\n",
    "5. Compute the probabilities:\n",
    "\n",
    " $P(\\text{year}|\\text{NN})$ \n",
    " \n",
    " $P(\\text{year}|\\text{DT})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_emission_model(tagged_words):\n",
    "    \"\"\"Build and sample Conditional{Freq->Prob}Dist for word given tag from a list of tagged words using MLE\n",
    "    \n",
    "    :param tagged_words: tagged words (word,tag)\n",
    "    :type tagged_words: list(tuple(str,str))\n",
    "    :return: Conditional Freq dist of word given tag, top 10 words with tag NN,\n",
    "             Conditional Prob dist of word given tag, P('year'|'NN'), P('year'|'DT')\n",
    "    :rtype: tuple(nltk.probability.ConditionalFreqDist,list(tuple(str,int)),nltk.probability.ConditionalProbDist,float,float)\"\"\"\n",
    "    \n",
    "    # in the previous labs we've seen how to build a freq dist\n",
    "    # we need conditional distributions to estimate the transition and emission models\n",
    "    # in this exercise we estimate the emission model\n",
    "    \n",
    "    # TODO: prepare the data\n",
    "    # the data object should be a list of tuples of conditions and observations\n",
    "    # in our case the tuples should be of the form (tag,word) where words are lowercased\n",
    "    # data =\n",
    "\n",
    "    # TODO: compute a Conditional Frequency Distribution for words given their tags using our data\n",
    "    emission_FD = None\n",
    "\n",
    "    # TODO: find the top 10 most frequent words given the tag NN\n",
    "    top_NN = ''\n",
    "\n",
    "    # TODO: Compute the Conditional Probability Distribution using the above Conditional Frequency Distribution. \n",
    "    #       Use nltk.probability.MLEProbDist estimator.\n",
    "    emission_PD = None\n",
    "\n",
    "    # TODO: compute the probabilities of P(year|NN) and P(year|DT)\n",
    "    p_NN = 0.0\n",
    "    p_DT = 0.0\n",
    "\n",
    "    return emission_FD, top_NN, emission_PD, p_NN, p_DT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your implementation by running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_emission_model():\n",
    "    tagged_words = brown.tagged_words(categories='news')\n",
    "    (emission_FD, top_NN, emission_PD, p_NN, p_DT) = my_emission_model(tagged_words)\n",
    "    print('Frequency of words given the tag *NN*: ')\n",
    "    pp.pprint(top_NN)\n",
    "    print('P(year|NN) = ', p_NN)\n",
    "    print('P(year|DT) = ', p_DT)\n",
    "\n",
    "test_emission_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the estimated probabilities. Why is P(year|DT) = 0 ? \n",
    "\n",
    "What are the problems with having zero (0) probabilities and what can be done to\n",
    "avoid this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4: Transition Model\n",
    "\n",
    "In this exercise we will estimate the transition model. In order to compute the\n",
    "Conditional Probability Distribution of $P (tag_{i+1} |tag_i )$ we first have to compute\n",
    "the Conditional Frequency Distribution of a tag at position $i + 1$ given the previous tag.\n",
    "\n",
    "The constructor of the `ConditionalFreqDist` class takes as input a list of tuples, each tuple consisting of a condition and an observation. For the transition\n",
    "model, the conditions are tags at position i and the observations are tags at\n",
    "position $i + 1$. The template of the function that you have to implement takes\n",
    "as argument the list of tagged sentences from the Brown corpus.\n",
    "\n",
    "1. Build the dataset to be passed to the `ConditionalFreqDist()` constructor. Each item in your data should be a pair of condition and observation: $(tag_i,tag_{i+1})$\n",
    "2. Compute the Conditional Frequency Distribution of a tag at position $i + 1$ given the previous tag.\n",
    "3. Compute the Conditional Probability Distribution for the above Conditional Frequency Distribution. Use the `MLEProbDist` estimator when calling the `ConditionalProbDist` constructor.\n",
    "4. Compute the probabilities \n",
    "   \n",
    "   $P(\\text{NN}|\\text{VBD})$ \n",
    "   \n",
    "   $P(\\text{NN}|\\text{DT})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_transition_model(tagged_sentences):\n",
    "    \"\"\"Build and sample Conditional{Freq->Prob}Dist for tag given preceding tag from a list of tagged words using MLE\n",
    "    \n",
    "    :param tagged_sentences: Tagged sentences for training and testing\n",
    "    :type tagged_sentences: list(list(tuple(str,str)))\n",
    "    :return: Conditional Freq dist of tag given preceding tag,\n",
    "             Conditional Prob dist of tag given preceding tag, P('NN'|'VBD') and P('NN'|'DT')\n",
    "    :rtype: tuple(nltk.probability.ConditionalFreqDist,nltk.probability.ConditionalProbDist,float,float)\"\"\"\n",
    "    \n",
    "    # TODO: prepare the data\n",
    "    # the data object should be an array of tuples of conditions and observations\n",
    "    # in our case the tuples will be of the form (tag_(i),tag_(i+1))\n",
    "    # data =\n",
    "\n",
    "    # TODO: compute a Conditional Frequency Distribution for a tag given the previous tag\n",
    "    transition_FD = None\n",
    "\n",
    "    # TODO: compute a Conditional Probability Distribution for the\n",
    "    # transition probability P(tag_(i+1)|tag_(i)) using an MLEProbDist\n",
    "    # to estimate the probabilities\n",
    "    transition_PD = None\n",
    "\n",
    "    # TODO: compute the probabilities of P('NN'|'VBD') and P('NN'|'DT')\n",
    "    p_VBD_NN = 0.0\n",
    "    p_DT_NN = 0.0\n",
    "\n",
    "    return transition_FD, transition_PD, p_VBD_NN, p_DT_NN\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your implementation by running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_transition_model():\n",
    "    tagged_sentences = brown.tagged_sents(categories='news')\n",
    "    (transition_FD, transition_PD, p_VBD_NN, p_DT_NN) = my_transition_model(tagged_sentences)\n",
    "    print('P(NN|VBD) = ', p_VBD_NN)\n",
    "    print('P(NN|DT) = ', p_DT_NN)\n",
    "    \n",
    "test_transition_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are the results what you would expect? The sequence NN DT seems very probable. \n",
    "\n",
    "How will this affect the sequence tagging?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Going further\n",
    "\n",
    "Modify your code for exercise 3 to use a different estimator, to introduce some\n",
    "smoothing, and compare the results with the original.\n",
    "In exercise 4 we didn’t do anything about the boundaries. Modify your code for\n",
    "exercise 4 to use `<s>` at the beginning of every sentence and `</s>` at the end.\n",
    "\n",
    "Explore the resulting conditional probabilities. What is the most likely tag at\n",
    "the beginning of a sentence? At the end?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
