Parts of Speech Tagging with NLTK

0 0

In corpus linguistics, POS Tagging (Parts of Speech Tagging) also called grammaticaltaggingis a process of marking up words in text format for a particular part of a speech based on its definition and context. It is responsible for text reading in a language and assigning some specific token (Parts of Speech) to each word. POS-tagging algorithms fall into two distinctive groups: rule-based and stochastic. E. Brill’s tagger, one of the first and most widely used English POS-taggers, employs rule-based algorithms. Parts of speech tagging can be important for syntactic and semantic analysis.

Rule Based POS Tagging

One of the oldest techniques of tagging is rule-based POS tagging. Rule-based taggers use a dictionary or lexicon for getting possible tags for each word. If the word has more than one possible tag, then rule-based taggers use hand-written rules to identify the correct tag. Disambiguation can also be performed in rule-based tagging by analyzing the linguistic features of a word along with its preceding as well as following words. For example, suppose if the preceding word of a word is an article then word must be a noun.

Stochastic POS Tagging

Another technique of tagging is Stochastic POS Tagging. Now, the question that arises here is which model can be stochastic. The model that includes frequency or probability (statistics) can be called stochastic. Any number of different approaches to the problem of part-of-speech tagging can be referred to as stochastic tagger. The simplest stochastic tagger applies the following approaches for POS tagging:

Word Frequency Approach

In this approach, the stochastic taggers disambiguate the words based on the probability that a word occurs with a particular tag. We can also say that the tag encountered most frequently with the word in the training set is the one assigned to an ambiguous instance of that word.

Tag Sequence Probalities

It is another approach of stochastic tagging, where the tagger calculates the probability of a given sequence of tags occurring. It is also called n-gram approach. It is called so because the best tag for a given word is determined by the probability at which it occurs with the n previous tags.

Part-of-speech tagging is harder than just having a list of words and their parts of speech, because some words can represent more than one part of speech at different times, and because some parts of speech are complex or unspoken, a large percentage of word-forms are ambiguous. For example, even “dogs”, which is usually thought of as just a plural noun, can also be a verb:

The sailor dogs the hatch

Correct grammatical tagging will reflect that “dogs” is here used as a verb, not as the more common plural noun. Grammatical context is one way to determine this; semantic analysis can also be used to infer that “sailor” and “hatch” implicate “dogs” as 1) in the nautical context and 2) an action applied to the object “hatch” (in this context, “dogs” is a nautical term meaning “fastens (a watertight door securely.

So, for something like the sentence above the word can has several semantic meanings. One being a model for question formation, another being a container for holding food or liquid, and yet another being a verb denoting the ability to do something.

Let’s learn with a NLTK Part of Speech example:

POS tag list:

CC coordinating conjunction

CD cardinal digit

DT determiner

IN preposition/subordinating conjunction

JJ adjective ‘big’

JJR adjective, comparative ‘bigger’

JJS adjective, superlative ‘biggest’

MD modal could, will

NN noun, singular ‘desk’

NNS noun plural ‘desks’

NNP proper noun, singular ‘Harrison’

NNPS proper noun, plural ‘Americans’

PRP personal pronoun I, he, she

PRP$ possessive pronoun my, his, hers

RB adverb very, silently,

RBR adverb, comparative better

RBS adverb, superlative best

UH interjection errrrrrrrm

VB verb, base form take

WRB wh-abverb where, when

Why Part of Speech Tagging

Part-of-Speech tagging in itself may not be the solution to any particular NLP problem. It is however something that is done as a prerequisite to simplify a lot of different problems. Let us consider a few applications of POS tagging in various NLP tasks.

As you can see on line 5 of the code above, the .pos_tag() function needs to be passed a tokenized sentence for tagging. The tagging is done by way of a trained model in the NLTK library. The included POS tagger is not perfect but it does yield pretty accurate results.

Import NLTK
token = nltk.word_tokenize ("Can you please buy me an Arizona Ice Tea? It's $0.99.")
print("part of speech: ", nltk.pos_tag(tokens))

[(‘Can’, ‘MD’), (‘you’, ‘PRP’), (‘please’, ‘VB’), (‘buy’, ‘VB’), (‘me’, ‘PRP’), (‘an’, ‘DT’), (‘Arizona’, ‘NNP’), (‘Ice’, ‘NNP’), (‘Tea’, ‘NNP’), (‘?’, ‘.’), (‘It’, ‘PRP’), (“‘s”, ‘VBZ’), (‘$’, ‘$’), (‘0.99’, ‘CD’), (‘.’, ‘.’)]

Parts of speech tagging can be important for syntactic and semantic analysis. So, for something like the sentence above the word can has several semantic meanings. One being a model for question formation, another being a container for holding food or liquid, and yet another being a verb denoting the ability to do something. Giving a word such as this a specific meaning allows for the program to handle it in the correct manner in both semantic and syntactic analyses.

Part-of-speech Tagset for Arabic

Modern Standard Arabic grammar defines a detailed part-of-speech hierarchy which is applied to words and morphological segments both. Fundamentally, a word may be classified as a nominal ism (اسم), verb fiʿil (فعل) or a particle ḥarf (حرف). The set of nominals include nouns, pronouns, adjectives and adverbs. The particles include prepositions, conjunctions and interrogatives, as well as many others. Morphological annotation in the Arabic corpus divides words into multiple segments. Each segment is assigned a part-of-speech tag. These tags are detailed in the following sections. In addition to part-of-speech tags, each segment is annotated using a set of multiple morphological features.

About Post Author

Chakir Mahjoubi

https://lexsense.net

Happy

0 0 %

Sad

0 0 %

Excited

0 0 %

Sleepy

0 0 %

Angry

0 0 %

Surprise

0 0 %