Sentiment analysis helps you extract an author's feelings towards a subject. This exercise will give you a taste of what's to come!

We created text_df representing a conversation with person and text columns.

Use qdap's polarity() function to score text_df. polarity() will accept a single character object or data frame with a grouping variable to calculate a positive or negative score.

In this example you will use the magrittr package's dollar pipe operator %$%. The dollar sign forwards the data frame into polarity() and you declare a text column name or the text column and a grouping variable without quotes.

text_data_frame %$% polarity(text_column_name)

To create an object with the dollar sign operator:

polarity_object <- text_data_frame %$% 
  polarity(text_column_name, grouping_column_name)

More specifically, to make a quantitative judgement about the sentiment of some text, you need to give it a score. A simple method is a positive or negative value related to a sentence, passage or a collection of documents called a corpus. Scoring with positive or negative values only is called "polarity." A useful function for extracting polarity scores is counts() applied to the polarity object. For a quick visual call plot() on the polarity() outcome.

In the Text Mining: Bag of Words course you learned that a corpus is a set of texts, and you studied some functions for preprocessing the text. To recap, one way to create & clean a corpus is with the functions below. Even though this is a different course, sentiment analysis is part of text mining so a refresher can be helpful.

• Turn a character vector into a text source using VectorSource().
• Turn a text source into a corpus using VCorpus().
• Remove unwanted characters from the corpus using cleaning functions like removePunctuation() and stripWhitespace() from tm, and replace_abbreviation() from qdap.

In this exercise a custom clean_corpus() function has been created using standard preprocessing functions for easier application.

clean_corpus() accepts the output of VCorpus() and applies cleaning functions. For example:

processed_corpus <- clean_corpus(my_corpus)

Now let's create a Document Term Matrix (DTM). In a DTM:

• Each row of the matrix represents a document.
• Each column is a unique word token.
• Values of the matrix correspond to an individual document's word usage.

The DTM is the basis for many bag of words analyses. Later in the course, you will also use the related Term Document Matrix (TDM). This is the transpose; that is, columns represent documents and rows represent unique word tokens.

You should construct a DTM after cleaning the corpus (using clean_corpus()). To do so, call DocumentTermMatrix() on the corpus object.

tm_dtm <- DocumentTermMatrix(tm_clean)

If you need a more in-depth refresher check out the Text Mining with Bag-of-Words in R course. Hopefully these two exercises have prepared you well enough to embark on your sentiment analysis journey!

Be aware that this is real data from Twitter and as such there is always a risk that it may contain profanity or other offensive content (in this exercise, and any following exercises that also use real Twitter data).

As discussed in the video a lexicon is a list of words. What is the purpose of a subjectivity lexicon?

Although Zipf observed a steep and predictable decline in word usage you may not buy into Zipf's law. You may be thinking "I know plenty of words, and have a distinctive vocabulary". That may be the case, but the same can't be said for most people! To prove it, let's construct a visual from 3 million tweets mentioning "#sb". Keep in mind that the visual doesn't follow Zipf's law perfectly, the tweets all mentioned the same hashtag so it is a bit skewed. That said, the visual you will make follows a steep decline showing a small lexical diversity among the millions of tweets. So there is some science behind using lexicons for natural language analysis!

In this exercise, you will use the package metricsgraphics. Although the author suggests using the pipe %>% operator, you will construct the graphic step-by-step to learn about the various aspects of the plot. The main function of the package metricsgraphics is the mjs_plot() function which is the first step in creating a JavaScript plot. Once you have that, you can add other layers on top of the plot.

An example metricsgraphics workflow without using the %>% operator is below:

metro_plot <- mjs_plot(data, x = x_axis_name, y = y_axis_name, show_rollover_text = FALSE)
metro_plot <- mjs_line(metro_plot)
metro_plot <- mjs_add_line(metro_plot, line_one_values)
metro_plot <- mjs_add_legend(metro_plot, legend = c('names', 'more_names'))
metro_plot

So far you have learned the basic components needed for assessing positive or negative intent in text. Remember the following points so you can feel confident in your results.

• The subjectivity lexicon is a predefined list of words associated with emotions or positive/negative feelings.
• You don't have to list every word in a subjectivity lexicon because Zipf's law describes human expression.

A quick way to get started is to use the polarity() function which has a built-in subjectivity lexicon.

The function scans the text to identify words in the lexicon. It then creates a cluster around an identified subjectivity word. Within the cluster valence shifters adjust the score. Valence shifters are words that amplify or negate the emotional intent of the subjectivity word. For example, "well known" is positive while "not well known" is negative. Here "not" is a negating term and reverses the emotional intent of "well known." In contrast, "very well known" employs an amplifier increasing the positive intent.

The polarity() function then calculates a score using subjectivity terms, valence shifters and the total number of words in the passage. This exercise demonstrates a simple polarity calculation. In the next video we look under the hood of polarity() for more detail.

Of course just positive and negative words aren't enough. In this exercise you will learn about valence shifters which tell you about the author's emotional intent. Previously you applied polarity() to text without valence shifters. In this example you will see amplification and negation words in action.

Recall that an amplifying word adds 0.8 to a positive word in polarity() so the positive score becomes 1.8. For negative words 0.8 is subtracted so the total becomes -1.8. Then the score is divided by the square root of the total number of words.

Consider the following example from Frank Sinatra:

• "It was a very good year"

"Good" equals 1 and "very" adds another 0.8. So, 1.8/sqrt(6) results in 0.73 polarity.

A negating word such as "not" will inverse the subjectivity score. Consider the following example from Bobby McFerrin:

• "Don't worry Be Happy"

"worry is now 1 due to the negation "don't." Adding the "happy", +1, equals 2. With 4 total words, 2 / sqrt(4) equals a polarity score of 1.

Even with Zipf's law in action, you will still need to adjust lexicons to fit the text source (for example twitter versus legal documents) or the author's demographics (teenager versus the elderly). This exercise demonstrates the explicit components of polarity() so you can change it if needed.

In Trey Songz "Lol :)" song there is a lyric "LOL smiley face, LOL smiley face." In the basic polarity() function, "LOL" is not defined as positive. However, "LOL" stands for "Laugh Out Loud" and should be positive. As a result, you should adjust the lexicon to fit the text's context which includes pop-culture slang. If your analysis contains text from a specific channel (Twitter's "LOL"), location (Boston's "Wicked Good"), or age group (teenagers' "sick") you will likely have to adjust the lexicon.

In this exercise you are not adjusting the subjectivity lexicon or qdap dictionaries containing valence shifters. Instead you are examining the existing word data frame objects so you can change them in the following exercise.

We've created text containing two excerpts from Beyoncé's "Crazy in Love" lyrics for the exercise.

Here you will adjust the negative words to account for the specific text. You will then compare the basic and custom polarity() scores.

A popular song from Twenty One Pilots is called "Stressed Out". If you scan the song lyrics, you will observe the song is about youthful nostalgia. Overall, most people would say the polarity is negative. Repeatedly the lyrics mention stress, fears and pretending.

Let's compare the song lyrics using the default subjectivity lexicon and also a custom one.

To start, you need to verify the key.pol subjectivity lexicon does not already have the term you want to add. One way to check is with grep(). The grep() function returns the row containing characters that match a search pattern. Here is an example used while indexing.

data_frame[grep("search_pattern", data_frame$column), ]

After verifying the slang or new word is not already in the key.pol lexicon you need to add it. The code below uses sentiment_frame() to construct the new lexicon. Within the code sentiment_frame() accepts the original positive word vector, positive.words. Next, the original negative.words are concatenated to "smh" and "kappa", both considered negative slang. Although you can declare the positive and negative weights, the default is 1 and -1 so they are not included below.

custom_pol <- sentiment_frame(positive.words, c(negative.words, "hate", "pain"))

Now you are ready to apply polarity and it will reference the custom subjectivity lexicon!

What is the philosophical basis for the Plutchik's wheel of emotion?

The tidyverse is a collection of R packages that share common philosophies and are designed to work together. This chapter covers some tidy functions to manipulate data. In this exercise you will compare a DTM to a tidy text data frame called a tibble.

Within the tidyverse, each observation is a single row in a data frame. That makes working in different packages much easier since the fundamental data structure is the same. Parts of this course borrow heavily from the tidytext package which uses this data organization.

For example, you may already be familiar with the %>% operator from the magrittr package. This forwards an object on its left-hand side as the first argument of the function on its right-hand side.

In the example below, you are forwarding the data object to function1(). Notice how the parentheses are empty. This in turn is forwarded to function2(). In the last function you don't have to add the data object because it was forwarded from the output of function1(). However, you do add a fictitious parameter, some_parameter as TRUE. These pipe forwards ultimately create the object.

object <- data %>% 
           function1() %>%
           function2(some_parameter = TRUE)

To use the %>% operator, you don't necessarily need to load the magrittr package, since it is also available in the dplyr package. dplyr also contains the functions inner_join() (which you'll learn more about later) and count() for tallying data. The last function you'll need is mutate() to create new variables or modify existing ones.

object <- data %>%
  mutate(new_Var_name = Var1 - Var2)

or to modify a variable

object <- data %>%
  mutate(Var1 = as.factor(Var1))

You will also use tidyr's spread() function to organize the data with each row being a line from the book and the positive and negative values as columns.

To change a DTM to a tidy format use tidy() from the broom package.

tidy_format <- tidy(Document_Term_Matrix)

This exercise uses text from the Greek tragedy, Agamemnon. Agamemnon is a story about marital infidelity and murder. You can download a copy here.

So far you have used a single lexicon. Now we will transition to using three, each measuring sentiment in different ways.

The tidytext package contains a function called get_sentiments which along with the [textdata] package allows you to download & interact well researched lexicons. Here is a small section of the loughran lexicon.

This lexicon contains 4150 terms with corresponding information. We will be exploring other lexicons but the structure & method to get them is similar.

Let's use tidytext with textdata to explore other lexicons' word labels!

Now that you understand the basics of an inner join, let's apply this to the "Bing" lexicon. Keep in mind the inner_join() function comes from dplyr and the lexicon object is obtained using tidytext's get_sentiments() function'.

The Bing lexicon labels words as positive or negative. The next three exercises let you interact with this specific lexicon. To use get_sentiments() pass in a string such as "afinn", "bing", "nrc", or "loughran" to download the specific lexicon.

The inner join workflow:

• Obtain the correct lexicon using get_sentiments().
• Pass the lexicon and the tidy text data to inner_join().
• In order for inner_join() to work there must be a shared column name. If there are no shared column names, declare them with an additional parameter, by equal to c with column names like below.

object <- x %>% 
    inner_join(y, by = c("column_from_x" = "column_from_y"))

• Perform some aggregation and analysis on the table intersection.

In this exercise you will apply another inner_join() using the "bing" lexicon.

Then you will manipulate the results with both count() from dplyr and spread() from tidyr to learn about the text.

The spread() function spreads a key-value pair across multiple columns. In this case the key is the sentiment & the values are the frequency of positive or negative terms for each line. Using spread() changes the data so that each row now has positive and negative values, even if it is 0.

The last Bing lexicon exercise! In this exercise you will use the pipe operator (%>%) to create a timeline of the sentiment in Moby Dick. In the end you will also create a simple visual following the code structure below. The next chapter goes into more depth for visuals.

ggplot(spread_data, aes(index_column, polarity_column)) +
  geom_smooth()

Now we transition to the AFINN lexicon. The AFINN lexicon has numeric values from 5 to -5, not just positive or negative. Unlike the Bing lexicon's sentiment, the AFINN lexicon's sentiment score column is called value.

As before, you apply inner_join() then count(). Next, to sum the scores of each line, we use dplyr's group_by() and summarize() functions. The group_by() function takes an existing data frame and converts it into a grouped data frame where operations are performed "by group". Then, the summarize() function lets you calculate a value for each group in your data frame using a function that aggregates data, like sum() or mean(). So, in our case we can do something like

data_frame %>% 
    group_by(book_line) %>% 
    summarize(total_value = sum(book_line))

In the tidy version of Huckleberry Finn, line 9703 contains words "best", "ever", "fun", "life" and "spirit". "best" and "fun" have AFINN scores of 3 and 4 respectively. After aggregating, line 9703 will have a total score of 7.

In the tidyverse, filter() is preferred to subset() because it combines the functionality of subset() with simpler syntax. Here is an example that filter()s data_frame where some value in column1 is equal to 24. Notice the column name is not in quotes.

filter(data_frame, column1 == 24)

The afinn object contains the AFINN lexicon. The huck object is a tidy version of Mark Twain's Adventures of Huckleberry Finn for analysis.

Line 5400 is All the loafers looked glad; I reckoned they was used to having fun out of Boggs. Stopwords and punctuation have already been removed in the dataset.

Last but not least, you get to work with the NRC lexicon which labels words across multiple emotional states. Remember Plutchik's wheel of emotion? The NRC lexicon tags words according to Plutchik's 8 emotions plus positive/negative.

In this exercise there is a new operator, %in%, which matches a vector to another. In the code below %in% will return FALSE, FALSE, TRUE. This is because within some_vec, 1 and 2 are not found within some_other_vector but 3 is found and returns TRUE. The %in% is useful to find matches.

some_vec <- c(1, 2, 3)
some_other_vector <- c(3, "a", "b")
some_vec %in% some_other_vector

Another new operator is !. For logical conditions, adding ! will inverse the result. In the above example, the FALSE, FALSE, TRUE will become TRUE, TRUE, FALSE. Using it in concert with %in% will inverse the response and is good for removing items that are matched.

!some_vec %in% some_other_vector

We've created oz which is the tidy version of The Wizard of Oz along with nrc containing the "NRC" lexicon with renamed columns.

You are given a stack of 10 employee surveys and told to figure out the team's sentiment. The two question survey has 1 question with a numeric scale (1-10) where employees answer how inspired they are at work and a second question for free form text.

You are asked to perform a sentiment analysis on the free form text. Would performing sentiment analysis on the text be appropriate?

Sometimes you want to track sentiment over time. For example, during an ad campaign you could track brand sentiment to see the campaign's effect. You saw a few examples of this at the end of the last chapter.

In this exercise you'll recap the workflow for exploring sentiment over time using the novel Moby Dick. One should expect that happy moments in the book would have more positive words than negative. Conversely dark moments and sad endings should use more negative language. You'll also see some tricks to make your sentiment time series more visually appealing.

Recall that the workflow is:

1. Inner join the text to the lexicon by word.
2. Count the sentiments by line.
3. Reshape the data so each sentiment has its own column.
4. (Depending upon the lexicon) Calculate the polarity as positive score minus negative score.
5. Draw the polarity time series.

This exercise should look familiar: it extends Bing tidy polarity: Call me Ishmael (with ggplot2)!.

One of the easiest ways to explore data is with a frequency analysis. Although not difficult, in sentiment analysis this simple method can be surprisingly illuminating. Specifically, you will build a barplot. In this exercise you are once again working with moby and bing to construct your visual.

To get the bars ordered from lowest to highest, you will use a trick with factors. reorder() lets you change the order of factor levels based upon another scoring variable. In this case, you will reorder the factor variable term by the scoring variable polarity.

Now that you have seen how polarity can be used to divide a corpus, let's do it! This code will walk you through dividing a corpus based on sentiment so you can peer into the information in subsets instead of holistically.

Your R session has oz_pol which was created by applying polarity() to "The Wonderful Wizard of Oz."

For simplicity's sake, we created a simple custom function called pol_subsections() which will divide the corpus by polarity score. First, the function accepts a data frame with each row being a sentence or document of the corpus. The data frame is subset anywhere the polarity values are greater than or less than 0. Finally, the positive and negative sentences, non-zero polarities, are pasted with parameter collapse so that the terms are grouped into a single corpus. Lastly, the two documents are concatenated into a single vector of two distinct documents.

pol_subsections <- function(df) {
  x.pos <- subset(df\$text, df\$polarity > 0)
  x.neg <- subset(df\$text, df\$polarity < 0)
  x.pos <- paste(x.pos, collapse = " ")
  x.neg <- paste(x.neg, collapse = " ")
  all.terms <- c(x.pos, x.neg)
  return(all.terms)
}

At this point you have omitted the neutral sentences and want to focus on organizing the remaining text. In this exercise we use the %>% operator again to forward objects to functions. After some simple cleaning use comparison.cloud() to make the visual.

In this exercise you go beyond subsetting on positive and negative language. Instead you will subset text by each of the 8 emotions in Plutchik's emotional wheel to construct a visual. With this approach you will get more clarity in word usage by mapping to a specific emotion instead of just positive or negative.

Using the tidytext subjectivity lexicon, "nrc", you perform an inner_join() with your text. The "nrc" lexicon has the 8 emotions plus positive and negative term classes. So you will have to drop positive and negative words after performing your inner_join(). One way to do so is with the negation, !, and grepl().

The "Global Regular Expression Print Logical" function, grepl(), will return a True or False if a string pattern is identified in each row. In this exercise you will search for positive OR negative using the | operator, representing "or" as shown below. Often this straight line is above the enter key on a keyboard. Since the ! negation precedes grepl(), the T or F is switched so the "positive|negative" is dropped instead of kept.

Object <- tibble %>%
  filter(!grepl("positive|negative", column_name))

Next you apply count() on the identified words along with spread() to get the data frame organized.

comparison.cloud() requires its input to have row names, so you'll have to convert it to a base-R data.frame, calling data.frame() with the row.names argument.

Another way to slice your text is to understand how much of the document(s) are made of positive or negative words. For example a restaurant review may have some positive aspects such as "the food was good" but then continue to add "the restaurant was dirty, the staff was rude and parking was awful." As a result, you may want to understand how much of a document is dedicated to positive vs negative language. In this example it would have a higher negative percentage compared to positive.

One method for doing so is to count() the positive and negative words then divide by the number of subjectivity words identified. In the restaurant review example, "good" would count as 1 positive and "dirty," "rude," and "awful" count as 3 negative terms. A simple calculation would lead you to believe the restaurant review is 25% positive and 75% negative since there were 4 subjectivity terms.

Start by performing the inner_join() on a unified tidy data frame containing 4 books, Agamemnon, Oz, Huck Finn, and Moby Dick. Just like the previous exercise you will use filter() and grepl().

To perform the count() you have to group the data by book and then sentiment. For example all the positive words for Agamemnon have to be grouped then tallied so that positive words from all books are not mixed. Luckily, you can pass multiple variables into count() directly.

Now that you learned about a kernel density plot you can create one! Remember it's like a smoothed histogram but isn't affected by binwidth. This exercise will help you construct a kernel density plot from sentiment values.

In this exercise you will plot 2 kernel densities. One for Agamemnon and another for The Wizard of Oz. For both you will perform an inner_join() with the "afinn" lexicon. Recall the "afinn" lexicon has terms scored from -5 to 5. Once in a tidy format, both books will retain words and corresponding scores for the lexicon.

After that, you need to row bind the results into a larger data frame using bind_rows() and create a plot with ggplot2.

From the visual you will be able to understand which book uses more positive versus negative language. There is clearly overlap as negative things happen to Dorothy but you could infer the kernel density is demonstrating a greater probability of positive language in the Wizard of Oz compared to Agamemnon.

We've loaded ag and oz as tidy versions of Agamemnon and The Wizard of Oz respectively, and created afinn as a subset of the tidytext "afinn" lexicon.

An easy way to compare multiple distributions is with a box plot. This code will help you construct multiple box plots to make a compact visual.

In this exercise the all_book_polarity object is already loaded. The data frame contains two columns, book and polarity. It comprises all books with qdap's polarity() function applied. Here are the first 3 rows of the large object.

This exercise introduces tapply() which allows you to apply functions over a ragged array. You input a vector of values and then a vector of factors. For each factor, value combination the third parameter, a function like min(), is applied. For example here's some code with tapply() used on two vectors.

f1 <- as.factor(c("Group1", "Group2", "Group1", "Group2"))
stat1 <- c(1, 2, 1, 2)
tapply(stat1, f1, sum)

The result is an array where Group1 has a value of 2 (1+1) and Group2 has a value of 4 (2+2).

Remember Plutchik's wheel of emotion? The NRC lexicon has the 8 emotions corresponding to the first ring of the wheel. Previously you created a comparison.cloud() according to the 8 primary emotions. Now you will create a radar chart similar to the wheel in this exercise.

A radarchart is a two-dimensional representation of multidimensional data (at least 3). In this case the tally of the different emotions for a book are represented in the chart. Using a radar chart, you can review all 8 emotions simultaneously.

As before we've loaded the "nrc" lexicon as nrc and moby_huck which is a combined tidy version of both Moby Dick and Huck Finn.

In this exercise you once again use a negated grepl() to remove "positive|negative" emotional classes from the chart. As a refresher here is an example:

object <- tibble %>%
  filter(!grepl("positive|negative", column_name))

This exercise reintroduces spread() which rearranges the tallied emotional words. As a refresher consider this raw data called datacamp.

If you applied spread() as in spread(datacamp, people, like) the data looks like this.

Often you will find yourself working with documents in groups, such as author, product or by company. This exercise lets you learn about the text while retaining the groups in a compact visual. For example, with customer reviews grouped by product you may want to explore multiple dimensions of the customer reviews at the same time. First you could calculate the polarity() of the reviews. Another dimension may be length. Document length can demonstrate the emotional intensity. If a customer leaves a short "great shoes!" one could infer they are actually less enthusiastic compared to a lengthier positive review. You may also want to group reviews by product type such as women's, men's and children's shoes. A treemap lets you examine all of these dimensions.

For text analysis, within a treemap each individual box represents a document such as a tweet. Documents are grouped in some manner such as author. The size of each box is determined by a numeric value such as number of words or letters. The individual colors are determined by a sentiment score.

After you organize the tibble, you use the treemap library containing the function treemap() to make the visual. The code example below declares the data, grouping variables, size, color and other aesthetics.

treemap(
  data_frame,
  index = c("group", "individual_document"),
  vSize = "doc_length",
  vColor = "avg_score",
  type = "value",
  title = "Sentiment Scores by Doc",
  palette = c("red", "white", "green")
)

The pre-loaded all_books object contains a combined tidy format corpus with 4 Shakespeare, 3 Melville and 4 Twain books. Based on the treemap you should be able to tell who writes longer books, and the polarity of the author as a whole and for individual books.

In this short exercise you will load and examine a small corpus of property rental reviews from around Boston. Hopefully you already know read.csv() which enables you to load a comma separated file. In this exercise you will also need to specify stringsAsFactors = FALSE when loading the corpus. This ensures that the reviews are character vectors, not factors. This may seem mundane but the point of this chapter is to get you doing an entire workflow from start to finish so let's begin with data ingestion!

Next you simply apply str() to review the data frame's structure. It is a convenient function for compactly displaying initial values and class types for vectors.

Lastly you will apply dim() to print the dimensions of the data frame. For a data frame, your console will print the number of rows and the number of columns.

Other functions like head(), tail() or summary() are often used for data exploration but in this case we keep the examination short so you can get to the fun sentiment analysis!

When starting a sentiment project, sometimes a quick polarity() will help you set expectations or learn about the problem. In this exercise (to save time), you will apply polarity() to a portion of the comments vector while the larger polarity object is loaded in the background.

Using a kernel density plot you should notice the reviews do not center on 0. Often there are two causes for this sentiment "grade inflation." First, social norms may lead respondents to be pleasant instead of neutral. This, of course, is channel specific. Particularly snarky channels like e-sports or social media posts may skew negative leading to "deflation." These channels have different expectations. A second possible reason could be "feature based sentiment". In some reviews an author may write "the bed was comfortable and nice but the kitchen was dirty and gross." The sentiment of this type of review encompasses multiple features simultaneously and therefore could make an average score skewed.

In a subsequent exercise you will adjust this "grade inflation" but here explore the reviews without any change.

In this exercise you will perform Step 3 of the text mining workflow. Although qdap isn't a tidy package you will mutate() a new column based on the returned polarity list representing all polarity (that's a hint BTW) scores. In chapter 3 we used a custom function pol_subsections which uses only base R declarations. However, in following the tidy principles this exercise uses filter() then introduces pull(). The pull() function works like works like [[ to extract a single variable.

Once segregated you collapse all the positive and negative comments into two larger documents representing all words among the positive and negative rental reviews.

Lastly, you will create a Term Frequency Inverse Document Frequency (TFIDF) weighted Term Document Matrix (TDM). Since this exercise code starts with a tidy structure, some of the functions borrowed from tm are used along with the %>% operator to keep the style consistent. If the basics of the tm package aren't familiar check out the Text Mining with Bag-of-Words in R course. Instead of counting the number of times a word is used (frequency), the values in the TDM are penalized for over used terms, which helps reduce non-informative words.

Since you learned about tidy principles this code helps you organize your data into a tibble so you can then work within the tidyverse!

Previously you learned that applying tidy() on a TermDocumentMatrix() object will convert the TDM to a tibble. In this exercise you will create the word data directly from the review column called comments.

First you use unnest_tokens() to make the text lowercase and tokenize the reviews into single words.

Sometimes it is useful to capture the original word order within each group of a corpus. To do so, use mutate(). In mutate() you will use seq_along() to create a sequence of numbers from 1 to the length of the object. This will capture the word order as it was written.

In the tm package, you would use removeWords() to remove stopwords. In the tidyverse you first need to load the stop words lexicon and then apply an anti_join() between the tidy text data frame and the stopwords.

Here you will learn that differing sentiment methods will cause different results. Often you will simply need to have results align directionally although the specifics may be different. In the last exercise you created tidy_reviews which is a data frame of rental reviews without stopwords. Earlier in the chapter, you calculated and plotted qdap's basic polarity() function. This showed you the reviews tend to be positive.

Now let's perform a similar analysis the tidytext way! Recall from an earlier chapter you will perform an inner_join() followed by count() and then a spread().

Lastly, you will create a new column using mutate() and passing in positive - negative.

Often authors will use more words when they are more passionate. For example, a mad airline passenger will leave a longer review the worse (the perceived) service. Conversely a less impassioned passenger may not feel compelled to spend a lot of time writing a review. Lengthy reviews may inflate overall sentiment since the reviews will inherently contain more positive or negative language as the review lengthens. This coding exercise helps to examine effort and sentiment.

In this exercise you will visualize the relationship between effort and sentiment. Recall your rental review tibble contains an id and that a word is represented in each row. As a result a simple count() of the id will capture the number of words used in each review. Then you will join this summary to the positive and negative data. Ultimately you will create a scatter plot that will visualize author review length and its relationship to polarity.

This exercise will create a common visual for you to understand term frequency. Specifically, you will review the most frequent terms from among the positive and negative collapsed documents. Recall the TermDocumentMatrix all_tdm you created earlier. Instead of 1000 rental reviews the matrix contains 2 documents containing all reviews separated by the polarity() score.

It's usually easier to change the TDM to a matrix. From there you simply rename the columns. Remember that the colnames() function is called on the left side of the assignment operator as shown below.

colnames(OBJECT) <- c("COLUMN_NAME1", "COLUMN_NAME2")

Once done, you will reorder the matrix to see the most positive and negative words. Review these terms so you can answer the conclusion exercises!

Lastly, you'll visualize the terms using comparison.cloud().

Recall the "grade inflation" of polarity scores on the rental reviews? Sometimes, another way to uncover an insight is to scale the scores back to 0 then perform the corpus subset. This means some of the previously positive comments may become part of the negative subsection or vice versa since the mean is changed to 0. This exercise will help you scale the scores and then re-plot the comparison.cloud(). Removing the "grade inflation" can help provide additional insights.

Previously you applied polarity() to the bos_reviews$comments and created a comparison.cloud(). In this exercise you will scale() the outcome before creating the comparison.cloud(). See if this shows something different in the visual!

Since this is largely a review exercise, a lot of the code exists, just fill in the correct objects and parameters!

Refer to the following plot from the exercise "Comparison Cloud":

Its not surprising that the most common positive words for rentals included "walk", "restaurants", "subway" and "stations". In contrast, top negative terms included "condition", "dumpsters", "hygiene", "safety" and "sounds".

If you were looking to rent your clean apartment and it was close to public transit and good restaurants would it get a favorable review?

Refer to the following plot from the exercise "Scaled Comparison Cloud":

For your rental, should you use an automated posting?