Due: October 9, 2012Problem 1 (30 points)Services like Twitter allow short, real-time commentary about whatever users feel like talking about, and this type of commentary is creating data of great interest for sentiment analysis. Often, there is interest in automatically determining whether a given tweet is positive, negative or neutral toward a specific topic, person, company, etc. For this problem, you will be working with tweets about health care reform that were collected at the time the US Congress was debating and voting on new health care legislation. Note: to do this problem, you must open one
of the Google spreadsheets that was shared with you as links in an
email. You may work with any of them while doing this problem. They are
Important: if for some reason you cannot access any of these, contact the instructor or the TA. Part A [10 pts] Annotate 20 tweets on the spreadsheet you've chosen (please work with only one of the spreadsheets). Follow the directions carefully! Step 1. Open the spreadsheet. You should see something like this: There are 1500 tweets, with one tweet per row of the spreadsheet. Each tweet has the following basic attributes:
Your job is to fill in the rest of the annotated
attributes for 20 tweets, similarly to what the instructor (Jason
Baldridge) did for the first thirty tweets in the list. Also, you'll see
annotations from many others -- these include students of past classes
and PhD students at UT Austin. The attributes are:
What
to do for each of these attributes will be described in detail below.
Note that there are actually three groups of these attributes for each
tweet. That is to allow one to have multiple targets per tweet. For
example, consider the tweet:
This is positive for the target "hcr" (health care reform) and negative for the target "gop" (Republicans). Step 2. Choose an annotator id. Your
annotator id can be the first letter of your first name followed by
your last name (for example, the id for "Jason Baldridge" is
"jbaldridge"). You can also use something else (e.g. wolverine,
kingtut), provided it isn't the same as any other ids already being
used. We will use this to find the tweets you annotated when considering
your answers to the other parts of this question. In your submission
for this part of the problem, write down what your annotator id is. Step 3. Claim 20 tweets by putting your annotator id next to them. Before
actually annotating any of the tweets, you must "claim" 20 of them by
putting your annotator id in the "annotator id" column for the tweets.
This will make sure that no one else who is editing the document at the
same time will accidentally write over one of your annotations. Note
that you can of course be annotating a new target in a tweet that
already has one or two targets annotated (three is the maximum number of
targets for a tweet on these spreadsheets). Step 4. Annotate each of the 20 tweets. For each tweet, read the text and decide what sentiment it expresses and what that sentiment is being expressed about. The possible values for sentiment are:
Sometimes it might be hard to make an exact determination, so it is fine to use the label "unsure" if you can't decide. The possible values for target are:
A few tips and comments:
Part B [3 pts] Calculate the probability of positive out of the positive and negative tweets in your 20 tweets. Part C [5 pts] Calculate
the probability of positive for tweets about health care reform
(target=hcr) from your 20 tweets plus the first 30 tweets annotated by
jbaldridge in HCR Tweets - Set3. As in part B, calculate the probability based only on
positive and negative tweets (about hcr). Part D [6 pts] Name
the three most difficult tweets you had to annotate and describe what
made them difficult. Try to identify things like polarity flipping, use
of discourse markers, sarcasm, etc. that were discussed in the slides. Part E [6 pts] Look
at tweets that were annotated by others (this can include jbaldridge)
and see if you disagree with the annotation of either the sentiment or
the target of any of these (including any "unsure" annotations). If you
do disagree, make a note in the dispute cell for the tweet on the
spreadsheet. In your homework, describe the tweet and how you would
annotate it differently and what information you think is necessary to
use. For example, is it necessary to know whether a hashtag corresponds
to a particular position, like #handsoff (against health care reform) and #p2 (for health care reform), or to read the content of a page that the tweeter linked to? If
you agree with all the annotations, you can instead write down any
general problems you think there are with annotating tweets in this way.
For example, are there any important target values missing, or are many
tweets just too short for you to make a determination? Problem 2 (30 points)
The band The Flaming Lips has asked you to build a system that monitors Twitter during the Austin City Limits festival and classify tweets as positive or negative about the band. Here is an example:
In answering the questions below, be sure to use dataset 1 and dataset 2 appropriately for computing the probabilities, as indicated above. (a) [2 pts] What is the probability of negative? (Show your work). (b) [3 pts] What is the probability of best given negative? (Show your work) (c) [15 pts] What is the probability of positive given best? (Show your work) (d) [10 pts] You realize it might be a good idea to use both (a) the positive and negative tweets and (b) the response of the positive and negative concert-goers to get a better estimate. What is the probability of negative given best if you use the tweets in addition to the data from the concert-goers to compute the prior probabilities of positive and negative? (Show your work) Problem 3 (25 points)
In a dataset of tweets annotated for polarity, there were 125 tweets annotated as neutral, 149 as positive, and 304 as negative. I create a subjectivity classifier that identifies tweets as
objective (neutral) or subjective (either positive or negative). It
labels 148 tweets as objective and the rest as subjective. Of the 148,
only 97 were actually objective. Fill in the following table based on the data given above
Part B [18 points] Now, answer the following questions about the classifier's ability to identify objective tweets. (a) [1 pt] How many false negatives did the classifier produce as a detector of objective tweets?
(b) [4 pts] What is the precision of the classifier as a detector of objective tweets? (c) [4 pts] What is the recall of the classifier as a detector of objective tweets? Next, answer the following questions about the classifier's ability to identify subjective tweets. (d) [1 pt] How many false positives did the classifier produce as a detector of subjective tweets? (e) [4 pts] What is the precision of the classifier as a detector of subjective tweets? (f) [4 pts] What is the recall of the classifier as a detector of subjective tweets? Problem 4 (15 points)The goal of a subjectivity classifier is to identify tweets that
should be labeled as positive or negative by a polarity classifier. The
polarity classifier is what will be used to understand the overall
positivity or negativity toward health care reform on new tweets, and it
will be used to present positive and negative examples to policy
makers. Recall that probabilistic classifiers provide not only a label,
but also a confidence value (the probability), and that probability can
be used as a threshold. For example, rather than saying that the best candidate label is: You can instead choose the label based on a threshold: where θ is a threshold value between 0 and 1. If θ = 0.5, then the decision is the same as the argmax above, but if θ =0.8, then we are saying that we will only consider tweets subjective if they have a probability of being subjective greater than .8, and so on. This allows us to use the confidence value to select more or fewer tweets as subjective. (At the extremes: if θ = 0, everything is subjective, and if θ = 1, nothing is.) This is important because when a classifier is more confident, it is typically more correct. So, if we have a higher threshold, it means that precision is higher because each decision is more likely to be right. However, if we have too high a threshold, we might fail to label many instances, so this can damage recall. This question asks to you to consider whether precision or recall is more important for identifying subjective tweets for different contexts. Another way of thinking about this question is this: is it more important (a) to catch all of the subjective tweets at the risk of including many objective ones and thus have better recall (by using values for θ less than 0.5), or (b) to select fewer tweets that the subjectivity classifier is more confident about as subjective at the risk of not providing many subjective tweets to the polarity classifier and thus have better precision (by using values for θ greater than 0.5)? With that in mind, and also considering that there may be 100,000 or more tweets per day that are relevant to your topic of interest, say whether you think precision or recall for subjective tweets is more important for the following purposes:
|
Assignments >