Stock Prediction [4]: article analysis (2/2) – Sentiment Analysis
An example of Positive Sentiment
It’s time for the fun part – the “special sauce” for our special algorithm, which we can always improve in the future. In this week’s tutorial of Deep Learning for Stock Market Prediction, we create our Sentiment Analysis algorithm to detect if an article talks about a stock in a positive or negative manner. Let’s get started.
Purpose: Define a function to figure out it’s sentiment – is it talking about the subject in a positive or negative manner?
Functions Defined: getSentiment()
Packages Used: Your Brain!
Explanation
Sentiment Analysis is a common application used in Deep Learning. However, figuring out how to do it is hard if you are just a beginner. One of the easiest methods is to just count the number of positive or negative words in a text. That’s what we’ll be doing today. In the future, we can make our process even better and eliminate any errors or bad interpretations made by the machine. Companies like Lexalytics and IBM’s Watson do this well.
Let’s start by defining our function, getSentiment, passing in a Text variable which stores the text of an article. We’ll also create two lists, for positive and negative words – this will help us not count words that have appeared more than once. It is, however, debatable if doing this makes the analysis better or worse – it’s your choice whether you’d like to add this in.
How do we count the number of positive and negative words? We will take every word in the article and check it against our Positive and Negative dictionaries, counting every time a positive/negative word appears.
Let’s start with positive words. We start by creating a variable for Positive Word Count and open our Positive dictionary. We then take every word of the article and check it against our dictionaries that we have just opened. Remember that our article’s text is in a listed format, which makes it easier to use Python’s for loop.
Remember that every word in our dictionary is in the format __WORD__. If the article’s word is in the Positive dictionary and (optional) has not already appeared in the text, then it counts as a positive word. Checking if a word is part of a dictionary can be done with Python’s If-In condition.
Summary:
– Open Positive Dictionaries
– For every word in article, check if in dictionary
– (optional) Check if that word hasn’t been analyzed already
– YES: Add +1, Add word to the “Used” list
– NO: Move On
– Repeat for Negative Words!
Play Around!
I implore you to try it out! A simple test is as follows:
text = “This amazing Deep Learning tutorial by Ethan Shaotran teaches beginners how to successfully analyze the news in order to predict the stock market. Stock Prediction is a tricky yet satisfying field of Artificial Intelligence (AI) research, but we can only get better at doing it!” print(getSentiment(text))
If you are even more adventurous, you can try analyzing an article with our previously defined GetNewsURLs() and URLtoText():
URLList = GetNewsURLs(insert_subject_here) url = URLList[0] articletext, articledate = URLtoText(url) ListedText = articletext.split() #Turn into list sentiment = getSentiment(ListedText) #Get Sentiment print(sentiment)
We’ll be doing this in Part 6 and Part 8, however!
Any questions can be left in the comments! 🙂
Is it as binary as positive or negative? Are there degrees of positive or negative sentiments?
Hi there. Our algorithm’s bag of words sentiment analysis model assigns each article text a score, which is simply calculated by subtracting the number of negative words from the number of positive words. A positive result means the article is most likely positive, while a negative result means the article is most likely negative. A very high positive/negative score means the article feels very strongly for or against the stock.