Stock Prediction [2]: article collection & interpretation

Welcome to Part 2 of our tutorial series on using Deep Learning for Stock Market Prediction. In this tutorial we’ll start the exiting part of coding our news-collecting functions.

Purpose: Collect Articles about a Stock and Get Their Body Text

Functions Defined: GetNewsURLs(), URLtoText(), and SourceInt()

Packages Used: News Corpus Builder, Newspaper

        For the sake of having a well organized dataset, it would be optimal to have them categorizable by source – in other words, we should define a list of sources that are trusted and ONLY collect articles about those sources. Here’s a list of sources we have found that generally write good content about stocks:

Let’s write a function that, inputting a URL, will return what source the URL is from:

def SourceInt (url):
    url = url.lower()

    if ('cnn' in url):
        sourceint = 1
    elif ('fool.com' in url):
        sourceint = 2
    elif ('finance.yahoo.com' in url):
        sourceint = 3
    elif ('investopedia.com' in url):
        sourceint = 4
    elif ('businessinsider.com' in url):
        sourceint = 5
    elif ('marketwatch.com' in url):
        sourceint = 6
    elif ('forbes.com' in url):
        sourceint = 7
    elif ('thestreet.com' in url):
        sourceint = 8
    elif ('fortune.com' in url):
        sourceint = 9
    elif ('cnbc.com' in url):
        sourceint = 10
    elif ('nasdaq.com' in url):
        sourceint = 11
    elif ('investorplace.com' in url):
        sourceint = 12
    elif ('seekingalpha.com' in url):
        sourceint = 13
    else: sourceint = 0

    return sourceint

Note that code will give each “Trusted Source” an integer from 1 to 13, and any other source a 0. This function is MultiPurpose, as we will be using in 2 different places for developing our model.

        We need a simple way to, inputting any Stock Ticker, get a list of recent articles written about that company, and hopefully a financial evaluation of it. The easiest way to get this information is Google News, which cumulates articles from all over the internet – and all you need is a search term! News Corpus Builder (Documentation HERE) has proven to be the best Python package suited for the job, so let’s start our function here:

def GetNewsURLs(Ticker):
    links = NewsGen.google_news_search(Ticker+' stocks',100)

Given the ticker TSLA, NCB will search “TSLA stocks” and return the top 100 articles, which should all be written in the last couple of days. We then need a way to store these URLs in a URL List – but caution: we only want articles from the following companies, which we defined above. Using our SourceInt() function, if the URL returns a score of 0, we know it is not on the “Trusted List” – and thus we won’t include it in our URL List.

The following code does this perfectly:

    URLList=[]

    for i in range(0,len(links)-1):
        URLLink = links[i][1]

        if not (SourceInt(URLLink) == 0):
            URLList.append(URLLink)

    return URLList

        Finally, we want a function that takes each News Article URL and return the body text of the article, as well as the date it was published (for creating our Training Dataset). The Newspaper package (Documentation HERE) is one of the best packages suited for this job. It’s quite simple too:

def URLtoText (url):
    article = Article(url)
    article.download()
    article.parse()
    return (article.text, article.publish_date)

Not much explaining to do here. article.text conveniently provides you with the body text, and article.publish_date gets you the date it was written!

By this point you should have 3 working functions: SourceInt(), GetNewsURLs(), and URLtoText(). We’ve got our Article Data. Now what do we do? In the next part of the tutorial, we will learn how to create some pretty awesome dictionaries to be able to classify our Articles as Positive or Negative!

Any questions can be left below, for Ethan to answer!

Want to Follow This Series?