Skip to main content
Security Informatics
Security Informatics Cover Image
Download PDF

Estimating the sentiment of social media content for security informatics applications

Security Informatics volume 1, Article number: 3 (2012) Cite this article

Abstract

Inferring the sentiment of social media content, for instance blog posts and forum threads, is both of great interest to security analysts and technically challenging to accomplish. This paper presents two computational methods for estimating social media sentiment which address the challenges associated with Web-based analysis. Each method formulates the task as one of text classification, models the data as a bipartite graph of documents and words, and assumes that only limited prior information is available regarding the sentiment orientation of any of the documents or words of interest. The first algorithm is a semi-supervised sentiment classifier which combines knowledge of the sentiment labels for a few documents and words with information present in unlabeled data, which is abundant online. The second algorithm assumes existence of a set of labeled documents in a domain related to the domain of interest, and leverages these data to estimate sentiment in the target domain. We demonstrate the utility of the proposed methods by showing they outperform several standard techniques for the task of inferring the sentiment of online movie and consumer product reviews. Additionally, we illustrate the potential of the methods for security informatics by estimating regional public opinion regarding two events: the 2009 Jakarta hotel bombings and 2011 Egyptian revolution.

1. Introduction

There is increasing recognition that the Web represents a valuable source of security-relevant intelligence and that computational analysis offers a promising way of dealing with the problem of collecting and analyzing data at Web scale [e.g., [1–4]]. As a consequence, tools and algorithms have been developed which support various security informatics objectives [3, 4]. To cite a specific example, we have recently shown that blog network dynamics can be exploited to provide reliable early warning for a class of extremist-related, real-world protest events [5].

Monitoring social media to spot emerging issues and trends and to assess public opinion concerning topics and events is of considerable interest to security professionals; however, performing such analysis is technically challenging. The opinions of individuals and groups are typically expressed as informal com-munications and are buried in the vast, and largely irrelevant, output of millions of bloggers and other online content producers. Consequently, effectively exploiting these data requires the development of new, automated methods of analysis [3, 4]. Although helpful computational analytics have been derived for traditional forms of written content, less has been done to develop techniques that are well-suited to the particular characteristics of the content found in social media.

This paper considers one of the central problems in the new field of social media analytics: deciding whether a given document, such as a blog post or forum thread, expresses positive or negative opinion to-ward a particular topic. The informal nature of social media content poses a challenge for language-based sentiment analysis. While statistical learning-based methods often provide good performance in unstructured settings like this [e.g., [6–13]], obtaining the required labeled instances of data, such as a collection of "exemplar" blog posts of known sentiment polarity, is usually an expensive and time-consuming undertaking.

We present two new computational methods for inferring sentiment orientation of social media content which address these challenges. Each method formulates the task as one of text classification, models the data as a bipartite graph of documents and words, and assumes that only limited prior information is available regarding the sentiment orientation of any of the documents or words of interest. The first algorithm adopts a semi-supervised approach to sentiment classification, combining knowledge of the sentiment polarity for a few documents and a small lexicon of words with information present in a corpus of unlabeled documents; note that such unlabeled data are readily obtainable in online applications. The second algorithm assumes existence of a set of labeled documents in a domain related to the domain of interest, and provides a procedure for transferring the sentiment knowledge contained in these data to the target domain. We demonstrate the utility of the proposed algorithms by showing they outperform several standard methods for the task of inferring the sentiment polarity of online reviews of movies and consumer products. Additionally, we illustrate the potential of the methods for security informatics through two case studies in which sentiment analysis of Arabic, Indonesian, and Danish (language) blogs is used to estimate regional public opinion regarding the 2009 Jakarta hotel bombings and 2011 Egyptian revolution.

2. Preliminaries

We approach the task of estimating the sentiment orientations of a collection of documents as a text classification problem. Each document of interest is represented as a "bag of words" feature vector x∈ℜ|V|, where the entries of × reflect some measure of the frequency with which the words in the vocabulary set V appear in the document. For example, the elements of × can be simple word-counts, or × can be normalized in various ways [6]. In this paper the elements xi of × are defined to indicate the presence (xi = 1) or absence (xi = 0) of the corresponding words in the document; however, specifying × using word-counts yields similar results.) We wish to learn a vector c∈ℜ|V| such that the classifier orient = sign(cTx) accurately estimates the sentiment orientation of document x, returning +1 (-1) for documents expressing positive (negative) sentiment about the topic of interest.

Knowledge-based classifiers leverage prior domain information to construct the vector c. One way to obtain such a classifier is to assemble lexicons of positive words V+⊆V and negative words V-⊆V, and then to set ci= +1 if word i∈V+, ci= -1 if i∈V-, and ci=0 if i is not in either lexicon; this classifier simply sums the positive and negative sentiment words in the document and assigns document orientation accordingly. While this scheme can provide acceptable performance in certain settings, it is unable to improve its performance or adapt to new domains, and it is usually labor-intensive to construct lexicons which are sufficiently complete to enable useful sentiment classification performance to be achieved.

Alternatively, learning-based methods attempt to generate the classifier vector c from examples of positive and negative sentiment. To obtain a learning-based classifier, one can begin by assembling a set of nl labeled documents {(xi, di)}, where di∈{+1, -1} is the sentiment label for document i. The vector c then can be learned through "training" with the set {(xi, di)}, for instance by solving the following set of equations for c:

[ X T X+ γ I | V| ] c=X T d,
(1)

where matrix X∈ℜnl×|V| has document vectors for rows, d∈ℜnl is the vector of document labels, I|V| denotes the |V|×|V| identity matrix, and gγ0 is a constant; this corresponds to regularized least squares (RLS) learning [14]. Many other strategies can be used to compute c, including Naïve Bayes (NB) statistical inference [6]. Learning-based classifiers have the potential to improve their performance and to adapt to new situations, but standard methods for realizing these capabilities require that fairly large training sets of labeled documents be obtained and this is usually expensive.

Sentiment analysis of social media content for security informatics applications is often characterized by the existence of only modest levels of prior knowledge regarding the domain of interest, reflected in the availability of a few labeled documents and small lexicon of sentiment-laden words, and by the need to rapidly learn and adapt to new domains. As a consequence, standard knowledge-based and learning-based sentiment analysis methods are typically ill-suited for security informatics. In order to address this challenge, the sentiment analysis methods developed in this paper enable limited labeled data to be combined with readily available "auxiliary" information to produce accurate sentiment estimates. More specifically, the first proposed method is a semi-supervised algorithm [e.g., [9, 10]] which leverages a source of supplementary data which is abundant online: unlabeled documents and words. Our second algorithm is a novel transfer learning method [e.g., [11]] which permits the knowledge present in data that has been previously labeled in a related domain (say online movie reviews) to be transferred to a new domain (e.g., reviews of consumer products).

Each of the algorithms proposed in this paper assumes the availability of a modest lexicon of sentiment-laden words. This lexicon is encoded as a vector w∈ℜ|Vl|, where Vl = V+∪V- is the sentiment lexicon and the entries of w are set to +1 or -1 according to the polarity of the corresponding words. The development of the algorithms begins by modeling the problem data as a bipartite graph Gb of documents and words (see Figure 1). It is easy to see that the adjacency matrix A for graph Gb is given by

Figure 1
figure1

Cartoon of bipartite graph model Gb, in which documents (red vertices) are connected to the words (blue vertices) they contain and the link weights (black edges) can reflect word frequencies.

Full size image
A = 0 X X T 0
(2)

where the matrix X∈ℜn×|V| is constructed by stacking the document vectors as rows, and each '0' is a matrix of zeros. In both the semi-supervised and transfer learning algorithms, integration of labeled and "auxiliary" data is accomplished by exploiting the relationships between documents and words encoded in the bipartite graph model. The basic idea is to assume that, in the bipartite graph Gb, positive documents will tend to be connected to (contain) positive words, and analogously for negative documents/words.

3. Semi-Supervised Sentiment Analysis

We now derive our first sentiment estimation algorithm for social media content. Consider the common situation in which only limited prior knowledge is available about the way sentiment is expressed in the domain of interest, in the form of small sets of documents and words for which sentiment labels are known, but where abundant unlabeled documents can be easily collected (e.g., via Web crawling). In this setting it is natural to adopt a semi-supervised approach, in which labeled and unlabeled data are combined and leveraged in the analysis process. In what follows we present a novel bipartite graph-based approach to semi-supervised sentiment analysis.

Assume the initial problem data consists of a corpus of n documents, of which nl << n are labeled, and a modest lexicon Vl of sentiment-laden words, and suppose that this label information is encoded as vectors d∈ℜnl and w∈ℜ|Vl|, respectively. Let dest∈ℜn be the vector of estimated sentiment orientations for the documents in the corpus, and define the "augmented" classifier caug = [destT cT]T∈ℜn+|V| which estimates the polarity of both documents and words. Note that the quantity caug is introduced for notational convenience in the subsequent development and is not directly employed for classification. More specifically, in the proposed methodology we learn caug, and therefore c, by solving an optimization problem involving the labeled and unlabeled training data, and then use c to estimate the sentiment of any new document of interest with the simple linear classifier orient = sign(cTx). We refer to this classifier as semi-supervised because it is learned using both labeled and unlabeled data. Assume for ease of notation that the documents and words are indexed so the first nl elements of dest and |Vl| elements of c correspond to the labeled data.

We wish to learn an augmented classifier caug with the following three properties: 1.) if a document is labeled, then the corresponding entry of dest should be close to this ±1 label; 2.) if a word is in the sentiment lexicon, then the corresponding entry of c should be close to this ±1 sentiment polarity; and 3.) if there is an edge Xij of Gb that connects a document × and a word v∈V and Xij possesses significant weight, then the estimated polarities of × and v should be similar. These objectives are encoded in the following minimization problem:

min c aug  c aug T L c aug + β 1 ∑ i = 1 n l ( d est,i  -  d i ) 2 + β 2 ∑ i = 1 V l ( c i  -  w i ) 2
(3)

where L = D - A is the graph Laplacian matrix for Gb, with D the diagonal degree matrix for A (i.e., Dii = ∑j Aij), and β1, β2 are nonnegative constants. Minimizing (3) enforces the three properties we seek for caug, with the second and third terms penalizing "errors" in the first two properties. To see that the first term enforces the third property, observe that this expression is a sum of components of the form Xij(dest,i - cj)2. The constants β1, β2 can be used to balance the relative importance of the three properties.

The caug which minimizes the objective function (3) can be obtained by solving the following set of linear equations:

L 11 + β 1 I nl L 12 L 13 L 14 L 21 L 22 L 23 L 24 L 31 L 32 L 33 + β 2 I V 1 L 34 L 41 L 42 L 43 L 44 c aug = β 1 d 0 β 2 w 0
(4)

where the Lij are matrix blocks of L of appropriate dimension. The system (4) is sparse because the data matrix × is sparse, and therefore large-scale problems can be solved efficiently. Note that in situations where the set of available labeled documents and words is very limited, say less than a couple hundred documents, sentiment classifier performance can be improved by replacing L in (4) with the normalized Laplacian Ln=D-1/2LD-1/2, or with a power of this matrix Lnk (for k a positive integer). The Appendix of this paper demonstrates that replacing L with Lnk serves to "smooth" the polarity estimates assigned to the vertices of Gb, thereby reducing the possibility for over-fitting and increasing the capability for generalization.

We summarize this discussion by sketching an algorithm for learning the proposed semi-supervised classifier:

Algorithm SS

  1. 1.

    Construct the set of equations (4), possibly by replacing the graph Laplacian L with Lnk.

  2. 2.

    Solve equations (4) for caug = [ destT cT ]T (for instance using the Conjugate Gradient method).

  3. 3.

    Estimate the sentiment orientation of any new document × of interest as: orient = sign(cTx).

The utility of Algorithm SS is now examined through a case study involving a standard sentiment analysis task: estimation of the sentiment polarity of online movie reviews (an exercise which is known to be diffi-cult).

4. Case Study One: Movie Reviews

This case study examines the performance of Algorithm SS for the problem of estimating sentiment of online movie reviews. The data used in this study is a publicly available set of 2000 movie reviews, 1000 positive and 1000 negative, collected from the Internet Movie Database and archived at the website [15]. The Lemur Toolkit [16] was employed to construct the data matrix × and vector of document labels d from these reviews. A lexicon of ~1400 domain-independent sentiment-laden words was obtained from [17] and employed to build the lexicon vector w.

This study compares the movie review orientation classification accuracy of Algorithm SS with that of three other schemes: 1.) lexicon-only, in which the lexicon vector w is used as the classifier as summarized in Section II, 2.) a classical NB classifier obtained from [18], and 3.) a well-tuned version of the RLS classifier (1). Algorithm SS is implemented with the following parameter values: β1 = 0.1, β2 = 0.5, and k = 10. A focus of the investigation is evaluating the extent to which good sentiment estimation performance can be achieved even if only relatively few labeled documents are available for training; thus we examine training sets which incorporate a range of numbers of labeled documents: nl = 50, 100, 150, 200, 300, 400, 600, 800, 1000.

Sample results from this study are depicted in Figure 2. Each data point in the plots represents the average of ten trials. In each trial, the movie reviews are randomly partitioned into 1000 training and 1000 test documents, and a randomly selected subset of training documents of size nl is "labeled" (i.e., the labels for these reviews are made available to the learning algorithms). As shown in Figure 2, Algorithm SS outperforms the other three methods. Note that, in particular, the accuracy obtained with the proposed approach is significantly better than the other techniques when the number of labeled training documents is small. It is expected that this property of Algorithm SS will be of considerable value in security informatics applications that involve social media data.

Figure 2
figure2

Results for the movie reviews case study. The plot shows how sentiment estimation accuracy (vertical axis) varies with number of available labeled movie reviews (horizontal axis) for four different classifiers: lexicon only (black), NB (magenta), RLS (blue), and Algorithm SS (red).

Full size image

5. Transfer Learning Sentiment Analysis

This section develops the second proposed sentiment estimation algorithm for social media content. Many security informatics applications are characterized by the presence of limited labeled data for the domain of interest but ample labeled information for a related domain. For instance, an analyst may wish to ascertain the sentiment of online discussions about an emerging topic of interest, and may have in hand a large set of labeled examples of positive and negative posts regarding other topics (e.g., from previous studies). In this setting it is natural to adopt a transfer learning approach, in which knowledge concerning the way sentiment is expressed in one domain, the so-called source domain, is transferred to permit sentiment estimation in a new target domain. In what follows we present a new bipartite graph-based approach to transfer learning sentiment analysis.

Assume that the initial problem data consists of a corpus of n = nT + nS documents, where nT is the (small) number of labeled documents available for the target domain of interest and nS >> nT is the number of labeled documents from some related source domain; in addition, suppose that a modest lexicon Vl of sentiment-laden words is known. Let this label data be encoded as vectors dT∈ℜnT, dS∈ℜnS, and w∈ℜ|V|, respectively. Denote by dT,est∈ℜnT, dS,est∈ℜnS, and c∈ℜ|Vl| the vectors of estimated sentiment orientations for the target and source documents and the words, and define the augmented classifier as caug = [dS,estT dT,estT cT]T ∈ ℜn+|V|. Note that the quantity caug is introduced for notational convenience in the subsequent development and is not directly employed for classification.

In what follows we derive an algorithm for learning caug, and therefore c, by solving an optimization problem involving the labeled source and target training data, and then use c to estimate the sentiment of any new document of interest via the simple linear classifier orient = sign(cTx). This classifier is referred to as transfer learning-based because c is learned, in part, by transferring knowledge about the way sentiment is expressed from a domain which is related to (but need not be identical to) the domain of interest.

We wish to learn an augmented classifier caug with the following four properties: 1.) if a source document is labeled, then the corresponding entry of dS,est should be close to this ±1 label; 2.) if a target document is labeled, then the corresponding entry of dT,est should be close to this ±1 label, and the information encoded in dT should be emphasized relative to that in the source labels dS,; 3.) if a word is in the sentiment lexicon, then the corresponding entry of c should be close to this ±1 sentiment polarity; and 4.) if there is an edge Xij of Gb that connects a document × and a word v∈V and Xij possesses significant weight, then the estimated polarities of × and v should be similar.

The four objectives listed above may be realized by solving the following minimization problem:

min c aug  c aug T L c aug + β 1 d S,est  -  k S d S 2 + β 2 d T,est  -  k T d T 2 + β 3 c - w 2
(5)

where L = D - A is the graph Laplacian matrix for Gb, as before, and β1, β2, β3, kS, and kT are nonnegative constants. Minimizing (5) enforces the four properties we seek for caug. More specifically, the second, third, and fourth terms penalize "errors" in the first three properties, and choosing β2 >β1 and kT >kS favors target label data over source labels. To see that the first term enforces the fourth property, note that this expression is a sum of components of the form Xij (dT,est,i - cj)2 and Xij (dS,est,i - cj)2.The constants β1, β2, β3 can be used to balance the relative importance of the four properties.

The caug which minimizes the objective function (5) can be obtained by solving the following set of linear equations:

L 11 + β 1 I nS L 12 L 13 L 21 L 22 + β 2 I nT L 23 L 31 L 32 L 33 + β 3 I V 1 c aug = β 1 k S d S β 2 k T d T β 3 w
(6)

where the Lij are matrix blocks of L of appropriate dimension. The system (6) is sparse because the data matrix × is sparse, and therefore large-scale problems can be solved efficiently. In applications with very limited labeled data, sentiment classifier performance can be improved by replacing L in (6) with the nor-malized Laplacian Ln or with a power of this matrix Lnk.

Note that developing systematic methods for characterizing how similar the source and target domains must be to enable useful transfer learning, or for selecting an appropriate source domain given a target set of interest, remain open research problems. Some helpful guidance for these tasks is provided in [12].

We summarize the above discussion by sketching an algorithm for constructing the proposed transfer learning classifier:

Algorithm TL

  1. 1.

    Construct the set of equations (6), possibly by replacing the graph Laplacian L with Lnk.

  2. 2.

    Solve equations (6) for caug = [dS,estT dT,estT cT]T.

  3. 3.

    Estimate the sentiment orientation of any new document × of interest as: orient = sign(cTx).

The utility of Algorithm TL is now examined through a case study involving online consumer product reviews.

6. Case Study Two: Product Reviews

This case study examines the performance of Algorithm TL for the problem of estimating sentiment of online product reviews. The data used in this study is a publicly available set of 1000 reviews of electronics products, 500 positive and 500 negative, and 1000 reviews of kitchen appliances, 500 positive and 500 negative, collected from Amazon and archived at the website [19]. The Lemur Toolkit [16] was employed to construct the data matrix × and vectors of document labels dS and dT from these reviews. A lexicon of 150 domain-independent sentiment-laden words was constructed manually and employed to form the lexicon vector w.

This study compares the product review sentiment classification accuracy of Algorithm TL with that of four other strategies: 1.) lexicon-only, in which the lexicon vector w is used as the classifier as summarized in Section II, 2.) a classical NB classifier obtained from [18], 3.) a well-tuned version of the RLS classifier (1), and 4.) Algorithm SS. Algorithm TL is implemented with the following parameter values: β1 = 1.0, β2 = 3.0, β3 = 5.0, kS = 0.5, kT = 1.0, and k = 5. A focus of the investigation is evaluating the extent to which the knowledge present in labeled reviews from a related domain, here kitchen appliances, can be transferred to a new domain for which only limited labeled data is available, in this case electronics. Thus we assume that all 1000 labeled kitchen reviews are available to Algorithm TL (the only algorithm which is designed to exploit this information), and examine training sets which incorporate a range of numbers of labeled documents from the electronics domain: nT = 20, 50, 100, 200, 300, 400.

Sample results from this study are depicted in Figure 3. Each data point in the plots represents the average of ten trials. In each trial, the electronics reviews are randomly partitioned into 500 training and 500 test documents, and a randomly selected subset of reviews of size nT is extracted from the 500 labeled training instances and made available to the learning algorithms. As shown in Figure 3, Algorithm TL outperforms the other four methods. Note that, in particular, the accuracy obtained with the transfer learning approach is significantly better than the other techniques when the number of labeled training documents in the target domain is small. It is expected that the ability of Algorithm TL to exploit knowledge from a related domain to quickly learn an effective sentiment classifier for a new domain will be of considerable value in security informatics applications involving social media data.

Figure 3
figure3

Results for the consumer product reviews case study. The plot shows how sentiment estimation accuracy (vertical axis) varies with number of available labeled electronics reviews (horizontal axis) for five different classifiers: lexicon only (orange), NB (black), RLS (magenta), Algorithm SS (blue), and Algorithm TL (red).

Full size image

7. Case Study Three: Jakarta Hotel Bombings

On 17 July 2009 the JW Marriott and Ritz-Carlton Hotels in Jakarta, Indonesia were hit by suicide bombing attacks within five minutes of each other. A little over a week later, on 26 July 2009, a document claiming responsibility for the attacks and allegedly written by N.M. Top was posted on the blog [20]; see Figure 4 for a screenshot of a portion of this blog post. In subsequent discussions we will refer to this post as the "Top post" for convenience, with the understanding that the authorship of the post is uncertain. At this time, senior U.S. intelligence and security officials expressed interest in understanding sentiment in the region regarding the bombings and the alleged claim of responsibility by a well-known extremist [personal communications, Senior U.S. Intelligence and Security Officials, July 2009]. Among other things, officials felt that characterizing this sentiment might provide insight into Indonesian public opinion concerning violent extremist organizations.

Figure 4
figure4

Screenshot of blog post, allegedly by N.M. Top, claiming responsibility for the July 2009 bombings of two hotels in Jakarta, Indonesia.

Full size image

To enable a preliminary assessment along these lines, we collected two sets of social media data related to the Top post: 1.) the ~3000 comments made to the post during the two week period immediately follow-ing the post, and 2.) several hundred posts made to other Indonesian language blogs in which the Top post was discussed. We manually labeled the sentiment of a small subset of these documents, and also translated into Indonesian the generic sentiment lexicon used in Case Study One (in this paper all language translation was performed using the tool available at http://translate.google.com). Observe that this approach to constructing a sentiment lexicon is far from perfect. However, because our proposed algorithms employ several sources of information to estimate the sentiment of content, it is expected that they will exhibit robustness to imperfections in any single data source. This study therefore offers the opportunity to explore the utility of a very simple approach to multilingual sentiment analysis: translate a small lexicon of sentiment-laden words into the language of interest and then apply Algorithm SS or Algorithm TL directly within that language (treating words as tokens). The capability to perform automated, multilingual content analysis is of substantial interest in many security-related applications.

We implemented Algorithm SS to estimate the sentiment expressed in the corpus of comments made to the Top post [20] and in the set of related discussions posted at other blogs. Sample sentiment estimation results for the comments made to the Top post are shown at the top of Figure 5. These comments are almost universally negative, condemning both the bombings and the justification for the bombings given in the Top post. Manual examination of a subset of the comments confirms the results provided by Algorithm SS. For example, a typical negative comment reads, in part (translated from the Indonesian):

Figure 5
figure5

Sample results for blog analysis case study. The top plot shows the estimated sentiment of comments made directly to the blog [20] in response to the Top post (blue is negative, red is "conspiracy-related", with red data multiplied by a factor of 10 to be visible on the graph). The bottom plot gives estimated sentiment of blog posts discussing the Top post (total is blue, positive is red, olive is neutral, and purple is negative, respectively).

Full size image

" ... a savage kind - Noordin M. Top does not deserve to live in this world, he only claims to serve Islam but actually he is in truth a disbeliever. You are all cowards, you terrorists reversing Islam, and Noordin Top is just a stupid terrorist who escaped Malaysia just to inflict violence and hatred ... ."

Interestingly, by slightly reformulating the classifier it is possible to discover that while almost all comments are negative, there is a thread of comments which puts forth various conspiracies associated with the origins of the bombings and/or the Top post. The following comment is illustrative of this theme:

" ... the only explanation for the sophistication of the attacks is that the bombers were aided by the CIA or Mossad."

Sample results obtained through sentiment analysis of relevant posts made to other blogs are presented at the bottom of Figure 5. These posts also express largely negative sentiments about the Top post, although they are not as consistently negative as are the comments made directly on the Top post blog site. It should be noted that the "neutral" post sentiment depicted in the plot at the bottom of Figure 5 are mainly news articles. Again, manual evaluation of a subset of these posts confirm the results obtained via auto-mated classification with Algorithm SS.

8. Case Study Four: Egyptian Revolution

Beginning on 25 January 2011, a popular uprising swept across Egypt in the form of massive demonstrations and rallies, labor strikes in various sectors, and violent clashes between protestors and security forces, ultimately leading to the resignation of Egyptian President Hosni Mubarak on 11 February. National security analysts and officials have expressed interest in understanding public sentiment regarding the Egyptian revolution generally and Mubarak specifically, especially 1.) in the weeks before the protests and 2.) for different regions of the globe.

To enable a preliminary assessment along these lines, we collected three sets of blog posts which are related to Egyptian unrest and Mubarak and were posted during the two week period immediately before the protests began on 25 January: 1.) 100 Arabic posts, 2.) 100 Indonesian posts, and 3.) 100 Danish posts. We manually labeled the sentiment of a small subset of these documents, and translated into the appropriate language the generic sentiment lexicon used in Case Study One for implementation in this study (using the translation tool available at http://translate.google.com). Observe that this approach to constructing a sentiment lexicon is far from perfect. However, because lexical information is only one of the data sources used in the proposed approaches to sentiment estimation, it is expected that overall performance will exhibit robustness to imperfections in any one source of data; indeed, this study provides a simple test of this expected robustness.

We used Algorithm SS to estimate the sentiment expressed in the three sets of blog posts noted above, classifying the posts as either 'negative' or 'positive/neutral'. The analysis reveals that, while the sentiment expressed by the bloggers in the sample is largely negative toward Mubarak, the fraction of negative posts varies by post language (and thus possibly by geographic region). In particular, as shown in Figure 6, Arabic language posts are the most negative, followed by Indonesian posts, with the Danish posts in our sample actually being slightly more positive/neutral than negative. Manual inspection of a subset of the comments confirms the results provided by Algorithm SS.

Figure 6
figure6

Results for Egyptian revolution case study. The bar chart shows that, while blog sentiment toward former Egyptian President Hosni Mubarak in the weeks leading up to the protests was largely negative, this sentiment varies with the language of posts. More specifically, the fraction of posts expressing negative sentiment regarding Mubarak is 0.80 for Arabic posts, 0.58 for Indonesian posts, and 0.44 for Danish posts.

Full size image

9. Summary

Sentiment analysis of social media content for security informatics applications is often characterized by the existence of only modest levels of prior knowledge regarding the domain of interest and the need to rapidly adapt to new domains; consequently, standard content analysis methods typically perform poorly in this setting. This paper presents two new computational methods for inferring the sentiment expressed in social media which address these challenges. Each method formulates the task as one of text classification, models the data as a bipartite graph of documents and words, and enables prior knowledge concerning the sentiment orientation of documents or words of interest to be effectively combined with "auxiliary" information to produce accurate sentiment estimates. The first proposed method is a semi-supervised algorithm that leverages a source of supplementary data which is abundant online: unlabeled documents and words. The second algorithm is a novel transfer learning method which permits the knowledge present in data that has been previously labeled in a related domain to be transferred to a new domain. We demonstrate the utility of the proposed algorithms by showing they outperform several standard methods for the task of inferring the sentiment polarity of online reviews of movies and consumer products. Additionally, we illustrate the potential of the methods for security informatics by estimating regional public opinion regarding two security-relevant events: the 2009 Jakarta hotel bombings and the 2011 Egyptian revolution.

The proposed algorithms are complementary in that they exploit different sources of auxiliary information (each of which is frequently available in security informatics applications). For instance, Algorithm SS is able to extract useful information from unlabeled documents and words, which are typically abundant online. This capability is particularly valuable in applications for which it is difficult or expensive to acquire any form of labeled data. Alternatively, if previous analysis has produced labeled information in a domain related to the current domain of interest, Algorithm TL can be used to effectively leverage this related data; observe that this situation is common in intelligence analysis settings.

Appendix

As summarized in Section 3, solving the optimization problem

balances three objectives:

min c aug  c aug T L n c aug + β 1 ∑ i = 1 n l ( d est,i  -  d i ) 2 + β 2 ∑ i = 1 V l ( c i  -  w i ) 2

  • clustering: reducing cTaugLncaug ensures that polarity estimates for documents, dest,i, and words, cj, are assigned to reduce the magnitude of terms of the form

    X ij d est,i D 1,ii 1 / 2 - c j D 2,jj 1 / 2 2

where D1,ii1/2 = (∑j Xij)1/2 and D2,jj1/2 = (∑i Xij)1/2;

  • prior knowledge on documents: if document i is labeled di then the polarity estimate dest,i should be close to this label;

  • prior knowledge on words: if word i is labeled wi then the polarity estimate ci should be close to this label.

It is shown in this paper that this procedure enables accurate sentiment classifiers to be learned. However, in situations in which very little labeled data is available, this approach can produce numerous isolated polarity clusters around labeled instances on the graph Gb, resulting in over-fitted solutions with little power for generalization. Here we show that replacing the term ϕ = caugTLncaug with ϕk = caugTLkncaug , where k a positive integer, smoothes the sentiment polarity estimates on Gb and resolves this difficulty.

Note first that Ln can be expressed as

L n = ∑ i = 1 n + | V | λ i z i z i T ,

where (λi, zi) are eigenvalue-eigenvector pairs for Ln, and similarly that Lnk can be written

L n k = ∑ i = 1 n + | V | λ i k z i z i T .

Next observe that the quantity

ϕ = c aug T L n c aug = ∑ i ∑ j X ij d est,i D 1,ii 1 / 2 - c j D 2,jj 1 / 2 2

measures the smoothness of the document-word polarity assignment specified by caug. If the eigenvalues λi are ordered so that 0 = λ1≤ λ2≤ ... ≤ λn+|V| then, because zjT Ln zj = λj, it is seen that the eigenvectors zi of Ln are ordered by their smoothness. Then, since {z1, ..., zn+|V|} is an orthonormal basis (Ln is sym-metric), caug can be expanded as caug = ∑i αi zi and ϕ becomes ϕ = ∑i λiαi2.

Analogously, ϕk = cTaugLkncaug can be written

ϕ k = ∑ i α i z i ∑ j λ j k z j z j T ∑ m α m z m = ∑ i λ i k α i 2 .

It follows that minimizing ϕk instead of ϕ results in smoother polarity specifications on Gb, because the former imposes a more aggressive penalization of polarity assignments that include the larger (and less smooth) eigenvalue terms.

References

  1. 1.

    US Committee on Homeland Security and Government Affairs, Violent Extremism, the Internet, and the Homegrown Terrorism Threat 2008.

  2. 2.

    Bergin A, Osman S, Ungerer C, Yasin N: Countering Internet Radicalization in Southeast Asia. ASPI Special Report 2009.

    Google Scholar 

  3. 3.

    Chen H, Yang C, Chau M, Li S (Eds): Intelligence and Security Informatics. Lecture Notes in Computer Science, Springer, Berlin; 2009.

    Google Scholar 

  4. 4.

    Proc 2010 IEEE International Conference on Intelligence and Security Informatics, Vancouver. 2010.

  5. 5.

    Colbaugh R, Glass K: Early warning analysis for social diffusion events. Proc 2010 IEEE International Conference on Intelligence and Security Informatics, Vancouver 2010.

    Google Scholar 

  6. 6.

    Pang B, Lee L: Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2008, 2: 1–135. 10.1561/1500000011

    Article  Google Scholar 

  7. 7.

    Dhillon I: Co-clustering documents and words using bipartite spectral graph partitioning. In Proc ACM International Conference on Knowledge Discovery and Data Mining. San Francisco; 2001.

    Google Scholar 

  8. 8.

    Kim S, Hovy E: Determining the sentiment of opinions. Proc International Conference on Computational Linguistics 2004.

    Google Scholar 

  9. 9.

    Sindhwani V, Melville P: Document-word co-regularization for semi-supervised sentiment analysis. Proc 2008 IEEE International Conference on Data Mining, Pisa 2008.

    Google Scholar 

  10. 10.

    Colbaugh R, Glass K: Estimating sentiment orientation in social media for intelligence monitoring and analysis. Proc 2010 IEEE International Conference on Intelligence and Security Informatics, Vancouver 2010.

    Google Scholar 

  11. 11.

    Pan S, Yang Q: A survey on transfer learning. IEEE Trans Knowledge and Data Engineering 2010, 22: 1345–1359.

    Article  Google Scholar 

  12. 12.

    Blitzer J, Dredze M, Perieia F: Biographies, bollywood, boom-boxes, and blenders: Domain adaptation for sentiment classification. Proc 45th Annual Meeting of the ACL, Prague 2007.

    Google Scholar 

  13. 13.

    He J, Liu Y, Lawrence R: Graph-based transfer learning. In Proc 18th ACM Conference on Information and Knowledge Management. Hong Kong; 2009.

    Google Scholar 

  14. 14.

    Hastie T, Tibshirani R, Friedman J: The Elements of Statistical Learning. Second edition. Springer, New York; 2009.

    Google Scholar 

  15. 15.

    accessed 2009, [http://www.cs.cornell.edu/People/pabo/movie-review-data/] accessed 2009

  16. 16.

    accessed Dec 2009, [http://www.lemurproject.org/] accessed Dec 2009

  17. 17.

    Ramakrishnan G, Jadhav A, Joshi A, Chakrabarti S, Bhattacharyya P: Question answering via Bayesian inference on lexical relations. In Proc 41st Annual Meeting of the ACL. Sapporo; 2003.

    Google Scholar 

  18. 18.

    accessed Dec. 2009, [http://www.borgelt.net/bayes.html] accessed Dec. 2009

  19. 19.

    accessed Dec. 2010, [http://www.cs.jhu.edu/~mdredze/] accessed Dec. 2010

  20. 20.

    accessed Aug. 2009, [http://www.mediaislam-bushro.blogspot.com] accessed Aug. 2009

Download references

Acknowledgements

This work was supported by the U.S. Department of Defense and the Laboratory Directed Research and Development Program at Sandia National Laboratories. Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-AC04-94AL85000.

Author information

Affiliations

  1. Institute for Complex Additive Systems Analysis, New Mexico Institute of Mining and Technology, Socorro, USA

    Kristin Glass

  2. Analytics and Cryptography Department, Sandia National Laboratories, Albuquerque, USA

    Richard Colbaugh

Authors
  1. Kristin Glass
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Richard Colbaugh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kristin Glass.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

KG and RC designed the research, KG and RC developed the computational algorithms, KG conducted the empirical tests, and RC wrote the paper. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Glass, K., Colbaugh, R. Estimating the sentiment of social media content for security informatics applications. Secur Inform 1, 3 (2012). https://doi.org/10.1186/2190-8532-1-3

Download citation

Keywords

  • sentiment analysis
  • social media
  • security informatics
  • machine learning