# Detecting Suicidal Ideation in Chinese Microblogs with Psychological
Lexicons

Xiaolei Huang${}^{\textrm{1,2}}$ Lei Zhang${}^{\textrm{1}}$ Tianli
Liu${}^{\textrm{3}}$ David Chiu${}^{\textrm{4}}$ Xin Li${}^{\textrm{*2}}$
Tingshao Zhu${}^{\textrm{*1,5}}$ ${}^{\textrm{1}}$ Institute of Psychology,
Chinese Academy of Sciences(CAS), China  ${}^{\textrm{2}}$ China Networking
Information Center, CAS, China  ${}^{\textrm{3}}$ Institute of Population
Research, Peking University, China  ${}^{\textrm{4}}$ Department of
Mathematics and Computer Science, University of Puget Sound, USA
${}^{\textrm{5}}$ Key Lab of Intelligent Information Processing, Institute of
Computing Technology, CAS, China

###### Abstract

Suicide is among the leading causes of death in China. However, technical
approaches toward preventing suicide are challenging and remaining under
development. Recently, several actual suicidal cases were preceded by users
who posted microblogs with suicidal ideation to Sina Weibo, a Chinese social
media network akin to Twitter. It would therefore be desirable to detect
suicidal ideations from microblogs in real-time, and immediately alert
appropriate support groups, which may lead to successful prevention. In this
paper, we propose a real-time suicidal ideation detection system deployed over
Weibo, using machine learning and known psychological techniques. Currently,
we have identified 53 known suicidal cases who posted suicide notes on Weibo
prior to their deaths. We explore linguistic features of these known cases
using a psychological lexicon dictionary, and train an effective suicidal
Weibo post detection model. $6714$ tagged posts and several classifiers are
used to verify the model. By combining both machine learning and psychological
knowledge, SVM classifier has the best performance of different classifiers,
yielding an F-measure of $68.3\%$, a Precision of $78.9\%$, and a Recall of
$60.3\%$.

##  I Introduction

The World Health Organization (WHO) recently reported that $800,000$ suicidal
deaths occurred worldwide in 2012 [1], which translates to a person committing
suicide every 40 seconds. This unfortunate report places suicide among the
leading causes of death worldwide, particularly for young people ages 15 to
29. In recent years, the suicide problem in China has drawn much attention.
Former research [2] reported that around $290,000$ Chinese people died due to
suicide every year, accounting for roughly $36\%$ of all suicidal cases
worldwide. Although the number has decreased during to past several years, WHO
[1] reported that there were still $120,730$ Chinese people committed suicide
in 2012.

For years, psychologists have been working on suicidal prevention, they found
that suicidal ideation can be clearly identified from victims’ writings and
posts. Wong, et al. analyzed suicide notes obtained from the coroner’s records
in Hong Kong [3]. In this study, they found several characteristics separating
suicidal from non-suicidal people and indicated reasons for the increasing
suicide rates in Hong Kong, including media and social changes. However,
because this data is only obtainable from key players in clinical
investigations, the volume of pre-suicidal data is very limited.

Now in the era of Web 2.0, a multitude of Social Network Sites (SNS), e.g.,
Twitter and Facebook, have emerged as essential platforms for communicating
with others in real-time. Users often express their emotions and opinions by
posts on SNS, which becomes their preferable choices. Jashinsky, et al.
reported suicidal behaviors in Twitter, they pointed out that the association
between social media and suicide has become a public health concern [4].

Due to various reasons, people may not have an appropriate way to alleviate
their depressions in real life. Previous study [5] indicated that some people
preferred to understate symptoms to avoid negative consequences, or some
people lacked the self awareness to report to others, in addition, they might
be lack of trust in others. In several real cases, users chose to post
microblog notes to describe their suicidal ideation on Weibo. However, these
suicidal behaviors on Weibo were not detected, or had simply not raised enough
attention. We believe that early detection of suicidal behaviors on SNS may
help prevent deaths.

In this paper, we build a suicide ideation detection framework to analyze
microblogs with various features, and explore the possibility of monitoring
suicidal behaviors. The current suicidal detection initiatives are very labor-
intensive and ineffective, because they rely heavily on simple keyword
matching and questionnaire. In collaboration with psychologists, we leverage
advanced text analysis and mining integrated with psychological techniques to
build an efficient suicidal ideation detection system over SNS. With Natural
Language Processing (NLP) techniques, we propose to use text analysis
techniques to study latent emotions of users, and then we design, develop, and
experiment a system that can identify suicidal posts in real-time.

Our research has three main contributions, highlighted below.

  * •

First, to the best of our knowledge, there is a lack of formal suicidal
research on Chinese social media. Our research is a unique and exploratory
research on real data set. We have verified 53 suicidal users on Weibo post-
mortem and collected over $30,000$ of their posts.

  * •

Second, we explore NLP techniques with psychological knowledge to find an
approach to predict suicidal posts and achieved F-measure of $68.3\%$. We also
propose an approach to handle imbalanced data between suicidal posts and non-
suicidal posts.

  * •

Third, interdisciplinary research between machine learning and psychology is
nontrivial. We use machine learning skills to verify and prove psychological
hypothesises.

The remainder of this paper is organized as follows. Section II presents the
related work. The background and description of our multi-hierarchical model
will be discussed in Section III. Section IV presents the experimental design
and test results of our proposed system. Finally, future work directions and
conclusion will be discussed in Section V.

##  II Related Works

In this section, we give an overview on suicide detection in psychology, and
then we briefly introduce current research on suicidal ideation detection.

Suicide has been intensively studied in the field of psychology. Traditional
research methods mainly rely on questionnaires. Chen, et al., reported that
“Media Propagation” had significant effects on suicidal behaviors [6]. They
found that simply due to seeing news reports of charcoal burning suicides, the
rate of charcoal burning suicide increased. Li, et al., studied one 13-year-
old youth’s 193 blog entries by using the Chinese Linguistic Inquiry and Word
Count (CLIWC), and found several key features in his suicidal notes, including
posting frequency, progressive self-referencing, and positive to negative
emotion words ratio [7].

In another study, Delgado-Gomez, et al., acquired 849 cases from Spain
hospital and scored them manually by psychological standards or questionnaires
[8]. They used these scores to evaluate four classifiers, including Boosting,
Linear Discriminate Analysis, Fisher Linear Discriminate Analysis and Support
Vector Machine (SVM). They found that the score, which was analyzed manually
by doctors, could be used to identify suicidal person using classifiers.

Opinion mining and sentiment analysis techniques have also been applied in
various fields [9], and increasing numbers of psychologists use social media
search engines to find interesting patterns in suicidal entries, such as
sentiment phrases, geographic location, etc. Vincent, et al., explored the
relationship between suicide and sexual orientation [10]. Jared, et al.,
tracked data from Twitter to find out suicide risk difference between states
[4].

Applying machine learning techniques in suicide identification has gained
traction in recent years. Linguistic Inquiry and Word Count Version 2007
(LIWC2007) evaluates various emotional, cognitive, and structural English
words or phrases presented in individuals’ verbal and written sentences [11].
LIWC could be used to identify a post’s emotional tendency. Ramirez-Esparza,
et al., worked on linguistic markers and themes of depression discussion by
collecting information both in depressed and non-depressed individuals from
legacy Internet forums using bulletin board systems (BBS) [12]. They analyzed
the text with the help of LIWC, and they found that online depressed writers
used more first person singular pronouns but less first-person plural pronouns
in both English and Spanish forums. They also found that depressed people who
wrote in Spanish were more likely to relational concerns; whereas depressed
people who wrote in English were more likely to mention medical concerns.

Huang, et al., used basic keywords matching (similar to regular expressions)
over Myspace.com to discover whether users have an intent to commit suicide
[13]. Their method is simple and recognized 14% of the known suicidal users.
Li, et al., later proposed a hybrid method [14]. They noted importance of
applying machine learning skills into emotional distress detection, like SVM.

Wang, et al., worked on classifying English clinical suicide notes into 16
different emotional categories, such as blame, fear, love, etc. [15]. Their
data is based on the fifth I2B2 (Informatics for Integrating Biology and the
Bedside) challenge [16], and they used SVM classifier to test different
features, with the result of F-measure around $50.04\%$, and showed the
possibility for applying machine learning into predicting suicide notes.
Liakata, et al. worked on the same data. They used a hybrid classifier, which
includes both SVM classifier and CRF classifier, with the best F-measure
$45.64\%$ [17]. Pestian, et al., applied NLP techniques on 66 suicide notes
and selected reading scores, words, concepts and other 63 features [18]. They
used over ten different methods in Weka [19], an useful machine learning tool
to classify their data, and they achieved an impressive precision of $74.4\%$,
compared to $51.0\%$ from psychiatric physician trainees and $60.9\%$ from
mental health professionals.

Wang, et al., proposed a Chinese depression detection system [20]. They used
three classifiers from Weka, and their analysis and experiment data included
one verified suicidal user’s data, which contained 215 posts. Their research
could achieve their best F-measure of $85\%$.

##  III Methods

In this section, we present the background and detailed description of our
system and the algorithms involved.

An overview of our three-level suicide detection framework is shown in Fig. 1.
Posts collected from our crawler are used to extract several basic attributes:
1) text contents; 2) user’s behaviors, such as posting time, mentions, posting
type; 3) social interactions, such as the users who comment or like this post.
With the help of linguistic dictionaries, we are able to extract a set of
features from those basic attributes as the first step, such as keywords,
negations, negative emotions, and so on. Then, we take a sample and assign
weights to process the data to solve imbalanced data. Imbalanced data refers
to any data set that exhibits an unequal distribution between its classes[21].
In our data set, none suicidal posts are much more than suicidal posts.
Finally, classifiers are applied to learn the suicidal features and build a
suicidal detection model.

Figure 1: Real-Time Suidical Detection Framework

###  III-A Data Collection and Tagging

Our data is collected through Sina Microblog Open Platform API [22] and our
Java-based crawler. We downloaded 53 verified suicidal users’ profiles and
over $30,000$ posts. In addition, we also collected $600,000$ posts from
$1,000$ thousand random non-suicidal users. We curated all suicidal users’
posts and obtained $614$ true suicidal posts. To perform a 90-10 test, we
randomly sampled $6,140$ posts from the set of non-suicidal users, for a total
of $6,754$ posts. After filtering some blank posts, we obtain $6,704$ posts.

###  III-B Lexicon Construction

Because no lexicons specifically for suicidal word or phrases are available,
we extend our psychological lexicons with help of the latest version of HowNet
[23], a Chinese-based emotional-words resource for sentiment analysis.
Psychological dictionaries contain some depressed words or phrases often used
in cases of suicide notes. We use those dictionaries to detect and count the
number of positive, negative, neutral words or phrases, and the ratio of
positive to negative words. Examples of Chinese words and phrases in HowNet
are depicted in Table I below, and examples of psychological dictionaries are
shown in Table II.

TABLE I: Emotional words in HowNet Emotion Words | Number | Example
---|---|---
Negative | 4370 | bad, sad, tragedy, coward
Positive | 4566 | happy, safe, amiable, calm
TABLE II: Psychological Dictionary Type | Number | Example
---|---|---
Terms | 3453 | dead, psychologist, insomnia, stilnox
|  | depression, estazolam, schizophrenia

###  III-C Feature Extraction and Modeling

We use Ansj [24], a Chinese word segmentation tool, to segment words and
remove punctuation and stop words. We choose three basic $N$-gram features:
Unigram, Bigram and Trigram. We use emotional words from lexicon to find out
its emotional polarity in the sentence, and we also compute the ratio of
negative to positive emotion words.

Li, et al., found that suicidal victims tend to write more about themselves
and less about others in posting entries, such as I, me, myself, we, etc. [7].
During our research, we found that victims also like to mention their parents
or grandparents, brothers or sisters, or even other suicidal victims. The
statistics of references is shown in Table III.

TABLE III: Personal Reference Statistics Type | Suicide Ratio | None Suicide Ratio | Example
---|---|---|---
Self-references | 68% | 35% | I, myself, me
Other-references | 28% | 5% | Dad, mother, grandpa

Syntactic features contains dependency relation, Part of Speech (POS) tagging
and sentence tense. We use Stanford Parser Core NLP [25] to obtain the
statistics of POS tag for each sentence. Previous research [18] showed suicide
notes have some distinct POS tag categories. Because the sentences on web are
usually complex and ambiguous, we only obtained three types of POS tags by
counting the numbers of words in the following: adjective, noun, verb.

We also found that temporal features matter. In our data, nearly $40\%$ of
suicidal posts are posted at night or early in the morning from 23:00 to 06:00
in the morning, whereas only $13\%$ of non-suicidal users posted during this
period. A plausible explanation might be some suicidal users suffer from
insomnia and post about hypnotic pills, such as tranquillizer or stilnox. In
contrast, only $13\%$ suicidal posts are posted in the morning from 07:00 to
13:00, and over $28\%$ of non-suicidal posts are posted during this same
period. Thus, we separate 24 hours into four time periods: 23:00 to 06:00,
07:00 to 13:00, 14:00 to 18:00, and 18:00 to 23:00. The more specific
information are shown in Fig. 2.

Figure 2: Posting Time Comparison Figure 3: Post’s Type Comparison

Posting type refers to original creation or forward posts, which is another
prominent factor. Although forwarding others’ posts (i.e., retweet or
propagation) has some effects on users’ behaviors, it appears in our data that
suicidal users prefer to post original thoughts. In our data, original posts
account for $87\%$ of tagged suicidal posts, and in contrast, original
creation only accounts for $33\%$ of non-suicidal users’ posts. The more
specific comparisons are shown in Fig. 3.

Another feature is posting frequency as time varies, which may be caused by
the users’ emotional variations. Research shows that the deceased posted less
when they were happier [7]. In addition, depressed users may post longer
emotional posts than people who commit suicide impulsively, and this
phenomenon is also reflected in our own research. Thus, for every post in our
data set, we compute its cycle’s frequency, and then compare it with its
users’ average posting frequency.

Since the data is skewed, we use over-sampling to solve this problem, and
design a function to compute and change automatically the initial value of
each feature. We consider the basic weight computation in three aspects.

  * •

First, the ratio of suicidal posts to none suicidal posts is chosen for
simulating real-world data sets. We denote the number of suicidal posts as
$N_{s}$ and the number of non-suicidal posts as $N_{n}$.

  * •

Second, the occurrence of each word or other features, such as occurrence of
original created posts and forward posts, reflect their significance in posts.
In contrast to TF-IDF [26] used in information retrieval, more frequent
occurrences may be more significant, if the words are self-references or
tagged suicidal. We consider its occurrence ratio between suicidal posts to
non-suicidal posts here. We denote occurrence in each suicidal post as $Q_{s}$
and occurrence in each none suicidal post as $Q_{n}$.

  * •

Third, we consider several properties of lexicon dictionaries. We consider two
properties: the polarity of dictionaries, such as positive or negative and its
lexicon size.

The final feature’s initial weight value $W_{o}$, such as one or zero, should
multiply an additional weight function, which is aim to solve skewed data
problem:

$\displaystyle\displaystyle
weight=W_{o}\times\log\left[\frac{\displaystyle\sum_{i=1}^{N_{s}}{Q_{s}}}{\displaystyle\sum_{i=1}^{N_{s}}{Q_{n}+1}}\right]\times\frac{N_{n}}{N_{s}}$
(1)

##  IV Experimental Evaluation

In this section, we describe our experiment approach and discuss the results.
Our experiment’s classification results are evaluated by “Precision,”
“Recall,” “F-measure,”, and “Accuracy”. We take whether the classifier could
correctly detect the suicidal posts for “Precision,” “Recall,” and
“F-measure,” respectively. “Accuracy” refers to all predictions match their
labels regardless whether it is suicidal posts or not. We use 10-fold-cross
validation to ensure proper separation of our $6704$ experimental data
entries, and use different classifiers to perform classification.

We use the SVM classifier from LibSVM [27] package for C and Java. The
classifier in our experiments used a Radial Basis Function (RBF) kernel. For
parameters, we use grid search with $C=16$ and $\gamma=3.0517578125e-5$. We
choose several classifiers from Weka [19]: NaiveBayes, LogisticRegression, J48
Classifier, RandomForest and SMO. All classifiers were run with default
parameter values, and the results are presented in Table IV.

TABLE IV: Cross-validation of performance on different classifiers | F-measure | Precision | Recall | Accuracy
---|---|---|---|---
NaiveBayes | $40.2\%$ | 79.0% | $26.9\%$ | $92.6313\%$
LogisticRegression | 39.3% | 55.8% | 26.9% | 88.0072%
J48 | 46.6% | 68.9% | 35.2% | 92.5865%
RandomForest | 37.4% | 74.0% | 25.0% | 92.3031%
SMO | 56.8% | 68.7% | 48.4% | 93.2279%
SVM | 68.3% | 78.9% | 60.3% | 94.0036%

Table IV presents the performance of each classifier. Clearly, we can see from
Table IV that the SVM classifier achieves the best “F-measure” with $68.3\%$,
“Recall” with $60.3\%$, and an “Accuracy” over $94\%$. If we take all posts
into consideration, the “Accuracy” reaches over $94\%$, which means the
possibility that this suicide detection system can assist psychologists in
identifying true suicidal posts. It will be a great help for suicide
prevention.

However, we find that $40\%$ of suicidal posts are not identified correctly.
This may mainly be due to the complexity and ambiguity of Chinese sentences on
the Internet. In addition, if we extracted more features and applied more
advanced text mining methods, we might achieve better results, which will be
in our future work.

From the feature selection Section III-C, we can find the preferences of
people with high risk of suicide.

  * •

Suicidal users prefer to post their suicidal posts at night.

  * •

Suicidal users prefer to post suicidal posts originally instead of forwarding
others’ posts, though others’ emotions will propagate and affect them.

  * •

Suicidal users prefer to mention themselves more frequently than none suicidal
users.

This research has a number of potential applications. The trained model can be
used to build an alert system, and the system is urgent for preventing and
intervening suicidal users. If this system is effective in detecting suicidal
posts from social networks, it may provide psychologists with advanced
decision support.

##  V Conclusion & Future Work

In this paper, we build a suicidal ideation detection system using
psychological lexicons. Based on psychological lexicons, we use machine
learning techniques to detect suicide ideation in posts from social networks,
which is a new approach to investigate people’s online behavior. Since the
social networks provide people with a new way of communication, to let them
express their emotion, which may include posts with suicide ideation, it is
very helpful to build a monitoring system that could help psychologists
analyze those posts, to identify psychological characteristics, such as
suicide. In our work, the precision of suicide ideation is $79\%$, and overall
the accuracy can reach $94\%$. The purpose of this research is to draw
people’s attention to problem of suicide and to show that how useful it is, if
we could use both machine learning and psychology knowledge for identifying
suicide ideation.

Our future work is undertaken in three directions: “Topic Model”, “Latent
Social Relationship”, and “Deep Learning”. Topic Model [28] is a very useful
tool to analyze text. It may find latent semantics in suicide posts, and it
also can find dependency on lexicons. We also plan to use latent social
network relationships to identify suicide individuals and groups. People who
have interactions in their posts might propagate information to others. Some
research have been done on this topic [6], and shed lights on this part in
media propagation. Deep learning is a very popular machine learning technique
in recent years. Applying deep learning techniques in detecting suicide post
text may help us build more efficient features to improve the performance of
prediction.

## Acknowledgments

The authors gratefully acknowledge the generous support from National High-
tech R&D Program of China (2013AA01A606), National Basic Research Program of
China(973 Program 2014CB744600), Key Research Program of CAS (KJZD-EW-L04) and
Strategic Priority Research Program (XDA06030800) from Chinese Academy of
Sciences. We would like to thank Hongbo Chen and Xiongcai Luo for insightful
discussions and suggestions, and special thank Google Summer of Code 2014 and
Portland State University for sponsoring the open-source development of this
project.

## References

  * [1] W. H. Organization, “Preventing Preventing suicide:A global imperative,” Website, 2014, http://www.who.int/mental_health/suicide-prevention/en/.
  * [2] R. Michael, X. Li, and Y. Zhang, “Suicide rates in china, 1995-1999,” _Lancet_ , vol. 359, no. 9309, pp. 835–840, 2002.
  * [3] P. W. Wong, A. W. Yeung, W. S. Chan, P. S. Yip, and A. K. Tang, “Suicide notes in hong kong in 2000,” _Death studies_ , vol. 33, no. 4, pp. 372–381, 2009\.
  * [4] J. Jashinsky, S. H. Burton, C. L. Hanson, J. West, C. Giraud-Carrier, M. D. Barnes, and T. Argyle, “Tracking suicide risk factors through twitter in the us.” 2013.
  * [5] S. E. Finn and J. H. Kamphuis, “The mmpi–2 restructured clinical (rc) scales and restraints to innovation, or” what have they done to my song?”,” _Journal of personality assessment_ , vol. 87, no. 2, pp. 202–210, 2006.
  * [6] Y.-Y. Chen, F. Chen, D. Gunnell, and P. S. Yip, “The impact of media reporting on the emergence of charcoal burning suicide in taiwan,” _PloS one_ , vol. 8, no. 1, p. e55000, 2013.
  * [7] T. M. Li, M. Chau, P. S. Yip, and P. W. Wong, “Temporal and computerized psycholinguistic analysis of the blog of a chinese adolescent suicide.” 2014\.
  * [8] D. Delgado-Gomez, H. Blasco-Fontecilla, A. A. Alegria, T. Legido-Gil, A. Artes-Rodriguez, and E. Baca-Garcia, “Improving the accuracy of suicide attempter classification,” _Artificial intelligence in medicine_ , vol. 52, no. 3, pp. 165–168, 2011.
  * [9] B. Pang and L. Lee, “Opinion mining and sentiment analysis,” _Foundations and trends in information retrieval_ , vol. 2, no. 1-2, pp. 1–135, 2008.
  * [10] V. Silenzio, P. R. Duberstein, W. Tang, N. Lu, X. Tu, and C. M. Homan, “Connecting the invisible dots: Reaching lesbian, gay, and bisexual adolescents and young adults at risk for suicide through online social networks,” _Social Science & Medicine_, vol. 69, no. 3, pp. 469–474, 2009\.
  * [11] C. Chung and J. W. Pennebaker, “The psychological functions of function words,” _Social communication_ , pp. 343–359, 2007.
  * [12] N. Ramirez-Esparza, C. K. Chung, E. Kacewicz, and J. W. Pennebaker, “The psychology of word use in depression forums in english and in spanish: Texting two text analytic approaches.” in _ICWSM_ , 2008.
  * [13] Y.-P. Huang, T. Goh, and C. L. Liew, “Hunting suicide notes in web 2.0-preliminary findings,” in _Multimedia Workshops, 2007. ISMW’07. Ninth IEEE International Symposium on_. IEEE, 2007, pp. 517–521.
  * [14] T. M. Li, M. Chau, P. W. Wong, and P. S. Yip, “A hybrid system for online detection of emotional distress,” in _Intelligence and Security Informatics_. Springer, 2012, pp. 73–80.
  * [15] W. Wang, L. Chen, M. Tan, S. Wang, and A. P. Sheth, “Discovering fine-grained sentiment in suicide notes,” _Biomedical informatics insights_ , vol. 5, no. Suppl 1, p. 137, 2012.
  * [16] J. P. Pestian, P. Matykiewicz, M. Linn-Gust, B. South, O. Uzuner, J. Wiebe, K. B. Cohen, J. Hurdle, and C. Brew, “Sentiment analysis of suicide notes: A shared task,” _Biomedical informatics insights_ , vol. 5, no. Suppl 1, p. 3, 2012.
  * [17] M. Liakata, J.-H. Kim, S. Saha, J. Hastings, and D. Rebholz-Schuhmann, “Three hybrid classifiers for the detection of emotions in suicide notes,” _Biomedical informatics insights_ , vol. 5, no. Suppl 1, p. 175, 2012.
  * [18] J. Pestian, H. Nasrallah, P. Matykiewicz, A. Bennett, and A. Leenaars, “Suicide note classification using natural language processing: A content analysis,” _Biomedical informatics insights_ , vol. 2010, no. 3, p. 19, 2010\.
  * [19] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The weka data mining software: an update,” _ACM SIGKDD explorations newsletter_ , vol. 11, no. 1, pp. 10–18, 2009.
  * [20] X. Wang, C. Zhang, Y. Ji, L. Sun, L. Wu, and Z. Bao, “A depression detection model based on sentiment analysis in micro-blog social network,” in _Trends and Applications in Knowledge Discovery and Data Mining_. Springer, 2013, pp. 201–213.
  * [21] H. He and E. A. Garcia, “Learning from imbalanced data,” _Knowledge and Data Engineering, IEEE Transactions on_ , vol. 21, no. 9, pp. 1263–1284, 2009\.
  * [22] Sina, “Sina Weibo open platform,” Website, 2014, http://open.weibo.com/.
  * [23] Z. Dong and Q. Dong, “Hownet-a hybrid language and knowledge resource,” in _Natural Language Processing and Knowledge Engineering, 2003. Proceedings. 2003 International Conference on_. IEEE, 2003, pp. 820–824.
  * [24] J. Sun, “Ansj Chinese Lexical Analysis System,” Website, 2014, http://ansj.org/.
  * [25] K. Toutanova, D. Klein, C. D. Manning, and Y. Singer, “Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network,” in _North American Chapter of the Association for Computational Linguistics_ , 2003.
  * [26] G. Salton and M. J. McGill, “Introduction to modern information retrieval,” 1983\.
  * [27] C.-C. Chang and C.-J. Lin, “Libsvm: a library for support vector machines,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 2, no. 3, p. 27, 2011.
  * [28] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” _the Journal of machine Learning research_ , vol. 3, pp. 993–1022, 2003.