PowerPoint

: 6647 : .

Social media

Twitter 140 310 500 tweets Top10 sites

: 2

+1 +

2

3

3

4

Web Scraping5

Web Scraping6 HTML

Web Scraping7 HTML

Web Scraping

8 Document

9

Named EntitiesStanford NER

10

Proper Nounsarktweetnlp

11

Adjective - NounStanford POS Tagger

12

Noun SequencesStanford POS Tagger

13

NounsStanford POS Tagger

14

Noun PhrasesOenNLPTagger

15

KeywordsJsoup16

17 Document

18

queriesThree Spanish journalists were released after being abducted around Aleppo, Syria, and held hostage by the Al-Nusra Front since 13 July 2015."july" AND "abducted""spanish journalists" AND "aleppo""july" AND "al-nusra""spanish journalists" AND "abducted""july" AND "front""spanish journalists" AND "al-nusra""journalists" AND "aleppo""aleppo" AND "13 july""journalists" AND "13 july""aleppo" AND "abducted""journalists" AND "al-nusra""abducted" AND "al-nusra""aleppo syria" AND "spanish journalists""abducted" AND "front""aleppo syria" AND "abducted""abducted" AND "13 july"

19 Queries

term1 AND term1

term1 term2 term3 AND term1

20

Twitter REST API Java Library Twitter4jTweets 7 450 requests / 15min

PagingNo RetweetsLanguage : English

Tweets

21News Date# NewsAverage Queries/NewsAverage Tweets/NewsDB Size06/05/201612464,760328444.98 MB08/05/201611101,5736855.92 MB

22

Apache Lucene 5.5.0Inverted index

Indexing

23

24

25Information Retrieval

Vector Space Model26

27Lucene Scoring

BM25 . tf idf27

28Language models

28

PRF; :1 IR (Pseudo) Relevance Set (Pseudo) Relevance Set (Pseudo) Relevance Feedback29Pseudo-Relevance Feedback

30Relevance Models

tweets31

*******TFIDFBM25AverageMaxscoreSumAverageMaxscoreSum10.950.95111120.750.9010.900.951311111140.950.90111150.85110.9011611111170.60110.951180.200.200.450.250.300.60911111110111111110.100.100.100.100.100.10Average Precision0.7640.8240.8680.8270.8500.882

tweets32

#TweetRelevant1travelling to Taif, Saudi ArabiaNo2Trophee Hassan Policeman Killed in Shootout With MilitantsYes3Saudi policeman killed in shootout with militants: Saudi Arabia says a policeman was killed in a shootout with https://t.co/p2cY4KWocmYes4Saudi Policeman Killed in Shootout With Militants: Saudi Arabia says a policeman was killed in a shootout wit... https://t.co/BxP885EDBQYes5Saudi Policeman Killed in Shootout With Militants: Saudi Arabia says a policeman was killed in a shootout with https://t.co/lB1ePd2lcvYes6Saudi policeman killed in shootout with militants - Saudi Arabia says a policeman was killed in a shootout with... https://t.co/iORDNawewqYes7Saudi Policeman Killed in Shootout With Militants - Saudi Arabia says a policeman was killed in a shootout with... https://t.co/iM6gzcyTGSYes8Saudi Policeman Killed in Shootout With Militants: Saudi Arabia says a policeman was killed in a shootout with militants in the weste...Yes9#Rainbow over #Taif, #Saudi, the weather!No10Saudi Arabia says a policeman was killed in a shootout with militants in the western city of Taif Yes

tweetsA policeman is killed in a shootout with militants in Taif, Saudi Arabia.

TFIDF,MAXSCORE

33

tweets : AllTweets: tweets : tweetsK: AllTweetsSimilar: tweets : FilTweets: Top 100 tweets duplicates: = 1 = 1FilTweets[N] = AllTweets[K] = +1Similar = i 1 CosSim( AllTweets[K] , FilTweets[i] ) > 0,5 Similar = __ Similar == FilTweets[N]=AllTweets[K]_{