Ζησόπουλος...
date post
18-Jan-2017Category
Software
view
16download
0
Embed Size (px)
Transcript of Ζησόπουλος...
PowerPoint
: 6647 : .
Social media
Twitter 140 310 500 tweets Top10 sites
: 2
+1 +
2
3
3
4
Web Scraping5
Web Scraping6 HTML
Web Scraping7 HTML
Web Scraping
8 Document
9
Named EntitiesStanford NER
10
Proper Nounsarktweetnlp
11
Adjective - NounStanford POS Tagger
12
Noun SequencesStanford POS Tagger
13
NounsStanford POS Tagger
14
Noun PhrasesOenNLPTagger
15
KeywordsJsoup16
17 Document
18
queriesThree Spanish journalists were released after being abducted around Aleppo, Syria, and held hostage by the Al-Nusra Front since 13 July 2015."july" AND "abducted""spanish journalists" AND "aleppo""july" AND "al-nusra""spanish journalists" AND "abducted""july" AND "front""spanish journalists" AND "al-nusra""journalists" AND "aleppo""aleppo" AND "13 july""journalists" AND "13 july""aleppo" AND "abducted""journalists" AND "al-nusra""abducted" AND "al-nusra""aleppo syria" AND "spanish journalists""abducted" AND "front""aleppo syria" AND "abducted""abducted" AND "13 july"
19 Queries
term1 AND term1
term1 term2 term3 AND term1
20
Twitter REST API Java Library Twitter4jTweets 7 450 requests / 15min
PagingNo RetweetsLanguage : English
Tweets
21News Date# NewsAverage Queries/NewsAverage Tweets/NewsDB Size06/05/201612464,760328444.98 MB08/05/201611101,5736855.92 MB
22
Apache Lucene 5.5.0Inverted index
Indexing
23
24
25Information Retrieval
Vector Space Model26
27Lucene Scoring
BM25 . tf idf27
28Language models
28
PRF; :1 IR (Pseudo) Relevance Set (Pseudo) Relevance Set (Pseudo) Relevance Feedback29Pseudo-Relevance Feedback
30Relevance Models
tweets31
*******TFIDFBM25AverageMaxscoreSumAverageMaxscoreSum10.950.95111120.750.9010.900.951311111140.950.90111150.85110.9011611111170.60110.951180.200.200.450.250.300.60911111110111111110.100.100.100.100.100.10Average Precision0.7640.8240.8680.8270.8500.882
tweets32
#TweetRelevant1travelling to Taif, Saudi ArabiaNo2Trophee Hassan Policeman Killed in Shootout With MilitantsYes3Saudi policeman killed in shootout with militants: Saudi Arabia says a policeman was killed in a shootout with https://t.co/p2cY4KWocmYes4Saudi Policeman Killed in Shootout With Militants: Saudi Arabia says a policeman was killed in a shootout wit... https://t.co/BxP885EDBQYes5Saudi Policeman Killed in Shootout With Militants: Saudi Arabia says a policeman was killed in a shootout with https://t.co/lB1ePd2lcvYes6Saudi policeman killed in shootout with militants - Saudi Arabia says a policeman was killed in a shootout with... https://t.co/iORDNawewqYes7Saudi Policeman Killed in Shootout With Militants - Saudi Arabia says a policeman was killed in a shootout with... https://t.co/iM6gzcyTGSYes8Saudi Policeman Killed in Shootout With Militants: Saudi Arabia says a policeman was killed in a shootout with militants in the weste...Yes9#Rainbow over #Taif, #Saudi, the weather!No10Saudi Arabia says a policeman was killed in a shootout with militants in the western city of Taif Yes
tweetsA policeman is killed in a shootout with militants in Taif, Saudi Arabia.
TFIDF,MAXSCORE
33
tweets : AllTweets: tweets : tweetsK: AllTweetsSimilar: tweets : FilTweets: Top 100 tweets duplicates: = 1 = 1FilTweets[N] = AllTweets[K] = +1Similar = i 1 CosSim( AllTweets[K] , FilTweets[i] ) > 0,5 Similar = __ Similar == FilTweets[N]=AllTweets[K]_{