User:Bohye Woo/nltk-Terms of Service
Virtual Environment
To create a virtual environment
cd to the place you want to make it and...
python3 -m venv venv
To activate a virtual environment
cd to the folder where "venv" is and...
source venb/bin/activate
NLTK
I started using FaceApp's ToS for NLTK experiments. Original Terms of Service from FaceApp:
Any questions, comments, suggestions, ideas, original or creative materials or other information you submit about FaceApp or our products or Services (collectively, “Feedback”), is non-confidential and we have no obligations (including without limitation obligations of confidentiality) with respect to such Feedback. You hereby grant to FaceApp a fully paid, royalty-free, perpetual, irrevocable, worldwide, non-exclusive, and fully sublicensable right and license to use, reproduce, perform, display, distribute, adapt, modify, re-format, create derivative works of, and otherwise commercially or non-commercially exploit in any manner, any and all Feedback, and to sublicense the foregoing rights, in connection with the operation and maintenance of the Services and/or FaceApp’s business.
- If you choose to login to the Services via a third-party platform or social media network, you will need to use your credentials (e.g., username and password) from a third-party online platform. You must maintain the security of your third party account and promptly notify us if you discover or suspect that someone has accessed your account without your permission. If you permit others to use your account credentials, you are responsible for the activities of such users that occur in connection with your account.
- 4. User Content
Our Services may allow you and other users to create, post, store and share content, including photos, videos, messages, text, software and other materials (collectively, “User Content”). User Content does not include user-generated filters. Subject to this Agreement and the Privacy Policy, you retain all rights in and to your User Content, as between you and FaceApp. Further, FaceApp does not claim ownership of any User Content that you post on or through the Services. You grant FaceApp a nonexclusive, royalty-free, worldwide, fully paid license to use, reproduce, modify, adapt, create derivative works from, distribute, perform and display your User Content during the term of this Agreement solely to provide you with the Services.
You acknowledge that some of the Services are supported by advertising revenue and may display advertisements and promotions, and you hereby agree that FaceApp may place such advertising and promotions on the Services or on, about, or in conjunction with your User Content. The manner, mode and extent of such advertising and promotions are subject to change without specific notice to you. You acknowledge that we may not always identify paid services, sponsored content, or commercial communications as such.
You represent and warrant that: (i) you own or otherwise have the right to use the User Content modified by you on or through the Services in accordance with the rights and licenses set forth in this Agreement; (ii) you agree to pay for all royalties, fees, and any other monies owed by reason of User Content you stylize on or through the Services; and (iii) you have the legal right and capacity to enter into this Agreement in your jurisdiction.
You may not create, post, store or share any User Content that violates this Agreement or for which you do not have all the rights necessary to grant us the license described above. Although we have no obligation to screen, edit or monitor User Content, we may delete or remove User Content at any time and for any reason.
FaceApp is not a backup service and you agree that you will not rely on the Services for the purposes of User Content backup or storage. FaceApp will not be liable to you for any modification, suspension, or discontinuation of the Services, or the loss of any User Content.
Tokenize
>>> import nltk
>>> text = "If you choose to login to the Services via a third-party platform or social media network, you will need to use your credentials."
>>> token = nltk.word_tokenize(text)
>>> token
['If', 'you', 'choose', 'to', 'login', 'to', 'the', 'Services', 'via', 'a', 'third-party', 'platform', 'or', 'social', 'media', 'network', ',', 'you', 'will', 'need', 'to', 'use', 'your', 'credentials', '.']
sort
>>> token.sort()
>>> token
[',', '.', 'If', 'Services', 'a', 'choose', 'credentials', 'login', 'media', 'need', 'network', 'or', 'platform', 'social', 'the', 'third-party', 'to', 'to', 'to', 'use', 'via', 'will', 'you', 'you', 'your']
collections
>>> import collections
>>> collections.Counter(token)
Counter({'to': 3, 'you': 2, ',': 1, '.': 1, 'If': 1, 'Services': 1, 'a': 1, 'choose': 1, 'credentials': 1, 'login': 1, 'media': 1, 'need': 1, 'network': 1, 'or': 1, 'platform': 1, 'social': 1, 'the': 1, 'third-party': 1, 'use': 1, 'via': 1, 'will': 1, 'your': 1})
Concordance
nltk.download("stopwords")
file=open('faceapp.txt','r')
raw=file.read()
tokens = nltk.word_tokenize(raw)
faceapp = nltk.Text(tokens)
faceapp.concordance('services')
//////result///////
Displaying 11 of 11 matches:
t about FaceApp or our products or Services ( collectively , “ Feedback ” ) ,
e operation and maintenance of the Services and/or FaceApp ’ s business . - If
. - If you choose to login to the Services via a third-party platform or soci
connection with your account . Our Services may allow you and other users to c
nt that you post on or through the Services . You grant FaceApp a nonexclusive
ent solely to provide you with the Services . You acknowledge that some of the
. You acknowledge that some of the Services are supported by advertising reven
advertising and promotions on the Services or on , about , or in conjunction
at we may not always identify paid services , sponsored content , or commercia
modified by you on or through the Services in accordance with the rights and
tent you stylize on or through the Services ; and ( iii ) you have the legal r
Similar
nltk.download("stopwords")
file=open('faceapp.txt','r')
raw=file.read()
tokens = nltk.word_tokenize(raw)
faceapp = nltk.Text(tokens)
faceapp.concordance('services')
//////result///////
rights operation
Output top 50 words
import sys
import codecs
import nltk
from nltk.corpus import stopwords
# NLTK's default English stopwords
default_stopwords = set(nltk.corpus.stopwords.words('english'))
#read stop words from a file (one stopword per line, UTF-8)
stopwords_file = './stopwords.txt'
custom_stopwords = set(codecs.open('stopwords.txt', 'r', 'utf-8').read().splitlines())
all_stopwords = default_stopwords | custom_stopwords
file = open('faceapp.txt','r')
raw = file.read()
tokens = nltk.word_tokenize(raw)
faceapp = nltk.Text(tokens)
# Remove single-character tokens (mostly punctuation)
tokens = [word for word in tokens if len(word) > 1]
# Remove numbers
tokens = [word for word in tokens if not word.isnumeric()]
# Lowercase all words (default_stopwords are lowercase too)
tokens = [word.lower() for word in tokens]
# Remove stopwords
tokens = [word for word in tokens if word not in all_stopwords]
# Calculate frequency distribution
fdist = nltk.FreqDist(tokens)
# Output top 50 words
for word, frequency in fdist.most_common(10):
print(u'{};{}'.format(word, frequency))
//////result///////
content;16
user;14
services;13
faceapp;9
may;6
use;5
agreement;5
create;4
rights;4
account;4
feedback;3
without;3
grant;3
fully;3
paid;3
right;3
license;3
display;3
post;3
advertising;3
promotions;3
agree;3
materials;2
collectively;2
obligations;2
including;2
hereby;2
royalty-free;2
worldwide;2
reproduce;2
perform;2
distribute;2
adapt;2
modify;2
derivative;2
works;2
otherwise;2
manner;2
connection;2
third-party;2
platform;2
credentials;2
us;2
users;2
store;2
share;2
subject;2
acknowledge;2
reason;2
backup;2
Using a Tagger: Pos_tagger
>>> from nltk import sent_tokenize, word_tokenize, pos_tag
>>> text = "You hereby grant to FaceApp a fully paid, royalty-free, perpetual, irrevocable, worldwide, non-exclusive, and fully sublicensable right and license to use, reproduce, perform, display, distribute, adapt, modify, re-format, create derivative works of, and otherwise commercially or non-commercially exploit in any manner, any and all Feedback, and to sublicense the foregoing rights, in connection with the operation and maintenance of the Services and/or FaceApp’s business."
>>> sent_tokenize(text)
['You hereby grant to FaceApp a fully paid, royalty-free, perpetual, irrevocable, worldwide, non-exclusive, and fully sublicensable right and license to use, reproduce, perform, display, distribute, adapt, modify, re-format, create derivative works of, and otherwise commercially or non-commercially exploit in any manner, any and all Feedback, and to sublicense the foregoing rights, in connection with the operation and maintenance of the Services and/or FaceApp’s business.']
>>> [word_tokenize(sent) for sent in sent_tokenize(text)]
[['You', 'hereby', 'grant', 'to', 'FaceApp', 'a', 'fully', 'paid', ',', 'royalty-free', ',', 'perpetual', ',', 'irrevocable', ',', 'worldwide', ',', 'non-exclusive', ',', 'and', 'fully', 'sublicensable', 'right', 'and', 'license', 'to', 'use', ',', 'reproduce', ',', 'perform', ',', 'display', ',', 'distribute', ',', 'adapt', ',', 'modify', ',', 're-format', ',', 'create', 'derivative', 'works', 'of', ',', 'and', 'otherwise', 'commercially', 'or', 'non-commercially', 'exploit', 'in', 'any', 'manner', ',', 'any', 'and', 'all', 'Feedback', ',', 'and', 'to', 'sublicense', 'the', 'foregoing', 'rights', ',', 'in', 'connection', 'with', 'the', 'operation', 'and', 'maintenance', 'of', 'the', 'Services', 'and/or', 'FaceApp', '’', 's', 'business', '.']]
>>> [pos_tag(word_tokenize(sent))for sent in sent_tokenize(text)]
[[('You', 'PRP'), ('hereby', 'VBP'), ('grant', 'JJ'), ('to', 'TO'), ('FaceApp', 'VB'), ('a', 'DT'), ('fully', 'RB'), ('paid', 'VBN'), (',', ','), ('royalty-free', 'JJ'), (',', ','), ('perpetual', 'JJ'), (',', ','), ('irrevocable', 'JJ'), (',', ','), ('worldwide', 'JJ'), (',', ','), ('non-exclusive', 'JJ'), (',', ','), ('and', 'CC'), ('fully', 'RB'), ('sublicensable', 'JJ'), ('right', 'NN'), ('and', 'CC'), ('license', 'NN'), ('to', 'TO'), ('use', 'VB'), (',', ','), ('reproduce', 'VB'), (',', ','), ('perform', 'VB'), (',', ','), ('display', 'NN'), (',', ','), ('distribute', 'NN'), (',', ','), ('adapt', 'NN'), (',', ','), ('modify', 'VB'), (',', ','), ('re-format', 'JJ'), (',', ','), ('create', 'JJ'), ('derivative', 'JJ'), ('works', 'NNS'), ('of', 'IN'), (',', ','), ('and', 'CC'), ('otherwise', 'RB'), ('commercially', 'RB'), ('or', 'CC'), ('non-commercially', 'RB'), ('exploit', 'NNS'), ('in', 'IN'), ('any', 'DT'), ('manner', 'NN'), (',', ','), ('any', 'DT'), ('and', 'CC'), ('all', 'DT'), ('Feedback', 'NNP'), (',', ','), ('and', 'CC'), ('to', 'TO'), ('sublicense', 'VB'), ('the', 'DT'), ('foregoing', 'NN'), ('rights', 'NNS'), (',', ','), ('in', 'IN'), ('connection', 'NN'), ('with', 'IN'), ('the', 'DT'), ('operation', 'NN'), ('and', 'CC'), ('maintenance', 'NN'), ('of', 'IN'), ('the', 'DT'), ('Services', 'NNPS'), ('and/or', 'NN'), ('FaceApp', 'NNP'), ('’', 'NNP'), ('s', 'NN'), ('business', 'NN'), ('.', '.')]]
CC coordinating conjunction CD cardinal digit DT determiner EX existential there (like: “there is” … think of it like “there exists”) FW foreign word IN preposition/subordinating conjunction JJ adjective ‘big’ JJR adjective, comparative ‘bigger’ JJS adjective, superlative ‘biggest’ LS list marker 1) MD modal could, will NN noun, singular ‘desk’ NNS noun plural ‘desks’ NNP proper noun, singular ‘Harrison’ NNPS proper noun, plural ‘Americans’ PDT predeterminer ‘all the kids’ POS possessive ending parent’s PRP personal pronoun I, he, she PRP$ possessive pronoun my, his, hers RB adverb very, silently, RBR adverb, comparative better RBS adverb, superlative best RP particle give up TO, to go ‘to’ the store. UH interjection, errrrrrrrm VB verb, base form take VBD verb, past tense, took VBG verb, gerund/present participle taking VBN verb, past participle is taken VBP verb, sing. present, known-3d take VBZ verb, 3rd person sing. present takes WDT wh-determiner which WP wh-pronoun who, what WP$ possessive wh-pronoun whose WRB wh-adverb where, when