|
Jupyter at Bryn Mawr College |
|
|
| Public notebooks: /services/public/dblank / Experiments / Debate1 | |||
US Presidential Debate Sep 26 2016¶
This is a Jupyter notebook detailing an analysis of the language used in the first debate.
Prepared by Doug Blank, Bryn Mawr College
For full discussion, see: http://blankversusblank.blogspot.com/2016/09/post-debate-analysis.html
Data from http://www.nytimes.com/2016/09/27/us/politics/transcript-debate.html
There were some errors that I corrected, so you can use the version here first_debate.txt.
First, we read the text into an array of lines:
In [165]:
text = [line.strip().replace("\n", " ").replace(".", " ").replace("?", " ")
.replace("“", " ").replace("”", " ").replace(":", " ")
.replace(",", " ").replace("—", " ").replace("-", " ")
for line in open("first_debate.txt").readlines()]
text_all = " ".join(text)
A sample to see what it looks like:
In [166]:
text[0]
Out[166]:
Now, we break it down by speaker:
In [167]:
holt = ""
clinton = ""
trump = ""
current = None
for line in text:
if not line:
continue
elif line in ["(APPLAUSE)", "(CROSSTALK)", "(LAUGHTER)"]:
continue
elif line.startswith("HOLT"):
current = "HOLT"
holt += line[4:] + " "
elif line.startswith("TRUMP"):
current = "TRUMP"
trump += line[5:] + " "
elif line.startswith("CLINTON"):
current = "CLINTON"
clinton += line[7:] + " "
else:
if current == "HOLT":
holt += line + " "
elif current == "TRUMP":
trump += line + " "
elif current == "CLINTON":
clinton += line + " "
else:
raise Exception("No speaker?!")
holt = holt.lower()
clinton = clinton.lower()
trump = trump.lower()
clinton = clinton.strip()
while " " in clinton:
clinton = clinton.replace(" ", " ")
holt = holt.strip()
while " " in holt:
holt = holt.replace(" ", " ").strip()
trump = trump.strip()
while " " in trump:
trump = trump.replace(" ", " ").strip()
Characters:¶
In [168]:
len(holt), len(trump), len(clinton)
Out[168]:
And split the text into words:
In [169]:
clinton_words = clinton.split(" ")
trump_words = trump.split(" ")
holt_words = holt.split(" ")
Number of total "words" spoken¶
In [170]:
len(clinton_words), len(trump_words), len(holt_words)
Out[170]:
Number of unique words spoken¶
In [171]:
clinton_set = set(clinton_words)
trump_set = set(trump_words)
holt_set = set(holt_words)
In [172]:
len(clinton_set), len(trump_set)
Out[172]:
Number of each word spoken, ranked from highest to lowest¶
In [173]:
def make_dict(words):
d = {}
for word in words:
count = d.get(word, 0)
d[word] = count + 1
return d
In [174]:
clinton_dict = make_dict(clinton_words)
trump_dict = make_dict(trump_words)
In [175]:
common_words = ["the", "to", "and", "or", "that", "of", "a", "in", "have", "it", "be",
"am", "are", "was", "were", "been", "be", "being", "is", "do", "would",
"but", "what", "so", "with", "about", "at", "on", "has", "can", "as",
"because", "when", "by", "an", "for", "this"]
In [184]:
for pair in sorted([items for items in clinton_dict.items() if items[0] not in common_words and
items[1] > 2],
key=lambda pair: pair[1], reverse=True):
print("%s: %s" % pair)
In [177]:
clinton_dict["china"], clinton_dict["plan"]
Out[177]:
In [178]:
trump_dict["china"], trump_dict["plan"]
Out[178]:
In [185]:
for pair in sorted([items for items in trump_dict.items() if items[0] not in common_words and
items[1] > 2],
key=lambda pair: pair[1], reverse=True):
print("%s: %s" % pair)
How often did they speak¶
In [180]:
text_all.count("TRUMP")
Out[180]:
In [181]:
text_all.count("CLINTON")
Out[181]: