Democrats have often criticised Trump for being negative and failing to provide hope. I wanted to test this by doing sentiment analysis on Trumps tweets since being elected. Then comparing that to his pre-election tweets. As always the code I used is in my code blog, here.

Analysis of Trump's pre election tweets

total positive tweets = 196 (12%)
total neutral tweets = 1330 (79%)
total negative tweets = 150 (9%)

More than 3 out of 4 are neutral. And the majority of the non neutral tweets are positive.

Analysis of Trumps' post election tweets

total positive tweets = 95 (34%)
total neutral tweets = 135 (48%)
total negative tweets = 49 (18%)

So almost half are neutral, one in 3 are positive and less than one in five are negative. The sample size for post election tweets is smaller but both data sets are in some agreement - the majority of tweets are neutral and positive tweets out-number negative tweets.

In the code blog I explain the technical aspects of this analysis and some of the potential weaknesses in the approach. In a future post I will describe an alternative approach using machine learning.

In conclusion I don't agree with commentators who claim Trump's message is mostly negative at least not on his twitter account. If anything his message has a neutral sentiment. However I think this approach to sentiment analysis is a little simplistic becuase it fails to consider the individual words in the context of the whole tweet.

During the UK Brexit campaign the Remain side failed to put forward many positive reasons for the UK remaining in the EU. Instead they decided to focus on negative predictions about the economy in the event of a vote to leave. One of these warnings focused on the UK housing market, they predicted house prices would drop significantly leaving home owners in negative equity. To test this prediction I scraped some house price data back in June 2016 about one week before the referendum. Now that we are six months on from the referendum I ran the same code to gather up to date data and compared the two datasets.
These are small data sets so require some caution when drawing conclusions but they do indicate a definite trend which gives me some confidence in the data.

Area 1 = Carrickfergus, Northern Ireland

Prperty Type                June Price             December Price            %change
apartment                     87713                    108899                           +24
bungalow                      142125                  165572                           +16
detached house            213521                  244081                           +14
semi-detached house   108414                  116005                            + 7
terrace house                85176                    80359                             - 6              
​ 
(prices are in pounds sterling)

Area  = Glengormley, Northern Ireland

Prperty Type                June Price             December Price            %change
apartment                     92227                    96550                              +5
bungalow                      153709                  163346                            +6
detached house            192498                  197158                            +2
semi-detached house   117517                   114188                            - 3
terrace house                95246                    96156                              +1             

Overall the change is a positive increase in the average price for most house types. The average house price from these figures has incresed from 128814 pounds to 138231pounds, approximately a 7% increase.

These figures are in quite good agreement with the overall trend reported else where, for example the Belfast Telegraph.

My approach to this question was to analyse data from around 50 current job advertisements in the UK and the US. I looked at the required skills and compared these for data analysts and data scientists.

Summary of findings - differences between data analysts and data scientists

• There wasn't much difference between the skills required of data analysts in the UK and the US, the same was true for data scientist jobs.
• Typical data analyst skills include data mining especially SQL skills and NoSQL skills, data cleansing and data quality, statistics, a degree in a mathmatical subject, working with unstructured data, data visualisation, python or R, data scraping, one skill sometimes quoted which is unique to data analyst is Excel. 
• Data science jobs may include all of the above skills as well as the following: machine learning, algorithms, hadoop, linear algebra, scikit-learn, one or two also mention the need for other programming skills esprcially C++ and Java.
• Data scientists earn about 1.5 to 2.0 times what a data analyst might earn for a similar level job i.e. comparing an entry level Data Analyst job to an entry level Data Science job and so on.
• Many, but not all, employers of data scientists want higher degrees - Masters or PhD whereas Data Analysts usually only require a degree in a mathmatical subject.

My conclusions 

Data Analysts and Data Scientists have similar core skill sets but data scientists require more skills than data analysts. While the skill sets are similar, job roles are less so. Data analyst roles may focus on data quality and obtaining data, possibly generating reports. Data science roles may require more programming skills and will almost certainly involve working with big data rather than just data. The data science roles focus on using the data whereas data analyst roles may focus more on describing/summarising the data. To get the best data science jobs you may need a higher level degree.
​
Other roles that can be similar are Business Intelligence Developer, data quality engineer and data engineer, these may be more technical and may require knowledge of ETL and data warehousing as well as BI tools such as Tableau.

Part 1 used the frequency distribution of characters to identify languages. This time I'm using key words for the three target languages. The code is available in my code blog.

test reults

for 20 word inputs
test    dutch_score    english_score    basque_score
Basque       0                       0                         3
English        0                      5                          0
Dutch          6                      0                          0

for 9 words
Basque       0                       0                         1
English        0                       1                         0
Dutch          2                       0                         0

for 2 words
Basque       0                       0                          0
English        0                       0                         0
Dutch          0                       0                         0

Summary:

Version 2 scored 100% for both 20 and 9 word inputs but failed miserably for the 2 word inputs scoring 0% for each of the three languages. I think a combination of both approaches would produce the most accurate results but it will always be difficult for very short inputs.

I used two sources of data for this post, the first covers the period from 1970, killings actually started in 1969, also the first data source is missing data from 1993: 

"National Consortium for the Study of Terrorism and Responses to Terrorism (START). (2016). Global Terrorism Database [Data file]. Retrieved from https://www.kaggle.com/START-UMD/gtd"

The second source:

https://www.theguardian.com/news/datablog/2010/jun/10/deaths-in-northern-ireland-conflict-data

As always the code used to produce graphics is available in my code blog.

This is not an attempt to give a full historical account of the Northern Ireland conflict, there are many resources availbale to people who are interested. I grew up during the 'Troubles' as they were euphemistically referred to so I am aware of the stories behind the statistics, it is important to remember that the statistics presented here were real people who were murdered in horrific ways. The data sources do not to mention the unarmed civilians murdered by the British Army in events like Bloody Sunday and the Ballymurphy masacre. It is also important to remember that the conflict, despite what some people think, is still ongoing. There are still groups, particularly on the Republican side, willing to use guns and bombs to 'advance' their political point of view i.e. they want to force everyone in Northern Ireland to be part of a single Irish state. The sectarian divide is perhaps more entrenched in Northern Ireland society than at any time in history. I have already mentioned in this blog that I believe the political side to the 1990s peace agreement is flawed and has helped to perpetuate the sectarian divide that in part gave rise to the conflict back in the 1960s and 70s. However it is clear from the graphic below that the number of casualties dropped significantly after the peace agreement. This is to be welcomed, my concern is that without tackling the underlying causes of the conflict the various groups involved have failed to create a lasting peace. There are indications that the violence is returning which could mean more people will lose their lives in the future.

All casualties per year (includes all groups: civialians, government forces and members of paramilitary groups)

This graphic shows an encouraging trend towards zero casualties however the next graph shows an increase in the number of attacks:

2015 is particularly worrying as it shows an increase in both Republican and Loyalist violence. There have also been warnings from the PSNI about possible terrorist attacks, for example, BBC and from the UK gov: UK gov site. If violent attacks continue to increase in number then it is only a matter of time until the number of casualties begins to increase.

Historical Statistics:

1970s

​attacks:

Geographical distribution of attacks:

1980s

​attacks:

the most striking difference is the proportion of Loyalist to Republican violence, Loyalist groups were less active during the 1980s compared to the 1970s.

Geographical distribution

1990s

​attacks

Geographical distribution, There was a definite shift in the distribution of attacks, with the majority being in the Eastern half of Northern Ireland:

2000s

attacks

There was an increase in attacks after the 1998 peace agreement mainly due to groups opposed to any settlement.

​Geographical distribution

2010s

attacks

Geographical distribution

Many attacks in Derry/Londonderry and Belfast. The cluster of attacks in Derry means the number of violent attacks has incresed for the fisrt time since the 1990s, also the number of attacks has obviously increased.

I believe the Stormont government has failed to address the following issues:

• Poverty, unemployement and poor education in the working class areas most likely to produce the next generation of terrorists. This leads to alienation of young men who have nothing to lose from becoming involved with paramilitaries.
• Sectarianism, Belfast is still physically divided by 'peace walls' and children are still segregated by religion for education from the age of 5. This divide perpetuates the 'us and them' mentatlity which may be a contributing factor in continued sectarianism.

Paramilitaries - particularly on the Republican side are increasing violence, they are trying to make the violent alternative more acceptable. On the Loyalist side the violence tends to be more crime related than political.

I have been teaching myself various different data analytics/data science tools and I wanted a project to try out some of these tools. Language identification is, for me, an interesting and non-trivial project. By language I mean human language such as English, French, Russian etc. Some languages can be identified in the written form because they use unique scripts, for example Hangul for Korean, Hiragana and Katakana for Japanese, Hanzi for Chinese and so on. I wanted to build a tool that can distinguish between written forms of languages that all use the Roman alphabet.
The requirements for the tool are that it can take a sentence or two of input language, parse it and make a decision on the language involved. The goal is not to create a commercially viable product but rather to have fun building a product which allows me to employ some of the tools I have been learning. I will use python and some of the libraries available to me such as NLTK.
There are a number of different approaches that I can take, the first one I want to investigate uses the frequency distribution of characters to identify the language. The code used to determine the frequency distribution of letters in some languages is in my code blog. The initial target languages are English, Dutch and Basque.
Once I generated the frequency distribution graphs for the three languages (shown below) I looked for some identifying features:

Frequency distribution of letters in a sample text written in Basque.

Frequency distribution of letters in a sample text written in Dutch.

In English the vowels 'e' and 'o' are more common than 'a' also t and y are used often. 'j', 'q' and 'x' are not common but this does not help because they are not common in Basque or Dutch either.
In Dutch 'e' and 'a' are more common than 'o'. 'y' is not common. 'v', 'w' and 'z' are used with almost equal frequency whereas in English 'w' is much more common than either 'v' or 'z' and in Basque 'z' is very common whereas 'w' and 'v' are not.
Basque uses 'a' more than 'e'. 'k' is more common in Basque than in either Dutch or English.

So the initial algorithm will use the usage frequency of: 
'a' and 'e'
'k', 'l' and 'm'
'y'
'v', 'w' and 'z'
to decide the languge being used

​The code for the language identifier is in my code blog. The test results:

I tested the tool using input text from online news sources in the target language. Each test used one or two short sentences as input, I chose input that contains only the target language, trying to identify the main language from input that contains examples of other languages is a more complex task.

test reults

for 20 word inputs
test    dutch_score    english_score    basque_score
Basque       1                       0                         4
English        2                      4                          0
Dutch          4                      2                          1

for 9 words
Basque       1                       0                         4
English        3                       3                         0
Dutch          4                       2                         1

for 2 words
Basque       3                       1                          2
English        2                       2                         1
Dutch          4                       2                         1

The results are not surprising, given that the identification is based on a statistical analysis, the tool works best for longer test inputs so inputs of at least 20 words (100% accurate), less well for the shorter 9 word input (67%) and quite poor for the two word input - just 33%.

Summary
The algorithm works best for longer inputs. It is better at recognising Basque and is poor at distinguishing English from Dutch. The approach I took can handle 3 languages, but if the number of languages increased beyond 10 the algorithm would become much more complicated. Next time I will use NLTK to identify stop words then use those stop words to identify the language.