Blog Archives

Second EU referendum - code used to parse json file

6/26/2016

The following code was used to parse data from a UK government website. The analysis is available on my Data Analysis Blog.

import json
import collections
import re

with open('eu.json', 'r') as f:
d = json.load(f)

dataDict = {}
for item in d['data']['attributes']['signatures_by_country']:
info = re.search(r"name': u'.+", str(item))
#print info.group()
name = re.sub("', u.+",'',info.group())
name = str(name[9:])
count = re.search(r"signature_count.+", info.group())
count = str(count.group())
count = count[18:]
count = re.sub('}', '', count)
dataDict[name] = int(count)

sortedData = collections.Counter(dataDict)
print sortedData.most_common(10)

0 Comments

Using Python to check if data in one csv file exists in a second file

6/15/2016

0 Comments

I have two files, the first is a list of names, and the second is a directory with over 100 .txt files (the same data on first names in the US that I used in the last post). The problem to solve is how do I check that the names in the first file exist in the second file(s). I could manually do a search or use the search function available in the csv and text files. But if the amount of data in the files and/or the number of files is large then that solution becomes less workable. An alternative is to use Python and Pandas to do the work for me - the code takes less than one second to run so is much faster than any manual search. Note - at this stage I don't care how many times the names appear in the files I am only interested in if they appear or not.

The code:

import pandas as pd

cols2 = ['name', 'gen', 'num']
cols1 = ['name']

ib_names = pd.read_csv('irishBoysNames.csv', names=cols1)
ig_names = pd.read_csv('irishGirlsNames.csv', names=cols1)
df = pd.read_csv('yob2010.txt', names=cols2)
boys = df[df.gen == 'M']
girls = df[df.gen == 'F']
bnames = boys[boys.columns[0]]
gnames = girls[girls.columns[0]]

for name in ig_names.name:
#for name in ib_names.name:
exists = name in gnames.values
#exists = name in bnames.values
print(name + ': ' + str(exists))

The output for the yob2010.txt file:

Aoife: True
Caoimhe: True
Saoirse: True
Ciara: True
Niamh: True
Erin: True
Roisin: True
Clodagh: True

Explanation:

The files have no header rows so I define two lists, one for each file, which contain the column names. I then import the data and create some new dataframes using this data. I then loop through the data and for each name in the list of names in file one I check if it is in file 2 (in the above file 2 = yob2010.txt).

0 Comments

Counting occurrences of characters in csv files

6/11/2016

0 Comments

Data used: a directory containing over 100 csv files, each file covers data for one year between 1880 and 2015. The data lists given names for babies born in the US and the number of babies given that name. I obtained the data from Kaggle, but the same data is available form several sources.

Analysis: for analysis please see the post for June 11 2016 in the Data Analysis Blog.

The code:

import pandas as pd
import collections
import numpy as np
import matplotlib.pyplot as plt

cols = ['name', 'gen', 'num']

df = pd.read_csv('yob2015.txt', names=cols)
boys = df[df.gen == 'M']
girls = df[df.gen == 'F']

names = df[df .columns[0]]
bnames = boys[boys.columns[0]]
gnames = girls[girls.columns[0]]

namestr = ''
#for name in bnames:
#namestr = namestr + name
for name in gnames:
namestr = namestr + name

alphabet = 'abcdefghijklmnopqrstuvwxyz'

c = collections.Counter(namestr)

occurenceDict = {}
for letter in alphabet:
#print '%s : %d' % (letter, c[letter])
occurenceDict[letter] = c[letter]

plt.bar(range(len(occurenceDict)), occurenceDict.values(), align='center')
plt.xticks(range(len(occurenceDict)), occurenceDict.keys())

plt.show()

This produces graphs such as:

Distribution of characters in male given names in the United States, year = 1880

0 Comments

Visualize data using D3

6/6/2016

0 Comments

I am using the same data as the previous post - PSNI data on anti-social behaviour in Northern Ireland, but this time I'm using D3 rather than Basemap and Python. The code is based on sample code from the book: Interactive Data Visualization for the web, by Scott Murray (O'Reilly).
The code:

<!DOCTYPE html>
<html lang="en">
   <head>
       <meta charset="utf-8">
       <title>Anti social crime March 2016</title>
       <script type="text/javascript" src="d3.min.js"></script>
   </head>
   <body style="background-color:#AABB77;">
       <script type="text/javascript">

           //Width and height
           var w = 1000;
           var h = 1000;

           var projection = d3.geo.mercator().scale((28000) / Math.PI).translate([w*1.3, h*11]);

           var path = d3.geo.path()
                           .projection(projection);

           var svg = d3.select("body")
                       .append("svg")
                       .attr("width", w)
                       .attr("height", h);

           d3.json("county.geojson", function(json) {
               svg.selectAll("path")
               .data(json.features)
               .enter()
               .append("path")
               .attr("d", path)
               .style("fill", "#6EA171");


               d3.csv("2016_03_ni_crime.csv", function(data) {

                       svg.selectAll("circle")
                       .data(data)
                       .enter()
                       .append("circle")
                       .attr("cx", function(d) {
                           return projection([d.Longitude, d.Latitude])[0];
                       })
                       .attr("cy", function(d) {
                           return projection([d.Longitude, d.Latitude])[1];
                       })
                       .attr("r", 3)
                       .style("fill", "#424756")
                       .style("opacity", 0.75);
               });

           });

       </script>
   </body>
</html>

The output:

0 Comments

Visualize geographic data using Python and Basemap

6/1/2016

0 Comments

The data: crime recorded by the police for March 2016 in N.Ireland. The data is available from here. The data came in a CSV file. I kept all crimes in the 'Anti Social' category and deleted the rest.

The code:

import numpy as np
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt

incidents = np.genfromtxt("2016_03_ni_crime.csv",
delimiter=',',
dtype=[('Longitude', np.float32), ('Latitude', np.float32)],
usecols=(0, 1))

fig = plt.figure()
themap = Basemap(projection = 'merc',
llcrnrlon = -8.5,
llcrnrlat = 54,
urcrnrlon = -5,
urcrnrlat = 55.3,
resolution = 'h',
area_thresh = 40000.0)

themap.drawcoastlines()
themap.drawcountries()
themap.fillcontinents(color = '#83a81d')
themap.drawmapboundary(fill_color ='#37a7d5')
x, y = themap(incidents['Longitude'], incidents['Latitude'])
themap.plot(x, y,
'o',
color='#464646',
markersize=4
)

plt.show()

the output:

0 Comments

Second EU referendum - code used to parse json file

Using Python to check if data in one csv file exists in a second file

Counting occurrences of characters in csv files

Visualize data using D3

Visualize geographic data using Python and Basemap

This blog includes:

Archives