The Iraq Inquiry documents on DocumentCloud

I’ve uploaded all 2.6million words of the Chilcot Report (also known as the Iraq Inquiry) to the document analysis service DocumentCloud. The documents can be found on the site here.

You’ll need a DocumentCloud account to analyse the documents by entity or timeline (more on that here), but if you don’t have an account you can also search all the documents using the embedded view below.

Help Me Investigate is on ice (but I’ll still help you investigate)

Help Me Investigate

I’m putting Help Me Investigate, my collaborative investigation project, on ice.

This year has seen the launch of a number of impressive crowdfunded and crowdsourced projects on platforms including Beacon and Contributoria – plus OpenCorporates Missions and the enormously impressive Bellingcat. Their rise, for me, confirms that there is no longer a need for the original mission that Help Me Investigate took on way back in 2009. Now I think it’s time to rip it up and start again. Continue reading

Former HMI Health editor named best new journalist of the year at British Journalism Awards

Former Help Me Investigate Health editor Tom Warren has been named best new journalist of the year at British Journalism Awards.

Tom, who joined the Bureau of Investigative Journalism last year, won the award for stories on payday loan lenders, party political funding, and the flotation of shares in Royal Mail. He said:

“Before I was a professional journalist HMI taught me skills and made me contacts that I couldn’t have picked up anywhere else. It was a great place to cut my teeth – and I’m not the only one. I can think of several former HMI’ers who have gone on to top jobs in journalism. It’s been a real powerhouse in that respect.”

Some of the data for the payday lenders story was collected by Help Me Investigate.

The Bureau of Investigative Journalism report on their site that judges said:

“Tom made great use of data and technology to unearth stories and details. He picked targets that no-one else was looking at to bring new information to light on matters of real public interest.

“He used excellent detective work to reveal the privileged bidders who profiteered from the Royal Mail flotation, forcing the Government to release the full list.”

The award came the day after another former HMI contributor, Beth Ashton, won an award for her work at the Manchester Evening News.

Beth Ashton wins social media award

Former Help Me Investigate Education contributor Beth Ashton has won an award for her work in the social media team at Trinity Mirror.

Beth wrote for Help Me Investigate Education in 2013 about a ‘leak’ that she stumbled across while investigating care homes.

She now works as Social Media Editor for the Manchester Evening News. It was her work on the newspaper’s campaign to raise money for Manchester Dogs’ Home that led to the award.

Beth accepted the award at a ceremony on Monday, at the Trinity Mirror Editorial Conference in Birmingham.

Help Me Investigate in 2013 – and 2014

As Help Me Investigate starts to plan new investigations in 2014, it’s a good time to look back at what we’ve been up to over the last year across the four sites (not including this blog). It’s easy to forget how much you do in a year – thanks to the many contributors who made everything below happen. As always, we’ve learned a lot and hit the new year wiser.

HMI Health

Help Me Investigate Health kicked off the year by organising an event on reporting the new health system with the BBC College of JournalismIn April we took part in and reported on a masterclass on health reporting organised by the National Union of Journalists and in May we spoke at and liveblogged a Medical Journalists Association event on the reorganised NHS.

HSJ logo

HMI Health editor Tom Warren and HMI Education editor Matt Burgess worked with the Health Service Journal to compile a list of FOI emails for clinical commissioning groups – the new bodies controlling health spending.

And we scraped the data that helped Scotland’s Sunday Post report on councils abandoning elderly people because they couldn’t afford the care at home and a 7% increase in absenteeism in Scottish authorities.

We also reported on Barnet, Enfield and Hillingdon referring fewer than 4% of depressed patientsNHS Merseyside spending £65,000 on re-hiring staff who worked for its predecessor, and the worst times to go to A&E in the Midlands. Continue reading

Finding data in Scotland

I’ve delivered data journalism training in Scotland twice in the past few months, and thought I’d share some tips on what data is available there, given that most guidance on data journalism focuses on data in England or the US.

General statistics and data

Scottish Neighbourhood Statistics has a ‘data download’ page which allows you to download data on a range of topics, each of which has dozens or hundreds of indicators, from ‘Physical environment’ (24 indicators) to ‘Economic activity, benefits and tax credits’ (377 indicators).

Subjects covered include health, housing, access to services, community care, crime, education and ‘special interest’ reports (these include sea fisheries, poverty and cultural event attendance).

Once you’ve downloaded the data you may find that it doesn’t name each local authority but instead uses a series of codes. These are ONS (Office for National Statistics) codes – you can find a master list on Wikipedia.

The Scottish Government Statistics page also provides access to a range of topics, including agriculture, business, tourism, transport and travel.

The Equality Evidence Finder is particularly interesting: this is an attempt to show the equality of different industries across measures such as gender, disability, income and so on. This, for example, is the page summarising the picture for ethnicity in Scottish business, enterprise and tourism.

The General Register Office for Scotland has broad statistics about births, deaths and marriages (known as ‘vital events‘), life expectancy, migration and elections.

And of course don’t overlook, where you can subscribe to an RSS feed for new datasets mentioning ‘Scotland’.

Health data in Scotland

ISD Scotland – the Information Services Division of NHS National Services Scotland – provides a range of health data from the right hand column of its home page, from specific conditions including cancer and mental health, through to the NHS workforce, eye care and dentistry, maternity, and prescriptions.

Rather helpfully, NHS Scotland has its own domain at, which means you can a search for Scottish health data on Google by adding to any search terms. Using your particular health board (e.g. ‘Grampian’) will also help.

NHS National Services Scotland is another useful domain at – adding to your search (and, for example filetype:xls) might also bring up some useful data – I found data on gifts and hospitality, payments to GP practices, and cancer waiting times using those.

Health Protection Scotland has regular data reports and publications. And there’s the Scottish Health Survey and Inpatient Patient Experience Survey on the Scottish Government statistics pages.

Education data in Scotland

Education Scotland publishes school reports, publications, and provides school-level information through the Scottish Schools Online portal. Some of this data is available as datasets through the Scottish Government statistics page too, such as this page on school meals.

Environment data in Scotland

Scottish Environment Statistics Online (SESO) publishes datasets on a wide range of indicators from recycling and conservation to air quality and noise pollution.

The Scottish Environment Protection Agency monitors environmental conditions and waste, most of which is linked from here.

And Scottish Natural Heritage has a page linking to statistics and reports.

If you are interested in data which has any sort of environmental impact (this can include things like housing and construction) then it may be worth looking into the Environmental Information (Scotland) Regulations (EIR).

This is similar to Freedom of Information laws, but have fewer exceptions and apply to a wider range of public bodies. The book EIR without the Lawyer is a useful guide for journalists.

Data on crime, justice and fire

The Scottish Government statistics pages on crime and justice, covering everything from perceptions of crime to numbers of police, racist incidents, the prison population and re-offending.

They also have data on fires and fire and rescue servicesAudit Scotland also has performance data on the fire and rescue services.

The Judiciary of Scotland has been more open than its English counterpart, with data including judicial expenses, and judgments and sentences – although these are provided as PDFs and feeds rather than more easily downloadable data.

Local government data in Scotland

Audit Scotland’s Council performance pages are a fantastic resource if you’re interested in the performance of services under local authority control. This includes schools, housing, social work and hundreds of ‘performance indicators’, with a spreadsheet for every council.

You can also look at the same data for all councils by service on the services page.

Local government finance data can be found on the Scottish Government statistics site – including council tax collection and budgets.

On housing some local authorities have passed over some or all of their housing responsibilities to housing associations. You can find data on these on the Scottish Housing Regulator site including inspection reports, accounts, performance returns and performance profile, regulation and rules.

The Scottish Parliament

MPs’ allowances are published on the Scottish Parliament website. It’s not particularly easy to download and compare. In these situations it’s often worth searching to see if you can find someone who’s tackled the problem – and Owen Boswarva did (at least, he did in 2011/12), providing a link to the bulk data. If he did it once, he may do it again, if you ask nicely…

Freedom of Information in Scotland

Scotland has its own Freedom of Information Act and its own Information Commissioner, who has a reputation for enforcing the Act more strongly than his equivalent south of the border.

To complicate things, many bodies operating in Scotland will also be subject to the UK FOI Act and the UK Information Commissioner.

You can follow public FOI requests mentioning ‘Scotland’ made using WhatDoTheyKnow, or of course specify a different region like ‘Dundee’ or a related term like ‘Scottish’. And you can look for Scottish public bodies’ disclosure logs, where they may be publishing the latest FOI requests answered.

Scotland also has the INSPIRE (Scotland) Regulations, which provide access to mapping – “spatial” – data.

And more broadly you may be interested in Public Contracts Scotland, which provides access to data on contracts awarded by local authorities, health bodies, education agencies, emergency services and national government.

I’ll be adding to and updating this post as I think of other sources. If you know of any please let me know.

Reporting the water industry – key performance indicators

Severn Trent Water Key Performance Indicators 2012

Severn Trent Water Key Performance Indicators 2012

If you’re interested in learning more about how the water industry works, Ofwat’s pages on regulating the industry provide information on the industry itself, what water and sewerage companies have to do to keep their licences, and how they are regulated. You can also find links to the water companies’ annual reports, including their ‘key performance indicatorsThe section of Key Indicators explains these (also called KPIs): it includes a link to a spreadsheet (XLSX) which does the following: Continue reading