Tag Archives: foi

FOIA Without the Lawyer – review and highlights

FOIA Without the Lawyer

I’ve been meaning to review FOIA Without the Lawyer for almost a year now. A natural companion to Heather Brooke’s introductory Your Right To Know, this takes the challenges that come after the FOI is submitted: the niggling exemptions and excuses used by public bodies to avoid supplying information requested under the Act.

In the process it details numerous ways of anticipating and responding to them, including various references to official guidance, tips from FOI officers, and experiences of journalists and others using FOI, all of which are hugely helpful. I’ve tried to summarise some of them here: Continue reading

How to get scoops from local councils


Since 2011, all councils have been required to publish expenditure on items over £500. At the CIJ Summer School this year, Paul Francis and Ted Jeory explained how to turn this information into a story… Continue reading

HMI Health compiles clinical commissioning group FOIs for health sector magazine

Over on Help Me Investigate Health, we’ve published a list of Freedom of Information emails for clinical commissioning groups (CCGs) – the new bodies controlling local health spending in England.

The list was compiled two months ago for Health Service Journal by Tom Warren and Matt Burgess, shortly after the new bodies took control, but is only now being published by HMI.

As part of the process, Matt and Tom also compiled a spreadsheet of CCG websites and FOI webpages.

Following FOI requests – making the most of WhatDoTheyKnow

Over on Help Me Investigate Health, Tom Warren has written a guide to using the Freedom of Information tool WhatDoTheyKnow. Key tips include:

  • Consider sending an FOI directly to the authority if you want to keep it private – but also remember that doing so publicly can be a great way to find contacts and ideas
  • You can use the site to find out an authority’s FOI email address
  • Use advanced search techniques in Google to search just that site

Read the post in full here.

What to do when an FOI response is not provided in the form asked for

Heard the one about the FOI request for data to be provided in an Excel spreadsheet? The authority printed it out, scanned it, and sent a PDF version of the scan.

You’ll forgive the journalist for being suspicious when an authority goes to such extremes to make it hard to interrogate their data.

In cases like these it’s worth looking at the Information Commissioner’s awareness guidance 29 on ‘means of communication’ (PDF):

This quotes Section 11(1) of the Freedom of Information Act, which stipulates that authorities should comply with your preference for “a copy of the information in [a] form acceptable to the applicant … so far as is reasonably practicable”

The key phrase here is “reasonably practicable”. In the example above, there is no excuse that simply sending the original Excel file – instead of a scanned PDF – was not “reasonably practicable”. 

What then? Well, you should ask for an explanation, and make a formal complaint to the authority quoting the ICO guidance and Section 11(1) or the FOI Act. If that doesn’t get any results, write to the ICO. Here’s the full passage from the guidance:

“If a public authority decides that it is not reasonably practicable to provide the information in the form preferred by the applicant … the authority must tell the applicant and give its reasons. The duty on the public authority is then to provide the information by any means which are reasonable in the circumstances. 

“If the applicant is not satisfied with the decision and wants to make a complaint, they must complete the public authority’s complaints procedure (if there is one). Once this process is complete, if the applicant remains dissatisfied, they may write to the ICO.”

If you are unlucky to deal with an authority which is regularly uncooperative in this manner, it may be worth quoting the awareness guidance 29 and Section 11(1) of the Act in your request for the information to be provided in spreadsheet form, for example:

“I would like this information to be provided in spreadsheet format (xls or csv) in line with Section 11(1) of the FOI Act and ICO Awareness Guidance 29.

Also useful more broadly when looking at the way a request is handled are the guidelines on ‘Request handling’ on the ICO website.

The best way to set up a spreadsheet for inputting FOI responses

On Help Me Investigate the Olympics we’ve been sending FOI requests to all of the local authorities which hosted the Olympic torch relay. 

That’s the easy part. The difficult bit is compiling the results into something meaningful. To do this, it’s helpful to create a spreadsheet using Google Docs so that all members can input the results. In the process, however, we’ve used a number of techniques to both speed up and future-proof the results.

Drop-down menus 

Firstly, it helps to make sure that you will be able to match the data you’re inputting with other data, such as the population of a local authority, or the political party in charge.

You can do this in your Google spreadsheet by creating a drop-down menu for the ‘matching’ column (normally an authority name or code) which draws from another list. This is how:
  1. Select the column you want your drop-down lists to appear in (normally A)
  2. Click on Data>Validation…
  3. On the window that appears, click on the drop-down menu for Criteria and select Items from a list.
  4. You can now either type in the list yourself, or select a range of cells that contain the items for your list (Create list from a range). It’s best to do the latter: click on the grid button to the right of this box, go to the sheet containing your ‘official’ list (which you’ll need to have copied into another sheet in your spreadsheet – here’s a good one to use from OpenlyLocal) and select the column or row containing your items. You should end up with a reference that looks like this: SheetName!A:A. Click OK. And click Save to apply the drop-down.
  5. Back on your data entry sheet each cell in the column you first selected should now have a small drop-down icon to the right. Users can select from this to make sure they’re using the same names for authorities as your ‘official’ list.

Input data at the most granular level

If you’re inputting data that isn’t consistently categorised across different respondents (as is most often the case), you need to input the data so that this doesn’t cause a problem.

The simplest way to do this – and also the fastest way – is to avoid imposing a particular classification system (either one authority’s, or your own) onto your data as you’re inputting it.

Imposing a classification system slows down the inputting process (the user has to make a decision where to categorise each piece of data), can introduce inconsistency (one user makes a different decision to another), and add extra work later on (what if you realise you decide to add a new category halfway through?)

Much simpler is to input the data in as basic a fashion as possible.

For example, we’re requesting information on spending. Instead of having one column per local authority, and having columns for each type of spending, like this:

COUNCIL A – 100 ——- 10000 —- 10100

It’s simpler to have one row per item like so:

COUNCIL A – Lights –10000 ——— “Blah blah”

This allows you to add extra columns when required (for example, if a third party contributed funding, or if you want a column for answers to a particular question).

It also means you don’t have to think about classification now. You can impose this on the data later by, for example, filtering it to look for rows which mention “light”, “lighting”, and other terms you’ve come across while inputting.

You can also generate a per-council view of the data with a straightforward pivot table.

It’s a good idea to have a ‘NOTEWORTHY?’ column so if you do come across interesting individual items you can simply put YES in that column and come back to them later, rather than get distracted by them. 

Use spreadsheet techniques to split pasted data

Even if your data has been supplied in a word document, you can paste it into your spreadsheet and use formulae to split it up as you need, rather than having to manually input each item separately.

It makes sense to have your spreadsheet set up in the same order as data is normally provided, i.e. the ‘item’ and ‘cost’ columns coming in that order, next to each other, so that these formulae work best.

Here are some typical techniques, depending on how it’s been provided:

Item and cost are separated by a space: if the items are always only one word (no spaces), then followed by the cost, you can paste them into another column in your spreadsheet and type this formula in your ‘item’ cell to grab and split them: 

=SPLIT(E34,” “)

If item and cost are separated by a colon, e.g. “Item: £200” then you can adapt it as follows:


Note: change E34 to the first cell your raw data has been pasted into.

This will split the contents of that cell wherever there is a space, so anything before that space will be pulled into your ‘item’ cell, and anything after will be pulled into the cell next to it (ideally your ‘cost’ cell)

if it works, copy that formula down as many rows as you need to apply it to all the data you originally pasted.

Finally, you’ll probably want to delete the raw data – but if you do this the formulae will stop working (there’ll be nothing for them to split), so before you do that select all the cells containing the results of your formulae (both columns). Click on Copy > Paste special… and select Values only.

This will paste the results of your formulae on top of the same cells that previously contained them. That means you can safely delete anything that the now-gone formulae used.

Other formats: the SPLIT function can also be used where your costs always begin with a pound sign, like so:  


This will remove the pound sign, but you don’t need it.

Where there’s no pound sign and the data has multiple spaces, but the figure is always at the end of each line, try using =RIGHT to grab it like so:


The number (6 in the example above) specifies how many characters you want. ‘6’ will grab any amount at the end of a line up to 99,999 (note that the comma is counted as a character). If you are likely to have higher amounts, change it to 7, or 8, etc.

This formula will also grab any other characters if your amounts are small, so with the line “Lighting 100” it will give you “ng 100”. So you will probably have to combine this formula with the =SPLIT function explained above, and then just use the second cell that you get from that.

There are other techniques you can use, but these are the most commonly useful.

Six questions to ask before you submit an FOI

David Higgerson suggests six questions that journalists can ask to improve the end results of a possible FOI request – or save you sending one entirely. they are:

  1. Is this information available elsewhere?
  2. Will they release the information to me without going through FOI?
  3. Is there another way of getting this information?
  4. Do I need to think about jargon in my FOI request?
  5. Are there examples of the information being released elsewhere?
  6. What reasons for refusal could a public body come up with?

It’s nice to see the post using an example from Help Me Investigate’s very early days – Heather Brooke’s FOI request to Birmingham City Council.

FOI and redaction costs – why they don’t count

If you come across a public authority which claims that it cannot meet your FOI request because of the costs involved in redacting data, quote the following decision by the Information Tribunal (PDF) between South Yorkshire Police and the the Information Commission (Appeal Number: EA/2009/0029).

As Public Partners put it:

Under the Appropriate Limit and Fees Regulations no public authority need spend more than £600 on complying with a request, with most having a ceiling of £450 – this is called the appropriate limit. The limit is made up of the costs involved in locating, retrieving and extracting information. All the time spent on these activities can be added together at a rate of £25 per hour until the limit is reached. In this case South Yorkshire Police refused to disclose the whole of a 187 page document because redacting intelligence information line by line would cost them well over £450.

The [tribunal] ruled very firmly that the Police were wrong. Their decision couldn’t be much clearer: “we find that a public authority cannot include the time cost of redaction when estimating its costs”. All the Police’s arguments were dismissed and the rule now seems conclusive.


A search engine for data from FOI responses

Tony Hirst?has created two basic tools that allow you to search for data supplied in response to FOI requests: this search tool for local councils; and this one for universities?(ignore the word 'council' – it's an error).

The data is limited to requests made via WhatDoTheyKnow (which accounts for around 10% of FOI requests) and responses with spreadsheets attached (rather than PDFs, for example) – but it's still a useful tool.

His?post about his experiment?provides more detail, including possible further developments:

"It strikes me that if I crawled the response pages, I could build my own index of data files, catalogued according to FOI request titles, in effect generating a ?fake? data.gov.uk or data.ac.uk opendata catalogue as powered by FOI requests?? (What would be really handy in the local council requests would be if the responses were tagged with with appropriate?LGSL code or IPSV terms?(indexing on the way out) as a form of useful public metadata that can help put the FOI released data to work??)
"Insofar as the requests may or may not be useful as signaling particular topic areas as good candidates as ?standard? open data releases, I still need to do some text analysis on the request titles.
"[…] PS via a post on HelpMeInvestigate, I came across this list of?FOI responses to requests made to the NHS Prescription Pricing Division. From a quick skim, some of the responses have ?data? file attachments, though in the form of PDFs rather than spreadsheets/CSV. However, it would be possible to scrape the pages to at least identify ones that do have attachments (which is a clue they may contain data sets?)"

7 ways to follow a field you want to investigate

Here’s a part by part guide to how you can follow different ‘streams’ of information as a journalist to understand what’s going on in a particular field, and how they can inform your real-world digging. Most of them involve using an RSS reader like Google Reader to follow feeds to keep in touch with developments.

1. Prepackaged news

While much is made of the ‘exclusive’ in journalism, and students will be harangued for recycling work done by other journalists, the truth is that the first thing most journalists do every day is check out their competitors, and get a feel for the current news agenda. A journalist has to balance being ‘on top’ of developments that others are covering (“Why don’t we have something on this story?”), while also reporting information that others don’t have. Continue reading