The Database State, a report commissioned by the Joseph Rowntree Trust has just been published. For anyone unconvinced that we are waking up in a database society rather than sleepwalking towards one, it makes sober reading. It also reminds us that some of the more trivial responses to the latest announcement that the government is to prowl social network sites fail to contextualise it in wider (moral, economic, legal and operational) issues.

It’s executive summary, assessing the 46 databases in use and development by government departments, pulls few punches:

• A quarter of the public-sector databases reviewed are almost certainly illegal under human rights or data protection law; they should be scrapped or substantially redesigned.
• More than half have significant problems with privacy or effectiveness and could fall foul of a legal challenge.
• Fewer than 15% of the public databases assessed in this report are effective, proportionate and necessary, with a proper legal basis for any privacy intrusions. Even so, some of them still have operational problems.
• Britain is out of line with other developed countries, where records on sensitive matters like healthcare and social services are held locally. In Britain, data is increasingly centralised, and shared between health and social services, the police, schools, local government and the taxman.
• The benefits claimed for data sharing are often illusory. Sharing can harm the vulnerable, not least by leading to discrimination and stigmatisation.
• The UK public sector spends over £16 billion a year on IT. Over £100 billion in spending is planned for the next five years, and even the Government cannot provide an accurate figure for cost of its Transformational Government’ programme. Yet only about 30% of government IT projects succeed.

The report offers an interesting insight. The twin imperatives of democratic governance (the supportive, public service agenda and the coercive, enforcing agenda) have been conflated and allowed to be governed by (private) technological ‘solutions’. The joined-up thinking is threatening to link chains.

Interestingly, the DIUS’s new database, Managing Information Across Partners (MIAP) which ‘will create a lifelong, online record of each person’s education and training from the age of 14 and maintain a register of learning provision’, whilst not a major threat to privacy (though threatening the possibility of ‘the creative CV’), is deemed by the authors as unecessary.

More worrying is the Communications Database and the Interceptions Modernisation Programme which will, in part, maintain it. The kinds of data it includes are:

subscriber information, records of numbers dialled, and the location of mobile phones. It may include headers of e-mails sent and received and information about websites accessed. Voice-over-IP operators such as Skype that operate centralised directory services are also able to log users and calls. The UK’s intelligence agencies, 52 police forces, HM Revenue and Customs, prisons and 510 public authorities can all demand access to communications data. 519,260 such requests were made in 2007.98 From 15 March 2009 ISPs and phone companies will be required to retain specified communications data for 12 months.

No opt out. No unfettered access. Embed the social networking prowling into that and it takes on a very different meaning. There is a consultation process planned for the introduction of the Communication Database in March 2009. After all the noise of the Digital Britain and Open Source reports and the imperative of ‘digital engagement’, it will be interesting to monitor this process. The Open Rights Group has a wiki which is doing some of this here.

The Database State report assesses 46 databases in some detail and it’s worth a read. It then begins to open debate on the consequences of their operation and makes the following recommendations with regard to privacy and human rights:

• Sensitive personal information should normally only be collected and shared with the subject’s consent – and where practical people should opt in rather than opting out.
• Government should compel the provision or sharing of sensitive personal data only for strictly defined purposes, and in almost all cases, sensitive data should be kept on local rather than national systems.
• Individuals should be able to enforce their privacy in court on human-rights grounds without being liable for costs – the state has massive resources to contest cases while the individual does not.
• Citizens should have the right to access most public services anonymously. We have been moving from a world in which departments had to take a positive decision to collect data, to one where they have to take a positive decision not to. This needs to be challenged.

A good watchdog report. More debate is needed though and more engagement in the drafting processes that lead to policy on and implementation of state databases. And perhaps not only state databases. There must be a similar analysis of private databases and data mining practices that I imagine is more chilling than even those explored in this report.