WIRED:The first reactions to the Wikileaks data fell into three camps, which should all be debated

I HAVE written a great deal in this column about “open source software”, a term coined by technologist Christine Peterson in 1998 to describe software that was openly developed and freely modifiable. “Open source”, though, has an older meaning in a different context.

“Open source intelligence” is a term in the spy community to describe obtaining and analysing information that is publicly available – as opposed to secret knowledge obtained through exclusive sources, such as informants and espionage.

Last Sunday, a huge chunk of military sources switched abruptly from covert to open source.

Wikileaks, the whistleblower website, published, in tandem with the New York Times, Der Spiegeland the Guardian, a cache of more than 75,000 previously secret intelligence documents on Afghanistan. Wikileaks received the documents and the newspapers worked to analyse and draw out stories and overviews from the data.

One of the reasons Wikileaks handed the information over to three separate media companies was to ensure the entirety of the story came out, and to protect the leak from legal attack from one jurisdiction or another. The datadump wasn’t absolute.

The newspapers chose to redact some of their content, and Wikileaks itself withheld a large amount of data (some 15,000 documents) as part of a “harm minimisation process demanded by our source”.

The main difference between the newspapers’ coverage of the data and Wikileaks’s own was that the whistleblower site provided the leak in as raw a form as one could imagine. It offered up formats for download that, while indigestible for humans, was eminently processable by computers – like SQL (usable to create standalone, searchable and minable databases), CSV (used by spreadsheets) and KML (a format created for Google Earth and intended for geographical analysis).

This was the raw material that fuelled the stories in the papers. The true significance of the Wikileaks downloads though may come in the next few months, as researchers, hobbyists and political groups pounce on this lump of data for their own purposes and ideas.

How much did the military’s reporting of casualties in Afghanistan vary from the figures that they were themselves receiving? What were the regional hotspots at various points, and where, for instance, were the US military’s “embedded reporters” in relation to those hotspots at the time? Were particular military groups more closely attached to civilian errors than others?

As the Guardiannoted, much of the rumours and reports within the cache of intelligence reports seem vague and contradictory. Even the White House defensively noted that much of this material was low-grade, basic intelligence that would be rejected later.

Both quoted a senior US officer who said: “As someone who had to sift through thousands of these reports, I can say that the chances of finding any real information are pretty slim.”

This is the advantage of the open source method. US military intelligence is well-trained in such matters, I am sure, but it works in isolation, and in secrecy. As the invasion of Iraq showed, it is entirely capable of running down ratholes and leaping to erroneous conclusions.

Instead, many eyes will now pore over this data from many different directions, looking for patterns and attempting to eliminate the noise, disinformation and fog of war.

Many will look to it to criticise and condemn the US presence in Afghanistan, but if those on the other side – those who support such military incursions – have any sense, they too will use it to understand better the war in which they find themselves and adapt their counsel to fit more accurately the facts on the ground.

That’s the benefit, usually, of an open society. We get to triangulate on the truth by gathering facts in the public space, then providing them to all sides to chew over. We use this against our own illusions and those of more closed societies who can only view the world through one narrow perspective.

Current wars, and many past ones, often lack the benefit of open debate. In our attempts to cover up the horror and awful moral ambiguities of war, we tie our strongest hand behind our back.

The first reactions to the early Wikileaks data seem to have fallen into three broad camps: those who were shocked by its revelations, those who felt the leak was an inappropriate breach of national security, and those who felt that it told us nothing new about the ugly nature of war.

All three can be true.

Some of us were unaware of mundane, horrific truths. Others feel that we need to sacrifice our open debate as a necessity of war, and finally, some of us, in the absence of that debate, suspected the worst. Well, the data is out there in the open now.

For good or bad, we should delve into it and see whether our instincts are correct, and trim them if not. It’s our duty as citizens of an open, imperfect society.