How to Fix Government V: Smarter Transparency

The Information Diet concludes around this theme: “Washington isn’t the land of vast, radical changes, it’s a battleship waiting to be nudged in the right direction. Let the legions of information-obese fight on the front lines, and join me in nudging the small nuts and bolts that hold the ship together.” This week, I’m writing a post a day talking about those nuts and bolts. I hope you’ll join the discussion.

It’s typical that governmental bodies release ethics related information, or information they don’t think anybody will care about on Friday afternoon or even late evenings. It’s called the “Friday News Dump”. So in honor of transparency’s special time in the news cycle, I’m releasing my last post in the How To Fix Government series on smart transparency.

We need to get a grasp on what effective transparency looks like, and up our game – both inside the government and outside, too. All too often, the claim is made that transparency alone is some kind of antidote to corruption and it isn’t. It’s a bizarre and cynical shortcut – when a system needs regular monitoring and policing in order to stay honest, politicians say “we’ll open it up to the public, so that everyone can see what’s going on, and that will keep everyone honest.”

The problem is, it doesn’t keep everyone honest. As Sunlight Labs’ Director Tom Lee said at South by Southwest, it’s not as though people come home from work, eat dinner with their family and say “I’m going to go fight waste, fraud and abuse on” It’s a preposterous and dangerous notion that signs the public up for a job they neither understand nor are committed to, and it lets government off the hook for policing itself.

We need to get beyond this idea that either publishing the data, or making the data pretty or more usable is good enough. We need to engage a bit more in “transparency design” to make sure that the data that’s being released is useful, and that it’s release is helping solve the problem that it’s intended to solve.

Here are some suggestions for a well designed transparency initiative inside the government, or product outside the government:

Know the intent

Know that transparency can have many desired outcomes, and it’s important to discuss what those are before fully planning out a transparency or open data stragegy. Agency employee: it’s really not important that you spend a whole lot of time on this, just a little consideration. A meeting’s worth of time, not six meetings worth of time. Just understand if the intent of the data is either to help fight corruption or increase accountability, create jobs, support interagency efficiency, spur industry growth, or even just experimentation. All or valid. All are useful. It’s just worth discussion up front.

Anomaly Detection and Advanced Sorting

For accountability data especially, it’s useful to design some form of anomaly detection into the system, and while government may not need to do this for bulk data releases, if you’re a web developer looking to increase transparency and accountability, it’s useful to think about this.

Instead of just putting “Here are Obama’s top donor employers” on a web page somewhere, tell me which employers surged in their giving for a given day, month, or quarter. Instead of saying “here are the top lobbying firms for the quarter” tell me which lobbying firms have increased their lobbying spending in the past quarter. Instead of just telling me who visited the White House, tell me whehter or not White House visits are increasing or decreasing, and whether or not a given person is visiting more frequently than they used to.

This is a kind of anomaly detection, and it’d be a great feature to add to most oversight products because I think it’d really tip off journalists in a new way. As they continue to have tighter deadlines and increased concern about revenues, it’d be nice to make it easier for reporters to connect the dots so that media can do its job more easily.


Coupled with anomaly detection, it’s useful to put data into context. If the intent of the story you’re trying to tell with obesity data is that people are getting fat, telling people that the obesity rate in their county went up by 2% for the year isn’t nearly as powerful as telling people that the obesity rate in their county went up by 2% while the surrounding counties went down by 3%. Or that their county is the thinnest in the country.

Likewise, telling people that Barack Obama took $25,000 from employees of the UPS isn’t nearly as useful or important as putting that number into some form of context. What percent of Obama’s fundrasing draw was that for the quarter? How does that compare with donations to other political candidates? How does UPS rank on the list of the top 5,000 donors to Barack Obama?


A good transparency system comes coupled with education about the system. A long time ago I wrote a blog post calling for a Github for Data, Derek Willis noted a slight objection:

Derek’s right and I’d add the problem of importing is nothing compared to the problem of understanding nuance of data. That’s why I think the entire transparency community: government agencies, the non-profit transparency community, the press and the civic hackers, need to do a much better job of both showing their work and explaining the data.

The mantra of the open source community is to build on the shoulders of giants, and the transparency community ought to be no different – but that cannot be done unless we start making it easier for people to get up and running. How many times has an organization spent time “understanding” campaign finance data when much of that time could have been saved by reading some documentation written by someone that’s already gone through the process? How much tacit knowledge of the nuance of the data behind lies within organizations like OMBWatch and the Sunlight Foundation? That knowledge needs to be more widely documented, available and shared.

The Real Value

If we want transparency to be useful for people, we have to stop designing it in such a way where we think that putting the data out there and making it pretty is the end-game. Software can do more than this.

Get the book