Public records archive
The PUC Papers
A searchable archive of more than 100,000 documents released under the Public Records Act during the California Public Utilities Commission corruption scandal.
The California Public Utilities Commission corruption scandal put hundreds of thousands of documents into the public domain through the Public Records Act. The PUC Papers is where you can actually read them.
Published by Consumer Watchdog's Capitol Watchdog project, it is a full-text searchable archive of more than 100,000 emails and filings covering the relationship between California's energy regulators and the industry they regulate — PG&E, the San Bruno explosion, Aliso Canyon, the commissioners themselves, and regulatory capture as a pattern rather than an incident.
Why an archive is the story
A press release about 100,000 documents is a press release. A searchable archive of 100,000 documents is an instrument other people can use, and that difference is the entire point of building it.
When the archive launched it produced its own coverage: the Los Angeles Times on the cosy emails between regulators and utility executives, KPBS on the database of officials' correspondence. Neither of those stories was written by us. They were written by reporters who searched the archive and found something — which is the outcome you are actually building for.
What that demands of the build
Search has to be genuinely good, because the alternative is a pile. A reporter arrives knowing a name, a date or a facility and nothing else, and the archive has to turn that into the eight documents that matter out of a hundred thousand. Full-text indexing across a corpus this size, with results fast enough that somebody keeps typing rather than giving up.
The documents also have to stay put. A citation in a published story points at a URL, and if that URL moves the story quietly loses its evidence. Stable addressing is not a nicety on an archive that journalism now depends on.
The unglamorous half
Public records arrive as whatever the agency felt like exporting: inconsistent formats, scanned pages, duplicated threads, attachments detached from their emails. Turning that into something indexable is most of the work and none of the credit, and it is redone every time a new release lands.
The archive is still being added to. The last update was this month.
The point
Consumer Watchdog obtained the documents. The archive is what makes them usable — by reporters, by researchers, by anyone who wants to check a claim rather than take one. That is the shape most of my advocacy work takes: the organisation does the investigation, and I build the thing that lets everybody else do theirs.