Systematic archiving of state web pages?

In an op-ed titled DNR Censors All Climate Change Info, published on the site Urban Milwaukee on December 26, 2016, author James Rowen calls attention to extensive changes to a DNR website now called The Great Lakes and a changing world.   When the Internet Archive’s Wayback Machine crawled the site on October 30, 2016, the site was called Climate Change and Wisconsin´s Great Lakes (see that iteration of the page via the Wayback machine here).  While the page’s url has not changed, the url is now the only part of the page that contains the phrase “climate change.”

On the federal level, various libraries, including the Library of Congress, the University of North Texas Libraries, and the California Digital Library, along with the Internet Archive, have been conducting an End of Term Archive since 2008, to capture and save U.S. Government websites at the end of presidential administrations.  (Information about the 2016 End of Web Presidential Harvest here; deadline to nominate sites to be crawled is January 13, 2017.)

The column about the changes to the DNR website got me wondering about what systematic efforts are happening at the state level to periodically crawl or harvest state websites for long-term preservation.  We’ve got the wonderful Wisconsin Digital Archives, but it is busy enough with harvesting and preserving what we think of as discrete publications.

Of course, whenever we talk about harvesting and preserving electronic information, the same issues pop up.  Here are a few of them:

  • Are “web pages” discrete, separate documents in and of themselves, like old-fashioned print publications?  Should there be a historical trail of them?  Or are they more like blackboards, used to post short-term messages?  (This question has now been around for a couple of decades!  Have we settled it yet?)
  • If we decide to archive web pages, how often should we do it?  The Wisconsin DNR example from this week illustrates that substantial changes (say, beyond correcting typos) don’t just happen when there’s a change in administrations.
  • It could end up being a massive project!  Who would do the work, and how would we set priorities?

I admit, the issues are daunting.  But the current Wisconsin Document Depository Program wasn’t built in a year (Wisconsin Document Depository Program Historical Background).  Based on my 18 years as a government information librarian, I feel safe in saying that in the future, students and scholars will be interested in things like when a state government’s outlook on climate change shifted.  And it’s crucial that we, as government information specialists, be thinking about these issues.  We’re unique in serving as links between government agencies and current and future students and scholars.  So let’s put our heads together!

–Beth Harper, Memorial Library, University of Wisconsin-Madison