As I read it, the developers are wondering whether XML is too inefficient compared to other approaches, such as Google's "protocol buffers."

This is a good question for them to be asking, but XML is the right choice. The goal is presumably to make transcripts broadly available to people, rather than enable super-fast machine-to-machine communication. So processing efficiency is not critical (and in any case, though XML can be overkill for some tasks, it's a good fit for marking up texts). The key issue is accessibility, and here XML has a huge advantage: it's widely understood by developers, and even a novice can read an XML document and make a guess at what it means.

Conveniently, the overview page on the site they mention has a clear, geek-friendly explanation. A quote:

However, protocol buffers are not always a better solution than XML – for instance, protocol buffers would not be a good way to model a text-based document with markup (e.g. HTML), since you cannot easily interleave structure with text. In addition, XML is human-readable and human-editable; protocol buffers, at least in their native format, are not. XML is also – to some extent – self-describing. A protocol buffer is only meaningful if you have the message definition (the .proto file).

So one way to respond to the developers would be to outline these reasons and point to this geek-friendly explanation. To make a "best practice" argument you could also show examples of how other governments are doing this. I don't know whether a U.S. example would be helpful in this situation, but if so then http://xml.house.gov/ might be a good reference.

Martin Wattenberg

80

What Martin said. XML, while having a bad rep, is exactly the format for text documents (as opposed to data, where it is justly criticized.)

A simple XML schema should be defined: one which requires the minimum of pre-processing before the documents are made public digitally. Schema design can become an awful drag as people seek the "perfect" structural abstraction.

I'd also ask "why not just plain text?" It's worth knowing the answers to that question instead of assuming that they're out there somewhere: sometimes plain text is just fine.

Perhaps the only fundamental benefit XML carries is 100% explicit definition of how characters are encoded in the file. XML could also make it clearer how the document breaks down into sections, although a well defined directory and filename structure can sometimes provide that.

There are more things which would be lovely to have, but if they would require big budget for more staff or system upgrades, or if they'd delay things with infinite debates, they should be logged and deferred to a future cycle.

My main concern: keep it simple. That's good advice in any case, and if the government respondent's first language is English, then you weren't getting "heavy geek," you were getting gibberish.

Joe Germuska

351

+1 on opting for the XML.

(I'd bet Ory's next question is one that's partially addressed in the "What are the best tools for “scraping” data off a Web page for analysis in Excel or other software?" thread.)

John Heasly

472