Dear BaseX developers,
I noticed in example 3 under http://docs.basex.org/wiki/CSV_Module#Examples that csv:parse() with option { 'format': 'map' } returns a map of maps, with hardcoded row numbers:
map { 1: map { "City": "Newton", "Name": "John" }, 2: map { "City": "Oldtown", "Name": "Jack" } }
Using maps, which are unordered, to represent something ordered like rows in a CSV, hardcoded row numbers are necessary for reassembling the map in document order. I assume this was a necessary approach when the module was developed in the map-only world of XQuery 3.0. Now that 3.1 supports arrays, might an array of maps be a closer fit for CSV parsing?
array { map { "City": "Newton", "Name": "John" }, map { "City": "Oldtown", "Name": "Jack" } }
I'm also curious, do you know of any efforts to create an EXPath spec for CSV? Putting spec and CSV in the same sentence is dangerous, since CSV is a notoriously under-specified format: "The CSV file format is not standardized" (see https://en.wikipedia.org/wiki/Comma-separated_values). But perhaps there is a common enough need for CSV parsing that such a spec would benefit the community? I thought I'd start by asking here, since BaseX's seems to be the most developed (or only?) CSV module in XQuery.
Then there's the question of how to approach implementations of such a spec. While XQuery is probably capable of parsing and serializing small enough CSV, CSVs do get large and naive processing with XQuery would tend to run into memory issues (as I found with xqjson). This means implementations would tend to write in a lower-level language. eXist, for example, uses Jackson for fn:parse-json(). I see Jackson has a CSV extension too: https://github.com/FasterXML/jackson-dataformat-csv. Any thoughts on the suitability of XQuery for the task?
Joe