Question about fast local disk IO

Users
1

What's the recommended way to achieve best sequential IO performance reading and writing really large files (10s or 100s of GB) concurrently on a fast/modern Linux server (assuming 20+GB/sec hardware disk IO speed / 16+ CPU cores / 64+ GB RAM / Chapel 2.2 or 2.3)?
Any code snippets that may help with that?
Thanks!

2

Hi I_M —

Welcome to Chapel's Discourse!

I think the answer may depend on a few things:

  • What kind of data are you reading?
  • What kind of file format are you reading from? (e.g., raw binary, Parquet, HDF5, NetCDF, Zarr, etc.)
  • What kind of data structure would you like it to end up being stored in? (e.g., array, array of records, list, …)

Also, when you say "sequential IO" and "concurrently", are you thinking of having several tasks / cores / nodes executing, each of which is reading a distinct file sequentially?

Thanks for any additional context,
-Brad

3

Hi Brad,
Data file (10s of GB, could be bigger than RAM size) may consist of either raw binary records (fixed format known in advance) or be a TSV text file (ASCII or UTF8) and the output file(s) will use the same formats.
I assume that appropriate data structure for processing (fairly simple and not expected to be CPU intensive) would be an array of records (unless there's a better option).
My goal is to have several tasks and cores ( but not nodes) reading in parallel from different non overlapping parts of the same local input file and then writing results to one or more output files.
Thanks!

4

Hi @I_M –

Thanks for the additional details! I agree that an array of records sounds like a good match for storing the output of these cases. I haven't done a lot of parallel file IO in Chapel lately, so let me give you some quick pointers and then I'll see if someone with more recent experience can jump in for further help and/or more concrete examples.

For the TSV case, my head goes to the ParallelIO package module that was added in Chapel 2.0 and which has seen further improvements since then. I have not had the opportunity to use it myself, but my understanding is that you can specify (1) a deserializer for your record and (2) a delimiter and have it "do the right thing" in terms of using multiple cores, sewing the results together into a single coherent array, etc. With a quick glance, the readDelimitedAsArray routine looks like what I'm thinking of. Beyond the documentation in the module itself, there's a test in the test system that exercises it to read an array of color records here.

That said, if the file size exceeds the available memory—and you're not able/willing to use multiple nodes / distributed memory with readDelimitedAsBlockArray—I'm not sure this routine has a way to restrict it to a subset of the file such that you could stripe chunks of the file in, process them, and then move on to the next chunk (I think it's "all or nothing"). The readDelimited iterator is a finer-grained alternative that would yield a record at a time (in parallel); or you may need to fall back to the more explicit file/fileReader features that I'm about to describe for binary IO:

For the binary case (assuming your record type doesn't result in an obvious and unambiguous delimiter between values that would permit it to use ParallelIO as well), I think you'll want to:

  • define a binaryDeserializer for your record
  • open the file
  • create a task per core using something like coforall tid in 0..<here.maxTaskPar { … }, where each task...
    • computes its subset of the file offsets and array indices (if reading into a coherent array; alternatively, each task could create its own local array to store its chunk of the file)
    • creates its own fileReader with disjoint region arguments to focus on its personalized chunk of the file
    • uses fileReader.read() to read into the array [slice] storing their [sub-array of] values

In this binary case, I've assumed you'll either know a priori how many records the file contains or that you can easily determine it using the file's size and record's binary representation size.

These are obviously just sketches. Again, I'll see if someone with more familiarity with these features can point you to an example or provide one with more detail than my words.

Thanks for your interest in Chapel,
-Brad

1 Like
5

An alternative to writing your own binaryDeserializer would be to write your own implementation of the deserialize method specialized for binaryDeserializer:


use IO;

record rec : serializable {
  var x : int;
  var y : real;

  // modifies existing record in-place
  proc ref deserialize(reader: fileReader(false, binaryDeserializer),
                       ref deserializer: binaryDeserializer) {
    // customize with whatever binary I/O operations you need
    this.x = reader.read(int);
    this.y = reader.read(real);
  }

  // Because we wrote a 'deserialize' method, the compiler no longer generates
  // a default implementation for 'serialize', so you'd need to write your
  // own if you want to print records for debugging purposes:
  proc serialize(writer: fileWriter(?), ref serializer) {
    writer.write("(", x, ", ", y, ")");
  }
}

proc main() {
  var temp = openMemFile();

  // manually write out some dummy values
  {
    var w = temp.writer(false, serializer=new binarySerializer());
    w.write(42);
    w.write(5.0);
  }

  // read them back in
  var x : rec;
  var r = temp.reader(false, deserializer=new binaryDeserializer());
  r.read(x);
  writeln(x);
}

This program prints:

(42, 5.0)
1 Like
6

Hi @I_M —

Since wrapping up the Chapel 2.3 release last week, I've been meaning to check back in here and see whether you were able to make progress with this task, or needed more help in doing so. Most of us are in the process of heading off for winter vacation holidays, but we'd be happy to help with additional sample codes or details in the new year if that'd be helpful.

Best wishes for the remainder of the year,
-Brad