CDF5 documentation lacking and a question

[czender]

A user applying NCO arithmetic to large data files in the "new" CDF5 format is encountering issues that result (silently) in bad answers. I'm trying to track down the problem. I have not found the netCDF documentation on CDF5 format limitations on this page and so one purpose of this issue to request the addition of CDF5 limits there (assuming that's the right place for it). The table also needs reformatting.

This PnetCDF page says

Amount of Individual Requests is Limited to 2GB: In PnetCDF, a single get/put request is limited to the amount of 2GiB....A solution for PnetCDF users is to break the request to multiple get/put calls so that each is less than amount of 2 GiB.

Forgetting about MPI, and only considering serial netCDF environments, does the 2GiB put/get request limit above apply to any netCDF file formats, including the CDF5 format? NCO does not limit its request sizes, so I wonder if that could be the problem...

[DennisHeimbigner]

Need to clarify a bit. Is a CDF5 file being accessed using NC_PNETCDF or by using
NC_CDF5 mode flag?

[czender]

Via NC_CDF5 flag

[wkliao]

Do you mean "silent" the unexpected values appear in files for write and vise versa for read?
And no error codes were returned...?

My understanding of NetCDF is it first finds the number of contiguous requests (in file layout) from arguments start and count, and then runs a loop to write one request at a time. I assume your case is to write a contiguous chunk of size > 2GB. I have never tried this before. It is likely some internal codes need to be adjusted to handle this kind of request.

[czender]

Yes, it would explain the problem my users are experiencing if requests for more than 2 GB from a CDF5 file do not return the requested amount. There is no problem doing this from a netCDF4 file. Again, this is just a hypothesis, I'm trying to track down a mysterious bug, and this could be it. Need confirmation.

[czender]

I have now replicated workflows that strongly suggest there are undocumented differences between put/get results for large (4-8 GB) variables between CDF5 and NETCDF4 files. I don't have proof that this is a netCDF issue rather than an NCO issue, though @wkliao suggests that CDF5 get/put in the netCDF library was never tested for requests > 2 GB (and PnetCDF explicitly does not support single requests > 2 GB). This issue prevents the DOE ACME MPAS model runs at high resolution (that are archived in CDF5 format) from being properly analyzed, so it affects users now. Is anyone (@wkliao ?) interested in verifying whether CDF5 has put/get limits?

My circumstantial methods and evidence are that a billion doubles all equal to one do average to one for NETCDF4 data, and don't for CDF5 data...

ncap2 -5 -v -O -s 'one=1;defdim("dmn_1e9",1000000000);dta_1e9[dmn_1e9]=1.0;two=2;' ~/nco/data/in.nc ~/foo_big.nc5
ncap2 -4 -v -O --cnk_dmn=dmn_1e9,100000000 -s 'one=1;defdim("dmn_1e9",1000000000);dta_1e9[dmn_1e9]=1.0;two=2' ~/nco/data/in.nc ~/foo_big.nc4
ncwa -O ~/foo_big.nc5 ~/foo_avg.nc5
ncwa -O ~/foo_big.nc4 ~/foo_avg.nc4
ncks -H ~/foo_avg.nc5
ncks -H ~/foo_avg.nc4

which results in

zender@skyglow:~$ ncks -H ~/foo_avg.nc5
netcdf foo_avg {

  variables:
    double dta_1e9 ;

    int one ;

    int two ;

  data:
    dta_1e9 = 0.536870912 ; 

    one = 1 ; 

    two = 2 ; 

} // group /
zender@skyglow:~$ ncks -H ~/foo_avg.nc4
netcdf foo_avg {

  variables:
    double dta_1e9 ;

    int one ;

    int two ;

  data:
    dta_1e9 = 1 ; 

    one = 1 ; 

    two = 2 ; 

} // group /

The first ncap2 command above requires the latest NCO snapshot to support CDF5. Note that these commands both place a few other variables, named "one" and "two", around the large variable "dta_1e9" in the output file, so dta_1e9 is not the only variable. When dta_1e9 is the only variable, the CDF5-based workflow yields the correct answer! So, if my hypothesis is correct, CDF5 variables larger than some threshold size (possibly 2 GB?) are not written and/or read correctly when the nc_put/get_var() is one request for the entire variable, and there are other variables in the dataset.

The behavior is identical on netCDF 4.4.1.1 and today's daily snapshot of 4.5.1-development.

[DennisHeimbigner]

Since the CDF5 code in libsrc came from pnetcdf originally,
it would not be surprising if the 2gb limit snuck in.
We are going to need a simple C program to test this.
I tried but ran out of memory.

[wkliao]

I tested a short netcdf program to mimic the I/O @czender described (the codes are shown below). It creates 3 variables, namely var, one, and two. var is a 1D array of type double of 2^9 elements. Variables one and two are scalars. The 3 variables are first written to a new file and then read back to calculate the average. However, I could not reproduce the problem with this program.

@czender could you use this program and modify it close to what you are doing with the NCO operations?

#include <stdio.h>
#include <stdlib.h>
#include <netcdf.h>

#define DIM 1073741824

#define ERR {if(err!=NC_NOERR){printf("Error at line %d in %s: %s\n", __LINE__,__FILE__, nc_strerror(err));nerrs++;}}

int main(int argc, char *argv[])
{
    int err, nerrs=0, ncid, dimid, varid[3], int_buf1, int_buf2;
    size_t i;
    double *buf, avg=0.0;

    err = nc_create("test.nc", NC_CLOBBER|NC_CDF5, &ncid); ERR
    err = nc_def_dim(ncid, "dim", DIM, &dimid); ERR
    err = nc_def_var(ncid, "var", NC_DOUBLE, 1, &dimid, &varid[0]); ERR
    err = nc_def_var(ncid, "one", NC_INT, 0, NULL, &varid[1]); ERR
    err = nc_def_var(ncid, "two", NC_INT, 0, NULL, &varid[2]); ERR
    err = nc_set_fill(ncid, NC_NOFILL, NULL); ERR
    err = nc_enddef(ncid); ERR

    buf = (double*) malloc(DIM * sizeof(double));
    for (i=0; i<DIM; i++) buf[i] = 1.0;

    err = nc_put_var_double(ncid, varid[0], buf); ERR
    int_buf1 = 1;
    err = nc_put_var_int(ncid, varid[1], &int_buf1); ERR
    int_buf2 = 2;
    err = nc_put_var_int(ncid, varid[2], &int_buf2); ERR
    err = nc_close(ncid); ERR

    err = nc_open("test.nc", NC_NOWRITE, &ncid); ERR
    err = nc_inq_varid(ncid, "var", &varid[0]); ERR
    err = nc_inq_varid(ncid, "one", &varid[1]); ERR
    err = nc_inq_varid(ncid, "two", &varid[2]); ERR
    for (i=0; i<DIM; i++) buf[i] = 0.0;
    err = nc_get_var_double(ncid, varid[0], buf); ERR
    int_buf1 = int_buf2 = 0;
    err = nc_get_var_int(ncid, varid[1], &int_buf1); ERR
    err = nc_get_var_int(ncid, varid[2], &int_buf2); ERR
    err = nc_close(ncid); ERR

    printf("get var one = %d\n",int_buf1);
    printf("get var two = %d\n",int_buf2);
    for (i=0; i<DIM; i++) avg += buf[i];
    avg /= DIM;
    printf("avg = %f\n",avg);
    free(buf);

    return (nerrs > 0);
}

% ./tst_cdf5 
get var one = 1
get var two = 2
avg = 1.000000

% ls -lh
total 8.1G
-rw------- 1 wkliao users 2.2K Aug 21 16:16 Makefile
-rw-r--r-- 1 wkliao users 8.1G Aug 21 16:29 test.nc
-rwxr-xr-x 1 wkliao users 1.1M Aug 21 16:27 tst_cdf5
-rw------- 1 wkliao users 1.8K Aug 21 16:27 tst_cdf5.c

% ncdump -h test.nc
netcdf test {
dimensions:
	dim = 1073741824 ;
variables:
	double var(dim) ;
	int one ;
	int two ;
}

[czender]

Thanks @wkliao. This is a good starting point. I made a few changes to more closely follow the NCO code path. However, I still get the same results you do. Will keep trying...

[czender]

@wkliao I notice something I do not understand about CDF5: I have a netCDF4 file with two variables whose on-disk sizes compute as 9 GB and 3 GB, respectively. By "computes as" I mean multiplying the dimension sizes times the size of NC_DOUBLE. And the netCDF4 (uncompressed) filesize is, indeed, 12 GB, as I expect. Yet when I convert that netCDF4 file to CDF5, the total size changes from 12 GB to 9 GB. In other words, it looks like CDF5 makes use of compression, or does not allocate wasted space for _FillValues, or something like that. If you understand what I'm talking about, please explain why CDF5 consumes less filespace than expected...

[wkliao]

The file size should be 12GB. Do you have a test program that can reproduce this?

[czender]

Uh oh. It's hard to wrap my head around all this. The mysterious issues only appear with huge files that are hard to manipulated. The issues when processing with NCO may be due to NCO, but in order to verify I have to use another toolkit. CDO does not yet support CDF5. And nccopy either fails (with -V) to extract the variables I want, or (with -v) extracts all the variables so I can't mimic the NCO workflow...

zender@skyglow:~$ nccopy -V one_dmn_rec_var,two_dmn_rec_var ~/nco/data/in.nc ~/foo.nc
NetCDF: Variable not found
Location: file nccopy.c; line 979
zender@skyglow:~$ nccopy -v one_dmn_rec_var,two_dmn_rec_var ~/nco/data/in.nc ~/foo.nc
zender@skyglow:~$ ncdump ~/foo.nc | m
netcdf foo {
dimensions:
        dgn = 1 ;
        bnd = 2 ;
        lat = 2 ;
        lat_grd = 3 ;
        lev = 3 ;
        rlev = 3 ;
        ilev = 4 ;
        lon = 4 ;
        lon_grd = 5 ;
        char_dmn_lng80 = 80 ;
        char_dmn_lng26 = 26 ;
        char_dmn_lng04 = 4 ;
        date_dmn = 5 ;
        fl_dmn = 3 ;
        lsmlev = 6 ;
        wvl = 2 ;
        time_udunits = 3 ;
        lon_T42 = 128 ;
        lat_T42 = 64 ;
        lat_times_lon = 8 ;
        gds_crd = 8 ;
        gds_ncd = 8 ;
        vrt_nbr = 2 ;
        lon_cal = 10 ;
        lat_cal = 10 ;
        Lon = 4 ;
        Lat = 2 ;
        time = UNLIMITED ; // (10 currently)
variables:
        int date_int(date_dmn) ;
                date_int:long_name = "Date (as array of ints: YYYY,MM,DD,HH,MM)" ;
        float dgn(dgn) ;
                dgn:long_name = "degenerate coordinate (dgn means degenerate, i.e., of size 1)" ;
...

Until I find another way to subset variables from a CDF5 file, I'm stuck. @WardF would you please instruct me how to use nccopy to subset certain variables from a CDF5 file? I think the above output demonstrates that nccopy (yes, the latest 4.5.x) has some breakage with the -v and -V options.