GSICS NetCDF Convention

GSICS has adopted the netCDF file format for data storage and this document prescribes the content of such files. Every GSICS netCDF file must adhere to the instructions given here.

Familiarity with the netCDF data model and the application programming interface is assumed in this document. Examples of file content in this convention use the Network Common Data Form Language (see Section 2.1.3 of the NetCDF User's Guide).

This document adopts the following netCDF conventions:

File Names

The exact form of GSICS netCDF file names is specified in the file naming convention.

Attributes Usage

It is recommended that information used to support the processing of data in the netCDF file shall be specified as an attribute. Global attributes provides meta-data information for the product and the product type it belongs, while variable attributes provide meta-data information for the contents of the associated variable array. Attributes should follow some agreed guideline, convention or standard as this improves the usability of the product as users can use the data in netCDF file or the associated array as they were intended by the product developer.

Usage of the comments

For information which is not processed by the user, rather to provide a clarification of the product, product type it belongs to, or the associated variable, the ":comment" attribute shall be used to hold this information. Product developers are encourage to "document" their products with comments to improve the usability of their products.

File Content Templates

The following templates prescribe the minimal content for each of the file types. Data producers can add additional data. Content in all capital letters is the description only and needs to be replaced by actual values.

GEO-LEO-IR Template

Global Attributes

 :Conventions = "CF-1.6";

 :Metadata_Conventions = "Unidata Dataset Discovery v1.0";

 :standard_name_vocabulary = "CF Standard Name Table (Version 19, 22 March 2012)";

 :project = "Global Space-based Inter-Calibration System <http://gsics.wmo.int>";

 :title = "TITLE MENTIONING THE PRODUCT TYPE, MONITORED, AND REFERENCE INSTRUMENTS";

 :summary = "SUMMARY TEXT";

 :keywords = "GSICS, satellite, remote sensing, inter-calibration, ONE OF ('reanalysis correction', 'near real time correction'), GEO-LEO-IR, MONITORED INSTRUMENT, REFERENCE INSTRUMENT, infrared";

 :references = "DOCUMENTATION URL";

 :institution = "NAME OF DATA PRODUCER'S ORGANIZATION";

 :license = "Information delivered as a GSICS product is generated in accordance with the GSICS principles and practices. GSICS products are public and may be used and redistributed freely. Any publication using GSICS products should acknowledge both GSICS and the relevant data creator organization. Neither the data creator, nor the data publisher, nor any of their employees or contractors, makes any warranty, express or implied, including warranties of merchantability and fitness for a particular purpose, or assumes any legal liability for the accuracy, completeness, or usefulness, of this information.";

 :naming_authority = "REVERSE DNS OF YOUR DOMAIN";

 :creator_name = "DATA PRODUCER'S DEPARTMENT/OFFICE";

 :creator_email = "DATA PRODUCER'S CONTACT EMAIL";

 :creator_url = "DATA PRODUCER'S WEBSITE";

 :wmo_data_category = 30; // type short

 :wmo_international_data_subcategory = "4" FOR NRTC DATA, "5" FOR RAC DATA; // short

 :local_data_subcategory = 1; // short

The geospatial set of attributes describes the applicability region of the file's data by defining a bounding box as a simple lat/lon rectangle.

 :geospatial_lat_min = SOUTHERNMOST LATITUDE; // float
 :geospatial_lat_max = NORTHERNMOST LATITUDE; // float
 :geospatial_lon_min = WESTERNMOST LONGITUDE; // float
 :geospatial_lon_max = EASTERNMOST LONGITUDE; // float
 :geospatial_lat_units = "degrees_north";
 :geospatial_lon_units = "degrees_east";

The next attribute combines two important information: the maturity level/processing phase and the version number. Further explanation of the MAJOR, MINOR, and REVISION numbers is given in the Data Versioning section.
 :processing_level = "ONE OF ('demonstration', 'preoperational', 'operational')/vMAJOR.MINOR.REVISION";

Monitored and reference instruments and their code from WMO common code table C-5 and C-8:
 :monitored_instrument = "SATELLITE INSTRUMENT";
 :monitored_instrument_code = "WMO_SATELLITE_CODE WMO_INSTRUMENT_CODE";

 :reference_instrument = "SATELLITE INSTRUMENT";
 :reference_instrument_code = "WMO_SATELLITE_CODE WMO_INSTRUMENT_CODE";

The date_created value the time when the file is created for the first time. It should never again change its value. The date_modified attribute's initial value should be the same as date_created. For any subsequent modification of file's data, only the date_modified value should be changed. The window_period represents the period of time for statistics (e.g. from :time_coverage_start to :time_coverage_end). In the case of GEO-LEO-IR NRTC (15 days window_period), this attribute should be expressed as "P-14D+0D". "0D" is the time of t=0.
 :date_created = "FILE CREATION TIME IN YYYY-MM-DDThh:mm:ssZ";
 :date_modified = "FILE MODIFICATION TIME IN YYYY-MM-DDThh:mm:ssZ";

 :history = "VALUE OF DATE_MODIFIED  PROCESSING SOFTWARE RUNTIME PARAMETERS";

 :id = "FILE'S NAME";

 :time_coverage_start = "MINIMUM OF THE VALIDITY_PERIOD VARIABLE AS YYYY-MM-DDThh:mm:ssZ";
 :time_coverage_end = "MAXIMUM OF THE VALIDITY_PERIOD VARIABLE AS YYYY-MM-DDThh:mm:ssZ";

 :window_period = "P-14D+14D" FOR GEO-LEO-IR RAC, "P-14D+0D" FOR GEO-LEO-IR NRTC (depends on each GPRC's definition); 

The following two sets of attributes are in support of the bias plotting application. Choose one that best suits your needs:

Option 1
 :planck_function_constant_c1 = 1.19104E-5

 :planck_function_constant_c1_unit = "mW(cm^-1)^-4 m^-2 sr^-1"

 :planck_function_constant_c2 = 1.43877

 :planck_function_constant_c2_unit = "K cm"

 :radiance_to_brightness_conversion_formula = "tb=((c2 *wnc)/ln(1.+(c1 * wnc^3)/radiance)-beta)/alpha"

 :brightness_to_radiance_conversion_formula = "radiance=(c1 * wnc^3) / ((exp(c2 * wnc / (alpha*tb+beta)))-1.)"

Option 2
 :planck_function_constant_c1 = 1.19104E-5

 :planck_function_constant_c1_unit = "mW(cm^-1)^-4 m^-2 sr^-1"

 :planck_function_constant_c2 = 1.43877

 :planck_function_constant_c2_unit = "K cm"

 :radiance_to_brightness_conversion_formula = "tb = a1 + a2 * ((c2*wnc)/ln(1.+(c1*wnc^3)/radiance)) + a3 * ((c2*wnc)/ln(1.+(c1*wnc^3)/radiance))^2"

 :brightness_to_radiance_conversion_formula = "radiance=(c1*wnc^3)/((exp(c2*wnc/(alpha*tb+beta)))-1.)"

Digital Object Identifiers - DOIs (Optional)

Note, there are no GSICS specific DOI names. DOIs are assigned by the the products owners' organisation (to do this, the owner's organisation must work with a DOI management agency and acts as one of their DOI datacentres). Assignment of DOIs to objects have the following conditions:
  1. The DOI system stands for quality, so DOI names should only be assigned to objects worthy of publication and citation.
  2. Assignment of a DOI to an object must comply with the \x93Rules of Good Scientific Practice\x94 i.e. the owner of the object for which a DOI is to be assigned must guarantee public access to the object and associated information for at least 10 years from the date that the DOI was first registrated.
:atbd_doi = "DOI_NAME"     // Note, there are no GSICS specific DOI names.
:product_doi = "DOI_NAME"  // DOI are assigned by the product owners's organisation and must satisfy the constrainits of 

Dimensions

   chan = <positive integer>;
   chan_strlen = <positive integer>;
   date = ONE OF (<positive integer>, "UNLIMITED");
   validity = 2;

Variables

   float wnc(chan);
     :units = "cm^-1";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";

   float beta(chan);
     :units = "K";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";

   float alpha(chan);
     :units = "1";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";

   char channel_name(chan, chan_strlen);
     :standard_name = "sensor_band_identifier" ;
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";

   float central_wavelength(chan);
     :units = "m";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :standard_name = "sensor_band_central_radiation_wavelength";

   double date(date);
     :units = "seconds since 1970-01-01T00:00:00Z";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :calendar = "gregorian";
     :standard_name = "time";

   double validity_period(date, validity);
     :units = "seconds since 1970-01-01T00:00:00Z";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :calendar = "gregorian";
     :standard_name = "time";

   float slope(date, chan);
     :units = "1";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = -2.0f; // float
     :valid_max = 2.0f; // float

   float slope_se(date, chan);
     :units = "1";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = -2.0f; // float
     :valid_max = 2.0f; // float

   float offset(date, chan);
     :units = "mW m-2 sr-1(cm-1)-1";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = -200.0f; // float
     :valid_max = 200.0f; // float

   float offset_se(date, chan);
     :units = "mW m-2 sr-1(cm-1)-1";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = -200.0f; // float
     :valid_max = 200.0f; // float

   float covariance(date, chan);
     :units = "mW m-2 sr-1(cm-1)-1";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = -200.0f; // float
     :valid_max = 200.0f; // float

   float std_scene_tb(chan);
     :units = "K";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = 230.0f; // float
     :valid_max = 290.0f; // float

   float std_scene_tb_bias(date, chan);
     :units = "K";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = 10.0f; // float
     :valid_max = -10.0f; // float

   float std_scene_tb_bias_se(date, chan);
     :units = "K";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = 10.0f; // float
     :valid_max = -10.0f; // float

   int number_of_collocations(date, chan);
     :units = "1";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :valid_min = 1; // int
     :valid_max = 2147483647; // int

The following three variables must exist if the Option 2 set of global attributes is used.

   float a1(chan);
     :units = "K";
     :long_name = "radiance to brightness temperature conversion band correction coefficients offset term";

   float a2(chan);
     :units = "1";
     :long_name = "radiance to brightness temperature conversion band correction coefficients first-order term";

   float a3(chan);
     :units = "1";
     :long_name = "radiance to brightness temperature conversion band correction coefficients second-order term";

GEO-LEO-VNIR Template

Global Attributes

The following part is the same as GEO-LEO-IR convention.
 :Conventions = "CF-1.6";
 :Metadata_Conventions = "Unidata Dataset Discovery v1.0";
 :standard_name_vocabulary = "CF Standard Name Table (Version 19, 22 March 2012)";
 :project = "Global Space-based Inter-Calibration System <http://gsics.wmo.int>";
 :title = "TITLE MENTIONING THE PRODUCT TYPE, MONITORED, AND REFERENCE INSTRUMENTS";
 :summary = "SUMMARY TEXT";
 :keywords = "GSICS, satellite, remote sensing, inter-calibration, ONE OF ('reanalysis correction', 'near real time correction'), GEO-LEO-VNIR, MONITORED INSTRUMENT, REFERENCE INSTRUMENT, infrared";
 :references = "DOCUMENTATION URL";
 :institution = "NAME OF DATA PRODUCER'S ORGANIZATION";
 :license = "Information delivered as a GSICS product is generated in accordance with the GSICS principles and practices. GSICS products are public and may be used and redistributed freely. Any publication using GSICS products should acknowledge both GSICS and the relevant data creator organization. Neither the data creator, nor the data publisher, nor any of their employees or contractors, makes any warranty, express or implied, including warranties of merchantability and fitness for a particular purpose, or assumes any legal liability for the accuracy, completeness, or usefulness, of this information.";
 :naming_authority = "REVERSE DNS OF YOUR DOMAIN";
 :creator_name = "DATA PRODUCER'S DEPARTMENT/OFFICE";
 :creator_email = "DATA PRODUCER'S CONTACT EMAIL";
 :creator_url = "DATA PRODUCER'S WEBSITE";
 :wmo_data_category = 30; // type short
 :wmo_international_data_subcategory = "4" FOR NRTC DATA, "5" FOR RAC DATA; // short
 :local_data_subcategory = "3" FOR GEO-LEO-VNIR, "4" FOR LEO-LEO-VNIR; // short,

 // The following geospatial set of attributes describes the applicability region of the file's data by defining a bounding box as a simple lat/lon rectangle.
 :geospatial_lat_min = SOUTHERNMOST LATITUDE; // float
 :geospatial_lat_max = NORTHERNMOST LATITUDE; // float
 :geospatial_lon_min = WESTERNMOST LONGITUDE; // float
 :geospatial_lon_max = EASTERNMOST LONGITUDE; // float
 :geospatial_lat_units = "degrees_north";
 :geospatial_lon_units = "degrees_east";

 // processing_level combines two important information: the maturity level/processing phase and the version number.
 // Further explanation of the MAJOR, MINOR, and REVISION numbers is given in the Data Versioning section.
 :processing_level = "ONE OF ('demonstration', 'preoperational', 'operational')/vMAJOR.MINOR.REVISION";

 // Monitored and reference instruments and their code from WMO common code table C-5 and C-8:
 :monitored_instrument = "SATELLITE INSTRUMENT";
 :monitored_instrument_code = "WMO_SATELLITE_CODE WMO_INSTRUMENT_CODE";
 :reference_instrument = "SATELLITE INSTRUMENT";
 :reference_instrument_code = "WMO_SATELLITE_CODE WMO_INSTRUMENT_CODE";

 :date_created = "FILE CREATION TIME IN YYYY-MM-DDThh:mm:ssZ"; // time when the file is created for the first time. It should never again change its value
 :date_modified = "FILE MODIFICATION TIME IN YYYY-MM-DDThh:mm:ssZ"; // attribute's initial value should be the same as date_created
 :history = "VALUE OF DATE_MODIFIED PROCESSING SOFTWARE RUNTIME PARAMETERS";
 :id = "FILE'S NAME";
 :time_coverage_start = "MINIMUM OF THE VALIDITY_PERIOD VARIABLE AS YYYY-MM-DDThh:mm:ssZ";
 :time_coverage_end = "MAXIMUM OF THE VALIDITY_PERIOD VARIABLE AS YYYY-MM-DDThh:mm:ssZ";

 // window_period represents the period of time for statistics (e.g. from :time_coverage_start to :time_coverage_end).
 :window_period = "P-XXD+XXD" FOR GEO-LEO-VNIR RAC, "P-XXD+0D" FOR GEO-LEO-VNIR NRTC; // Depends on each GPRC's definition

 :atbd_doi = "DOI_NAME"; // Optional
 :product_doi = "DOI_NAME"; // Optional

The following OPTIONAL global attributes are added for GEO-LEO-VNIR.

 :srf_url = "SRF (netCDF) URL"; // Optional

Dimensions

   chan = <positive integer>;
   chan_strlen = <positive integer>;
   method = <positive integer>;
   method_strlen = <positive integer>;
   date = ONE OF (<positive integer>, "UNLIMITED");
   validity = 2;

Variables

   float central_wavelength(chan);
     :standard_name = "sensor_band_central_radiation_wavelength" ;
     :long_name = "nominal channel central wavelength";
     :units = "m";
     :valid_max = 1.5e-05f ;
     :valid_min = 3.e-06f ;
     :_FillValue = -999999.f ;
     :valid_max = 1.5e-05f ;
     :valid_min = 3.e-06f ;

   char channel_name(chan, chan_strlen);
     :standard_name = "sensor_band_identifier" ;
     :long_name = "channel identifier";

   double date(date);
     :standard_name = "time";
     :long_name = "DESCRIPTION OF VARIABLE'S CONTENT";
     :units = "seconds since 1970-01-01T00:00:00Z";
     :calendar = "gregorian";
     :bounds = "validity_period" ;

   double validity_period(date, validity);
     :standard_name = "time";
     :long_name = "correction validity period";
     :units = "seconds since 1970-01-01T00:00:00Z";
     :calendar = "gregorian";

   float mon_official_offset(date, chan);
     :long_name = "official calibration offset";
     :units = "W m-2 sr-1 um-1";
     :_FillValue = -999.f ; // Depends on instruments
     :valid_max = 100.f ; // Depends on instruments
     :valid_min = -100.f ; // Depends on instruments

   float mon_official_slope(date, chan);
     :long_name = "official calibration slope";
     :units = "W m-2 sr-1 um-1";
     :_FillValue = -999.f ;
     :valid_max = 100.f ;
     :valid_min = -100.f ;

   char method_name(method, method_strlen);
     :long_name = "method identifier";

   float weight_method(date, chan, method);
     :long_name = "weight of the methods used for the blend in each channel";
     :units = "1";
     :valid_max = 1.f ;
     :valid_min = 0.f ;

   float mon_offset(date, chan, method);
     :long_name = "calibration offset with respect to reference instrument";
     :units = "W m-2 sr-1 um-1";
     :_FillValue = -999.f ;
     :valid_max = 100.f ;
     :valid_min = -100.f ;

   float mon_slope(date, chan, method);
     :long_name = "calibration slope with respect to reference instrument";
     :units = "W m-2 sr-1 um-1";
     :_FillValue = -999.f ;
     :valid_max = 100.f ;
     :valid_min = -100.f ;

Repository of Templates and Their Example Files

To help data producers in creating GSICS netCDF files, the table below contains the links to the template files. Also included are example files implementing the templates. To check if a GEO-LEO-IR products' contents is correct, the NCML version of the product shall be validated against this GEO-LEO-IR XML schema definition.

GSICS Correction

Type Template ExampleSorted ascending
GEO-LEO-VNIR gsics-geoleovnir-template.cdl  
GEO-LEO-IR gsics-goeleoir-template.xml EUMETSAT GPRC RAC File
GEO-LEO-IR gsics-geoleoir-template.cdl NOAA GPRC RAC File

Collocation Data

Type Template Example
GEO-LEO-IR gsics-geoleoircolloc-template.cdl  

Data Versioning

Important factor in increasing data's utility for users is its version information. Each GSICS data type, as defined by the data category, international data subcategory, and data producer, must have its own independent version identifier. This version identifier must appear in the processing_level global attribute.

Data version identifier will be of the form "vX.Y.Z" where X, Y, and Z are integer numbers. X is the major version number; Y is the minor version number; and Z is the revision number. All three numbers shall increase in ascending order independently of each other. For example: "v0.0.1", "v1.5.0", "v2.3.16", "v10.0.3".

When comparing version identifiers, the major version number takes precedence, then the minor version number, and lastly the revision number, i.e.: v0.0.1 < v1.5.0 < v2.3.16 < v10.0.3.

Semantic meaning for each of the three version number changes is as follows:

Revision number will increase for:

  • Any change that brings files and data in line with the published documentation. This includes: any global or variable attribute name and its value, variable's name, variable's data, or dimensions and their values.
Minor version number will increase for:

  • Any addition of new variables, global or variable attributes, or dimensions. Nothing already present in files can change.

  • ATBD changes that affect only the calculation of uncertainties by less than a factor of 2.
Major version number will increase for:

  • Any other ATBD change.

  • Any change in how some data is calculated even if that does not amount to a modification of the ATBD. For example: change of reference calibration targets, etc.

  • Any change of components except uncertainty estimates already present in files.

  • Any other change not covered by the minor or revision number changes.

Templates

  • GEO-LEO-IR.xml: GEO-LEO-IR EUMETSAT preoperational product (to be replaced by operational version in Q3/Q4 2015

  • GEO_LEO_IR.xsd: GEO-LEO-IR NCML Schema - basic version that performs a very simple check. It counts there are the correct number of global attributes, dimensions and variables. It also checked the expected names of these components.
Topic attachments
I Attachment Action Size Date Who Comment
GEO-LEO-IR.xmlxml GEO-LEO-IR.xml manage 11 K 27 Jul 2015 - 12:40 PeterMiu GEO-LEO-IR EUMETSAT preoperational product (to be replaced by operational version in Q3/Q4 2015
GEO_LEO_IR.xsdxsd GEO_LEO_IR.xsd manage 6 K 30 Jul 2015 - 10:38 PeterMiu GEO-LEO-IR NCML Schema - basic version that performs a very simple check. It counts there are the correct number of global attributes, dimensions and variables. It also checked the expected names of these components.
W_US-NESDIS-STARSATCALRACGEOLEOIRGOES13Imager-MetopAIASI_C_KNES_20100414120000_demo_03.cdlcdl W_US-NESDIS-STARSATCALRACGEOLEOIRGOES13Imager-MetopAIASI_C_KNES_20100414120000_demo_03.cdl manage 8 K 09 Jul 2012 - 17:21 UnknownUser GEO-LEO-IR NOAA GPRC Example File
gsics-geoleoir-template.cdlcdl gsics-geoleoir-template.cdl manage 6 K 09 Jul 2012 - 17:18 UnknownUser GEO-LEO-IR Template File
gsics-geoleoircolloc-template.cdlcdl gsics-geoleoircolloc-template.cdl manage 9 K 23 May 2016 - 07:42 MasayaTakahashi GEO-LEO-IR Collocation Template File
gsics-geoleovnir-template.cdlcdl gsics-geoleovnir-template.cdl manage 5 K 02 May 2016 - 05:38 MasayaTakahashi GEO-LEO-VNIR Template File
Topic revision: r104 - 21 Feb 2017, PeterMiu
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding GSICS Wiki? Send feedback