GIRO executable – first release

Purpose

The purpose of sharing this version of the GIRO executable which provides little functionality is to verify that the handling of NetCDF files by the executable works properly on different Linux machines (both 32 bits and 64 bits).

In the presentation [GIRO_VITICCHIE] one finds the proposal to share a first version of the GIRO executable to perform a first test of:

       reading of the input NetCDF files (observation + spectral response function),

       computation of the observables, and

       writing of a “dummy” lunar calibration file in which only the observables will be stored while rest of the information is left at _FillValue (for the definition of observables please refer to [HLD_GIRO_IO]).

These are the main functionalities of the executable files available in this archive.

Content of the GIRO_firstrelease.tar.gz archive

When unpacking the GIRO_firstrelease.tar.gz archive the user will find locally the following files and folders:

Name

Type

Description

GIRO_firstrelease32.exe

executable file

GIRO executable to be used on 32 bits Linux machines

GIRO_firstrelease64.exe

executable file

GIRO executable to be used on 64 bits Linux machines

./data/

folder

Folder containing three NetCDF lunar observation files from both MTSAT2 and MSG3

./SRFs/

folder

Folder containing the NetCDF spectral response function files of MTSAT2 and MSG3

./results/

folder

Empty

readme.pdf

.pdf document

This document

NetCDF.licence, zlib.licence, and hdf5.licence

text files

Copyrights of the static libraries of the executable

 

Such folder structuring must be respected when using either executable file. In detail, in the folder in which the executable will be lauched the three folders data/, SRFs/ and results/ must be available. During a run, the GIRO executable looks for the spectral response function file of a satellite in the folder SRFs/ and then in the folder data/ for lunar observation files to analyse. The naming of the files stored in both folders must respect the format convention described in [HLD_GIRO_IO]. Sub-folders are allowed in both the folders data/ and SRFs/ with the exception made for subfolders whose name contains the code-name of a satellite (see next section for the description of the code-name).

Running the GIRO executable

A GIRO calibration run can be launched with the command line (for a 32 bits machine):

$ ./GIRO_firstrelease32.exe SAT_NAME

where SAT_NAME is the code-name of the satellite that is used in the naming of the spectral response function NetCDF file of the satellite and the lunar observation NetCDF files from the same satellite. For example:

SAT_NAME = MSG3 (argument of the command line)

W_XX-EUMETSAT-Darmstadt,VIS+IR+SRF,MSG3+SEVIRI_C_EUMG.nc (MSG3 SRF file in SRFs/)

W_XX-EUMETSAT-Darmstadt,VISNIR+SUBSET+MOON,MSG3+SEVIRI_C_EUMG_20140318140112_01.nc (lunar observation file in data/)

The GIRO executable can be launched for a single satellite. Each GIRO run works on all the channels of a satellite that can be analysed using the GIRO executable. The selection of the analysable channels of the satellite is performed by referring to their spectral extension/coverage. In detail, in order to be analysed, a channel must be completely included in the spectral range 0.3 – 2.5 µm. As a result of this process, a list of analysable channels is defined and the executable looks into each lunar observation file to read exclusively the data related to the analysable channels. This means that the user can store the data from all the channels of a satellite in a lunar observation file; the GIRO executable will select the analysable part of such file. Please note that this selection is done by comparing the names of the channels as found in the spectral response function file and the same names as found in the lunar observation file. From this stems that the naming and codes indicating the satellite, the instrument, and the instrument channels must be consistent in the spectral response function file and in the lunar observation files.

In order to get a help on how to run the GIRO executable the user can use the command line:

$ ./GIRO_firstrelease32 –-help

Also in the case of a wrong command line the help is printed on screen.

Interpreting the output of the run

The files that have been included in the folders data/ and SRFs/ will allow the user to run the GIRO executable in the case he/she did not prepare any lunar observation NetCDF file. In detail, two different runs can be performed:

1.       Run on three lunar observation files from MTSAT2 (i.e., a satellite whose instrument has one analysable channel).

2.       Run on three lunar observation files from MSG3 (i.e., a satellite whose instrument has four analysable channels).

These files contain all the COMPULSORY, ADDITIONAL, and IMAGETTES data (see [GIRO_VITICCHIE], [GIRO_TAKAHASHI], and [HLD_GIRO_IO] for the definition of these data).

When launching:

$ ./GIRO_firstrelease32.exe MTSAT2

the executable will print the number of analysable channels for MTSAT2 (i.e., one) and the number of lunar observation files that have been found to be analysed (i.e., three).

The following messages will specify the paths to the files that are produced by the GIRO executable. Currently, two kinds of files are produced by the GIRO executable:

1.       Sanity check files: sanity_check_’SAT_NAME’+’Instrument’+’Channel’.info

these files (one file per analysed channel) collect the results of the comparison between the observables as provided by the user and the observables are derived by the imagette processing of the GIRO executable (see sections 3.3 and 5.2 in [HLD_GIRO_IO]) for each analysed file. For example, in the file sanity_check_MTSAT2+Imager+VIS.info on finds:


         W_JP-JMA-MSC,VISNIR+SUBSET+MOON,MTSAT2+Imager_C_RJTD_20100701062451_01.nc
         Quantity                                User                     Imgt  Processing          
         Observed Irradiance                     0.000702360438230500     0.000702360438649869     
         Observed DC                             15887136                 15887136                 
         DC Offset                               50.515755395683456186    50.515755395683456186    
         Total number of pixels                  82395                    82395                    

         W_JP-JMA-MSC,VISNIR+SUBSET+MOON,MTSAT2+Imager_C_RJTD_20110704163217_01.nc
         Quantity                                User                     Imgt Processing          
         Observed Irradiance                     0.000026484273576469     0.000026484273701312     
         Observed DC                             924069                   924069                   
         DC Offset                               48.963885088919290922    48.963885088919290922    
         Total number of pixels                  9607                     9607                     

         W_JP-JMA-MSC,VISNIR+SUBSET+MOON,MTSAT2+Imager_C_RJTD_20130725035138_01.nc
         Quantity                                User                     Imgt Processing          
         Observed Irradiance                     0.001303947964100031     0.001303947964754193     
         Observed DC                             27668717                 27668717                 
         DC Offset                               50.745347698334967390    50.745347698334967390    
         Total number of pixels                  116446                   116446

 

        

In this example the observed irradiance as provided by the user and the observed irradiance as derived by the GIRO executable via the imagette processing are slightly different even though the procedure employed in the computation of these quantities is exactly the same. The difference stems from the fact that when storing the imagettes in the lunar observation NetCDF file the user has performed a rounding of the radiance values in the imagette. This rounding could not be applied to the digital count (DC) imagette; for this reason the Observed DC, the DC Offset and the Total number of pixels are exactly the same in the two columns (these are derived from the DC imagette). This example shows the usefulness of the imagette processing combined to the sanity checks to quantify the impact, for example, of a method of storing the data on the final results of the processing.

2.       A single lunar calibration NetCDF file:

W_’wmo_LocationIndicator’,SATCAL+ROLOVISNIR,’SAT_NAME’+’Instrument’_C_’wmo_CCCC_Code’_’Observation_time_of_the_first_imagette_YYYYMMDDHHMMSS’_’major_version_number’.nc (for the content of this file refer to [HLD_GIRO_IO]).

 

When launching:

 

$ ./GIRO_firstrelease32.exe MSG3

 

the executable will print a total number of four analysable channels and of three lunar observation files to analyse. Also in this case the paths to the output files that are produced by the procedure are printed; in detail, five output files will be produced when analysing MSG3 data:

1.       sanity_check_MSG3+SEVIRI+VIS006.info

2.       sanity_check_MSG3+SEVIRI+VIS008.info

3.       sanity_check_MSG3+SEVIRI+HRVIS.info

4.       sanity_check_MSG3+SEVIRI+NIR016.info

5.  W_XX-EUMETSAT-Darmstadt,SATCAL+ROLOVISNIR,MSG3+SEVIRI_C_EUMG_20130101145644_00.nc

 

 

i.e., four sanity check files and one lunar calibration file. Besides this, also three error messages will be printed during the processing:

 

COMPULSORY INFORMATION Irr_OBS MISSING IN FILE ‘file name’

 

These messages follow the creation of the sanity check file of the HRVIS channel (i.e., when the procedure examines the content of the HRVIS channel in the lunar observation files), and point out that no irradiance value has been provided by the user. This is a normal output that a user must expect when analysing data from MSG satellites in which not in all lunar observations the Moon is available in all the VISNIR channels simultaneously. This stems from the fact that the HRVIS (high resolution) channel covers only a part of the field of view covered by the VIS06, VIS08, and NIR16 (low resolution) channels. By checking the content of the sanity check file produced for the HRVIS channels the user will find the _FillValue adopted for the lunar calibration NetCDF files: -999.

Analysing your own data

In the case in which the user had prepared his/her own lunar observation files following the format specifications in [HLD_GIRO_IO], in order to run the GIRO_firstrelease on such data he/she will need to prepare the SRF NetCDF file of the satellite with which the data were acquired. To produce the SRF file please contact us and we will made the file available in the GSICS wiki page from which the GIRO_firstrelease.tar.gz has been downloaded.

Recommendations for the users

Verifying that the GIRO executables run properly on your local machine is not the only goal of the exercise described in the previous section! We strongly recommend that the users spend some time in checking the content of both input and output NetCDF files. In detail, the user should compare the content of both the lunar observation files and the lunar calibration files against the content of the tables reported in [HLD_GIRO_IO]. It is worth pointing out that some additional/complementary information can be found in these NetCDF files (e.g., variable attributes like valid_min, valid_max); these were not described in the [HLD_GIRO_IO] for the sake of shortness. In spite of this, when producing his/her own dataset, the user should try to include these additional information in the NetCDF lunar observation files.

If the user is not familiar with the handling of NetCDF files this link will be useful: http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-tutorial.html

List of GUIs to read NetCDF files: http://nsidc.org/data/netcdf/tools.html

System requirements

OS

Linux 32 bits: gcc 4.6.2

Linux 64 bits: gcc 4.3.2

Required libraries/shared objects

libm.so.6, libdl.so.2, and libc.so.6

 

Release notes

Release

First Release

25th July 2014

 

Compilation Environment

Linux 32 bits: gcc 4.6.2

Linux 64 bits: gcc 4.3.2

Zlib 1.2.8

HDF 5-1.8.13

NetCDF-4.3.2

Tested on

Ubuntu 14.04 LTS 32 bits, openSUSE 11.1 64 bits, and openSUSE 12.1 32 bits

Licences

Please refer to the .licence files for the Copyrights of the static libraries used by the executable.

Points of contact

Bartolomeo Viticchie

bartolomeo.viticchie-at-eumetsat.int

Masaya Takahashi

masaya.takahashi-at-eumetsat.int

Sebastien Wagner

sebastien.wagner-at-eumetsat.int

References

[HLD_GIRO_IO]

https://gsics.nesdis.noaa.gov/pub/Development/20140624/GSICS_ROLO_HighLevDescript_IODefinition_wiki.pdf (please ignore the warning message)

[GIRO_VITICCHIE]

https://gsics.nesdis.noaa.gov/pub/Development/20140624/Lunar_Calibration_Toolbox_Implementation.pptx (please ignore the warning message)

[GIRO_TAKAHASHI]

https://gsics.nesdis.noaa.gov/pub/Development/20140624/20140624_Takahashi_LunarCal_DataFormt.pptx (please ignore the warning message)