GRWG/GDWG Web Meeting 2020-08-12

GSICS Web Meeting: Scoping ISCCP-NG - GSICS interactions

Agenda

• Quick overview of ISCCP-NG (Andy Heidinger)
• How to define deliverables needed from GSICS (Tim Hewison)
• ISCCP-NG Wish List for GSICS (Andy Heidinger)
  • What information to include in L1g to support the inclusion of the GSICS corrections as they exist today?
  • How to define milestones for deliverables
• Thoughts on ISCCP[-NG] & GSICS (Ken Knapp)
• Feedback from ISCCP on use of GSICS Products (Manik Bali)
• How to interact with ISCCP-NG (and IOGEO)
• Establishment of Task Team

Introduction

This meeting was organized in response to the action A.GWG.2020.1f.1: GSICS will hold a web meeting in June to establish a Task Force from GSICS and "ISCCP-NG" to identify requirements and set up pilot programs before a meeting at EUMETSAT in September.

Participants

Chair: Tim Hewison

EUMETSAT : Tim Hewison, Viju John

CMA: Chengli Qi

NOAA: Xiangqian(Fred) Wu, Manik Bali, Lawrence E Flynn, Fangfang Yu, Ken Knapp

NASA: Dave Doelling, Ben Scarino, Raj Bhatt

JMA: Arata Okuyama, Hideaki Tanaka

KMA: Minju Gu, Wonhyeong, Jin Woo

Summary of Key Outcomes

• ISCCP-NG is creating an excellent platform for testing and using GSICS products and algorithms. ISCCP-NG wants to be a GSICS customer

• ISCCP-NG will start to generate common cloud products from the common channels of the advanced GEO imagers operational in ~2023

• ISCCP-NG may include reprocessing of legacy sensors’ archive data later

• It is critical to GSICS that we receive feedback from ISCCP-NG on the definition and use of GSICS products

• All GSICS agencies operating advanced GEO imagers should present plans to develop calibration monitoring and correction for their common channels at the 2021 annual meeting

A.GDWG.20200812.1: GDWG to review existing product THREDDS directory structure to make it easier for ISCCP-NG users to access products.
A.GCC.20200812.2: GCC/NOAA to Update Product Catalog and provide link to User Guides explicitly
A.GRWG.20200812.3: GRWG to investigate 3.8 micron daytime calibration
A.GRWG.20200812.4: Ken Knapp (NOAA) to solicit ISSCP-NG comments on GEO imager calibration accuracy capabilities and requirements based on Table 1 of Hewison et al. 2020
D.GRWG.20200812.1: Dedicated meetings on specific topics. ISCCP and GSICS to participate in each other’s meetings.

ISCCP-NG Summary

By Andrew Heidinger (NOAA) Presented by Ken Knapp

The presentation began with the ISCCP-NG meeting in EUMETSAT. It was mentioned that ISCCP-NG (Next Generation) is a follow-on from the classical International Satellite Cloud Climatology Project (ISCCP), which aimed at producing cloud climatologies from the legacy satellites starting from AVHRR.

With the advent of the next generation GEO and LEO satellites, ISCCP-NG aims to develop new algorithms to exploit their new common channels to improve its long-term cloud products.

ISCCP-NG is a rethinking as we have common channels across multiple satellites.

Andy Heidinger is an advocate of the L1g ( L1 gridded ) data set.

Tim: Are you going to change the name of ISCCP-NG.
Ken: Similar to calling Advanced VHRR as in AVHRR the ISCCP becomes ISCCP-NG. But it is possible that it will as the program matures.

Tim: At the EUMETSAT ISCCP-NG meeting we discussed the remit of producing NRT or RAC, which one would be preferable?
Ken: This is up for discussion, as it relates to the latency in the processing chain

Tim: Lead time of GSICS product production is important.

Dave Doelling: There are two VIIRS (NPP and N20). The calibration for these two VIIRS are different. GSICS VIS/NIR recommends using N20-VIIRS as the VIS/NIR reference. When you do cloud properties, there is no azimuth angle dependence as in GEO. How do you address this? Which one takes precedence VIIRS or GEO?

Ken: The last meeting focused on using GEO as a reference however Its an ongoing discussion.

Defining Deliverables for ISCCP-NG

By Tim Hewison, EUMETSAT

Tim gave an overview of GSICS product status for GEO imagers, emphasizing that we still have work to do to be useful to ISCCP-NG. However, for the IR channels, this is already underway - e.g. KMA is preparing the KOMPSAT-2A product.

Tim then gave an overview of the ISCCP-NG discussion. A key point is the distinction between GSICS monitoring and applying/using GSICS correction. For instance, if you are monitoring the instrument, do you (still) need a correction if biases are small?

How should biases be corrected? (at the count level, radiance level or Brightness Temp level). How should the correction coefficients be distributed? (Should they be embedded in monitored instrument L1 data file e.g. MSG -x L1B ) or provided separately as a GSICS product data file or both.

Dave: How does ISSCP plan to achieve sensor homogenization?
Ken: We would do Harmonization instead of Homogenization.

Dave: CLARREO's launch will give a boost to tying measurements to traceable reference.

So far we have targeted to correct all GEO imagers for monitoring their biases

Tim: Will ISCCP-NG focus on all GEOs or just operational ones?
Ken: Operational ones to start with and perhaps later on merge with legacy

Tim: Do you need all channels?
Ken: Best is to start with common channels.

Tim: When is the next ISCCP-NG meeting?
Ken: At this point, not sure.

Dave: Is the goal of ISCCP-NG to get cloud properties?
Ken: Development of ISCCP-NG would be driven by mission for cloud products. Will likely start with a common cloud mask from common channels.

Dave: We should save all the output data. On older satellites, machine learning can be used to train newer satellites using older satellites.

Tim: We need inter-calibration products for all Solar channels

Tim: How do we archive data?
Ken: It is up to agencies to archive data.

Dave Doelling: How are counts and radiances related to new and old GEO?
Fred Wu: This relation is updated at every solar calibration. If we are calibrating to radiance then the solar spectrum matters. As long as this is not changed (e.g., if VIIRS and GEO use the same solar calibration spectrum).
Arata: We have two sets of coefficients: 1) count to radiance fixed 2) Correction coefficients to reflect solar diffuser calibration
Dave: Then we have to work on arriving at a common method
Arata: GSICS corrections we apply on the former (1), which uses corrections to the solar diffuser.

ISCCP-NG Wish list for GSICS

By Andy Heidinger, NOAA

Ken gave Andy’s presentation. The ISCCP-NG wish list is comprehensive. Some of the wish list items are already addressed. In summary, the wish list is levied on both GSICS as well as ISCCP-NG. That is what can/should GSICS deliver/provide to ISCCP-NG and conversely what should ISSCP-NG deliver to GSICS.

Some of the points listed in the wish list are

• We would like to have corrections as attributes in our L1g which operate on

  • Reflectances (0-1)

• We would like IR Radiance in the standard unit for all sensors

• We would like to have corrections that are of the same form for all sensors.

• We would like standard radiance to temperature parameters which we will include as attributes.

Dave: In the end, cloud retrieval is done in Brightness Temperature Space
Ken: Can GSICS make finding data files are easy?

Overall more interaction is needed to understand each other.

Thoughts on ISCCP[-NG] & GSICS

By Ken Knapp, NOAA

• Timeliness and latency

  • When are GSICS corrections available vs. ISCCP-NG needs.?

  • ISCCP-NG still needs to define Timeliness requirements

  • ISCCP-NG interim iCDR may be released first then a final ISCCP-NG products

• ISCCP-classic

  • Will continue as long as the team is supported at NCEI and providers send data

  • in the early stages of testing VIIRS integration

  • Should we try to run ISCCP using GSICS calibration corrections? (VIS and IR needed)

  • Has access to GSICS corrections changed?

• Radiance → Brightness temperature - How is that provided?

  • Equations? [Many satellite providers follow this approach]

  • LUT? [VIIRS does this]

• Risk

  • ISCCP historically has centralized the calibration.

  • A decentralized calibration approach increases risk

Ken also suggested a path forward:

• Agree on structure for corrections first

  • Answer many of Andy’s questions regarding correction type, channels, etc.

• Move out in parallel

  • ISCCP NG begins work on developing GEO Level 1g

  • GSICS expands to provide corrections for necessary channels

Tim: RAC's are generated with a latency of 2 weeks. For NRT it is faster (~1 day).
Fangfang: Latency for ABI product is RAC is a 2 week lead time. The IR and VIS products are different.
Dave : DCC Cal is updated on a monthly basis. One can interpolate to daily, noise is reduced with a running mean.

ISCCP-Classic would continue. There is a plan to use GSICS Coeff in the classic.

Radiance to BT conversion process needs to be harmonized

Fred: From a perspective of reprocessing, realized many calibration deficiencies during reprocessing. We don’t have a plan to correct the calibration. Would like to collaborate with ISCCP-NG to correct calibration.

Dave: Could one save L0 data and apply a calibration model.

Fred: Technically this approach is straightforward for radiometric calibration. But the magnitude of data is enormous so it is just a scale problem.

Dave: In the future as the computing power grows we would like to do this.

Fred: We didn’t use to archive L0 data. As we move to cloud this could be possible as a project.

Dave: This could be made as an on order biases and one can easily implement version control. One can make online. Do you think geometric updates are needed more frequently?

Fred: If you wish to apply for geometric as well as radiometric then it is time-consuming. it uses INR

Feedback from ISCCP on Use of GSICS products: GSICS Response

By Manik Bali and Larry Flynn, NOAA

The presentation was in response to the feedback Ken Knapp presented when he tried to access and use GSICS products at the 2019 GSICS Annual Meeting. Some of these can be summarized as

• Files behind a TDS make it difficult to perform bulk downloads

• It was not immediately clear what all the links were and which were useful and which were dead ends.

• ATBDs are prominent but are less useful to end users (initially).

• User guides are especially helpful but are more difficult to access.

• The adjustments provided do not also provide the calibration coefficients

  • Leaving the user wondering “Am I using the right calibration coefficients??”

• The GSICS websites appear to be developed for GSICS contributors not for users.

Manik Bali presented the process of accessing GSICS products. It was recommended that the products be accessed via the GSICS product catalog and not via the THREDDS server directory.

Scripts to download GSICS products have been developed at EUMETSAT (Python by Peter Miu) and NOAA (bash wget script Manik Bali).

Manik went on to show how to access and open GSICS product data files online. He recommended that in order to use GSICS products it is important to start with the User Guide. A good point to start is products with Pre-Op maturity.

All the products are updated daily.

Ken: Is there a way to get the status of update of products?
Manik: Yes there are two ways. 1 ) Quicklooks 2) registering on the Alert System here

Ken: The indication of maturity confuses the user
Manik: One can directly start using the product. User feedback is an important component of assignment of maturity to the products

Larry: After the meeting: We need to provide better guidance on what the maturity levels mean for users.

THREDDS Directory structure

Collaboration THREDDS server attempts to bring together over 66 inter-calibration products onto a single distribution system. We have had two versions of the directory structure. Each of these has advantages,

While the Product Catalog attempts to mitigate the complexity of accessing in a hierarchical structure,

How to interact with ISCCP-NG

By Tim Hewison, EUMETSAT

Plans for GSICS products for GEO imagers for ISCCP-NG

Almost complete spectral coverage by IASI. RSB Spectral coverage of GEO images in 2022/2023.

DCC method applicable to VIS channels, lunar inter-calibration and Ray-matching may be applicable to all common channels in the Reflected Solar Band, but need further development.

Dave: Blended approach is very good as we are taking the best from several approaches ( DCC, Raymatching , LC)

Dave: The 3.8 micron channel is critical for cloud products - How can we monitor its calibration?
Tim: Current GEO-LEO IR algorithm only monitors these channels at night, but this should be followed up by GRWG.
Fangfang: For ABI did not see any difference between day and night.
Fangfang: Sometimes due to stray light it could induce anomalies in night
Tim: Seems to not be a problem with day/night but due to dynamic range.

Tim: How to interact with ISCCP-NG  

?  

Do we need a task team?

Tim: It is a collective effort so no need for a task team.
Dave: If ISCCP-NG is going to meet twice a year, it is important to have a GSICS presentation in ISCCP-NG to get feedback from them.
Tim: Reciprocal attendance in GSICS meetings for ISCCP-NG is important too.

Tim presented a table of GEO imager calibration accuracy from Hewison et al. 2020 and proposed this provide a framework for discussion of ISCCP-NG requirements.

Larry: In terms of coverage of instruments, uniform geospatial coverage is where we are weakest.

Ken: From outside the table summarising the maturity of the GSICS products for GEO imagers seems to show differences of quality.

Manik: Users are encouraged to use the products and provide feedback.