Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix observation id naming for *FRINGE #5

Open
SharonGoliath opened this issue Jun 2, 2020 · 3 comments
Open

Fix observation id naming for *FRINGE #5

SharonGoliath opened this issue Jun 2, 2020 · 3 comments

Comments

@SharonGoliath
Copy link
Collaborator

Querying the observation ID GS-CAL20171119-8-001* shows that in this case the observation ID is set to GS-CAL20171119-8-001-RG-FRINGE for the unprocessed file and it is shown to be a ‘composite’.

It should be:

file id observation id
rgS20171119S0123_fringe GS-CAL20171119-8-001-RG-FRINGE
S20171119S0123 GS-CAL20171119-8-001
@dbohlender
Copy link

A query of Gemini's archive returns 115 processed fringe datasets. I think these are
missing in CAOM2 tables:

Data label File ID
GN-CAL20110313-900-188 N20110313S0188_fringe.fits
GN-CAL20120203-900-281 N20120203S0281_fringe.fits
GS-CAL20060208-1-001-G-FRINGE rgS20060208S0100_FRINGE.fits
GS-CAL20060208-2-001-G-FRINGE rgS20060208S0107_FRINGE.fits
GS-CAL20061129-3-001-G-FRINGE rgS20061129S0108_FRINGE.fits
GS-CAL20070913-2-001-RG-FRINGE rgS20070913S0050_FRINGE.fits
GS-CAL20100713-2-001-RG-FRINGE rgS20100713S0080_FRINGE.fits
GS-CAL20100715-1-001-G-FRINGE rgS20100715S0038_FRINGE.fits
GS-CAL20101027-1-001-G-FRINGE rgS20101027S0022_FRINGE.fits

@dbohlender
Copy link

The following are NOT processed fringe frames according to a Gemini archive query. Not sure if this provides a clue for incorrect product ID/observation ID combos. Likely not.

Observation ID Product ID #1 Product ID #2
GS-CAL20030903-16-001 rgS20030903S0090_FRINGE S20030903S0090
GS-CAL20070912-5-001 rgS20070912S0032_FRINGE S20070912S0032
GS-CAL20100213-2-001 rgS20100213S0048_FRINGE S20100213S0048
GS-CAL20171119-8-001-RG-FRINGE S20171119S0123 rgS20171119S0123_fringe

For some reason a query of the Gemini archive using the last example's observation ID returns nothing but a search by the second file name prefix does.

I think these are all identified as composites but not all have members in CAOM2.

@dbohlender
Copy link

dbohlender commented Jun 2, 2020

I've attached a file listing Gemini CAOM2 observations that are identified as composite observations but have no members. (The query used is at the top of the file.) The good news is that 9140 of more than 9500 of these are TReCS observations. I think these should NOT be composites. I believe processed TReCS files in Gemini's archive are derived by combining the NNODSETS x NSAVSETS contained within a single unprocessed image into a simpler image array.

Most TReCS files are being processed correctly except for the 'composite' algorithm. The exceptions might be those with data labels ending in -G. I may have only found examples with -R suffixes previously. The latter are correctly combined with the unprocessed observations (again, except for 'composite') whereas the -G versions show up as distinct observations.

There are 373 GMOS-N/S observations and a handful of hrwfs and michelle observations.

no_members.txt

SharonGoliath added a commit that referenced this issue Aug 26, 2020
* s2627 - handle a few more naming patterns during validation.

* t76247 - initial testing with proposed Gemini incremental harvesting URL.

* s2646 - regression testing.

* s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe.

* s2647 - regression testing cleanup.

* Remove spurious directory commit.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2468 - SITELLE regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile.

* Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution.

* Issue #3 - fix provenance handling.

* Issue #3 - integration testing.

* Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled.

* Issue #6 - replace execute_composable invocations with run_composable invocations.

* Issue #6 - use all caom2tools from github.

* Issue #6 - fix how recently public query handles netrc file authentication and authorization.

* Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator.

* Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'.

* s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files.

* s2748 + #5 - some fixes.

* s2748 - setting the cache for a file lookup.

* Issue #5 - rethinking the way command-line arguments are issued.

* Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name.

* Issue #5 - more cleanup of code that's no longer required.

* Issue #5 - scrape help.

* s2766 - OBJECT file spatial WCS refactoring.

* s2766 - make changes for s2770 available sooner.

* Update setup.cfg

* s2770 - GEMINI.

Co-authored-by: Sharon Goliath <[email protected]>
SharonGoliath added a commit that referenced this issue Aug 28, 2020
* s2627 - handle a few more naming patterns during validation.

* t76247 - initial testing with proposed Gemini incremental harvesting URL.

* s2646 - regression testing.

* s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe.

* s2647 - regression testing cleanup.

* Remove spurious directory commit.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2468 - SITELLE regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile.

* Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution.

* Issue #3 - fix provenance handling.

* Issue #3 - integration testing.

* Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled.

* Issue #6 - replace execute_composable invocations with run_composable invocations.

* Issue #6 - use all caom2tools from github.

* Issue #6 - fix how recently public query handles netrc file authentication and authorization.

* Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator.

* Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'.

* s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files.

* s2748 + #5 - some fixes.

* s2748 - setting the cache for a file lookup.

* Issue #5 - rethinking the way command-line arguments are issued.

* Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name.

* Issue #5 - more cleanup of code that's no longer required.

* Issue #5 - scrape help.

* s2766 - OBJECT file spatial WCS refactoring.

* s2766 - make changes for s2770 available sooner.

* Update setup.cfg

* s2770 - GEMINI.

* Issue #12 - GMOS keywords should be associated with instrument, not provenance.

Co-authored-by: Sharon Goliath <[email protected]>
SharonGoliath added a commit that referenced this issue Sep 3, 2020
* s2627 - handle a few more naming patterns during validation.

* t76247 - initial testing with proposed Gemini incremental harvesting URL.

* s2646 - regression testing.

* s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe.

* s2647 - regression testing cleanup.

* Remove spurious directory commit.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2468 - SITELLE regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile.

* Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution.

* Issue #3 - fix provenance handling.

* Issue #3 - integration testing.

* Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled.

* Issue #6 - replace execute_composable invocations with run_composable invocations.

* Issue #6 - use all caom2tools from github.

* Issue #6 - fix how recently public query handles netrc file authentication and authorization.

* Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator.

* Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'.

* s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files.

* s2748 + #5 - some fixes.

* s2748 - setting the cache for a file lookup.

* Issue #5 - rethinking the way command-line arguments are issued.

* Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name.

* Issue #5 - more cleanup of code that's no longer required.

* Issue #5 - scrape help.

* s2766 - OBJECT file spatial WCS refactoring.

* s2766 - make changes for s2770 available sooner.

* Update setup.cfg

* s2770 - GEMINI.

* Issue #12 - GMOS keywords should be associated with instrument, not provenance.

* Issue #5 - files with no prefix do not have 'fixed' data labels.

* s2748 - Add support for ALOPEKE/ZORRO observationID handling to GemName.

* s2748 - initial attempt at ALOPEKE observation.

* s2748 - Alopeke - 2 planes per set of files.

* s2748 - Add ZORRO test records.

* s2766 - side-effects of consistent keyword handling in fits2caom2.

* s2748 - add calibration files to Alopeke/Zorro test set.

Co-authored-by: Sharon Goliath <[email protected]>
SharonGoliath added a commit that referenced this issue Sep 8, 2020
* s2627 - handle a few more naming patterns during validation.

* t76247 - initial testing with proposed Gemini incremental harvesting URL.

* s2646 - regression testing.

* s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe.

* s2647 - regression testing cleanup.

* Remove spurious directory commit.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2468 - SITELLE regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile.

* Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution.

* Issue #3 - fix provenance handling.

* Issue #3 - integration testing.

* Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled.

* Issue #6 - replace execute_composable invocations with run_composable invocations.

* Issue #6 - use all caom2tools from github.

* Issue #6 - fix how recently public query handles netrc file authentication and authorization.

* Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator.

* Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'.

* s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files.

* s2748 + #5 - some fixes.

* s2748 - setting the cache for a file lookup.

* Issue #5 - rethinking the way command-line arguments are issued.

* Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name.

* Issue #5 - more cleanup of code that's no longer required.

* Issue #5 - scrape help.

* s2766 - OBJECT file spatial WCS refactoring.

* s2766 - make changes for s2770 available sooner.

* Update setup.cfg

* s2770 - GEMINI.

* Issue #12 - GMOS keywords should be associated with instrument, not provenance.

* Issue #5 - files with no prefix do not have 'fixed' data labels.

* s2748 - Add support for ALOPEKE/ZORRO observationID handling to GemName.

* s2748 - initial attempt at ALOPEKE observation.

* s2748 - Alopeke - 2 planes per set of files.

* s2748 - Add ZORRO test records.

* s2766 - side-effects of consistent keyword handling in fits2caom2.

* s2748 - add calibration files to Alopeke/Zorro test set.

* s2748 - fix attribute name in config reference.

* s2748 - regression testing - get scrape + modify to work with preview generation.

Co-authored-by: Sharon Goliath <[email protected]>
SharonGoliath added a commit that referenced this issue Sep 9, 2020
* s2627 - handle a few more naming patterns during validation.

* t76247 - initial testing with proposed Gemini incremental harvesting URL.

* s2646 - regression testing.

* s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe.

* s2647 - regression testing cleanup.

* Remove spurious directory commit.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2468 - SITELLE regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile.

* Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution.

* Issue #3 - fix provenance handling.

* Issue #3 - integration testing.

* Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled.

* Issue #6 - replace execute_composable invocations with run_composable invocations.

* Issue #6 - use all caom2tools from github.

* Issue #6 - fix how recently public query handles netrc file authentication and authorization.

* Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator.

* Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'.

* s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files.

* s2748 + #5 - some fixes.

* s2748 - setting the cache for a file lookup.

* Issue #5 - rethinking the way command-line arguments are issued.

* Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name.

* Issue #5 - more cleanup of code that's no longer required.

* Issue #5 - scrape help.

* s2766 - OBJECT file spatial WCS refactoring.

* s2766 - make changes for s2770 available sooner.

* Update setup.cfg

* s2770 - GEMINI.

* Issue #12 - GMOS keywords should be associated with instrument, not provenance.

* Issue #5 - files with no prefix do not have 'fixed' data labels.

* s2748 - Add support for ALOPEKE/ZORRO observationID handling to GemName.

* s2748 - initial attempt at ALOPEKE observation.

* s2748 - Alopeke - 2 planes per set of files.

* s2748 - Add ZORRO test records.

* s2766 - side-effects of consistent keyword handling in fits2caom2.

* s2748 - add calibration files to Alopeke/Zorro test set.

* s2748 - fix attribute name in config reference.

* s2748 - regression testing - get scrape + modify to work with preview generation.

Co-authored-by: Sharon Goliath <[email protected]>
SharonGoliath added a commit that referenced this issue Oct 27, 2020
* s2627 - handle a few more naming patterns during validation.

* t76247 - initial testing with proposed Gemini incremental harvesting URL.

* s2646 - regression testing.

* s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe.

* s2647 - regression testing cleanup.

* Remove spurious directory commit.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2648 - regression testing for OMM.

* s2468 - SITELLE regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing.

* s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile.

* Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution.

* Issue #3 - fix provenance handling.

* Issue #3 - integration testing.

* Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled.

* Issue #6 - replace execute_composable invocations with run_composable invocations.

* Issue #6 - use all caom2tools from github.

* Issue #6 - fix how recently public query handles netrc file authentication and authorization.

* Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator.

* Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'.

* s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files.

* s2748 + #5 - some fixes.

* s2748 - setting the cache for a file lookup.

* Issue #5 - rethinking the way command-line arguments are issued.

* Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name.

* Issue #5 - more cleanup of code that's no longer required.

* Issue #5 - scrape help.

* s2766 - OBJECT file spatial WCS refactoring.

* s2766 - make changes for s2770 available sooner.

* Update setup.cfg

* s2770 - GEMINI.

* Issue #12 - GMOS keywords should be associated with instrument, not provenance.

* Issue #5 - files with no prefix do not have 'fixed' data labels.

* s2748 - Add support for ALOPEKE/ZORRO observationID handling to GemName.

* s2748 - initial attempt at ALOPEKE observation.

* s2748 - Alopeke - 2 planes per set of files.

* s2748 - Add ZORRO test records.

* s2766 - side-effects of consistent keyword handling in fits2caom2.

* s2748 - add calibration files to Alopeke/Zorro test set.

* s2748 - fix attribute name in config reference.

* s2748 - regression testing - get scrape + modify to work with preview generation.

* caom2pipe refactoring - remove work.py and it's dependencies, refactor composable.py.

* ZORRO - files without a third axis.

* GEMINI Dockerfile changes - move from opencadc-metadata-curation to opencadc.

Co-authored-by: Sharon Goliath <[email protected]>
SharonGoliath added a commit to SharonGoliath/gem2caom2 that referenced this issue Jan 7, 2021
SharonGoliath added a commit to SharonGoliath/gem2caom2 that referenced this issue Jan 7, 2021
… was by observationID/data label. It's no longer required now that processing is by file name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants