-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix observation id naming for *FRINGE #5
Comments
A query of Gemini's archive returns 115 processed fringe datasets. I think these are
|
The following are NOT processed fringe frames according to a Gemini archive query. Not sure if this provides a clue for incorrect product ID/observation ID combos. Likely not.
For some reason a query of the Gemini archive using the last example's observation ID returns nothing but a search by the second file name prefix does. I think these are all identified as composites but not all have members in CAOM2. |
I've attached a file listing Gemini CAOM2 observations that are identified as composite observations but have no members. (The query used is at the top of the file.) The good news is that 9140 of more than 9500 of these are TReCS observations. I think these should NOT be composites. I believe processed TReCS files in Gemini's archive are derived by combining the NNODSETS x NSAVSETS contained within a single unprocessed image into a simpler image array. Most TReCS files are being processed correctly except for the 'composite' algorithm. The exceptions might be those with data labels ending in -G. I may have only found examples with -R suffixes previously. The latter are correctly combined with the unprocessed observations (again, except for 'composite') whereas the -G versions show up as distinct observations. There are 373 GMOS-N/S observations and a handful of hrwfs and michelle observations. |
* s2627 - handle a few more naming patterns during validation. * t76247 - initial testing with proposed Gemini incremental harvesting URL. * s2646 - regression testing. * s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe. * s2647 - regression testing cleanup. * Remove spurious directory commit. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2468 - SITELLE regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile. * Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution. * Issue #3 - fix provenance handling. * Issue #3 - integration testing. * Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled. * Issue #6 - replace execute_composable invocations with run_composable invocations. * Issue #6 - use all caom2tools from github. * Issue #6 - fix how recently public query handles netrc file authentication and authorization. * Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator. * Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'. * s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files. * s2748 + #5 - some fixes. * s2748 - setting the cache for a file lookup. * Issue #5 - rethinking the way command-line arguments are issued. * Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name. * Issue #5 - more cleanup of code that's no longer required. * Issue #5 - scrape help. * s2766 - OBJECT file spatial WCS refactoring. * s2766 - make changes for s2770 available sooner. * Update setup.cfg * s2770 - GEMINI. Co-authored-by: Sharon Goliath <[email protected]>
* s2627 - handle a few more naming patterns during validation. * t76247 - initial testing with proposed Gemini incremental harvesting URL. * s2646 - regression testing. * s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe. * s2647 - regression testing cleanup. * Remove spurious directory commit. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2468 - SITELLE regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile. * Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution. * Issue #3 - fix provenance handling. * Issue #3 - integration testing. * Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled. * Issue #6 - replace execute_composable invocations with run_composable invocations. * Issue #6 - use all caom2tools from github. * Issue #6 - fix how recently public query handles netrc file authentication and authorization. * Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator. * Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'. * s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files. * s2748 + #5 - some fixes. * s2748 - setting the cache for a file lookup. * Issue #5 - rethinking the way command-line arguments are issued. * Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name. * Issue #5 - more cleanup of code that's no longer required. * Issue #5 - scrape help. * s2766 - OBJECT file spatial WCS refactoring. * s2766 - make changes for s2770 available sooner. * Update setup.cfg * s2770 - GEMINI. * Issue #12 - GMOS keywords should be associated with instrument, not provenance. Co-authored-by: Sharon Goliath <[email protected]>
* s2627 - handle a few more naming patterns during validation. * t76247 - initial testing with proposed Gemini incremental harvesting URL. * s2646 - regression testing. * s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe. * s2647 - regression testing cleanup. * Remove spurious directory commit. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2468 - SITELLE regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile. * Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution. * Issue #3 - fix provenance handling. * Issue #3 - integration testing. * Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled. * Issue #6 - replace execute_composable invocations with run_composable invocations. * Issue #6 - use all caom2tools from github. * Issue #6 - fix how recently public query handles netrc file authentication and authorization. * Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator. * Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'. * s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files. * s2748 + #5 - some fixes. * s2748 - setting the cache for a file lookup. * Issue #5 - rethinking the way command-line arguments are issued. * Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name. * Issue #5 - more cleanup of code that's no longer required. * Issue #5 - scrape help. * s2766 - OBJECT file spatial WCS refactoring. * s2766 - make changes for s2770 available sooner. * Update setup.cfg * s2770 - GEMINI. * Issue #12 - GMOS keywords should be associated with instrument, not provenance. * Issue #5 - files with no prefix do not have 'fixed' data labels. * s2748 - Add support for ALOPEKE/ZORRO observationID handling to GemName. * s2748 - initial attempt at ALOPEKE observation. * s2748 - Alopeke - 2 planes per set of files. * s2748 - Add ZORRO test records. * s2766 - side-effects of consistent keyword handling in fits2caom2. * s2748 - add calibration files to Alopeke/Zorro test set. Co-authored-by: Sharon Goliath <[email protected]>
* s2627 - handle a few more naming patterns during validation. * t76247 - initial testing with proposed Gemini incremental harvesting URL. * s2646 - regression testing. * s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe. * s2647 - regression testing cleanup. * Remove spurious directory commit. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2468 - SITELLE regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile. * Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution. * Issue #3 - fix provenance handling. * Issue #3 - integration testing. * Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled. * Issue #6 - replace execute_composable invocations with run_composable invocations. * Issue #6 - use all caom2tools from github. * Issue #6 - fix how recently public query handles netrc file authentication and authorization. * Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator. * Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'. * s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files. * s2748 + #5 - some fixes. * s2748 - setting the cache for a file lookup. * Issue #5 - rethinking the way command-line arguments are issued. * Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name. * Issue #5 - more cleanup of code that's no longer required. * Issue #5 - scrape help. * s2766 - OBJECT file spatial WCS refactoring. * s2766 - make changes for s2770 available sooner. * Update setup.cfg * s2770 - GEMINI. * Issue #12 - GMOS keywords should be associated with instrument, not provenance. * Issue #5 - files with no prefix do not have 'fixed' data labels. * s2748 - Add support for ALOPEKE/ZORRO observationID handling to GemName. * s2748 - initial attempt at ALOPEKE observation. * s2748 - Alopeke - 2 planes per set of files. * s2748 - Add ZORRO test records. * s2766 - side-effects of consistent keyword handling in fits2caom2. * s2748 - add calibration files to Alopeke/Zorro test set. * s2748 - fix attribute name in config reference. * s2748 - regression testing - get scrape + modify to work with preview generation. Co-authored-by: Sharon Goliath <[email protected]>
* s2627 - handle a few more naming patterns during validation. * t76247 - initial testing with proposed Gemini incremental harvesting URL. * s2646 - regression testing. * s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe. * s2647 - regression testing cleanup. * Remove spurious directory commit. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2468 - SITELLE regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile. * Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution. * Issue #3 - fix provenance handling. * Issue #3 - integration testing. * Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled. * Issue #6 - replace execute_composable invocations with run_composable invocations. * Issue #6 - use all caom2tools from github. * Issue #6 - fix how recently public query handles netrc file authentication and authorization. * Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator. * Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'. * s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files. * s2748 + #5 - some fixes. * s2748 - setting the cache for a file lookup. * Issue #5 - rethinking the way command-line arguments are issued. * Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name. * Issue #5 - more cleanup of code that's no longer required. * Issue #5 - scrape help. * s2766 - OBJECT file spatial WCS refactoring. * s2766 - make changes for s2770 available sooner. * Update setup.cfg * s2770 - GEMINI. * Issue #12 - GMOS keywords should be associated with instrument, not provenance. * Issue #5 - files with no prefix do not have 'fixed' data labels. * s2748 - Add support for ALOPEKE/ZORRO observationID handling to GemName. * s2748 - initial attempt at ALOPEKE observation. * s2748 - Alopeke - 2 planes per set of files. * s2748 - Add ZORRO test records. * s2766 - side-effects of consistent keyword handling in fits2caom2. * s2748 - add calibration files to Alopeke/Zorro test set. * s2748 - fix attribute name in config reference. * s2748 - regression testing - get scrape + modify to work with preview generation. Co-authored-by: Sharon Goliath <[email protected]>
* s2627 - handle a few more naming patterns during validation. * t76247 - initial testing with proposed Gemini incremental harvesting URL. * s2646 - regression testing. * s2646 - refactor Gemini to make use of the run_composable, name_builder_composable, and data_source_composable abstractions introduced into caom2pipe. * s2647 - regression testing cleanup. * Remove spurious directory commit. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2648 - regression testing for OMM. * s2468 - SITELLE regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing. * s2650 - SPIRou regression testing - add importlib-metadata to Dockerfile. * Issue #4 - add CachingObsFileRelationship class to do a lookup of the observationID<=>file name relationship that is ordered by distance from execution. * Issue #3 - fix provenance handling. * Issue #3 - integration testing. * Issue #6 - replace execute_composable invocations with run_composable invocations. Refactor how file name to data label lookup is handled. * Issue #6 - replace execute_composable invocations with run_composable invocations. * Issue #6 - use all caom2tools from github. * Issue #6 - fix how recently public query handles netrc file authentication and authorization. * Issue #6 - got side-tracked and fixed (?) the WCS validation issue with the caom2.4 validator. * Issue #6 - header information from archive.gemini.edu may contain '--- PHU' OR '--- HDU 0'. * s2748 - do an initial test observation for ZORRO/ALOPEKE, and rename the expected xml files. * s2748 + #5 - some fixes. * s2748 - setting the cache for a file lookup. * Issue #5 - rethinking the way command-line arguments are issued. * Issue #5 - remove a lot of code that was used when processing was by observationID/data label. It's no longer required now that processing is by file name. * Issue #5 - more cleanup of code that's no longer required. * Issue #5 - scrape help. * s2766 - OBJECT file spatial WCS refactoring. * s2766 - make changes for s2770 available sooner. * Update setup.cfg * s2770 - GEMINI. * Issue #12 - GMOS keywords should be associated with instrument, not provenance. * Issue #5 - files with no prefix do not have 'fixed' data labels. * s2748 - Add support for ALOPEKE/ZORRO observationID handling to GemName. * s2748 - initial attempt at ALOPEKE observation. * s2748 - Alopeke - 2 planes per set of files. * s2748 - Add ZORRO test records. * s2766 - side-effects of consistent keyword handling in fits2caom2. * s2748 - add calibration files to Alopeke/Zorro test set. * s2748 - fix attribute name in config reference. * s2748 - regression testing - get scrape + modify to work with preview generation. * caom2pipe refactoring - remove work.py and it's dependencies, refactor composable.py. * ZORRO - files without a third axis. * GEMINI Dockerfile changes - move from opencadc-metadata-curation to opencadc. Co-authored-by: Sharon Goliath <[email protected]>
… was by observationID/data label. It's no longer required now that processing is by file name.
Querying the observation ID GS-CAL20171119-8-001* shows that in this case the observation ID is set to GS-CAL20171119-8-001-RG-FRINGE for the unprocessed file and it is shown to be a ‘composite’.
It should be:
The text was updated successfully, but these errors were encountered: