Extract 'Alignment' and 'Genotype' to be separate independent pipelines

[michael-harper]

See the context here in the background of the scoping document

Summary
This pull request proposes separating the alignment and genotyping stages into two distinct workflows within the production pipelines. These changes aim to enhance flexibility, control, and allow future changes to be on parity with industry best practices, particularly those established by Illumina's DRAGEN hardware. Additionally, version control, once implemented will be more easily implemented with separated alignment and genotyping from downstream workflows.

Proposed Changes

1. Separation of Pipelines:

• Alignment and Genotyping: Split into standalone pipelines to prevent versioning conflicts and ensure consistent inputs.
• Modularity: Allows independent updates and optimisations for each workflow without mutual disruption.

2. Pipeline Starting Points:

• Detection of CRAM and gVCF Files: Adjust pipelines to ensure clear and informative error messages for missing data, and allow manual triggering of the genotyping pipeline as needed.
• Resource Dependencies: Replace stage dependencies with resource dependencies, requiring a CPG-processed CRAM in Metamist before running the genotyping pipeline, as well as either fastq or CRAM files registered in Metamist prior to running the alignment pipeline.

3. Repository Structure:

• Current Limitations: The existing structure is not conducive to navigating and understanding independent pipelines.
• Proposed Structure: Consider separate repositories for the production pipelines API and actual pipelines, with clear folder structures for individual pipelines and shared resources. This PR implements an interim folder structure.
• This PR will provide an interim folder structure for the alignment and genotyping pipelines.

Considerations

• Integration with Custom Cohorts: Ensure that updates to cohorts with additional samples or sequencing groups are managed without disrupting the pipeline.
• User Responsibilities: Users must ensure that samples in custom cohorts have the required gVCF files before running the pipeline.
• Further Discussion: Topics such as repository restructuring and shared resource versioning require further discussion.
• Version control: Yet to be defined but this separation will hopefully help future efforts in this domain
• Breaking continuity: Production-pipelines is designed to automatically trigger stages when inputs do not exist. Extracting alignment and genotyping pipelines from downstream workflows breaks this continuity. Users will need to manually trigger alignment and/or genotyping pipelines to ensure that all sequencing groups have the correct input for downstream analysis. This break in continuity, although contrary to the design logic of production-pipelines, ensures the separation of pipelines for future version control efforts and prevents erroneous pipeline runs without the correct inputs for all samples.

[MattWellie]

I think this argument will be disappearing soon following the implementation of custom cohorts, @vivbak ?

[MattWellie]

Suggested change

	@stage(
	analysis_type='cram',
	analysis_keys=['cram'],
	)
	@stage(analysis_type='cram', analysis_keys=['cram'])

[MattWellie]

lots of unused imports here, not sure how this passed linting checks

[MattWellie]

not a fan of this at all. We're installing cpg_workflows as a package in the image we intend to run. Relative imports are garbage.

This is also the wrong import, we're using cpg_utils.hail_batch.get_batch

Suggested change

	from ... import get_batch
	from cpg_utils.hail_batch import get_batch

[MattWellie]

BLEH RELATIVE IMPORTS

[MattWellie]

I think it's worth having a conversation about how to segregate this properly - i.e. you've sub-divided these stages into cpg_workflows.stages.XXX, which feels like a very partial solution. That puts us in a weird position where we're trying to separately version a few files inside cpg_workflows, which is itself versioned at the top level.

There's a broader issue of how this pipeline is run - we start the driver image by doing a git clone of production-pipelines into the VM, then we run main.py. That's fine. But when we import cpg_workflows.XXX it imports from the relative path cpg_workflows.XXX that we just cloned into, not the installed version of the codebase (because AFAIK local filepaths are traversed first when Python locates modules). That leads to 'fun' issues where the version of the code being run is based on the current commit, up until we run a python job or a query_command - those are executed in a new VM without the git clone, so they read from the installed version of the codebase (potentially a different version of the codebase). I don't think that should really be addressed by this change, but we should resolve this ambiguity.

A blue sky thinking approach to this division:

- src
  - alignment_workflow
    - stages
      - align
      - cram_qc
    - version.py
  - genotype_workflow
    - stages
      - genotype
      - gvcf_qc
    - version.py
  - gatk_sv_workflow
    - stages
    - version.py
  - large_cohort_workflow
    - stages
    - version.py
  - rare_disease_workflow
    - stages
    - version.py
  - ...? other pipelines we want to version separately
  - cpg_workflows (the library which represents the abstract pipeline - interactions with metamist, setting up a workflow graph, etc.)
- main.py (imports from all the relevant INSTALLED pipelines and builds workflows from them)
- tests
- pyproject.toml

• This would enable us to ensure we're only using the installed version of the code (from cpg_workflows.stages.XXX import YYY is guaranteed to use the imported code, as there's no local path .cpg_workflows, so we know we're using the installed code)
• This would enable use to split out the pipeline framework from the stages that rely on it
• This would enable use to version each coherent pipeline separately

Undoubtably this is more effort, it would involve reorganising the production pipelines codebase and having to build a new version of the cpg_workflows image for each test run in GCP.

However, it would resolve a fundamental ambiguity in our production pipelines workflows - when we run our pipelines, we don't truly know which version of the code we're running. That is a very stupid problem to have, when this whole effort is around versioning our pipelines more specifically. We ARE using cloud compute, so we ARE using containerised pipelines. With our current structure, we're using some of the code in the container, and some pulled in from github at runtime.

This is kinda twinned with #647, a pet peeve of mine - we're aware, and kind of.. OK? with not really knowing exactly what code is going to run in our main pipeline.

n.b. this isn't a proposal so much as it is a kind of... rant. What we have is not good, I don't think this PR solves that (it covers physical separation of the stages, but doesn't address things we should probably address first, like "what code are we running")

^^ I actually don't know if any of this is feasible

[MattWellie]

As a side note, this is a huge leap considering that AFAIK the RFC/scoping doc here has not been reviewed by anyone in software, and there are a number of discussion points identifying that @vivbak and other members of data/software should be consulted before going ahead with implementation.