Skip to content

Latest commit

 

History

History
68 lines (62 loc) · 3.59 KB

README.md

File metadata and controls

68 lines (62 loc) · 3.59 KB

Template of a Cloud Computation

Template of a LEXIS workflow allowing to transfer a dataset from DDI to a Cloud Compute instance on which a computation will be done by running a script provided by the user:

The Run workflow is:

  • getting details on the input dataset (size, locations)
  • asking the Dynamic Allocation Module (DAM) to select the best Cloud infrastructure to compute these input data
  • transferring the input dataset from DDI to the selected Cloud Staging Area
  • creating a Cloud Compute instance
  • SSHFS-mounting the staging area on this compute instance (accessible to root only)
  • Copying the content of this mounted filesystem on the local filesystem accessible as non-root, by default /lexis_input/
  • running the user-provided script to perform a computation on these inputs
  • copying computation results to the cloud staging area
  • transferring of these results from the cloud staging area to DDI
  • replicating these results to other sites if specified
  • cleaning up the cloud staging area and releasing the cloud compute instance

Input properties

The template expects the following input properties (mandatory inputs in bold):

  • token: OpenID Connect access token
  • project_id: LEXIS project identifier
  • computation_dataset_path_input_path: Dataset containing input data
  • computation_script_content: Content of the script to execute
  • computation_output_directory: Path of the directory of results on the Compute Instance to store to DDI
  • computation_ddi_project_path: Path where to transfer the computation results in DDI
  • computation_environment_variables: Environment variables for the script
    • default: {}
  • computation_input_directory: Local directory on the compute instance where the input dataset is accessible to non-root user
    • default: /lexis_input
  • computation_compute_instance_image_name: Name of the Openstack image for the Compute Instance to create
    • default: Ubuntu-18.04
  • computation_compute_instance_user: User used to connect to the compute instance
    • default: ubuntu
  • computation_mount_point_input_dataset: Directory on the compute instance where to mount the dataset
    • default: /mnt/lexis_input
  • computation_decrypt_dataset_input: Should the input dataset be decrypted
    • default: false
  • computation_uncompress_dataset_input: the input dataset be uncompressed
    • default: false
  • computation_mount_point_input_dataset: Directory on the compute instance where to mount the dataset (accessible to root only)
    • default: /mnt/lexis_input
  • computation_output_directory: Computation output directory
    • default: /lexis/output
  • computation_metadata_dataset_result: Metadata for the computation results dataset to create in DDI
    • default:
      • creator:
        • Cloud Computation worflow
      • contributor:
        • Cloud Computation worflow
      • publisher:
        • Cloud Computation worflow
      • resourceType: Dataset
      • title: Cloud Computation workflow results
  • computation_encrypt_dataset_result: Encrypt the result dataset
    • default: false
  • computation_compress_dataset_result: Compress the result dataset
    • default: false
  • computation_result_dataset_replication_sites: List of sites where the result dataset should be available - WARNING: a replicated dataset can't be deleted - (example of values: it4i, lrz)
    • default: []

Ouput attribute

The following output attribute is provided:

  • attribute destination_path of component CloudToDDIJob: the path to the result dataset in DDI
  • attribute stdout of component ExecScript: script stdout
  • attribute stderr of component ExecScript: script stderr