[ideas] when error = "continue" do not stop the pipeline upon 'errorValue' int 5 | Timed out The {mirai} dispatcher is not running.

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

With big pipelines, out-of-memory error is part of the production, and it is the most common error.

I use error = "continue" so to get to the end of the pipeline with only failed jobs, and leave the pipeline running as much as possible overnight.

However, I get

34676  | 50311   | 352        | 2412      | 113     | 0      | 0        | 18:29 53.83Submitted batch job 18951449
Submitted batch job 18951450
Submitted batch job 18951451
34674  | 50311   | 354        | 2412      | 113     | 0      | 0        | 18:29 54.08Submitted batch job 18951452
Submitted batch job 18951453
34671  | 50311   | 357        | 2412      | 113     | 0      | 0        | 18:29 54.27Submitted batch job 18951454
Submitted batch job 18951455
34669  | 50311   | 359        | 2412      | 113     | 0      | 0        | 18:29 54.35Submitted batch job 18951456
Submitted batch job 18951457
Submitted batch job 18951458
34666  | 50311   | 362        | 2412      | 113     | 0      | 0        | 18:29 54.65Submitted batch job 18951459
Submitted batch job 18951460
Submitted batch job 18951461
34664  | 50311   | 364        | 2412      | 113     | 0      | 0        | 18:29 54.85Submitted batch job 18951462
Submitted batch job 18951463
Submitted batch job 18951464
34529  | 50444   | 360        | 2412      | 119     | 0      | 0        | 18:34 12.50
Warning messages:
1: replacing previous import ‘broom::tidy’ by ‘tidySingleCellExperiment::tidy’ when loading ‘HPCell’ 
2: replacing previous import ‘tidySingleCellExperiment::tidy’ by ‘tidySummarizedExperiment::tidy’ when loading ‘HPCell’ 
3: replacing previous import ‘tidySummarizedExperiment::tidy’ by ‘tidyseurat::tidy’ when loading ‘HPCell’ 
4: replacing previous import ‘tidySingleCellExperiment::join_transcripts’ by ‘tidyseurat::join_transcripts’ when loading ‘HPCell’ 
5: replacing previous import 'magrittr::set_names' by 'purrr::set_names' when loading 'HPCell' 
6: trust_object_timestamps was deprecated in tar_option_set(). Use the more unified trust_timestamps instead. 
Error:
! targets::tar_make() error
── Debugging ───────────────────────────────────────────────────────────────────────────────────────────────────────
    • tar_errored()
    • tar_meta(fields = any_of("error"), complete_only = TRUE)
    • tar_workspace()
    • tar_workspaces()
── How to ──────────────────────────────────────────────────────────────────────────────────────────────────────────
    • Debug: https://books.ropensci.org/targets/debugging.html
    • Help: https://books.ropensci.org/targets/help.html
── Last error message ──────────────────────────────────────────────────────────────────────────────────────────────
    'errorValue' int 5 | Timed out
 The {mirai} dispatcher is not running. If you are using {crew} without {targets}, be sure to call the start() method of the controller before doing anything else. If you already did, or if you are using {targets}, then the dispatcher process probably started and then failed.
 Please also try upgrading R packages {nanonext}, {mirai}, and {crew} to their latest versions on CRAN. (Likewise with {targets} if you are using it.) Upgrading these packages solves many kinds of errors. 
Another possibility is an out-of-memory error. The dispatcher process can run out of memory if it is overwhelmed with data objects too large or too many to comfortably fit inside a single R process. Please read https://wlandau.github.io/crew/articles/logging.html to learn about proactive resource usage logging which can provide useful data in the event of a crash. 
If memory consumption is a problem, then as a workaround, you could make each task save/load files instead of using/returning large R objects in memory. If you are using {targets}, you might consider storage = "worker", retrieval = "worker", and/or cloud storage as documented at https://books.ropensci.org/targets/performance.html and https://books.ropensci.org/targets/cloud-storage.html.
── Last error traceback ────────────────────────────────────────────────────────────────────────────────────────────
    base::tryCatch(base::withCallingHandlers({ NULL base::saveRDS(base::do.c...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), na...
    doTryCatch(return(expr), name, parentenv, handler)
    tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    base::withCallingHandlers({ NULL base::saveRDS(base::do.call(base::do.ca...
    base::saveRDS(base::do.call(base::do.call, base::c(base::readRDS("/tmp/R...
    base::do.call(base::do.call, base::c(base::readRDS("/tmp/RtmpcUJMBC/call...
    (function (what, args, quote = FALSE, envir = parent.frame()) { if (!is....
    (function (targets_function, targets_arguments, options, envir = NULL, s...
    tryCatch(out <- withCallingHandlers(targets::tar_callr_inner_try(targets...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    withCallingHandlers(targets::tar_callr_inner_try(targets_function = targ...
    targets::tar_callr_inner_try(targets_function = targets_function, target...
    do.call(targets_function, targets_arguments)
    (function (pipeline, path_store, names_quosure, shortcut, reporter, seco...
    crew_init(pipeline = pipeline, meta = meta_init(path_store = path_store)...
    self$run_crew()
    self$iterate()
    self$conclude_worker_task()
    self$controller$pop(scale = TRUE, throttle = TRUE)
    controller$pop(scale = scale, throttle = throttle, error = error)
    .subset2(self, "scale")(throttle = throttle)
    private$.launcher$scale(demand = self$unresolved(), throttle = throttle)
    self$tally()
    daemons %|||% daemons_info(name = private$.name, seconds_interval = priv...
    daemons_info(name = private$.name, seconds_interval = private$.seconds_i...
    if_any(valid, daemons, daemons_error(daemons, name))
    daemons_error(daemons, name)
    crew_error(paste(message, info))
    crew_stop(message = message, class = c("crew_error", "crew"))
    rlang::abort(message = message, class = class, call = emptyenv())
    signal_abort(cnd, .file)
Execution halted

I would like to avoid this crash, as it makes so much harder to conclude the pipeline, forcing me to restart the pipeline 40 times, and execute few successful targets at a time, before crashing.

Thanks!

[wlandau]

This error happens at the level of mirai (in crew) and cannot be recovered from in the same way as error = "continue".

[stemangiola]

So there is no way to make the pipeline resilient for out-of-memory issue? for example if a worker crashes from mirai go on in a way the worker (and offending target) be ignored, so the pipeline continues until the end, expect for incomplete targets?

This seems an important aspects for scaling up pipelines.

[wlandau]

The error happens when the mirai dispatcher crashes, not when a worker crashes. The dispatcher is the local process that sends tasks to workers. There is only one dispatcher, and if it crashes, there is no way to continue. But if a worker crashes, then crew simply launches a new one and mirai retries the task.

[stemangiola]

Would implementing a memory limit within the script that would fail gracefully before the mirai dispatcher does, mitigate the problem?

how about rlimit

I believe future is able to fail gracefully for memory issue with options(future.globals.maxSize = 8 * 1024^3)

[wlandau]

You're welcome to try, but you'll get more information about resource usage with autometric: https://wlandau.github.io/crew/articles/logging.html

The message mentions the possibility of an out-of-memory issue because it's one that has popped up, but that's not necessarily where the error is coming from.