[ideas] meake it easier to debug large pipelines

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

If I have 10K targets, debugging is quite hard. Currently, not only I have to wait few minutes to skip all targets, but many other targets execute, and because I have callr_function = NULL my session crashes because get memory starved.

I suggest

• If debug = "abc", then the target "abc" is run first without going through all other targets, no matter if they were completed or to be still executed.

Here I am debugging debug = "empty_tbl_tier_1_36c66b571bdb5cae" but other unrelated steps are being queued and executed starving my session, and so I am not able to debug the step I want.

• in tar_meta, there should be a column with all the input target names (dependencies). For example if target "c" depends on targets "a" and "b", it would be nice to have a column

dependent_targets = c("b", "c")

So can run the function completely manually, and debug very quickly.

Thanks!

[wlandau]

It sounds like workspace-based debugging would fit better with your situation: https://books.ropensci.org/targets/debugging.html#workspaces. The manual demonstrates workspace_on_error = TRUE, but you can also use the workspaces argument of tar_option_set() to specify one or more individual branches: https://docs.ropensci.org/targets/reference/tar_option_set.html#arg-workspaces

[stemangiola]

Thanks I will try