Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to cache XNAT data #134

Open
laurencejackson opened this issue Jan 17, 2023 · 0 comments
Open

Option to cache XNAT data #134

laurencejackson opened this issue Jan 17, 2023 · 0 comments

Comments

@laurencejackson
Copy link
Collaborator

Importing a project's data from XNAT for each run is a large fraction of each iteration of training runs. This is because on each run the data is downloaded from the server to memory and then discarded when the script exits.

We should implement a caching system so that the downloaded data is stored in an ordered way on the local disk.

The persistence of this data could be defined by.

  • time: a downloaded dataset is cached for a set amount of time and will be deleted after it expires
  • size: a cached dataset directory is defined and will not exceed a certain size, new data coming in will prompt the deletion of old data from the cache.

@GSTT-CSC/core tagging for visibility/input.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant