-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add slurm workers for calibration end-to-end test #3461
base: main
Are you sure you want to change the base?
Conversation
8957ff2
to
8cbe910
Compare
I'm not sure if I'm the best person to review this. And I think Charlie is out? Maybe some other people who are more familiar with calibration can take a look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add some docs (inline comments would be fine) to give some context to those who read this file and are not familiar with ClusterManagers?
33a96d3
to
a20a84c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there documentation about the WorkerBacked?
There is no documentation on the WorkerBackend at the moment, I am planning on updating the ClimaCalibrate as part of this PR but I can add some more information in this current PR as well. |
a20a84c
to
3accb17
Compare
This PR updates the calibration end-to-end test to use Distributed.jl and the updated ClimaCalibrate with task-based parallelism. The main two files changed are
calibration/model_interface.jl
andcalibration/test/e2e_test.jl
ClimaCalibrate v0.0.6 has three changes relevant to this PR
forward_model
instead ofset_up_forward_model
andrun_forward_model
addprocs(SlurmManager(n))
can be used to acquired Slurm workersforward_model
runs across Julia workersOther changes: