Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merged PR 750781: Fix retry logic for low disk space
When there is a low disk space, we cancel PipQueue and running pips by triggering SchedulerCancellationToken. The pips get cancelled and they have SandboxedProcessPipExecutionStatus.Canceled. However, we do not set RetryInfo for those pips, so Canceled sandboxed result is translated to PipResultStatus.Failed due to the lack of RetryInfo. The orchestrator receives pip results with Failed with no error logs, so we log DistributionPipFailedOnWorker errors. We should have set RetryInfo for those cancelled pips. Because we check environment.Context.CancellationToken instead of SchedulerCancellationToken, we skip setting RetryInfo. Context.CancellationToken is triggered when CTRL-C is pressed. SchedulerCancellationToken is triggered when we request termination in Scheduler. Related work items: #2121638
- Loading branch information