Update minimums when scheduling splits to full nodes
The TopologyAwareNodeSelector does not correctly choose the candidate with the fewest
queued splits because it never updates the minimum. This leads to an uneven distribution
and hotspots.
@@ -239,6 +239,7 @@ public class TopologyAwareNodeSelector
fullCandidatesConsidered++;
int totalSplitCount = assignmentStats.getQueuedSplitCountForStage(node);
if (totalSplitCount < min && totalSplitCount < maxPendingSplitsPerTask) {
+ min = totalSplitCount;
bestQueueNotFull = node;
}
}
