🌱 Priorities machine with remediate-machine anotation when selecting the next machine to be remediated

[Karthik-K-N]

What this PR does / why we need it:

This PR contains the changes to priorities the machine with cluster.x-k8s.io/remediate-machine annotation

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #11385

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign vincepri for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

This PR is currently missing an area label, which is used to identify the modified component when generating release notes.

Area labels can be added by org members by writing /area ${COMPONENT} in a comment

Please see the labels list for possible areas.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[fabriziopandini]

@Karthik-K-N thanks for taking up this work!
Let me know if you want some additional clarification

[Karthik-K-N]

Thanks for the clarification, I will update it accordingly. Thanks

[Karthik-K-N]

Updated with changes for KCP as suggested, Will do for remaining soon.

[Karthik-K-N]

Made changes for machine set controller as well, Please take a look when you get some time. Thanks.

[enxebre]

can we update the annotation description as well?

current:

// RemediateMachineAnnotation is the annotation used to mark machines that should be remediated by MachineHealthCheck reconciler.

I think it should be:

// RemediateMachineAnnotation request the MachineHealthCheck reconciler to mark a Machine as unhealthy. CAPI builtin remediation will prioritize Machines with the annotation to be remediated.

[Karthik-K-N]

RemediateMachineAnnotation

Updated. Thank you.

[fabriziopandini]

What about this alternative implementation

Suggested change

	// Returns the machines to be remediated in the following order
	// Machines with RemediateMachineAnnotation annotation if any,
	// Machines failing to come up first because
	// there is a chance that they are not hosting any workloads (minimize disruption).
	func getMachinesToRemediateInOrder(remediateMachines []*clusterv1.Machine) []*clusterv1.Machine {
	// From machines to remediate select the machines with RemediateMachineAnnotation annotation.
	annotatedMachines := collections.FromMachines(remediateMachines...).Filter(collections.HasAnnotationKey(clusterv1.RemediateMachineAnnotation))
	// Filter out the machines which are unique (machines which are not in annotated machines list)
	uniqueMachines := collections.FromMachines(remediateMachines...).Difference(annotatedMachines).UnsortedList()
	// Sort the machines from newest to oldest.
	sort.SliceStable(uniqueMachines, func(i, j int) bool {
	return uniqueMachines[i].CreationTimestamp.After(uniqueMachines[j].CreationTimestamp.Time)
	})
	// combine the annotated machines along with machines to remediate.
	return append(annotatedMachines.UnsortedList(), uniqueMachines...)
	}
	// Returns the machines to be remediated in the following order
	// - Machines with RemediateMachineAnnotation annotation if any,
	// - Machines failing to come up first because
	// there is a chance that they are not hosting any workloads (minimize disruption).
	func sortMachinesToRemediate(machines []*clusterv1.Machine) {
	sort.SliceStable(machines, func(i, j int) bool {
	_, iHasRemediateAnnotation := machines[i].Annotations[clusterv1.RemediateMachineAnnotation]
	_, jHasRemediateAnnotation := machines[j].Annotations[clusterv1.RemediateMachineAnnotation]
	if iHasRemediateAnnotation && !jHasRemediateAnnotation {
	return true
	}
	if !iHasRemediateAnnotation && jHasRemediateAnnotation {
	return false
	}
	return machines[i].CreationTimestamp.After(machines[j].CreationTimestamp.Time)
	})
	}

(trying to have something easier to read/that sticks to the fact that this is a sorting func)

[Karthik-K-N]

Yeah makes sense, I think I took the complicated way, I just updated as suggested but still kept in a seperate function to easily test, Let me know if I need to make it inplace and enhance the existing test itself.

[fabriziopandini]

If possible, I would keep the existing name, it seems more clear to me