✨ Add bootCommands to cloud-init file generation #11271

davidumea · 2024-10-08T14:17:23Z

What this PR does / why we need it:
Adds the ability to provide bootcmd commands to cloud-init via the KubeadmConfig or KubeadmConfigTemplates custom resources. (Same for KubeadmControlPlane)

One thing that I know that does not work at the time of creating this PR is validating/mutating webhooks for KubeadmConfig and KubeadmConfigTemplates. Some pointers to where to change that would be appreciated.

Which issue(s) this PR fixes
Fixes #

k8s-ci-robot · 2024-10-08T14:17:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-10-08T14:17:32Z

Welcome @davidumea!

It looks like this is your first PR to kubernetes-sigs/cluster-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-10-08T14:17:33Z

Hi @davidumea. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-10-08T14:21:48Z

Are there version constraints around bootcmd? Does it already exist in cloud-init for a long time?

What happens if we render it but the cloud-init version in the machine does not support it? Does it make cloud-init fail as a whole or is it just ignored?

Edit: looks like it is already there for a long time (from the start?) at 0.7.7 https://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

davidumea · 2024-10-09T12:28:47Z

Are there version constraints around bootcmd? Does it already exist in cloud-init for a long time?

Edit: looks like it is already there for a long time (from the start?) at 0.7.7 https://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

Exactly I don't think that should be an issue as bootcmd has been part of cloud-init for a long time.

What happens if we render it but the cloud-init version in the machine does not support it? Does it make cloud-init fail as a whole or is it just ignored?

During my testing, if the bootcmd module fails e.g. due to invalid commands, cloud-init still continues with the following modules. I haven't been able to test running this with a cloud-init version that doesn't support bootcmd.

davidumea · 2024-10-17T12:30:16Z

@chrischdi Is there anything I can do to get this reviewed?

fabriziopandini · 2024-10-18T08:37:00Z

/ok-to-test

cc @enxebre @vincepri @randomvariable

randomvariable

Bootcmd module has been in cloud-init for as long as I can remember (which is back to 2010), so don't see an issue there.

I would like the documentation to be more explicit though, particularly about how this will have no effect on Ignition. Have suggested some changes.

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go

neolit123 · 2024-10-18T15:05:09Z

internal/apis/bootstrap/kubeadm/v1alpha3/kubeadmconfig_types.go

+	// BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
+	// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
+	// +optional
+	BootCommands []BootCommand `json:"bootCommands,omitempty"`


shouldn't it be added only to v1beta1 which is the actively maintained API version?

If I remove this and run make verify, it removes bootCommands from the CRDs.

Might be an issue in the verify logic, what do you think?

diff --git a/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml b/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml index 79c3b19e0..4e1852f77 100644 --- a/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml +++ b/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml @@ -47,17 +47,6 @@ spec: KubeadmConfigSpec defines the desired state of KubeadmConfig. Either ClusterConfiguration and InitConfiguration should be defined or the JoinConfiguration should be defined. properties: - bootCommands: - description: |- - BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd - module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition. - items: - description: BootCommand defines input for each bootcmd command - in cloud-init. - items: - type: string - type: array - type: array clusterConfiguration: description: ClusterConfiguration along with InitConfiguration are the configurations necessary for the init command

etc

New fields should not be added to older API versions.
Conversions should be implemented instead.
See eg

cluster-api/internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

Lines 47 to 56 in 3db18fb

dst.Files = restored.Files

dst.Users = restored.Users

if restored.Users != nil {

for i := range restored.Users {

if restored.Users[i].PasswdFrom != nil {

dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom

}

}

}

I reverted my changes to the v1alpha3 and v1alpha4 versions and added conversions instead a9c426a

If I remove this and run make verify, it removes bootCommands from the CRDs.

Disregard this, I didn't realize that all versions of the crds were in the same manifest.

fabriziopandini · 2024-10-21T08:36:03Z

internal/apis/bootstrap/kubeadm/v1alpha3/kubeadmconfig_types.go

+	// BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
+	// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
+	// +optional
+	BootCommands []BootCommand `json:"bootCommands,omitempty"`


New fields should not be added to older API versions.
Conversions should be implemented instead.
See eg

cluster-api/internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

Lines 47 to 56 in 3db18fb

dst.Files = restored.Files

dst.Users = restored.Users

if restored.Users != nil {

for i := range restored.Users {

if restored.Users[i].PasswdFrom != nil {

dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom

}

}

}

fabriziopandini · 2024-10-21T08:37:14Z

bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go

@@ -34,6 +34,7 @@ func TestNewInitControlPlaneAdditionalFileEncodings(t *testing.T) {
 	cpinput := &ControlPlaneInput{
 		BaseUserData: BaseUserData{
 			Header:              "test",
+			BootCommands:        nil,


Can we have a test for BootCommands not nil for init, join CP, join workers?

I added non-nil values for BootCommands in TestNewInitControlPlaneCommands and TestNewJoinControlPlaneAdditionalFileEncodings, did you want me to add more tests or is this enough? I couldn't find one for "join workers" daf4f97

WRT to changes in TestNewInitControlPlaneCommands, TestNewJoinControlPlaneAdditionalFileEncodings I have dedicated comments.

We still need test coverage for join and join CP (if they are missing, kindly add them so we pay down some tech debt before adding more on top)

fabriziopandini · 2024-10-21T08:39:02Z

controlplane/kubeadm/internal/filters_test.go

@@ -548,7 +548,7 @@ func TestMatchInitOrJoinConfiguration(t *testing.T) {
    },
    JoinConfiguration: nil,
    Files:             nil,
-    ... // 10 identical fields
+    ... // 11 identical fields


I need to take some time to dig into impact of this change on existing clusters, because we need to make sure this change does not trigger rollouts
However, I'm not sure when I will get to it 😓

Do you think it's feasible to have this included in the next minor release?

Sorry if I missed this notification before.
Unfortunately code freeze is tomorrow, but let's first focus on getting the PR right first, then we can think about what options do we have

I took some time to look into this, and it should not trigger a rollout even if message in the logs could change after we deploy this change.

@chrischdi could you kindly double check this assumption

Should be fine. Especially clusterctl upgrade tests should catch this if it happens and we are wrong.

Edit: sorry for the long round-trip, forgot to send the review / comment... (was pending)

… PostKubeadmCommands

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go

fabriziopandini · 2024-11-25T15:23:40Z

bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go

@@ -94,6 +95,9 @@ func TestNewInitControlPlaneCommands(t *testing.T) {

 	cpinput := &ControlPlaneInput{
 		BaseUserData: BaseUserData{
+			BootCommands: []bootstrapv1.BootCommand{
+				{"echo", "hello"},


In order to make this test to be useful to validate the code actually works/to prevent regressions, you should also modify test assertions down below that boot commands actually show up in the generated output (and possibly, ensuring that command show up in the right section of the generated file)

I wasn't sure how to ensure that the commands show up in the right section of the generated file, but I added a check that the bootcommands show up in the generated file and also that the bootcmd: string is there (which is omitted when bootCommands are unset)

a4f3bdb

What do you think?

I would suggest to test if the out put has the expected bootcmd and runcmd block vs testing the output has single lines.

Something like

expectedRunCmd := `runcmd: - "\"echo $(date) ': hello world!'\"" - 'kubeadm init --config /run/kubeadm/kubeadm.yaml && echo success > /run/cluster-api/bootstrap-success.complete' - "echo $(date) ': hello world!'"` g.Expect(out).To(ContainSubstring(expectedRunCmd))

Should work; if you want to make it fancy, you can use regex, but up to you (similar for bootCmd)
Might be, to improve readability, let's also replace hello world! with something more specific like "hello preKubeadmCommands!" and "hello postKubeadmCommands!"

Thanks for the suggestion!

6fb468d

bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go

fabriziopandini · 2024-11-25T15:29:09Z

bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go

@@ -34,6 +34,7 @@ func TestNewInitControlPlaneAdditionalFileEncodings(t *testing.T) {
 	cpinput := &ControlPlaneInput{
 		BaseUserData: BaseUserData{
 			Header:              "test",
+			BootCommands:        nil,


WRT to changes in TestNewInitControlPlaneCommands, TestNewJoinControlPlaneAdditionalFileEncodings I have dedicated comments.

We still need test coverage for join and join CP (if they are missing, kindly add them so we pay down some tech debt before adding more on top)

fabriziopandini · 2024-11-25T15:34:41Z

controlplane/kubeadm/internal/filters_test.go

@@ -548,7 +548,7 @@ func TestMatchInitOrJoinConfiguration(t *testing.T) {
    },
    JoinConfiguration: nil,
    Files:             nil,
-    ... // 10 identical fields
+    ... // 11 identical fields


Sorry if I missed this notification before.
Unfortunately code freeze is tomorrow, but let's first focus on getting the PR right first, then we can think about what options do we have

I took some time to look into this, and it should not trigger a rollout even if message in the logs could change after we deploy this change.

@chrischdi could you kindly double check this assumption

internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

internal/apis/bootstrap/kubeadm/v1alpha4/conversion.go

chrischdi · 2024-11-25T16:59:34Z

/test help

k8s-ci-robot · 2024-11-25T16:59:37Z

@chrischdi: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-build-main
/test pull-cluster-api-e2e-blocking-main
/test pull-cluster-api-e2e-conformance-ci-latest-main
/test pull-cluster-api-e2e-conformance-main
/test pull-cluster-api-e2e-latestk8s-main
/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-mink8s-main
/test pull-cluster-api-e2e-upgrade-1-31-1-32-main
/test pull-cluster-api-test-main
/test pull-cluster-api-test-mink8s-main
/test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-apidiff-main

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-apidiff-main
pull-cluster-api-build-main
pull-cluster-api-e2e-blocking-main
pull-cluster-api-test-main
pull-cluster-api-verify-main

In response to this:

/test help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-11-25T17:00:02Z

/test pull-cluster-api-e2e-main

chrischdi · 2024-11-26T12:22:09Z

/test pull-cluster-api-e2e-main

chrischdi · 2024-11-27T12:34:56Z

/retest

Looks like a flake in creating the docker container and overlapping ports

chrischdi · 2024-12-02T13:36:28Z

/retest

Just to make sure its really not a flake

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label labels Oct 8, 2024

k8s-ci-robot requested a review from elmiko October 8, 2024 14:17

k8s-ci-robot requested a review from enxebre October 8, 2024 14:17

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 8, 2024

randomvariable suggested changes Oct 18, 2024

View reviewed changes

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go Outdated Show resolved Hide resolved

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go Outdated Show resolved Hide resolved

davidumea requested a review from randomvariable October 18, 2024 13:45

fabriziopandini added the area/provider/bootstrap-kubeadm Issues or PRs related to CAPBK label Oct 18, 2024

k8s-ci-robot removed the do-not-merge/needs-area PR is missing an area label label Oct 18, 2024

neolit123 reviewed Oct 18, 2024

View reviewed changes

fabriziopandini reviewed Oct 21, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 1, 2024

davidumea added 6 commits November 5, 2024 09:16

Add bootCommands cloud-init file generation

31406da

fixup: go lint

97311d6

fixup: lint: added missing newline

4367074

fixup: add changes from verify-gen

0c23af8

fixup: filter tests

504dfd5

fixup: clearer documentation for BootCommands, PreKubeadmCommands and…

19d52df

… PostKubeadmCommands

davidumea added 4 commits November 5, 2024 09:17

fixup: verify: reflect new documentation in the crds

51068a2

fixup: restore v1alpha3,v1alpha4 and add conversions

8ed0171

fixup: non-nil values for bootcommands in cloudinit tests

332e78d

fixup: lint

28edd6c

davidumea force-pushed the david/bootcommand-upstream branch from ebc85c2 to 28edd6c Compare November 5, 2024 08:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 5, 2024

fixup: verify crds

8918ff3

davidumea requested review from fabriziopandini and neolit123 November 18, 2024 08:38

fabriziopandini reviewed Nov 25, 2024

View reviewed changes

davidumea added 2 commits November 25, 2024 17:38

fixup: use yaml names in go doc comments

94835dc

fixup: removed extra linebreak

84e06c6

davidumea added 4 commits November 26, 2024 08:53

fixup: remove bootcommands from testing AdditionalFileEncodings

bd63a05

fixup: make sure bootcommands show up in generated output

a4f3bdb

fixup: add test for join cp commands

4673fb0

fixup: add test for join node commands

c059138

davidumea requested a review from fabriziopandini November 26, 2024 11:23

fixup: ensure commands show up in the right section

6fb468d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Add bootCommands to cloud-init file generation #11271

✨ Add bootCommands to cloud-init file generation #11271

davidumea commented Oct 8, 2024 •

edited

Loading

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

chrischdi commented Oct 8, 2024 •

edited

Loading

davidumea commented Oct 9, 2024 •

edited

Loading

davidumea commented Oct 17, 2024

fabriziopandini commented Oct 18, 2024

randomvariable left a comment

neolit123 Oct 18, 2024

davidumea Oct 21, 2024

fabriziopandini Oct 21, 2024

davidumea Oct 22, 2024

davidumea Oct 22, 2024 •

edited

Loading

fabriziopandini Oct 21, 2024

fabriziopandini Oct 21, 2024

davidumea Oct 22, 2024 •

edited

Loading

fabriziopandini Nov 25, 2024 •

edited

Loading

fabriziopandini Oct 21, 2024

davidumea Nov 5, 2024 •

edited

Loading

fabriziopandini Nov 25, 2024

chrischdi Nov 28, 2024

fabriziopandini Nov 25, 2024

davidumea Nov 26, 2024

fabriziopandini Nov 28, 2024

davidumea Dec 3, 2024

fabriziopandini Nov 25, 2024 •

edited

Loading

fabriziopandini Nov 25, 2024

chrischdi commented Nov 25, 2024

k8s-ci-robot commented Nov 25, 2024

chrischdi commented Nov 25, 2024

chrischdi commented Nov 26, 2024

chrischdi commented Nov 27, 2024

chrischdi commented Dec 2, 2024

	dst.Files = restored.Files

	dst.Users = restored.Users
	if restored.Users != nil {
	for i := range restored.Users {
	if restored.Users[i].PasswdFrom != nil {
	dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom
	}
	}
	}

✨ Add bootCommands to cloud-init file generation #11271

Are you sure you want to change the base?

✨ Add bootCommands to cloud-init file generation #11271

Conversation

davidumea commented Oct 8, 2024 • edited Loading

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

chrischdi commented Oct 8, 2024 • edited Loading

davidumea commented Oct 9, 2024 • edited Loading

davidumea commented Oct 17, 2024

fabriziopandini commented Oct 18, 2024

randomvariable left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidumea Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidumea Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

fabriziopandini Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidumea Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrischdi commented Nov 25, 2024

k8s-ci-robot commented Nov 25, 2024

chrischdi commented Nov 25, 2024

chrischdi commented Nov 26, 2024

chrischdi commented Nov 27, 2024

chrischdi commented Dec 2, 2024

davidumea commented Oct 8, 2024 •

edited

Loading

chrischdi commented Oct 8, 2024 •

edited

Loading

davidumea commented Oct 9, 2024 •

edited

Loading

davidumea Oct 22, 2024 •

edited

Loading

davidumea Oct 22, 2024 •

edited

Loading

fabriziopandini Nov 25, 2024 •

edited

Loading

davidumea Nov 5, 2024 •

edited

Loading

fabriziopandini Nov 25, 2024 •

edited

Loading