MINOR: upgrade ubuntu from trusty to bionic #18286

chia7712 · 2024-12-20T18:03:06Z

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

jolshan · 2024-12-20T22:33:20Z

run is here: https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-12-20--001.85d9ae90-e619-4689-a395-6d4075645b53--1734723292--chia7712--add_log_pid--acd9853691/report.html

Only the first had the pid issue this time.

jolshan · 2024-12-20T22:35:28Z

And unfortunately, looks like we didn't log this line. :(

ijuma · 2024-12-21T02:53:29Z

Is this only to debug the issue or is it generally useful?

chia7712 · 2024-12-21T03:26:03Z

Is this only to debug the issue or is it generally useful?

in fact, this PR is a no-op due to allow_fail=True - ducktape uses log instead of exception :_

https://github.com/confluentinc/ducktape/blob/master/ducktape/cluster/remoteaccount.py#L366C51-L366C70

chia7712 · 2024-12-22T11:49:07Z

@jolshan the root cause is that the output of ps ax is truncated. see following output

 5150 ?        Sl     0:04 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xlog:gc*:file=/opt/kafka-3.1.2/bin/../logs/kafkaServer-gc.log:time,tags:filecount=10,filesize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9192 -Dkafka.logs.dir=/opt/kafka-3.1.2/bin/../logs -Dlog4j.configuration=file:/mnt/kafka/log4j.properties -cp /opt/kafka-3.1.2/bin/../libs/activation-1.1.1.jar:/opt/kafka-3.1.2/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka-3.1.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka-3.1.2/bin/../libs/commons-cli-1.4.jar:/opt/kafka-3.1.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka-3.1.2/bin/../libs/connect-api-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-basic-auth-extension-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-json-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-mirror-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-mirror-client-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-runtime-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-transforms-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jackson-annotations-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-core-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-databind-2.12.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jackson-dataformat-csv-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-datatype-jdk8-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-jaxrs-base-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-jaxrs-json-provider-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-module-jaxb-annotations-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-module-scala_2.12-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka-3.1.2/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka-3.1.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka-3.1.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka-3.1.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka-3.1.2/bin/../libs/jersey-client-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-common-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-container-servlet-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-container-servlet-core-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-hk2-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-server-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jetty-client-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-continuation-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-http-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-io-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-security-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-server-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-servlet-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-servlets-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-util-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-util-ajax-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jline-3.12.1.jar:/opt/kafka-3.1.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka-3.1.2/bin/../libs/jose4j-0.7.8.jar:/opt/kafka-3.1.2/bin/../libs/kafka_2.12-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-clients-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-log4j-appender-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-metadata-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-raft-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-server-common-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-shell-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-storage-3.1.2.jar

there is no keyword "kafka.Kafka" as it is truncated.

the following output is from container

    534 ?        Sl     0:09 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xlog:gc*:file=/opt/kafka-3.1.2/bin/../logs/kafkaServer-gc.log:time,tags:filecount=10,filesize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9192 -Dkafka.logs.dir=/opt/kafka-3.1.2/bin/../logs -Dlog4j.configuration=file:/mnt/kafka/log4j.properties -cp /opt/kafka-3.1.2/bin/../libs/activation-1.1.1.jar:/opt/kafka-3.1.2/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka-3.1.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka-3.1.2/bin/../libs/commons-cli-1.4.jar:/opt/kafka-3.1.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka-3.1.2/bin/../libs/connect-api-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-basic-auth-extension-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-json-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-mirror-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-mirror-client-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-runtime-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-transforms-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jackson-annotations-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-core-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-databind-2.12.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jackson-dataformat-csv-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-datatype-jdk8-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-jaxrs-base-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-jaxrs-json-provider-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-module-jaxb-annotations-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-module-scala_2.12-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka-3.1.2/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka-3.1.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka-3.1.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka-3.1.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka-3.1.2/bin/../libs/jersey-client-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-common-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-container-servlet-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-container-servlet-core-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-hk2-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-server-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jetty-client-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-continuation-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-http-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-io-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-security-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-server-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-servlet-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-servlets-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-util-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-util-ajax-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jline-3.12.1.jar:/opt/kafka-3.1.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka-3.1.2/bin/../libs/jose4j-0.7.8.jar:/opt/kafka-3.1.2/bin/../libs/kafka-clients-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-log4j-appender-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-metadata-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-raft-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-server-common-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-shell-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-storage-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-storage-api-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-streams-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-streams-examples-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-streams-scala_2.12-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-streams-test-utils-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-tools-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka_2.12-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/lz4-java-1.8.0.jar:/opt/kafka-3.1.2/bin/../libs/maven-artifact-3.8.1.jar:/opt/kafka-3.1.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka-3.1.2/bin/../libs/metrics-core-4.1.12.1.jar:/opt/kafka-3.1.2/bin/../libs/netty-buffer-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-codec-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-common-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-handler-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-resolver-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-tcnative-classes-2.0.46.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-transport-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-transport-classes-epoll-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-transport-native-epoll-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-transport-native-unix-common-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka-3.1.2/bin/../libs/paranamer-2.8.jar:/opt/kafka-3.1.2/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka-3.1.2/bin/../libs/reflections-0.9.12.jar:/opt/kafka-3.1.2/bin/../libs/reload4j-1.2.19.jar:/opt/kafka-3.1.2/bin/../libs/rocksdbjni-6.22.1.1.jar:/opt/kafka-3.1.2/bin/../libs/scala-collection-compat_2.12-2.4.4.jar:/opt/kafka-3.1.2/bin/../libs/scala-java8-compat_2.12-1.0.0.jar:/opt/kafka-3.1.2/bin/../libs/scala-library-2.12.14.jar:/opt/kafka-3.1.2/bin/../libs/scala-logging_2.12-3.9.3.jar:/opt/kafka-3.1.2/bin/../libs/scala-reflect-2.12.14.jar:/opt/kafka-3.1.2/bin/../libs/slf4j-api-1.7.36.jar:/opt/kafka-3.1.2/bin/../libs/slf4j-reload4j-1.7.36.jar:/opt/kafka-3.1.2/bin/../libs/snappy-java-1.1.8.4.jar:/opt/kafka-3.1.2/bin/../libs/trogdor-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/zookeeper-3.6.3.jar:/opt/kafka-3.1.2/bin/../libs/zookeeper-jute-3.6.3.jar:/opt/kafka-3.1.2/bin/../libs/zstd-jni-1.5.0-4.jar -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/kafka/kafka_heap_dump.bin kafka.Kafka /mnt/kafka/kafka.properties

Log4j2 introduces new jars to the classpath, which leads to truncation issues.

I have some preliminary approaches to address this problem and will update this PR later.

chia7712 · 2024-12-23T03:06:19Z

the above comment is not correct :(

The true root cause is that the grep -i kafka.Kafka command can inadvertently match paths like kafka/kafka. Specifically, it matched Dlog4j.configuration=file:/mnt/kafka/kafka-log4j.properties even though the process name was truncated.

However, #17373 changes the log path from /mnt/kafka/kafka-log4j.properties to /mnt/kafka/log4j.properties. As a result, the grep command no longer matches any relevant lines, leading to the failure.

mumrah · 2024-12-23T14:53:46Z

@chia7712 #17881 adds a "triage" label to PRs from non-committers. Turns out this also affect committers if their membership visibility in the ASF GitHub org is not public. I added instructions for setting your membership visibility to public https://github.com/apache/kafka/blob/trunk/.github/workflows/README.md#pr-triage

chia7712 · 2024-12-23T15:14:28Z

Turns out this also affect committers if their membership visibility in the ASF GitHub org is not public.

done!!!

chia7712 · 2024-12-23T16:22:51Z

@jolshan Could you please test this patch? I am using Ubuntu 18.04 instead of 14.04, and it appears that the ps ux command output is no longer truncated.

vagrant@worker1:~$ ps ax | grep -i kafka.Kafka
 8473 ?        Sl     0:02 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xlog:gc*:file=/opt/kafka-3.1.2/bin/../logs/kafkaServer-gc.log:time,tags:filecount=10,filesize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9192 -Dkafka.logs.dir=/opt/kafka-3.1.2/bin/../logs -Dlog4j.configuration=file:/mnt/kafka/log4j.properties -cp /opt/kafka-3.1.2/bin/../libs/activation-1.1.1.jar:/opt/kafka-3.1.2/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka-3.1.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka-3.1.2/bin/../libs/commons-cli-1.4.jar:/opt/kafka-3.1.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka-3.1.2/bin/../libs/connect-api-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-basic-auth-extension-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-json-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-mirror-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-mirror-client-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-runtime-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/connect-transforms-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jackson-annotations-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-core-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-databind-2.12.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jackson-dataformat-csv-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-datatype-jdk8-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-jaxrs-base-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-jaxrs-json-provider-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-module-jaxb-annotations-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jackson-module-scala_2.12-2.12.6.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka-3.1.2/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka-3.1.2/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka-3.1.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka-3.1.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka-3.1.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka-3.1.2/bin/../libs/jersey-client-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-common-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-container-servlet-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-container-servlet-core-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-hk2-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jersey-server-2.34.jar:/opt/kafka-3.1.2/bin/../libs/jetty-client-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-continuation-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-http-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-io-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-security-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-server-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-servlet-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-servlets-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-util-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jetty-util-ajax-9.4.48.v20220622.jar:/opt/kafka-3.1.2/bin/../libs/jline-3.12.1.jar:/opt/kafka-3.1.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka-3.1.2/bin/../libs/jose4j-0.7.8.jar:/opt/kafka-3.1.2/bin/../libs/kafka-clients-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-log4j-appender-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-metadata-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-raft-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-server-common-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-shell-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-storage-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-storage-api-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-streams-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-streams-examples-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-streams-scala_2.12-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-streams-test-utils-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka-tools-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/kafka_2.12-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/lz4-java-1.8.0.jar:/opt/kafka-3.1.2/bin/../libs/maven-artifact-3.8.1.jar:/opt/kafka-3.1.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka-3.1.2/bin/../libs/metrics-core-4.1.12.1.jar:/opt/kafka-3.1.2/bin/../libs/netty-buffer-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-codec-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-common-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-handler-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-resolver-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-tcnative-classes-2.0.46.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-transport-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-transport-classes-epoll-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-transport-native-epoll-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/netty-transport-native-unix-common-4.1.73.Final.jar:/opt/kafka-3.1.2/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka-3.1.2/bin/../libs/paranamer-2.8.jar:/opt/kafka-3.1.2/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka-3.1.2/bin/../libs/reflections-0.9.12.jar:/opt/kafka-3.1.2/bin/../libs/reload4j-1.2.19.jar:/opt/kafka-3.1.2/bin/../libs/rocksdbjni-6.22.1.1.jar:/opt/kafka-3.1.2/bin/../libs/scala-collection-compat_2.12-2.4.4.jar:/opt/kafka-3.1.2/bin/../libs/scala-java8-compat_2.12-1.0.0.jar:/opt/kafka-3.1.2/bin/../libs/scala-library-2.12.14.jar:/opt/kafka-3.1.2/bin/../libs/scala-logging_2.12-3.9.3.jar:/opt/kafka-3.1.2/bin/../libs/scala-reflect-2.12.14.jar:/opt/kafka-3.1.2/bin/../libs/slf4j-api-1.7.36.jar:/opt/kafka-3.1.2/bin/../libs/slf4j-reload4j-1.7.36.jar:/opt/kafka-3.1.2/bin/../libs/snappy-java-1.1.8.4.jar:/opt/kafka-3.1.2/bin/../libs/trogdor-3.1.2.jar:/opt/kafka-3.1.2/bin/../libs/zookeeper-3.6.3.jar:/opt/kafka-3.1.2/bin/../libs/zookeeper-jute-3.6.3.jar:/opt/kafka-3.1.2/bin/../libs/zstd-jni-1.5.0-4.jar -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/kafka/kafka_heap_dump.bin kafka.Kafka /mnt/kafka/kafka.properties

brandboat · 2024-12-23T17:22:43Z

@jolshan the root cause is that the output of ps ax is truncated. see following output

I believe this issue is related to the 4096-byte limit of /proc/pid/cmdline, which was fixed in Linux version 4.2. Upgrading to Ubuntu 15.10 or later (which uses Linux v4.2) should resolve the issue.
https://stackoverflow.com/questions/57117038/how-to-get-the-maximum-value-for-ps-output-in-linux-ubuntu-redhat-centos

jolshan · 2024-12-23T17:39:57Z

Taking a look. Thanks!

jolshan · 2024-12-23T17:43:33Z

Thanks for the patch -- I had at one point tried a stab at upgrading ubuntu since 14 is already way past EOL. Makes sense that we should upgrade.

I think our infra unfortunately overrides the version set here, but I think I originally got stuck with the other changes in the base.sh files. Give me a moment to re-setup the testing on my end and I will let you know how it goes.

I do think this is generally the correct direction to go.

chia7712 · 2024-12-23T17:56:25Z

I think our infra unfortunately overrides the version set here, but I think I originally got stuck with the other changes in the base.sh files. Give me a moment to re-setup the testing on my end and I will let you know how it goes.

Another solution is to use jcmd instead of ps ax (KAFKA-18343) to avoid encountering the truncation issue.

jolshan · 2024-12-23T18:16:15Z

@chia7712 that may work for the short term as well. I tried a first pass with the new instance type and it didn't work quite right yet. I will keep trying a few things in the meantime.

jolshan · 2024-12-23T21:37:18Z

I tried a few difference instances but I'm seeing the kafka failed to start up in 60 seconds. However, this time it looks like a different issue related to the servers not finding the controller correctly. 🤔 I wonder if there is another issue here. I also don't see the pid error at all.

I think this was also the issue I had before when I tried to see if I could update the ubuntu image. It seems like there is no requests between the workers at all, as there are UnknownHostExceptions. This could just be due to our setup though.

jolshan · 2024-12-23T22:08:49Z

This was my old branch which had the ntpdate in a slightly different place (this branch also didn't work for me at the time though)
trunk...jolshan:kafka:ami-update

chia7712 · 2024-12-24T01:09:11Z

@jolshan maybe there are broken commands after upgrading the OS version. We can try the solution of https://issues.apache.org/jira/browse/KAFKA-18343 which is more compatible with all existent setup.

chia7712 · 2024-12-24T01:09:58Z

@brandboat do you have free cycle to file a patch? I'd like to keep this PR as it is.

brandboat · 2024-12-24T01:16:41Z

@brandboat do you have free cycle to file a patch? I'd like to keep this PR as it is.

I'll file a PR today, thanks for bring this up!

brandboat

LGTM! I ran ducktape tests/kafkatest/tests/core/transactions_upgrade_test.py and everything passed. This means we've successfully fixed the 'process id not found' issue, at least in the VirtualBox environment. Thanks Chia-Ping make the test even better.

SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id:       2024-12-24--013
run time:         63 minutes 44.715 seconds
tests run:        9
passed:           9
flaky:            0
failed:           0
ignored:          0

Test Log

[INFO:2024-12-24 11:29:40,031]: starting test run with session id 2024-12-24--013...
[INFO:2024-12-24 11:29:40,031]: running 9 tests...
[INFO:2024-12-24 11:29:40,031]: Triggering test 1 of 9...
[INFO:2024-12-24 11:29:40,037]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.1.2', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 11:29:40,040]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.1.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 11:29:40,040]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.1.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 11:29:40,041]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.1.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 11:34:34,466]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.1.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 11:37:24,367]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.1.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 11:37:24,367]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.1.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
[INFO:2024-12-24 11:37:24,469]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-12-24 11:37:24,469]: Triggering test 2 of 9...
[INFO:2024-12-24 11:37:24,475]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.2.3', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 11:37:24,478]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.2.3.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 11:37:24,478]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.2.3.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 11:37:24,478]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.2.3.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 11:40:59,730]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.2.3.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 11:43:50,670]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.2.3.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 11:43:50,670]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.2.3.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
[INFO:2024-12-24 11:43:50,773]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-12-24 11:43:50,773]: Triggering test 3 of 9...
[INFO:2024-12-24 11:43:50,778]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.3.2', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 11:43:50,779]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.3.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 11:43:50,780]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.3.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 11:43:50,780]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.3.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 11:48:50,583]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.3.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 11:51:42,484]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.3.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 11:51:42,485]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.3.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
[INFO:2024-12-24 11:51:42,587]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-12-24 11:51:42,587]: Triggering test 4 of 9...
[INFO:2024-12-24 11:51:42,591]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.4.1', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 11:51:42,593]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.4.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 11:51:42,593]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.4.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 11:51:42,593]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.4.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 11:56:05,994]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.4.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 11:58:57,325]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.4.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 11:58:57,325]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.4.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
[INFO:2024-12-24 11:58:57,427]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-12-24 11:58:57,427]: Triggering test 5 of 9...
[INFO:2024-12-24 11:58:57,432]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.5.2', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 11:58:57,434]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.5.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 11:58:57,434]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.5.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 11:58:57,435]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.5.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 12:02:42,192]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.5.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 12:05:38,416]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.5.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 12:05:38,417]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.5.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
[INFO:2024-12-24 12:05:38,519]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-12-24 12:05:38,519]: Triggering test 6 of 9...
[INFO:2024-12-24 12:05:38,523]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.6.2', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 12:05:38,524]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.6.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 12:05:38,524]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.6.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 12:05:38,524]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.6.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 12:09:53,911]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.6.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 12:12:45,588]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.6.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 12:12:45,588]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.6.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
[INFO:2024-12-24 12:12:45,690]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-12-24 12:12:45,691]: Triggering test 7 of 9...
[INFO:2024-12-24 12:12:45,695]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.7.2', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 12:12:45,698]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.7.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 12:12:45,698]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.7.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 12:12:45,698]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.7.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 12:17:01,914]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.7.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 12:19:53,210]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.7.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 12:19:53,211]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.7.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
[INFO:2024-12-24 12:19:53,313]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-12-24 12:19:53,314]: Triggering test 8 of 9...
[INFO:2024-12-24 12:19:53,319]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.8.1', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 12:19:53,321]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.8.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 12:19:53,321]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.8.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 12:19:53,322]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.8.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 12:24:00,810]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.8.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 12:26:52,393]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.8.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 12:26:52,394]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.8.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
[INFO:2024-12-24 12:26:52,497]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-12-24 12:26:52,497]: Triggering test 9 of 9...
[INFO:2024-12-24 12:26:52,502]: RunnerClient: Loading test {'directory': '/home/brandboat/Code/kafka/kafka/tests/kafkatest/tests/core', 'file_name': 'transactions_upgrade_test.py', 'cls_name': 'TransactionsUpgradeTest', 'method_name': 'test_transactions_upgrade', 'injected_args': {'from_kafka_version': '3.9.0', 'metadata_quorum': 'ISOLATED_KRAFT', 'use_new_coordinator': False, 'group_protocol': None}}
[INFO:2024-12-24 12:26:52,505]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.9.0.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: on run 1/1
[INFO:2024-12-24 12:26:52,506]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.9.0.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Setting up...
[INFO:2024-12-24 12:26:52,506]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.9.0.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Running...
[INFO:2024-12-24 12:30:33,409]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.9.0.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...
[INFO:2024-12-24 12:33:24,643]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.9.0.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: PASS
[INFO:2024-12-24 12:33:24,643]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.9.0.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Data: None
=======================================================================================================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id:       2024-12-24--013
run time:         63 minutes 44.715 seconds
tests run:        9
passed:           9
flaky:            0
failed:           0
ignored:          0
=======================================================================================================================================================================================
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.1.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   7 minutes 44.327 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.2.3.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   6 minutes 26.192 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.3.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   7 minutes 51.705 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.4.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   7 minutes 14.732 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.5.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   6 minutes 40.982 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.6.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   7 minutes 7.064 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.7.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   7 minutes 7.513 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.8.1.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   6 minutes 59.072 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.9.0.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None
status:     PASS
run time:   6 minutes 32.137 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

brandboat · 2024-12-24T16:13:33Z

gentle ping @jolshan, I just filed #18312 which use jcmd instead of ps ax to mitigate the output truncate issue, perhaps you can give it a try, thanks!

jolshan · 2024-12-24T17:24:05Z

Thanks @brandboat. I will take a look at that one.

jolshan · 2024-12-24T17:25:46Z

@jolshan maybe there are broken commands after upgrading the OS version.

I think there could also be something on our infra side that I'm not setting up correctly. I requested some folks on our side to take a look and due to the holidays, it may be a while. I will look into the second solution (jcmd) now. :)

chia7712 · 2024-12-25T01:06:22Z

@brandboat should we fix all other cases using "ps ux" to avoid similar issue in the future?

brandboat · 2024-12-25T02:03:05Z

@brandboat should we fix all other cases using "ps ux" to avoid similar issue in the future?

There is another ps us usage in

kafka/tests/kafkatest/utils/util.py

Line 60 in 4ed7f2c

    
           lines = [l for l in node.account.ssh_capture("ps ax | grep %s | grep -v grep" % proc_grep_string)]

. What's interesting about this one is that it parses the Kafka jar in the classpath to retrieve the version. However, if we switch to using java_pids, this functionality would no longer be possible. I'm afraid that we have to leave it as it is.

chia7712 · 2024-12-25T02:44:35Z

I'm afraid that we have to leave it as it is.

yes, +1 to keep it

github-actions bot added triage PRs from the community tests Test fixes (including flaky tests) small Small PRs labels Dec 20, 2024

chia7712 mentioned this pull request Dec 20, 2024

KAFKA-9366: Upgrade log4j to log4j2 #17373

Merged

3 tasks

MINOR: upgrade ubuntu from trusty to bionic

c6ae7cf

chia7712 force-pushed the add_log_pid branch from acd9853 to c6ae7cf Compare December 23, 2024 16:19

chia7712 changed the title ~~MINOR: print the error log if we can't list pids~~ MINOR: upgrade ubuntu from trusty to bionic Dec 23, 2024

fix build

e8e5e0b

github-actions bot added the tools label Dec 23, 2024

brandboat approved these changes Dec 24, 2024

View reviewed changes

github-actions bot removed the triage PRs from the community label Dec 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MINOR: upgrade ubuntu from trusty to bionic #18286

MINOR: upgrade ubuntu from trusty to bionic #18286

chia7712 commented Dec 20, 2024

jolshan commented Dec 20, 2024

jolshan commented Dec 20, 2024

ijuma commented Dec 21, 2024

chia7712 commented Dec 21, 2024

chia7712 commented Dec 22, 2024

chia7712 commented Dec 23, 2024

mumrah commented Dec 23, 2024

chia7712 commented Dec 23, 2024

chia7712 commented Dec 23, 2024

brandboat commented Dec 23, 2024

jolshan commented Dec 23, 2024

jolshan commented Dec 23, 2024

chia7712 commented Dec 23, 2024

jolshan commented Dec 23, 2024

jolshan commented Dec 23, 2024 •

edited

Loading

jolshan commented Dec 23, 2024 •

edited

Loading

chia7712 commented Dec 24, 2024

chia7712 commented Dec 24, 2024

brandboat commented Dec 24, 2024

brandboat left a comment

brandboat commented Dec 24, 2024

jolshan commented Dec 24, 2024

jolshan commented Dec 24, 2024

chia7712 commented Dec 25, 2024

brandboat commented Dec 25, 2024

chia7712 commented Dec 25, 2024

MINOR: upgrade ubuntu from trusty to bionic #18286

Are you sure you want to change the base?

MINOR: upgrade ubuntu from trusty to bionic #18286

Conversation

chia7712 commented Dec 20, 2024

Committer Checklist (excluded from commit message)

jolshan commented Dec 20, 2024

jolshan commented Dec 20, 2024

ijuma commented Dec 21, 2024

chia7712 commented Dec 21, 2024

chia7712 commented Dec 22, 2024

chia7712 commented Dec 23, 2024

mumrah commented Dec 23, 2024

chia7712 commented Dec 23, 2024

chia7712 commented Dec 23, 2024

brandboat commented Dec 23, 2024

jolshan commented Dec 23, 2024

jolshan commented Dec 23, 2024

chia7712 commented Dec 23, 2024

jolshan commented Dec 23, 2024

jolshan commented Dec 23, 2024 • edited Loading

jolshan commented Dec 23, 2024 • edited Loading

chia7712 commented Dec 24, 2024

chia7712 commented Dec 24, 2024

brandboat commented Dec 24, 2024

brandboat left a comment

Choose a reason for hiding this comment

brandboat commented Dec 24, 2024

jolshan commented Dec 24, 2024

jolshan commented Dec 24, 2024

chia7712 commented Dec 25, 2024

brandboat commented Dec 25, 2024

chia7712 commented Dec 25, 2024

jolshan commented Dec 23, 2024 •

edited

Loading

jolshan commented Dec 23, 2024 •

edited

Loading