ctr cann't export image pulled by crictl

[johnsonisjiang]

What happened:

when in containerd environment, if we use old version crictl and ctr, no matter the images pulled by ctr or crictl, the ctr can export the image successfully.
But when crictl >= 1.24.0, the issue happens, the image pulled by crictl, when use ctr export, it will failed; For comparison, images pulled by ctr have no problem.

ctr export image which pulled by crictl , will show error message

ctr -n k8s.io images export my_image.tar sha256:af282666faecdad41485bea589ff4c6f25af2abbf76c7b8dff24e7df2570798e
ctr: failed to get reader: content digest sha256:10d56570098259f40562e64b4d5ad9374c94af3e00688d62827e8c79c47d9617: not found

the crictl version: 1.25.0, ctr version 1.7.20

What you expected to happen:

I have another env (crictl:1.19.0; ctr:1.3.7), in this machine, images pulled by crictl can be exported by ctr successfully.
I think there maybe some changes between crictl versions, which changes the export perform.
Would you please tell me how to export images pulled by new version crictl?

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

the crictl version: 1.25.0, ctr version 1.7.20

[saschagrunert]

THank you for opening this report @johnsonisjiang. Both crictl v1.19.0 and v1.25.0 are end of life. crictl also just invokes the RPC to pull the image, so the main issue seems to be on the containerd side here.

[johnsonisjiang]

THank you for opening this report @johnsonisjiang. Both crictl v1.19.0 and v1.25.0 are end of life. crictl also just invokes the RPC to pull the image, so the main issue seems to be on the containerd side here.

Thanks, I got the reason, if the "/etc/containerd/config.toml"->"discard_unpacked_layers" was true, the image pulled by crictl may exported failed, but it have no effect on images pulled by ctr(ctr use contained api not grpc with containerd, default was false)

[saschagrunert]

@johnsonisjiang unfortunately I don't think there is much we can do from a cri-tools perspective.