[question] conan upload --package-only?

[hoyhoy]

What is your question?

We have a number of build machines with different platforms and architectures. At the end of our builds, we're doing a conan upload so that artifactory has the binaries that we need. What we really need is a way to upload a package only if it's different, and only the binaries. What's happening now is the conan upload on our build machines is overwriting the new recipes with the current one if a build is in-progress. Is there a way to only upload the packages without overwriting the recipes? Or, only uploading to artifactory if the package is different? Ideally, we'd like to have both options.

Or, is there a better way to cache platform-specific binaries without using conan upload?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @hoyhoy

Thanks for your question.

We have a number of build machines with different platforms and architectures. At the end of our builds, we're doing a conan upload so that artifactory has the binaries that we need. What we really need is a way to upload a package only if it's different, and only the binaries. What's happening now is the conan upload on our build machines is overwriting the new recipes with the current one if a build is in-progress.

Conan does not re-upload the recipe when the recipe is already in the server, so there is no overwrite. Same with the binaries, when the binaries are there, they are not re-uploaded even if you do conan upload, the messages should say so "skipping binary already in the server"

So when you are doing a conan upload of a new binary to the server where other binaries already exist, the recipe will not be overwritten.

Another issue is the concurrency. If 2 different jobs try to upload the exact same recipe at the same time, there can be a race condition. This is not necessarily an overwrite, but a race condition. The recommendation to avoid this is uploading the recipe earlier in the CI process, even before uploading the binaries and then launch the CI parallel jobs that build the different binaries.

[hoyhoy]

We're seeing manually updated recipes being overwritten by out-of-date recipes on our local artifactory server. Not sure of the exact cause, but we do have builds in progress while the recipes are being uploaded. I just assumed it was the build machines overwriting them. What I think is happening is that our build machine has the old recipe, artifactory has a new recipe, and then the previously downloaded recipe on our build machine is being uploaded.

I assumed this was happening....
(1) conan build started on a build machine
(2) conan recipe is uploaded via a developer on another machine
(3) conan upload is run on the build machine after the build has succeeded overwriting the updated recipe

[memsharded]

(3) conan upload is run on the build machine after the build has succeeded overwriting the updated recipe

This would need clear and detailed reproduction steps, including the exact commands to reproduce, exact versions and OS of the client and server, the commands, etc. Conan does not re-upload files that already exist in the server, there are 2 different layers to avoid that, first checking the recipe revision/package-id, then later with the Artifactory X-headers for deduplication of files.