Release action - Separate runner for amd64 and arm64 (main) (#5130)

* Release action - Separate runner for amd64 and arm64 (main) Previously arm64 images were created on a standard amd64 runner using qemu. qemu emulation is very slow and there have been recent issues. Native image creation is faster and more reliable. This change creates separate runners for amd64 and arm64. A new job is then needed to combine the image digests into a multi-architecture manifest that gets pushed to Docker Hub and ghcr. Signed-off-by: David Enyeart <[email protected]> * Update ubuntu-20.04 to ubuntu-22.04 in docs Update docs to reflect the update from ubuntu-20.04 to ubuntu-22.04. Signed-off-by: David Enyeart <[email protected]> --------- Signed-off-by: David Enyeart <[email protected]>
hyperledger · Feb 5, 2025 · 252649e · 252649e
1 parent 1325059
commit 252649e
Show file tree

Hide file tree

Showing 3 changed files with 114 additions and 37 deletions.
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -25,10 +25,10 @@ jobs:
     strategy:
       matrix:
         include:
-          - image: fabric-ubuntu-22.04
+          - image: ubuntu-22.04
             target: linux
             arch: amd64
-          - image: fabric-ubuntu-22.04
+          - image: ubuntu-22.04
             target: linux
             arch: arm64
           - image: macos-11
@@ -40,7 +40,7 @@ jobs:
           - image: fabric-windows-latest
             target: windows
             arch: amd64
-    runs-on: ${{ github.repository == 'hyperledger/fabric' && 'fabric-ubuntu-22.04' || 'ubuntu-22.04' }}
+    runs-on: ubuntu-22.04
     steps:
       - name: Checkout Fabric Code
         uses: actions/checkout@v4
@@ -62,9 +62,10 @@ jobs:
           # <path> of the artifact may include multiple files.
           path: release/${{ matrix.target }}-${{ matrix.arch }}/*.tar.gz
 
-  build-and-push-docker-images:
-    name: Build and Push
-    runs-on: ${{ github.repository == 'hyperledger/fabric' && 'fabric-ubuntu-22.04' || 'ubuntu-22.04' }}
+  # build native images using a different runner for each architecture (faster and more reliable than using qemu to build multi-architecture images on ubuntu-22.04)
+  build-and-push-native-docker-images:
+    name: Build and Push native images
+    runs-on: ${{ matrix.runner }}
 
     permissions:
       contents: read
@@ -73,9 +74,15 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        registry:
-          - docker.io
-          - ghcr.io
+
+        runner:
+          - ubuntu-22.04      # creates linux-amd64 images
+          - ubuntu-22.04-arm  # creates linux-arm64 images 
+
+        # Dynamic matrix
+        # If owner is 'hyperledger' run job for Docker Hub and ghcr, otherwise for personal forks just run job for ghcr
+        registry: ${{ fromJSON(github.repository_owner == 'hyperledger' && '["docker.io", "ghcr.io"]' || '["ghcr.io"]') }}
+
         component:
           - name: baseos
             context: images/baseos
@@ -87,71 +94,141 @@ jobs:
             context: .
 
     steps:
-      - name: Skip Docker Hub publish for forks
-        if: ${{ github.repository_owner != 'hyperledger' && matrix.registry == 'docker.io' }}
-        run: exit 1
-
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
-        with:
-          buildkitd-flags: --debug
-          buildkitd-config-inline: |
-            [worker.oci]
-              max-parallelism = 1
 
       - name: Checkout
         uses: actions/checkout@v4
-
+        
       - name: Login to the ${{ matrix.registry }} Container Registry
         uses: docker/login-action@v3
         with:
           registry: ${{ matrix.registry }}
           username: ${{ matrix.registry == 'docker.io' && secrets.DOCKERHUB_USERNAME || github.actor }}
           password: ${{ matrix.registry == 'docker.io' && secrets.DOCKERHUB_TOKEN    || secrets.GITHUB_TOKEN }}
 
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3       
+
       - name: Docker meta
         id: meta
         uses: docker/metadata-action@v5
         with:
           images: ${{ matrix.registry }}/${{ github.repository_owner }}/fabric-${{ matrix.component.name }}
-          tags: |
-            type=semver,pattern={{version}}
-            type=semver,pattern={{major}}.{{minor}}
-            type=semver,pattern={{major}}.{{minor}}.{{patch}}
 
       - name: Build and push ${{ matrix.component.name }} Image
-        id: push
-        uses: docker/build-push-action@v5
+        id: build-and-push
+        uses: docker/build-push-action@v6
         with:
           context: ${{ matrix.component.context }}
           file: images/${{ matrix.component.name }}/Dockerfile
-          platforms: linux/amd64,linux/arm64
-          tags: ${{ steps.meta.outputs.tags }}
-          push: ${{ github.event_name != 'pull_request' }}
           labels: ${{ steps.meta.outputs.labels }}
           build-args: |
             FABRIC_VER=${{ env.FABRIC_VER }}
             UBUNTU_VER=${{ env.UBUNTU_VER }}
             GO_VER=${{ env.GO_VER }}
             GO_TAGS=
+          outputs: type=image,"name=${{ matrix.registry }}/${{ github.repository_owner }}/fabric-${{ matrix.component.name }}",push-by-digest=true,name-canonical=true,push=true
+
+      - name: Export digest
+        run: |
+          mkdir -p ${{ runner.temp }}/digests/${{ matrix.registry }}/${{ matrix.component.name }}
+          digest="${{ steps.build-and-push.outputs.digest }}"
+          touch "${{ runner.temp }}/digests/${{ matrix.registry }}/${{ matrix.component.name }}/${digest#sha256:}"
+
+      - name: Upload digest
+        uses: actions/upload-artifact@v4
+        with:
+          name: digests-${{ matrix.registry }}-${{ matrix.component.name }}-${{ matrix.runner }}
+          path: ${{ runner.temp }}/digests/${{ matrix.registry }}/${{ matrix.component.name }}/*
+          if-no-files-found: error
+          retention-days: 1
+
+  # This job merges the architecture-specific digests for the images created above
+  # and creates a multi-architecture image manifest with user-friendly tags
+  merge-and-push-multi-arch-image:
+    name: Merge and Push multi-arch image
+    runs-on: ubuntu-22.04
+    needs:
+      - build-and-push-native-docker-images
+
+    permissions:
+      contents: read
+      packages: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+
+        # Dynamic matrix
+        # If owner is 'hyperledger' run job for Docker Hub and ghcr, otherwise for personal forks just run job for ghcr
+        registry: ${{ fromJSON(github.repository_owner == 'hyperledger' && '["docker.io", "ghcr.io"]' || '["ghcr.io"]') }}
+
+        component:
+          - name: baseos
+            context: images/baseos
+          - name: ccenv
+            context: images/ccenv
+          - name: peer
+            context: .
+          - name: orderer
+            context: .
+
+    steps:
+
+      - name: Download digests
+        uses: actions/download-artifact@v4
+        with:
+          path: ${{ runner.temp }}/digests/${{ matrix.registry }}/${{ matrix.component.name }}
+          pattern: digests-${{ matrix.registry }}-${{ matrix.component.name }}-*
+          merge-multiple: true
+
+      - name: Login to the ${{ matrix.registry }} Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ matrix.registry }}
+          username: ${{ matrix.registry == 'docker.io' && secrets.DOCKERHUB_USERNAME || github.actor }}
+          password: ${{ matrix.registry == 'docker.io' && secrets.DOCKERHUB_TOKEN    || secrets.GITHUB_TOKEN }}
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ matrix.registry }}/${{ github.repository_owner }}/fabric-${{ matrix.component.name }}
+          tags: |
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=semver,pattern={{major}}.{{minor}}.{{patch}}
+      
+      - name: Create manifest list and push # combines the downloaded amd64 and arm64 digests and pushes multi-architecture manifest with the tags specified above
+        working-directory: ${{ runner.temp }}/digests/${{ matrix.registry }}/${{ matrix.component.name }}
+        run: |
+          docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
+            $(printf '${{ matrix.registry }}/${{ github.repository_owner }}/fabric-${{ matrix.component.name }}@sha256:%s ' *)
+        
+      - name: Inspect image
+        run: |
+          docker buildx imagetools inspect ${{ matrix.registry }}/${{ github.repository_owner }}/fabric-${{ matrix.component.name }}:${{ steps.meta.outputs.version }}
 
   create-release:
     name: Create GitHub Release
     needs:
       - build-binaries
-      - build-and-push-docker-images
-    runs-on: ${{ github.repository == 'hyperledger/fabric' && 'fabric-ubuntu-22.04' || 'ubuntu-22.04' }}
+      - merge-and-push-multi-arch-image
+    runs-on: ubuntu-22.04
     permissions:
       contents: write
     steps:
       - name: Checkout Fabric Code
         uses: actions/checkout@v4
+
       - name: Download Artifacts
         id: download
         uses: actions/download-artifact@v4
+        with:
+          pattern: "release-*"
+
       - name: Release Fabric Version
         uses: ncipollo/release-action@v1
         with:

diff --git a/docs/source/prereqs.md b/docs/source/prereqs.md
@@ -128,7 +128,7 @@ path is to use WSL2 (Windows Subsystem for Linux version 2) to provide a native
 
 WSL2 may not be installed by default; you can check and install WSL2 by going into "Programs and Features", clicking on "Turn Windows features on or off" and ensuring that both "Windows Subsystem For Linux" and "Virtual Machine Platform" are selected.
 
-Next you will need to install a Linux distribution such as Ubuntu-20.04 and make sure it's set to using version 2 of WSL. Refer to [Install WSL](https://docs.microsoft.com/en-us/windows/wsl/install) for more information.
+Next you will need to install a Linux distribution such as Ubuntu-22.04 and make sure it's set to using version 2 of WSL. Refer to [Install WSL](https://docs.microsoft.com/en-us/windows/wsl/install) for more information.
 
 Finally, you need to ensure Docker Desktop has integration enabled for your distribution so it can interact with Docker elements, such as a bash command window. To do this, open the Docker Desktop gui and go into settings, select `Resources` and them `WSL Integration` and ensure the checkbox for enable integration is checked. You should then see your WSL2 linux distribution listed (if you don't then it is probably because it is still a WSL1 distribution and needs to be converted to WSL2) and you can then toggle the switch to enable integration for that distro. Refer to [Docker Desktop WSL2 backend](https://docs.docker.com/desktop/windows/wsl/) for more information
 

diff --git a/vagrant/Vagrantfile b/vagrant/Vagrantfile
@@ -7,7 +7,7 @@
 
 Vagrant.require_version ">= 1.7.4"
 Vagrant.configure('2') do |config|
-  config.vm.box = "bento/ubuntu-20.04"
+  config.vm.box = "bento/ubuntu-22.04"
   config.vm.synced_folder "..", "/home/vagrant/fabric"
   config.ssh.forward_agent = true