Issues with remote filebucket setup

[mdechiaro]

Describe the Bug

I am trying to get remote filebucket setup, and I believe I have the right config as per the documentation, but it doesn't appear to be working.

Expected Behavior

I expect filebuckets to be uploaded remotely to a single puppetserver. Instead, these files reside locally on the puppetserver that compiled the catalog.

Steps to Reproduce

The following config is in our manifests/00site.pp file.

filebucket { 'puppetca':
  server => 'puppetca.hostname.com',
  path   => false,
}

File {
  backup => 'puppetca',
}

The following config enabled in auth.conf on puppetserver.

        {
            # Allow nodes to access all file_bucket_files.  Note that access for
            # the 'delete' method is forbidden by Puppet regardless of the
            # configuration of this rule.
            match-request: {
                path: "/puppet/v3/file_bucket_file"
                type: path
                method: [get, head, post, put]
            }
            allow: "*"
            sort-order: 500
            name: "puppetlabs file bucket file"
        },

Environment

• OSS Puppet 8.9.0, running on Ubuntu 22.04
• OSS Puppetserver 8.6.2 running on Ubuntu 22.04

Additional Context

I don't see any errors in puppetserver.log.

In the puppet catalog, I do a see line showing a filebucket attempt to remote server, but I don't see these files in /opt/puppetlabs/server/data/puppetserver/bucket/ on the remote server.

Info: /Stage[main]/Files/Files::Control_files[/path/to/file.txt]/File[/path/to/file.txt]: Filebucketed /path/to/file.txt to puppetca with sum c9070ced9a7b62d49656c2d7efacee909484bbc4f986d3dc0b15b3df26ccb872

I do see these files locally on the puppetserver that compiled the catalog.

I confirmed that the puppet agent is able to reach the remote filebucket server. Currently we are using our "CA" server as this remote filebucket server.

Is there any other places to review in order to troubleshoot further? I would love some assistance to narrow down what the problem is. Thanks for your help.