[3006.x][s3fs] delete files from local cache when deleted from s3

[dead10ck]

What does this PR do?

Currently, s3 files are not deleted from the local cache when they are deleted from s3. See #65611

This is currently a draft, as it's based on #65610. The two are not directly related, but the latter adds some testing boilerplate for the file server, which this builds on.

What issues does this PR fix or reference?

Fixes: #65611

Merge requirements satisfied?

• Docs
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

[twangboy]

This needs to be rebased (not merged) and conflicts resolved.

[dead10ck]

Ok this was rebased and has a change log entry now.

[dwoz]

Asking for @whytewolf's input on this one.

[whytewolf]

my main concern would be people using large filesystems in s3 with this. lots of for loops isn't exactly speedy when crawling large filesystems. but hard to see a way around them without redesigning the whole thing.

no one should be relying on the cache item still being there. if they are that is undefined behavior and was never promised. so not sure we need that warning.

[dead10ck]

my main concern would be people using large filesystems in s3 with this. lots of for loops isn't exactly speedy when crawling large filesystems. but hard to see a way around them without redesigning the whole thing.

Yeah, though if you have a large fs, then simply doing the s3 object listing would become a bottleneck well before the local file system would. S3 isn't really meant to serve as a large scale NFS, so s3 wouldn't be a good choice of file server for such a use case anyway.

no one should be relying on the cache item still being there. if they are that is undefined behavior and was never promised. so not sure we need that warning.

Yeah, but my suspicion is that people who are relying on it are doing so unknowingly. This came about because I was still seeing states getting applied that I had deleted because I forgot to remove its invocation from top.sls. It's easy to imagine this change causing breakage for systems where s3fs has been in use for a long period of time.

[whytewolf]

my main concern would be people using large filesystems in s3 with this. lots of for loops isn't exactly speedy when crawling large filesystems. but hard to see a way around them without redesigning the whole thing.

Yeah, though if you have a large fs, then simply doing the s3 object listing would become a bottleneck well before the local file system would. S3 isn't really meant to serve as a large scale NFS, so s3 wouldn't be a good choice of file server for such a use case anyway.

no one should be relying on the cache item still being there. if they are that is undefined behavior and was never promised. so not sure we need that warning.

Yeah, but my suspicion is that people who are relying on it are doing so unknowingly. This came about because I was still seeing states getting applied that I had deleted because I forgot to remove its invocation from top.sls. It's easy to imagine this change causing breakage for systems where s3fs has been in use for a long period of time.

both of those are good reasons. and why i accepted in the review. @dwoz this is good to go.