[Bug]: An infinite loop occurs in merge optimization, and the number of input files is equal to the number of output files.

[MrDannyWu]

What happened?

When I used Amoro to merge the Icerberg tables, I found that one table (no longer written) input 6 files during the merge, but the output was still 6 files.

Affects Versions

master

What engines are you seeing the problem on?

Optimizer

How to reproduce

Relevant log output

No response

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wangtaohz]

Thanks for report this issue!

I have confirmed that the issue is caused by splitting task. The splitting task uses the self-optimizing.max-task-size-bytes , and when this property is smaller than self-optimizing.target-size, the task becomes too small to merge files into the target size, resulting in repeated merging.

Using the max value of these two properties can solve this problem, the related code is here com.netease.arctic.server.optimizing.plan.AbstractPartitionPlan.BinPackingTaskSplitter#splitTasks.

[MrDannyWu]

Okay, thanks a lot!