Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Assigned multiple different partition IDs to the same partition name and created successfully. #46082

Open
3 tasks done
XLPE opened this issue Dec 27, 2024 · 0 comments · May be fixed by #46083
Open
3 tasks done

Comments

@XLPE
Copy link

XLPE commented Dec 27, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

Version

2.1.5

What's Wrong?

When inserting data into an auto-partitioned table, it is possible for the same partition name to be assigned multiple different partition IDs, indirectly leading to the creation of a large number of redundant tablets. In my environment, an insertion that should have created 100 tablets ended up creating over 9,000 tablets, causing instability in the cluster.
table:

CREATE TABLE base_info_m (
    c_month varchar(255) NOT NULL
    -- 其他列...
)ENGINE=OLAP
DUPLICATE KEY(c_month)
AUTO PARTITION BY LIST(c_month)
(
    PARTITION p2024116 VALUES IN("202411")
)
DISTRIBUTED BY HASH('c_month') BUCKETS 100;

insert sql:

insert into base_info_m   select  * from t where t.c_month='202411'

fe log:

2024-12-18 18:18:44,131 INFO (thrift-server-pool-12189|559823) [InternalCatalog.createPartitionWithIndices():2106] succeed in creating partition[255404482-p2024116], table : [255230533-base_info_m]
2024-12-18 18:18:44,131 INFO (thrift-server-pool-12189|559823) [InternalCatalog.addPartition():1692] table[base_info_m] add partition[p2024116] which already exists
2024-12-18 18:18:44,133 INFO (thrift-server-pool-11499|499577) [InternalCatalog.createPartitionWithIndices():2106] succeed in creating partition[255298217-p2024116], table : [255230533-base_info_m]
2024-12-18 18:18:44,133 INFO (thrift-server-pool-11499|499577) [InternalCatalog.addPartition():1692] table[base_info_m] add partition[p2024116] which already exists
2024-12-18 18:18:44,150 INFO (thrift-server-pool-12021|559655) [InternalCatalog.createPartitionWithIndices():2106] succeed in creating partition[255316262-p2024116], table : [255230533-base_info_m]
2024-12-18 18:18:44,150 INFO (thrift-server-pool-12021|559655) [InternalCatalog.addPartition():1692] table[base_info_m] add partition[p2024116] which already exists
.....

One of the BE (Backend) logs

I20241218 18:18:42.194964 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298230, table_id=255230533, partition_id=255298217, replica_id=255298233, stores.size=12, first store=/data02/doris-storage
I20241218 18:18:42.195721 3360680 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298254, table_id=255230533, partition_id=255298217, replica_id=255298256, stores.size=12, first store=/data10/doris-storage
I20241218 18:18:42.196261 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298278, table_id=255230533, partition_id=255298217, replica_id=255298279, stores.size=12, first store=/data06/doris-storage
I20241218 18:18:42.196411 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298298, table_id=255230533, partition_id=255298217, replica_id=255298301, stores.size=12, first store=/data11/doris-storage
I20241218 18:18:42.197140 3360680 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298322, table_id=255230533, partition_id=255298217, replica_id=255298324, stores.size=12, first store=/data08/doris-storage
I20241218 18:18:42.197643 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298346, table_id=255230533, partition_id=255298217, replica_id=255298347, stores.size=12, first store=/data07/doris-storage
I20241218 18:18:42.197715 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298366, table_id=255230533, partition_id=255298217, replica_id=255298369, stores.size=12, first store=/data03/doris-storage
I20241218 18:18:42.198566 3360680 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298390, table_id=255230533, partition_id=255298217, replica_id=255298392, stores.size=12, first store=/data01/doris-storage
I20241218 18:18:42.198913 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298414, table_id=255230533, partition_id=255298217, replica_id=255298415, stores.size=12, first store=/data04/doris-storage
I20241218 18:18:42.199162 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298434, table_id=255230533, partition_id=255298217, replica_id=255298437, stores.size=12, first store=/data09/doris-storage
I20241218 18:18:42.200104 3360680 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298458, table_id=255230533, partition_id=255298217, replica_id=255298460, stores.size=12, first store=/data05/doris-storage
I20241218 18:18:42.200398 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298482, table_id=255230533, partition_id=255298217, replica_id=255298483, stores.size=12, first store=/data12/doris-storage
I20241218 18:18:42.200505 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298502, table_id=255230533, partition_id=255298217, replica_id=255298505, stores.size=12, first store=/data02/doris-storage
I20241218 18:18:42.201678 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298526, table_id=255230533, partition_id=255298217, replica_id=255298528, stores.size=12, first store=/data10/doris-storage
I20241218 18:18:42.201699 3360680 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298550, table_id=255230533, partition_id=255298217, replica_id=255298551, stores.size=12, first store=/data06/doris-storage
I20241218 18:18:42.201918 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298570, table_id=255230533, partition_id=255298217, replica_id=255298573, stores.size=12, first store=/data11/doris-storage
I20241218 18:18:42.202903 3360680 tablet_manager.cpp:272] begin to create tablet. tablet_id=255298594, table_id=255230533, partition_id=255298217, replica_id=255298596, stores.size=12, first store=/data08/doris-storage
I20241218 18:18:42.437721 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404483, table_id=255230533, partition_id=255404482, replica_id=255404485, stores.size=12, first store=/data02/doris-storage
I20241218 18:18:42.437844 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404507, table_id=255230533, partition_id=255404482, replica_id=255404508, stores.size=12, first store=/data10/doris-storage
I20241218 18:18:42.439217 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404527, table_id=255230533, partition_id=255404482, replica_id=255404530, stores.size=12, first store=/data06/doris-storage
I20241218 18:18:42.439227 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404551, table_id=255230533, partition_id=255404482, replica_id=255404553, stores.size=12, first store=/data11/doris-storage
I20241218 18:18:42.440524 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404575, table_id=255230533, partition_id=255404482, replica_id=255404576, stores.size=12, first store=/data08/doris-storage
I20241218 18:18:42.440696 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404595, table_id=255230533, partition_id=255404482, replica_id=255404598, stores.size=12, first store=/data07/doris-storage
I20241218 18:18:42.442117 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404643, table_id=255230533, partition_id=255404482, replica_id=255404644, stores.size=12, first store=/data01/doris-storage
I20241218 18:18:42.442091 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404619, table_id=255230533, partition_id=255404482, replica_id=255404621, stores.size=12, first store=/data03/doris-storage
I20241218 18:18:42.443413 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404663, table_id=255230533, partition_id=255404482, replica_id=255404666, stores.size=12, first store=/data04/doris-storage
I20241218 18:18:42.443660 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404687, table_id=255230533, partition_id=255404482, replica_id=255404689, stores.size=12, first store=/data09/doris-storage
I20241218 18:18:42.444985 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404711, table_id=255230533, partition_id=255404482, replica_id=255404712, stores.size=12, first store=/data05/doris-storage
I20241218 18:18:42.445149 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404731, table_id=255230533, partition_id=255404482, replica_id=255404734, stores.size=12, first store=/data12/doris-storage
I20241218 18:18:42.446177 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404755, table_id=255230533, partition_id=255404482, replica_id=255404757, stores.size=12, first store=/data02/doris-storage
I20241218 18:18:42.446630 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404779, table_id=255230533, partition_id=255404482, replica_id=255404780, stores.size=12, first store=/data10/doris-storage
I20241218 18:18:42.447506 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404799, table_id=255230533, partition_id=255404482, replica_id=255404802, stores.size=12, first store=/data06/doris-storage
I20241218 18:18:42.447930 3360681 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404823, table_id=255230533, partition_id=255404482, replica_id=255404825, stores.size=12, first store=/data11/doris-storage
I20241218 18:18:42.448966 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404847, table_id=255230533, partition_id=255404482, replica_id=255404848, stores.size=12, first store=/data08/doris-storage
I20241218 18:18:42.450199 3360682 tablet_manager.cpp:272] begin to create tablet. tablet_id=255404867, table_id=255230533, partition_id=255404482, replica_id=255404870, stores.size=12, first store=/data07/doris-storage

What You Expected?

partition p2024116 should not be recreated repeatedly.

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@XLPE XLPE linked a pull request Dec 27, 2024 that will close this issue
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant