Fix insert/load with timestamp of Long.MIN_VALUE and LONG.MAX_VALUE #14491

jt2594838 · 2024-12-18T11:17:34Z

No description provided.

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java

codecov · 2024-12-19T05:02:50Z

Codecov Report

Attention: Patch coverage is 75.78947% with 46 lines in your changes missing coverage. Please review.

Project coverage is 39.42%. Comparing base (55b37e0) to head (7916e79).
Report is 31 commits behind head on master.

Files with missing lines	Patch %	Lines
...a/org/apache/iotdb/db/utils/ModificationUtils.java	0.00%	7 Missing ⚠️
...te/utils/writer/AbstractCrossCompactionWriter.java	16.66%	5 Missing ⚠️
...orageengine/load/splitter/NonAlignedChunkData.java	0.00%	4 Missing ⚠️
...g/apache/iotdb/db/tools/TsFileResourcePrinter.java	0.00%	4 Missing ⚠️
...e/iotdb/db/queryengine/plan/parser/ASTVisitor.java	0.00%	3 Missing ⚠️
...tor/readchunk/AlignedSeriesCompactionExecutor.java	0.00%	3 Missing ⚠️
...n/planner/plan/node/load/LoadSingleTsFileNode.java	0.00%	2 Missing ⚠️
...gion/compaction/repair/RepairDataFileScanUtil.java	90.00%	2 Missing ⚠️
...region/utils/TsFileDeviceStartEndTimeIterator.java	0.00%	2 Missing ⚠️
...l/recover/file/UnsealedTsFileRecoverPerformer.java	50.00%	2 Missing ⚠️
... and 9 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14491      +/-   ##
============================================
- Coverage     39.60%   39.42%   -0.19%     
  Complexity      185      185              
============================================
  Files          4334     4341       +7     
  Lines        275866   275427     -439     
  Branches      33533    33647     +114     
============================================
- Hits         109267   108580     -687     
- Misses       166599   166847     +248

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jt2594838 · 2024-12-19T07:40:53Z

integration-test/src/test/java/org/apache/iotdb/session/it/IoTDBSessionSimpleIT.java

@@ -336,7 +344,7 @@ public void insertByObjAndNotInferTypeTest() {
      expected.add(TSDataType.TEXT.name());

      Set<String> actual = new HashSet<>();
-      SessionDataSet dataSet = session.executeQueryStatement("show timeseries root.**");
+      SessionDataSet dataSet = session.executeQueryStatement("show timeseries root.sg1.**");


This avoids conflicts with other tests.

jt2594838

Self-reviewed

shuwenwei · 2024-12-20T07:10:07Z

...a/org/apache/iotdb/db/storageengine/dataregion/compaction/repair/RepairDataFileScanUtil.java

The previousTime also needs to be modified. This value is used in checkPreviousTimeAndUpdate to check the order of timestamps. Perhaps we can add a boolean to determine whether to set initial values.

shuwenwei · 2024-12-20T07:17:35Z

These classes also need to be modified:

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/executor/readchunk/AlignedSeriesCompactionExecutor.java
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/executor/readchunk/ReadChunkAlignedSeriesCompactionExecutor.java
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/writer/AbstractCompactionWriter.java

shuwenwei · 2024-12-24T03:49:07Z

...e/dataregion/compaction/execute/utils/executor/readchunk/SingleSeriesCompactionExecutor.java

@@ -158,8 +161,10 @@ public void execute() throws IOException {
      flushChunkWriter();
    }
    fileWriter.checkMetadataSizeAndMayFlush();
-    targetResource.updateStartTime(device, minStartTimestamp);
-    targetResource.updateEndTime(device, maxEndTimestamp);
+    if (minStartTimestampSet) {


Can use minStartTimestamp <= minEndTimestamp to instead these booleans?

You mean still let the initial value of minStartTimestamp be Long.MAX and maxEndTimestamp be Long.MIN?
It is possible, but I think it may be more important not to leave the impression that Long.MAX and Long.MIN are treated as special timestamps and keep the consistency with other places.

shuwenwei · 2024-12-24T10:00:15Z

...b/db/storageengine/dataregion/compaction/selector/estimator/AbstractCompactionEstimator.java

@@ -140,8 +142,12 @@ protected int calculatingMaxOverlapFileNumInSubCompactionTask(List<TsFileResourc
      long maxEndTimeOfCurrentDevice = Long.MIN_VALUE;
      int overlapFileNumOfCurrentDevice = 0;
      for (ArrayDeviceTimeIndex resource : resourcesContainsCurrentDevice) {
-        long deviceStartTimeInCurrentFile = resource.getStartTime(device);
-        long deviceEndTimeInCurrentFile = resource.getEndTime(device);
+        if (resource.definitelyNotContains(device)) {


This check should have been done in line135

sonarqubecloud · 2024-12-24T10:25:45Z

Quality Gate failed

Failed conditions
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

jt2594838 added 3 commits December 18, 2024 16:15

add test

6ef78af

fix insert/load with Long.MIN/MAX

bffdbf2

Merge branch 'master' into fix_insert_long_min_max

160b6aa

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java

SteveYurongSu self-assigned this Dec 19, 2024

jt2594838 added 2 commits December 19, 2024 11:24

Fix tests

2445092

fix tests

8a1cef9

fix tests and use optional

43fed13

jt2594838 commented Dec 19, 2024

View reviewed changes

jt2594838 added 3 commits December 19, 2024 17:29

rename & self review

d083a71

fix test

057b399

avoid test confilct

5c4214d

shuwenwei reviewed Dec 20, 2024

View reviewed changes

fix review

6a67574

jt2594838 requested a review from shuwenwei December 21, 2024 06:48

shuwenwei reviewed Dec 24, 2024

View reviewed changes

shuwenwei approved these changes Dec 24, 2024

View reviewed changes

shuwenwei reviewed Dec 24, 2024

View reviewed changes

remove redundant check

7916e79

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix insert/load with timestamp of Long.MIN_VALUE and LONG.MAX_VALUE #14491

Fix insert/load with timestamp of Long.MIN_VALUE and LONG.MAX_VALUE #14491

jt2594838 commented Dec 18, 2024

codecov bot commented Dec 19, 2024 •

edited

Loading

jt2594838 Dec 19, 2024

jt2594838 left a comment

shuwenwei Dec 20, 2024

jt2594838 Dec 21, 2024

shuwenwei commented Dec 20, 2024

shuwenwei Dec 24, 2024

jt2594838 Dec 24, 2024

shuwenwei Dec 24, 2024

jt2594838 Dec 24, 2024

sonarqubecloud bot commented Dec 24, 2024

Fix insert/load with timestamp of Long.MIN_VALUE and LONG.MAX_VALUE #14491

Are you sure you want to change the base?

Fix insert/load with timestamp of Long.MIN_VALUE and LONG.MAX_VALUE #14491

Conversation

jt2594838 commented Dec 18, 2024

codecov bot commented Dec 19, 2024 • edited Loading

Codecov Report

jt2594838 Dec 19, 2024

Choose a reason for hiding this comment

jt2594838 left a comment

Choose a reason for hiding this comment

shuwenwei Dec 20, 2024

Choose a reason for hiding this comment

jt2594838 Dec 21, 2024

Choose a reason for hiding this comment

shuwenwei commented Dec 20, 2024

shuwenwei Dec 24, 2024

Choose a reason for hiding this comment

jt2594838 Dec 24, 2024

Choose a reason for hiding this comment

shuwenwei Dec 24, 2024

Choose a reason for hiding this comment

jt2594838 Dec 24, 2024

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 24, 2024

Quality Gate failed

codecov bot commented Dec 19, 2024 •

edited

Loading