Skip to content

HDDS-15664. ozone debug ldb display DeletedBlocksTransactionSummary info #10614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ChenSammi merged 2 commits into from
Jul 3, 2026
Merged

HDDS-15664. ozone debug ldb display DeletedBlocksTransactionSummary info #10614

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
35f9152
HDDS-15664. ozone debug ldb display DeletedBlocksTransactionSummary info
ChenSammi Jun 25, 2026
8c605c0
address comment
ChenSammi Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/ldb/DBScanner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
package org.apache.hadoop.ozone.debug.ldb;

import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.hadoop.hdds.scm.block.DeletedBlockLogStateManagerImpl.SERVICE_NAME;
import static org.apache.hadoop.hdds.scm.metadata.SCMDBDefinition.STATEFUL_SERVICE_CONFIG;

import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.JsonInclude;
Expand All @@ -28,6 +30,8 @@
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import com.google.protobuf.ByteString;
import com.google.protobuf.InvalidProtocolBufferException;
import java.io.BufferedWriter;
import java.io.File;
import java.io.IOException;
Expand All @@ -54,6 +58,7 @@
import java.util.regex.Pattern;
import org.apache.hadoop.hdds.cli.AbstractSubcommand;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.scm.container.ContainerID;
import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
import org.apache.hadoop.hdds.utils.IOUtils;
Expand Down Expand Up @@ -730,9 +735,9 @@ public Void call() {
// one, to ensure valid JSON format.
sb.append(", ");
}
Object key = dbColumnFamilyDefinition.getKeyCodec()
.fromPersistedFormat(byteArrayKeyValue.getKey());
if (withKey) {
Object key = dbColumnFamilyDefinition.getKeyCodec()
.fromPersistedFormat(byteArrayKeyValue.getKey());
if (schemaV3) {
int index =
DatanodeSchemaThreeDBDefinition.getContainerKeyPrefixLength();
Expand All @@ -758,9 +763,11 @@ public Void call() {
Object o = dbColumnFamilyDefinition.getValueCodec()
.fromPersistedFormat(byteArrayKeyValue.getValue());

o = parseStatefulServiceConfig(key, o, dbColumnFamilyDefinition);

if (valueFields != null) {
Map<String, Object> filteredValue = new HashMap<>();
filteredValue.putAll(getFieldsFilteredObject(o, dbColumnFamilyDefinition.getValueType(), fieldsSplitMap));
filteredValue.putAll(getFieldsFilteredObject(o, o.getClass(), fieldsSplitMap));
sb.append(writer.writeValueAsString(filteredValue));
} else {
sb.append(writer.writeValueAsString(o));
Expand Down Expand Up @@ -829,6 +836,20 @@ List<Object> getFieldsFilteredObjectCollection(Collection<?> valueObject, Map<St
}
}

private Object parseStatefulServiceConfig(Object key, Object value, DBColumnFamilyDefinition dbColumnFamily) {
if (dbColumnFamily.getName().equals(STATEFUL_SERVICE_CONFIG.getName()) && key.equals(SERVICE_NAME) &&
value instanceof ByteString) {
try {
return HddsProtos.DeletedBlocksTransactionSummary.parseFrom((ByteString) value);
} catch (InvalidProtocolBufferException e) {
LOG.error("Failed to parse {} for key {}", STATEFUL_SERVICE_CONFIG.getName(), SERVICE_NAME, e);
} catch (IOException e) {
LOG.error("Failed to parse {} for key {}", STATEFUL_SERVICE_CONFIG.getName(), SERVICE_NAME, e);

@adoroszlai adoroszlai Jun 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate catch blocks, please merge them.

@ChenSammi ChenSammi Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai , STATEFUL_SERVICE_CONFIG is a table which doesn't belong to a specific feature of SCM, it's used as a key-value store(String -> ByteString) for all SCM features to persist value which need persistent. Currently it's used by container balancer, and Root CA rotation manager, now storage capacity feature. So the element in this table is specific to a feature, not like other tables in SCM, which has the same value data structure.
Currently, DBScanner can display this table with it raw ByteString value, not a understandable format of human, equal to no use during the debug time.
This task aims to covert the ByteString value to a storage capacity feature data structure and display it, so DBScanner will be helpful in the debug of future storage capacity feature. I'm not sure if I explain it clearly enough?

@adoroszlai adoroszlai Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ChenSammi for the explanation.

I would like to suggest some refactoring to reduce hard-coding in DBScanner, and also support container balancer and Root CA rotation manager: adoroszlai@0d5dd70 I can also submit follow-up PR if you prefer.

@ChenSammi ChenSammi Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure @adoroszlai , you can go ahead for the the follow-up PR.

@adoroszlai adoroszlai Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created #10654.

}
}
return value;
}

private static class ByteArrayKeyValue {
private final byte[] key;
private final byte[] value;
Expand Down