HDDS-15616. ListBuckets API ignores MaxBuckets parameter and returns all buckets

[Gargi-jais11]

What changes were proposed in this pull request?

S3 test test_list_buckets_paginated is failing today as ListBuckets pagination using MaxBuckets and ContinuationToken on GET / (ListAllMyBuckets) is not supported.

creates 2 buckets and calls list_buckets(MaxBuckets=1).

Expected: 1 bucket per page + ContinuationToken when more exist.
Ozone: Ignores MaxBuckets and returns both buckets → assert len == 1 fails with 2.

Simple example:

Buckets: [bucket-A, bucket-B]
Client: list_buckets(MaxBuckets=1) 

AWS: { Buckets: [bucket-A], ContinuationToken: "..." }
Ozone: { Buckets: [bucket-A, bucket-B] } ← 2 items, no token 

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15616

How was this patch tested?

Added UT and IT.
Also tested manually.
Before Fix:

// no continuation token is given and send all the buckets stored.
bash-5.1$ aws s3api list-buckets --max-buckets 1 --endpoint-url http://s3g:9878 --output json
{
    "Buckets": [
        {
            "Name": "buck1",
            "CreationDate": "2026-06-30T07:04:27.544Z"
        },
        {
            "Name": "buck2",              -----------------------------> wrong behaviour
            "CreationDate": "2026-06-30T07:04:30.460Z"
        },
        {
            "Name": "buck3",       -----------------------------> wrong behaviour
            "CreationDate": "2026-06-30T07:04:35.556Z"
        },
        {
            "Name": "buck4",           -----------------------------> wrong behaviour
            "CreationDate": "2026-06-30T07:04:40.205Z"
        },
        {
            "Name": "buck5",        -----------------------------> wrong behaviour
            "CreationDate": "2026-06-30T07:04:43.890Z"
        }
    ],
    "Owner": {
        "DisplayName": "om",
        "ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
    }
}

After Fix:

bash-5.1$ aws s3api list-buckets --max-buckets 1 --endpoint-url http://s3g:9878/ --output json
{
    "Buckets": [
        {
            "Name": "buck1",
            "CreationDate": "2026-06-23T08:51:13.848Z"
        }
    ],
    "Owner": {
        "DisplayName": "om",
        "ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
    },
    "ContinuationToken": "000000056275636b31-f239795a4c8be1e364face563bdc554b5559cbc1978becf3086b5079c2c44850"
}

bash-5.1$ aws s3api list-buckets --max-buckets 2 --endpoint-url http://s3g:9878/ --output json
{
    "Buckets": [
        {
            "Name": "buck1",
            "CreationDate": "2026-06-23T08:51:13.848Z"
        },
        {
            "Name": "buck2",
            "CreationDate": "2026-06-23T08:51:16.495Z"
        }
    ],
    "Owner": {
        "DisplayName": "om",
        "ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
    }
}

bash-5.1$ ozone sh bucket create /s3v/buck3
bash-5.1$ aws s3api list-buckets --max-buckets 2 --endpoint-url http://s3g:9878/ --output json
{
    "Buckets": [
        {
            "Name": "buck1",
            "CreationDate": "2026-06-23T08:51:13.848Z"
        },
        {
            "Name": "buck2",
            "CreationDate": "2026-06-23T08:51:16.495Z"
        }
    ],
    "Owner": {
        "DisplayName": "om",
        "ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
    },
    "ContinuationToken": "000000056275636b32-0880ee15253ee89c74cbc13b73c026a3ecd9529d0682e4d61e85dde392eb3ed3"
}

bash-5.1$ aws s3api list-buckets --max-buckets 1 --continuation-token 000000056275636b32-0880ee15253ee89c74cbc13b73c026a3ecd9529d0682e4d61e85dde392eb3ed3 \
    --endpoint-url http://s3g:9878/ --output json
{
    "Buckets": [
        {
            "Name": "buck3",
            "CreationDate": "2026-06-23T08:52:25.003Z"
        }
    ],
    "Owner": {
        "DisplayName": "om",
        "ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
    }
}
bash-5.1$ aws s3api list-buckets --max-buckets 2 --endpoint-url http://s3g:9878/
{
    "Buckets": [
        {
            "Name": "buck1",
            "CreationDate": "2026-06-23T08:51:13.848Z"
        },
        {
            "Name": "buck2",
            "CreationDate": "2026-06-23T08:51:16.495Z"
        }
    ],
    "Owner": {
        "DisplayName": "om",
        "ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
    },
    "ContinuationToken": "000000056275636b32-0880ee15253ee89c74cbc13b73c026a3ecd9529d0682e4d61e85dde392eb3ed3"
}

[ayushtkn]

can we not do this logic inside the method itself validateMaxBuckets, actually this isn't validate, it is like getMaxBuckets, because it does

 return Math.min(maxBuckets, S3Consts.MAX_BUCKETS_LIMIT);

Maybe we can drop the ternary here and do all this logic inside the method

[ayushtkn]

could the listWithMaxBuckets be used here?

[ayushtkn]

the second condition is unnecessary, pagination should naturally terminate when there are no elements remaining.

We can maybe assert that it returns exactly 2 elements which are bucketA & bucketB

[ayushtkn]

maybe assertThat(found).containsExactlyInAnyOrder(bucketA, bucketB);

[ayushtkn]

is this a almost a copy from the V1 test, we should ideally refactor to make sure we are not having the same code both places.

unless that is some established practice for the particular case