Skip to content

Conversation

@ZRHann
Copy link

@ZRHann ZRHann commented Dec 11, 2025

Description / 描述

The S3 driver has been updated to implement the Getter interface. This addition introduces the Get method, which allows the retrieval of both files and directories from S3. The method works by first attempting to fetch the object as a file using HeadObject. If the object is not found (HTTP 404), it checks if the path corresponds to a directory by listing objects with the given prefix. The Get method handles two different object listing strategies, “v1” and “v2”, based on the configuration of the ListObjectVersion.

Motivation and Context / 背景

  • Improve performance for directly accessing multi-level directories.
  • Some S3 implementations do not allow listing objects without a prefix at the first level (i.e., for paths like bucket://), which can cause issues with the original List method. This can prevent access to paths like bucket://xxx/xxx (althrough with prefix). By using Get, which directly retrieves objects based on the path, this problem can be avoided.

How Has This Been Tested? / 测试

I added logging inside the newly implemented Get method, then created an S3 storage instance to verify that:

  • the Get method was indeed invoked, and
  • it returned the correct results for both files and multi-level directories.

This confirmed that the new logic behaves as expected.

Checklist / 检查清单

  • I have read the CONTRIBUTING document.
    我已阅读 CONTRIBUTING 文档。
  • I have formatted my code with go fmt or prettier.
    我已使用 go fmtprettier 格式化提交的代码。
  • I have added appropriate labels to this PR (or mentioned needed labels in the description if lacking permissions).
    我已为此 PR 添加了适当的标签(如无权限或需要的标签不存在,请在描述中说明,管理员将后续处理)。
  • I have requested review from relevant code authors using the "Request review" feature when applicable.
    我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
  • I have updated the repository accordingly (If it’s needed).
    我已相应更新了相关仓库(若适用)。

Comment on lines +281 to +322
// If HeadObject fails with 404, check if it's a directory
prefix := getKey(path, true)
var contents []*s3.Object
var commonPrefixes []*s3.CommonPrefix
switch d.ListObjectVersion {
case "v1":
listInput := &s3.ListObjectsInput{
Bucket: &d.Bucket,
Prefix: &prefix,
MaxKeys: aws.Int64(1), // Only need to check if at least one object exists
}
listResult, err := d.client.ListObjectsWithContext(ctx, listInput)
if err != nil {
return nil, errors.WithMessage(err, "failed to list objects with prefix")
}
contents = listResult.Contents
commonPrefixes = listResult.CommonPrefixes
case "v2":
listInput := &s3.ListObjectsV2Input{
Bucket: &d.Bucket,
Prefix: &prefix,
MaxKeys: aws.Int64(1),
}
listResult, err := d.client.ListObjectsV2WithContext(ctx, listInput)
if err != nil {
return nil, errors.WithMessage(err, "failed to list objects v2 with prefix")
}
contents = listResult.Contents
commonPrefixes = listResult.CommonPrefixes
default:
return nil, fmt.Errorf("unsupported ListObjectVersion: %s", d.ListObjectVersion)
}
if len(contents) > 0 || len(commonPrefixes) > 0 {
dirName := stdpath.Base(path + "/")
return &model.Object{
Name: dirName,
Modified: d.Modified,
IsFolder: true,
Path: path,
}, nil
}
return nil, errs.ObjectNotFound
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// If HeadObject fails with 404, check if it's a directory
prefix := getKey(path, true)
var contents []*s3.Object
var commonPrefixes []*s3.CommonPrefix
switch d.ListObjectVersion {
case "v1":
listInput := &s3.ListObjectsInput{
Bucket: &d.Bucket,
Prefix: &prefix,
MaxKeys: aws.Int64(1), // Only need to check if at least one object exists
}
listResult, err := d.client.ListObjectsWithContext(ctx, listInput)
if err != nil {
return nil, errors.WithMessage(err, "failed to list objects with prefix")
}
contents = listResult.Contents
commonPrefixes = listResult.CommonPrefixes
case "v2":
listInput := &s3.ListObjectsV2Input{
Bucket: &d.Bucket,
Prefix: &prefix,
MaxKeys: aws.Int64(1),
}
listResult, err := d.client.ListObjectsV2WithContext(ctx, listInput)
if err != nil {
return nil, errors.WithMessage(err, "failed to list objects v2 with prefix")
}
contents = listResult.Contents
commonPrefixes = listResult.CommonPrefixes
default:
return nil, fmt.Errorf("unsupported ListObjectVersion: %s", d.ListObjectVersion)
}
if len(contents) > 0 || len(commonPrefixes) > 0 {
dirName := stdpath.Base(path + "/")
return &model.Object{
Name: dirName,
Modified: d.Modified,
IsFolder: true,
Path: path,
}, nil
}
return nil, errs.ObjectNotFound
return nil, errs.NotSupport

既然调用一次列表,这里就直接返回errs.NotSupport让op.Get去op.List寻找,op.List有缓存

}

// try to get object as a file using HeadObject
key := getKey(path, false)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
key := getKey(path, false)
key := getKey(stdpath.Join(d.GetRootPath(), path), false)

Comment on lines +251 to +257
if utils.PathEqual(path, "/") {
return &model.Object{
Name: "Root",
IsFolder: true,
Path: "/",
}, nil
}
Copy link
Member

@j2rong4cn j2rong4cn Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if utils.PathEqual(path, "/") {
return &model.Object{
Name: "Root",
IsFolder: true,
Path: "/",
}, nil
}

已经不用Get获取根目录对象了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants