¶ Description
List files in a bucket in an Huwaei Blob Storage
¶ Parameters
¶ Parameters Tab
Parameters:
- Optional: folder path (prefix) to list files from
- Bucket name
- Bucket Region
- Access Key ID
- Secret Key
- Debug information level
- Optional: extra parameters for cURL
- Number of retries on connection error
¶ Description Tab
Parameters:
- Script name
- Short description
- Revision
- Decription
¶ Configuration Tab
See dedicated page for more information.
¶ About
This action also works when accessing the web through a PROXY server.
To use this action, you’ll need to get several parameters from the Huawei Website .
Once you have completed the “setup process” , you can use the parameter P1 to list the files in your Huwaei Blob Storage system.
¶ Huawei Blob Storage First Time Setup
Before using the Huawei Blob storage actions inside ETL, you need to get these 4 parameters:
- ETL Parameter P1: Bucket Region
- ETL Parameter P2: Bucket Name
- ETL Parameter P3: Access-Key
- ETL Parameter P4: Secret-Key
…from the Huwaei website.
Here are the steps to get these 4 parameters:
- Open https://www.huaweicloud.com and click on the “Log in” button:
- Enter your credentials and click the “Log In” button:
- Click on the “console” link inside the header:
- Click on the symbol at the top of the column on the left:
- …and select (click) “Object Storage Service” in the big menu:
- Click on the “Create Bucket” button on the top right:
- Fill-in all the required fields on click on the “Create Now” button on the bottom:
11. Fill-in the next screen:
Please make sure to keep the ETL Parameter P4 (Secret-Key) is a safe place since it cannot be retreived from the Huawei website later on.
¶ What this action does (expanded)
HuaweiOBSListFiles reads the object catalogue of a Huawei Object Storage Service (OBS) bucket and returns one row per object under an optional prefix.
Key characteristics:
- Read-only & idempotent — never modifies bucket contents.
- Region-aware — the
regionyou select is used to reach the correct OBS endpoint for the bucket. - Fully paginated — transparently follows OBS pagination (OBS lists up to 1,000 keys per page).
- Prefix-scoped — you can restrict the listing to any “folder” by setting
Optional: folder path (prefix) to list files from. - Proxy compatible — honors the platform’s proxy settings.
¶ Parameters (deep dive)
Names below match the Parameters tab.
¶ Optional: folder path (prefix) to list files from (optional)
- Filters the list to keys that start with this value.
- Treat it like a folder path. Slashes (
/) are literal; bothreports/2024/andreports/2024work. - Empty means “list the entire bucket.”
- Use prefixes to reduce scan time on buckets with millions of objects.
¶ bucket Name
- The exact OBS bucket name (DNS-style, lowercase, hyphens allowed).
- Must belong to the selected region; if you see
NoSuchBucket, verify the region.
¶ Bucket Region
- OBS region where the bucket resides (e.g.,
la-south-2,eu-west-0,ap-southeast-3, …). - This determines the API endpoint. A mismatched region often yields
PermanentRedirectorNoSuchBucket.
¶ Access key (Access Key ID) & Secret key (Secret Key)
- IAM user or temporary credentials with at least read/list permissions on the bucket.
- Store as secrets/variables; avoid hard-coding.
- Rotate routinely; prefer a dedicated, least-privilege access key for this integration.
¶ Debug information level
-
Log verbosity for the connector:
- nothing — quiet, recommended for production.
- basic — request summaries & high-level timing.
- verbose — low-level HTTP traces; use only when diagnosing.
¶ Optional: extra parameters for cURL (optional)
- Extra cURL options if you need special networking behavior (corporate proxy flags, timeouts, TLS tweaks).
- Use sparingly; prefer platform-level networking config.
- Do not override HTTP method or authorization headers.
¶ Number of retries on connection error
- Number of automatic retries on transient failures (network glitches, throttling).
- Retries use incremental back-off. Typical values:
3–7for stable links,7–12for flaky links.
¶ Output schema (what to expect)
The action yields one row per object (file) found under the selected scope. Columns can vary slightly by version, but typically include:
- Key / Path — full object key (e.g.,
folder/sub/file.csv) - SizeBytes — object size in bytes
- LastModifiedUtc — last modification timestamp (UTC)
- ETag — entity tag (content hash)
- StorageClass — e.g.,
STANDARD,WARM,COLD, etc. (if available)
Tip: Run a small listing first (set a narrow
Optional: folder path (prefix) to list files from) and inspect the Data panel to confirm the exact column names in your environment.
¶ Quick start (minimal steps)
- Set
bucket Nameto your OBS bucket. - Pick
regionthat matches the bucket’s region (e.g.,la-south-2). - Paste
Access key/Secret keyfor a user allowed to list the bucket. - (Optional) Set
Optional: folder path (prefix) to list files fromto restrict the listing. - Leave
Debug information level = nothing,Number of retries on connection error = 5(or your standard). - Run. Review the Data grid and downstream usage.
¶ Behavior & limits
- Pagination: Handled automatically; large buckets may take time. Narrowing
Optional: folder path (prefix) to list files fromdramatically improves performance. - Ordering: OBS returns keys in lexicographic order.
- Folders: OBS is flat storage; “folders” are prefix conventions. The action lists all keys that begin with the prefix; there’s no separate folder row unless your bucket contains zero-size “marker” objects.
- Eventual consistency: Newly uploaded or deleted objects are usually visible quickly, but there can be short propagation delays.
¶ Required permissions (least privilege)
Grant the IAM principal used by Access key/Secret key these OBS permissions on the target bucket (and its objects):
- List bucket (enumerate objects)
- Get bucket location (some SDKs/clients use this)
Optional (only if you read object metadata beyond the standard list response): Get object metadata.
Keep permissions restricted to the specific bucket (resource-scoped) whenever possible.
¶ Best practices
- Use prefixes to keep scans fast and predictable (e.g., date-partitioned paths).
- Keep
Debug information level = nothingin production; switch toverboseonly to capture a failing run. - Rotate credentials and store them in your platform’s secret store.
- Prefer regional proximity when chaining actions that read objects after listing.
- If you run from a corporate network, ensure OBS endpoints are allow-listed and TLS interception doesn’t break the connection.
¶ Notes on regions
The Bucket Region dropdown includes all public OBS regions (e.g., la-south-2, eu-west-0, multiple ap-southeast-* and cn-*). Always select the value that exactly matches your bucket’s region in the Huawei console; otherwise the service will redirect or reject the request.
