S3 enumerator

Lists the objects of an Amazon S3 bucket (or of an S3-compatible store such as MinIO), optionally filtered by key prefix, folder level, name pattern, age and an object tag, and emits one file info record per object. Use it with S3 reader to process the files of a bucket.

Pins

Pin	Kind	Types	Description
Input	input	any	Starts the enumeration (for example a timer).
Output	output	Viaducta.TypeSystem.Metadata.ViaFileInfo	Called once per object.
Continue	continue output	same as input	Called with the input record after the last object.

Configuration

Field	Type	Default	Description
Authentication	dropdown: 0 AccessKey, 1 DefaultChain, 2 AssumeRole	0	How to sign in.
AccessKeyId	string		Access key id. Required for 0. For 2 optional: with it the role is assumed with this key, without it with the default chain.
SecretAccessKey	password		Secret of AccessKeyId. Required with AccessKeyId.
SessionToken	password		Session token of temporary credentials; optional.
RoleArn	string		2: ARN of the role to assume, for example arn:aws:iam::123456789012:role/import. Required for 2.
ExternalId	string		2: external id the trust policy of the role requires; optional.
StsServiceUrl	string		2: STS endpoint. Empty: the regional AWS STS endpoint of Region. For an S3-compatible store with STS (MinIO) the same URL as ServiceUrl.
Region	string		AWS region of the bucket, for example eu-central-1. Required when ServiceUrl is empty; with ServiceUrl the signing region (empty: us-east-1).
ServiceUrl	string		Endpoint of an S3-compatible store or an S3 VPC endpoint, http(s) without query, for example http://minio:9000. Empty: AWS S3 in Region.
ForcePathStyle	boolean	false	true addresses <endpoint>/<bucket>/<key> instead of <bucket>.<endpoint>; most S3-compatible stores need it.
BucketName	string		Required. Bucket to list.
Prefix	string		Optional key prefix, for example incoming/2026/. Empty lists all objects.
SkipTag	string		Object tag name; objects that have this tag are not emitted (for example viaducta-processed, set by S3 reader post-processing 3 Tag). Costs one request per object (see Notes). Empty lists all.
TopLevelOnly	boolean	false	true lists only the objects directly under Prefix (end it with /), not those in deeper folders.
NamePattern	string	*	Only objects whose key after the last / matches; * and ? as wildcards, case-insensitive, for example *.csv.
MinAgeMinutes	integer	0	Only objects not modified for at least this many minutes, so objects still being written are skipped. 0 lists all.
OrderBy	dropdown: 0 Name, 1 LastModified	0	Emit order: by key (as S3 lists, UTF-8 binary order) or oldest modification first.
MaxObjects	integer	0	Emit at most this many objects per run, the first ones in OrderBy order. 0 emits all.

Behaviour

• The S3 client is created at flow start; missing or invalid sign-in fields fail the start. 1 DefaultChain uses the AWS default credential chain: environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), the shared credentials file and profile (AWS_PROFILE), web identity (AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN, as on EKS), container credentials (ECS, EKS Pod Identity) and the EC2 instance profile. 2 AssumeRole calls STS AssumeRole (session name viaducta) and renews the temporary credentials five minutes before they expire. With 1 and 2 the credentials are fetched with the first S3 call. The identity needs s3:ListBucket (and s3:GetObjectTagging with SkipTag).
• Per input the whole listing is read first (ListObjectsV2; all levels, or with TopLevelOnly only the objects directly under Prefix), folder markers (keys ending with /) are dropped, then the objects are filtered by NamePattern and MinAgeMinutes, ordered by OrderBy, checked for SkipTag and cut to MaxObjects; then they are emitted. Changes downstream components make in the bucket during the run (for example S3 reader moving objects) don't change the list.
• Each object is emitted as ViaFileInfo: FileName = key (including folders), FilePath = s3://<bucket>/<key> (every key segment URL-escaped, for example s3://imports/daily/my%20file.csv), Size = size in bytes, CreateDateTime = last modification (UTC; S3 keeps no creation time), LastAccessDateTime = DateTime.MinValue.
• The progress bar shows <emitted>/<total> objects (while a large bucket is listed: listing, <n> objects, while tags are read: reading tags, <n> objects); the run ends with the status <emitted>/<total> objects listed.
• After the last object Continue is called with the input record. A missing bucket raises an error. No record positions are set on the emitted records.

Notes

• Connect Output to S3 reader to read each listed object: the reader takes bucket and key from FilePath (leave its BucketName/Key empty). With post-processing in the reader (delete, move, tag + SkipTag) a timer-driven flow processes each object once.
• SkipTag cost: the listing carries no tags, so every candidate object costs one GetObjectTagging request (after the other filters, and only until MaxObjects objects are found). For large backlogs prefer reader post-processing 2 Move to a prefix outside Prefix, or a small MaxObjects.
• The listing is kept in memory for the run; use Prefix to narrow very large buckets. MaxObjects limits what is emitted, not what is listed.
• MaxObjects with OrderBy 1 and reader post-processing processes a backlog oldest first in slices, one slice per timer run.
• 1 DefaultChain in the platform: shared-profile entries that assume a role (role_arn) or use AWS IAM Identity Center (sso_session) are not supported; use 2 AssumeRole or environment credentials instead.
• Store secrets as expression settings and reference them (=Secrets.AwsSecretKey) instead of typing keys into the flow.

Example

{ "Authentication": 1, "Region": "eu-central-1", "BucketName": "imports", "Prefix": "daily/", "SkipTag": "viaducta-processed", "NamePattern": "*.csv", "MinAgeMinutes": 5, "OrderBy": 1, "MaxObjects": 100 }

Timer → S3 enumerator → S3 reader → CSV decoder → ...