GCS enumerator

Lists the objects of a Google Cloud Storage bucket, optionally filtered by name prefix, folder level, name pattern, age and a custom metadata entry, and emits one file info record per object. Use it with GCS reader to process the files of a bucket.

Pins

Pin	Kind	Types	Description
Input	input	any	Starts the enumeration (for example a timer).
Output	output	Viaducta.TypeSystem.Metadata.ViaFileInfo	Called once per object.
Continue	continue output	same as input	Called with the input record after the last object.

Configuration

Field	Type	Default	Description
Authentication	dropdown: 0 ServiceAccountKey, 1 ApplicationDefault	0	How to sign in.
ServiceAccountKey	password		0: the whole service-account JSON key file. Required for 0, except with ServiceUrl (see Behaviour).
ServiceUrl	string		JSON API endpoint of a GCS emulator or a private endpoint, http(s) without query, for example http://localhost:4443; without a path /storage/v1/ is added. Empty: Google Cloud Storage.
BucketName	string		Required. Bucket to list.
Prefix	string		Optional name prefix, for example incoming/2026/. Empty lists all objects.
SkipTag	string		Custom metadata key; objects that have this metadata entry are not emitted (for example viaducta-processed, set by GCS reader post-processing 3 Tag). Read from the listing, no extra requests. Empty lists all.
TopLevelOnly	boolean	false	true lists only the objects directly under Prefix (end it with /), not those in deeper folders.
NamePattern	string	*	Only objects whose name after the last / matches; * and ? as wildcards, case-insensitive, for example *.csv.
MinAgeMinutes	integer	0	Only objects whose content was not written for at least this many minutes, so objects still being written are skipped. 0 lists all.
OrderBy	dropdown: 0 Name, 1 LastModified	0	Emit order: by name (as GCS lists, UTF-8 binary order) or oldest content first.
MaxObjects	integer	0	Emit at most this many objects per run, the first ones in OrderBy order. 0 emits all.

Behaviour

• The storage client is created at flow start; missing or invalid sign-in fields fail the start. 0 ServiceAccountKey reads the JSON key (only service-account keys are accepted). 1 ApplicationDefault uses Google's Application Default Credentials: the key file named by GOOGLE_APPLICATION_CREDENTIALS, the gcloud user credentials, workload identity (GKE, Cloud Run) or the service account of the VM; none found fails the start. Access tokens are fetched with the first call and renewed by the client. The identity needs storage.objects.list (role Storage Object Viewer).
• With ServiceUrl set and Authentication 0 without a key, requests are sent without credentials. This is meant for emulators such as fake-gcs-server, which check no sign-in; Google Cloud Storage rejects such requests (401).
• Per input the whole listing is read first (all levels, or with TopLevelOnly only the objects directly under Prefix), folder placeholders (names ending with /) are dropped, then the objects are filtered by NamePattern, MinAgeMinutes and SkipTag, ordered by OrderBy and cut to MaxObjects; then they are emitted. Changes downstream components make in the bucket during the run (for example GCS reader moving objects) don't change the list.
• Each object is emitted as ViaFileInfo: FileName = object name (including folders), FilePath = gs://<bucket>/<name> (every name segment URL-escaped, for example gs://imports/daily/my%20file.csv), Size = size in bytes, CreateDateTime = when the current content was written (UTC, the creation of the object's current generation; metadata changes don't move it), LastAccessDateTime = DateTime.MinValue. MinAgeMinutes and OrderBy 1 use the same time.
• The progress bar shows <emitted>/<total> objects (while a large bucket is listed: listing, <n> objects); the run ends with the status <emitted>/<total> objects listed.
• After the last object Continue is called with the input record. A missing bucket raises an error (HTTP 404). No record positions are set on the emitted records.

Notes

• Connect Output to GCS reader to read each listed object: the reader takes bucket and name from FilePath (leave its BucketName/ObjectName empty). With post-processing in the reader (delete, move, tag + SkipTag) a timer-driven flow processes each object once.
• GCS has no object tags: "tag" in these plugins means a custom metadata entry (x-goog-meta-<name>). The listing carries the metadata, so SkipTag is free, but tagged objects are still listed and filtered in memory; for large backlogs prefer reader post-processing 2 Move to a prefix outside Prefix, or a small MaxObjects.
• The listing is kept in memory for the run; use Prefix to narrow very large buckets. MaxObjects limits what is emitted, not what is listed.
• MaxObjects with OrderBy 1 and reader post-processing processes a backlog oldest first in slices, one slice per timer run.
• Store the service-account key as an expression setting and reference it (=Secrets.GcsServiceAccountKey) instead of pasting it into the flow.

Example

{ "Authentication": 1, "BucketName": "imports", "Prefix": "daily/", "SkipTag": "viaducta-processed", "NamePattern": "*.csv", "MinAgeMinutes": 5, "OrderBy": 1, "MaxObjects": 100 }

Timer → GCS enumerator → GCS reader → CSV decoder → ...