---
id: google-cloud--gcs-enumerator
title: "GCS enumerator"
summary: "Lists the objects of a Google Cloud Storage bucket, optionally filtered by name prefix, folder level, name pattern, age and a custom metadata entry, and emits one file info record per object. Use it with *GCS reader* to process the files of a bucket."
audience: user
keywords: "input, output, continue, authentication, serviceaccountkey, serviceurl, bucketname, prefix, skiptag, toplevelonly, namepattern, minageminutes, orderby, maxobjects"
---
# GCS enumerator

Lists the objects of a Google Cloud Storage bucket, optionally filtered by name prefix, folder level, name pattern, age and a custom metadata entry, and emits one file info record per object. Use it with *GCS reader* to process the files of a bucket.

## Pins

| Pin | Kind | Types | Description |
|---|---|---|---|
| Input | input | any | Starts the enumeration (for example a timer). |
| Output | output | `Viaducta.TypeSystem.Metadata.ViaFileInfo` | Called once per object. |
| Continue | continue output | same as input | Called with the input record after the last object. |

## Configuration

| Field | Type | Default | Description |
|---|---|---|---|
| `Authentication` | dropdown: `0` ServiceAccountKey, `1` ApplicationDefault | `0` | How to sign in. |
| `ServiceAccountKey` | password | | `0`: the whole service-account JSON key file. Required for `0`, except with `ServiceUrl` (see Behaviour). |
| `ServiceUrl` | string | | JSON API endpoint of a GCS emulator or a private endpoint, `http(s)` without query, for example `http://localhost:4443`; without a path `/storage/v1/` is added. Empty: Google Cloud Storage. |
| `BucketName` | string | | Required. Bucket to list. |
| `Prefix` | string | | Optional name prefix, for example `incoming/2026/`. Empty lists all objects. |
| `SkipTag` | string | | Custom metadata key; objects that have this metadata entry are not emitted (for example `viaducta-processed`, set by *GCS reader* post-processing `3` Tag). Read from the listing, no extra requests. Empty lists all. |
| `TopLevelOnly` | boolean | `false` | `true` lists only the objects directly under `Prefix` (end it with `/`), not those in deeper folders. |
| `NamePattern` | string | `*` | Only objects whose name after the last `/` matches; `*` and `?` as wildcards, case-insensitive, for example `*.csv`. |
| `MinAgeMinutes` | integer | `0` | Only objects whose content was not written for at least this many minutes, so objects still being written are skipped. `0` lists all. |
| `OrderBy` | dropdown: `0` Name, `1` LastModified | `0` | Emit order: by name (as GCS lists, UTF-8 binary order) or oldest content first. |
| `MaxObjects` | integer | `0` | Emit at most this many objects per run, the first ones in `OrderBy` order. `0` emits all. |

## Behaviour

- The storage client is created at flow start; missing or invalid sign-in fields fail the start. `0` ServiceAccountKey reads the JSON key (only service-account keys are accepted). `1` ApplicationDefault uses Google's Application Default Credentials: the key file named by `GOOGLE_APPLICATION_CREDENTIALS`, the gcloud user credentials, workload identity (GKE, Cloud Run) or the service account of the VM; none found fails the start. Access tokens are fetched with the first call and renewed by the client. The identity needs `storage.objects.list` (role Storage Object Viewer).
- With `ServiceUrl` set and `Authentication` `0` without a key, requests are sent without credentials. This is meant for emulators such as fake-gcs-server, which check no sign-in; Google Cloud Storage rejects such requests (401).
- Per input the whole listing is read first (all levels, or with `TopLevelOnly` only the objects directly under `Prefix`), folder placeholders (names ending with `/`) are dropped, then the objects are filtered by `NamePattern`, `MinAgeMinutes` and `SkipTag`, ordered by `OrderBy` and cut to `MaxObjects`; then they are emitted. Changes downstream components make in the bucket during the run (for example *GCS reader* moving objects) don't change the list.
- Each object is emitted as `ViaFileInfo`: `FileName` = object name (including folders), `FilePath` = `gs://<bucket>/<name>` (every name segment URL-escaped, for example `gs://imports/daily/my%20file.csv`), `Size` = size in bytes, `CreateDateTime` = when the current content was written (UTC, the creation of the object's current generation; metadata changes don't move it), `LastAccessDateTime` = `DateTime.MinValue`. `MinAgeMinutes` and `OrderBy` `1` use the same time.
- The progress bar shows `<emitted>/<total> objects` (while a large bucket is listed: `listing, <n> objects`); the run ends with the status `<emitted>/<total> objects listed`.
- After the last object `Continue` is called with the input record. A missing bucket raises an error (HTTP 404). No record positions are set on the emitted records.

## Notes

- Connect `Output` to *GCS reader* to read each listed object: the reader takes bucket and name from `FilePath` (leave its `BucketName`/`ObjectName` empty). With post-processing in the reader (delete, move, tag + `SkipTag`) a timer-driven flow processes each object once.
- GCS has no object tags: "tag" in these plugins means a custom metadata entry (`x-goog-meta-<name>`). The listing carries the metadata, so `SkipTag` is free, but tagged objects are still listed and filtered in memory; for large backlogs prefer reader post-processing `2` Move to a prefix outside `Prefix`, or a small `MaxObjects`.
- The listing is kept in memory for the run; use `Prefix` to narrow very large buckets. `MaxObjects` limits what is emitted, not what is listed.
- `MaxObjects` with `OrderBy` `1` and reader post-processing processes a backlog oldest first in slices, one slice per timer run.
- Store the service-account key as an expression setting and reference it (`=Secrets.GcsServiceAccountKey`) instead of pasting it into the flow.

## Example

```json
{ "Authentication": 1, "BucketName": "imports", "Prefix": "daily/", "SkipTag": "viaducta-processed", "NamePattern": "*.csv", "MinAgeMinutes": 5, "OrderBy": 1, "MaxObjects": 100 }
```

Timer → **GCS enumerator** → GCS reader → CSV decoder → ...