Google Cloud

Read and write objects in Google Cloud Storage. The GCS enumerator lists the objects of a bucket, the GCS reader opens one as a file stream (one batch per object, then delete, move or tag it), the GCS writer uploads text or a file stream, and GCS operation deletes, copies, moves, tags or changes the storage class of an object.

Timer → GCS enumerator → GCS reader → CSV decoder → ... your logic

The reader emits the same file stream the local file plugins use, so every decoder works on GCS objects as it does on local files. Sign in with a service-account JSON key or with Application Default Credentials (workload identity, the VM's service account).

Plugins

Plugin	What it does	Pins
GCS enumerator	Lists the objects of a Google Cloud Storage bucket, optionally filtered by name prefix, folder level, name pattern, age and a custom metadata entry, and emits one file info record per object. Use it with GCS reader to process the files of a bucket.	Input, Output, Continue
GCS operation	Deletes, copies, moves, tags or changes the storage class of an object in Google Cloud Storage. The object comes from an incoming object file info (GCS enumerator) or from the configuration; the file info of the resulting object is passed on. Use it for housekeeping (archive or delete old objects) or to move objects between buckets.	Input, Output, NoData
GCS reader	Reads an object from Google Cloud Storage and emits it as a file stream, for example for a CSV or ASN.1 decoder. Every object is a batch: once it is processed and committed downstream, the object can be deleted, moved or tagged; a failed object can be moved to an error location. The object comes from an incoming object file info (GCS enumerator) or from the configuration. If the object does not exist, the input is passed to NoData.	Input, Output, NoData
GCS writer	Uploads text or a file stream as an object to Google Cloud Storage and passes the input on. Sets content type, custom metadata and storage class, can refuse to overwrite and can create the bucket. Streams are uploaded in chunks as they are read.	Input, Output