GCS operation

Deletes, copies, moves, tags or changes the storage class of an object in Google Cloud Storage. The object comes from an incoming object file info (GCS enumerator) or from the configuration; the file info of the resulting object is passed on. Use it for housekeeping (archive or delete old objects) or to move objects between buckets.

Pins

Pin	Kind	Types	Description
Input	input	Viaducta.TypeSystem.Metadata.ViaFileInfo	Triggers the operation. A ViaFileInfo whose FilePath is a gs:// URL selects the object; any other input uses BucketName/ObjectName.
Output	output	Viaducta.TypeSystem.Metadata.ViaFileInfo	Called after the operation with the object's file info: the target of a copy or move, otherwise the object itself.
NoData	no-data output	same as input	Called with the input record when the bucket or object does not exist (HTTP 404).

Configuration

Field	Type	Default	Description
Authentication	dropdown: 0 ServiceAccountKey, 1 ApplicationDefault	0	How to sign in.
ServiceAccountKey	password		0: the whole service-account JSON key file. Required for 0, except with ServiceUrl.
ServiceUrl	string		JSON API endpoint of a GCS emulator or a private endpoint, for example http://localhost:4443. Empty: Google Cloud Storage.
Operation	dropdown: 0 Delete, 1 Copy, 2 Move, 3 Tag, 4 ChangeStorageClass	0	What to do with the object.
BucketName	string		Bucket of the object, used when the input is not an object ViaFileInfo.
ObjectName	string		Object name including folders, used when the input is not an object ViaFileInfo.
TargetBucket	string		Copy/Move: target bucket (must exist); empty keeps the object's bucket.
TargetNamePattern	string	%SN%	Copy/Move: target name. %SN% source name with folders, %FN% its last part (after the last /), %BN% the last part without extension, %TS% yyyyMMdd-HHmmss, %DATE% yyyyMMdd, %TIME% HHmmss (UTC).
ExistingObjectOption	dropdown: 0 Fail, 1 Overwrite, 2 Rename	0	Copy/Move when the target exists: fail (HTTP 412), replace it, or write to <name>_yyyyMMdd_HHmmss<ext> (plus _1, _2, ... if that exists too).
Tag	string	viaducta-processed	Tag: custom metadata entry to set, name or name=value; without a value the UTC time is the value. Other metadata is kept.
StorageClass	dropdown: 1 Standard, 2 Nearline, 3 Coldline, 4 Archive	2	ChangeStorageClass: the new storage class.

Behaviour

• The storage client is created at flow start; missing sign-in fields, an unknown operation or existing-object option, Tag without a name or ChangeStorageClass without a class fail the start. Sign-in modes and the credential-less emulator access as in GCS enumerator. The identity needs the permissions of the operation (storage.objects.get always; storage.objects.delete; storage.objects.create for copies; storage.objects.update for Tag).
• Per input the object is chosen (from the ViaFileInfo gs:// URL or BucketName/ObjectName; with neither the input fails) and its metadata is read. A 404 calls NoData with the input record.
• 0 Delete deletes the object. 1 Copy copies it server-side (rewrite) to <TargetBucket or the same bucket>/<TargetNamePattern> with content type, metadata and storage class. 2 Move copies, then deletes the source. 3 Tag sets Tag as a custom metadata entry. 4 ChangeStorageClass rewrites the object onto itself with the new class (content, content type and metadata stay; a new generation is written); an object that already has the class is left as it is.
• Every operation only acts on the generation whose metadata was read: an object replaced meanwhile fails with HTTP 412.
• Output is called with a ViaFileInfo (FileName = name, FilePath = gs://<bucket>/<name>, size and the time the content was written): of the target for Copy and Move, of the object otherwise (for Delete the object that was deleted). GCS errors raise an error.

Notes

• Copy and move stay within the endpoint and sign-in; the target bucket must exist. Large objects are copied in several rewrite calls, which the client repeats until done.
• Google Cloud Storage checks ExistingObjectOption 0 and 2 atomically (ifGenerationMatch=0 on the copy) and the source generation (ifSourceGenerationMatch, ifGenerationMatch). The plugin also checks with a read right before, so emulators that ignore these conditions (fake-gcs-server) behave the same, with a short window.
• GCS has no object tags; Tag writes custom metadata (x-goog-meta-<name>), which GCS enumerator's SkipTag reads from its listing.
• Nearline, Coldline and Archive have minimum storage durations (30, 90, 365 days): moving an object out of such a class or deleting it early is charged for the rest of the period. Changing the class of a large object rewrites all its bytes.
• In a bucket with object versioning Delete and Move keep the old generation as a noncurrent version, and ChangeStorageClass adds one.
• Store the service-account key as an expression setting (=Secrets.GcsServiceAccountKey).

Example

{ "Authentication": 1, "Operation": 2, "TargetBucket": "archive", "TargetNamePattern": "%DATE%/%FN%", "ExistingObjectOption": 2 }

Timer → GCS enumerator (OrderBy 1, MinAgeMinutes 1440) → GCS operation → Log