GCS writer

Uploads text or a file stream as an object to Google Cloud Storage and passes the input on. Sets content type, custom metadata and storage class, can refuse to overwrite and can create the bucket. Streams are uploaded in chunks as they are read.

Pins

Pin	Kind	Types	Description
Input	input	Viaducta.Plugins.TextPlugin.Types.TextRecord, Viaducta.Plugins.Files.Types.ViaFileStream	Content to store: TextRecord.Value (UTF-8) or the stream.
Output	output	same as input	Called with the input record after a successful write.

Configuration

Field	Type	Default	Description
Authentication	dropdown: 0 ServiceAccountKey, 1 ApplicationDefault	0	How to sign in.
ServiceAccountKey	password		0: the whole service-account JSON key file. Required for 0, except with ServiceUrl.
ServiceUrl	string		JSON API endpoint of a GCS emulator or a private endpoint, for example http://localhost:4443. Empty: Google Cloud Storage.
BucketName	string		Target bucket.
CreateBucket	boolean	false	true creates the bucket when it does not exist. Needs ProjectId.
ProjectId	string		CreateBucket: id of the Google Cloud project the bucket is created in, for example my-project-123456. Required with CreateBucket.
BucketLocation	string		CreateBucket: location of a new bucket, for example EU, europe-west3. Empty: Google's default (US).
ObjectName	string		Target object name including folders, for example exports/daily.csv. Ending with / it is a folder and the record's name is appended (see Behaviour).
Overwrite	boolean	true	true replaces an existing object; false fails the write (HTTP 412) when the object exists.
ContentType	string		Content type of the object, for example text/csv. Empty: the ContentType of the TextRecord, otherwise from the extension of the name (.csv, .json, .jsonl, .xml, .txt, .pdf, .zip, .gz, .parquet, images, ...), otherwise application/octet-stream.
StorageClass	dropdown: 0 Default, 1 Standard, 2 Nearline, 3 Coldline, 4 Archive	0	Storage class of the object; 0 sends none (the bucket's default class).
Metadata	dictionary		Name/value pairs stored as custom metadata (x-goog-meta-<name>).

Behaviour

• The storage client is created at flow start; missing sign-in fields, an unknown storage class or CreateBucket without ProjectId fail the start. Sign-in modes and the credential-less emulator access as in GCS enumerator. The identity needs storage.objects.create (plus storage.objects.delete to overwrite, and storage.buckets.get and storage.buckets.create for CreateBucket).
• Object name: for a TextRecord, its Name replaces ObjectName, or is appended when ObjectName ends with /; without a Name, ObjectName is used. For a ViaFileStream, ObjectName is used; when it is empty the stream's Name is used, and when it ends with / the stream's Name is appended. An empty result fails the record.
• With CreateBucket the bucket is created in ProjectId at BucketLocation on the first write of the run if it is missing.
• The content is sent as a resumable upload in chunks as it is read (no length or seeking needed); the client checks the hash of the stored object. Content type, metadata and storage class are set on the object.
• With Overwrite false the write is create-only (ifGenerationMatch=0): an existing object fails it with HTTP 412 and stays unchanged.
• Then Output is called with the input record. Other input types fail with a "not supported" error. GCS errors (missing bucket, authentication, existing object with Overwrite false) raise an error.

Notes

• ViaFileStream input: the content of Stream is uploaded from its current position; the stream stays open (it belongs to the component that opened it).
• GCS objects can't be appended to, so there is no append mode as in Blob writer: every record writes a whole object. For logs or JSON lines write one object per record (ObjectName a folder, the record name unique) or collect the records into one stream first.
• Nearline, Coldline and Archive have minimum storage durations (30, 90, 365 days) and retrieval fees; deleting or replacing an object earlier is charged for the rest of the period.
• Bucket names are global across all of Google Cloud; CreateBucket fails when another project owns the name.
• Store the service-account key as an expression setting (=Secrets.GcsServiceAccountKey).

Example

{ "Authentication": 1, "BucketName": "exports", "CreateBucket": true, "ProjectId": "my-project-123456", "BucketLocation": "EU", "ObjectName": "daily/", "StorageClass": 2, "Metadata": { "source": "erp" } }

Timer → Sql query → Json serializer → GCS writer → Log (the TextRecord.Name of the serializer becomes the file in daily/)