S3 writer

Uploads text or a file stream as an object to Amazon S3 (or an S3-compatible store such as MinIO) and passes the input on. Sets content type, metadata and storage class, can refuse to overwrite and can create the bucket. Large streams are uploaded in parts.

Pins

Pin	Kind	Types	Description
Input	input	Viaducta.Plugins.TextPlugin.Types.TextRecord, Viaducta.Plugins.Files.Types.ViaFileStream	Content to store: TextRecord.Value (UTF-8) or the stream.
Output	output	same as input	Called with the input record after a successful write.

Configuration

Field	Type	Default	Description
Authentication	dropdown: 0 AccessKey, 1 DefaultChain, 2 AssumeRole	0	How to sign in.
AccessKeyId	string		Access key id. Required for 0. For 2 optional: with it the role is assumed with this key, without it with the default chain.
SecretAccessKey	password		Secret of AccessKeyId. Required with AccessKeyId.
SessionToken	password		Session token of temporary credentials; optional.
RoleArn	string		2: ARN of the role to assume. Required for 2.
ExternalId	string		2: external id the trust policy of the role requires; optional.
StsServiceUrl	string		2: STS endpoint. Empty: the regional AWS STS endpoint of Region.
Region	string		AWS region of the bucket, for example eu-central-1. Required when ServiceUrl is empty; with ServiceUrl the signing region (empty: us-east-1).
ServiceUrl	string		Endpoint of an S3-compatible store or an S3 VPC endpoint, http(s) without query. Empty: AWS S3 in Region.
ForcePathStyle	boolean	false	true addresses <endpoint>/<bucket>/<key>; most S3-compatible stores need it.
BucketName	string		Target bucket.
CreateBucket	boolean	false	true creates the bucket (in Region, private) when it does not exist.
Key	string		Target key including folders, for example exports/daily.csv. Ending with / it is a folder and the record's name is appended (see Behaviour).
Overwrite	boolean	true	true replaces an existing object; false fails the write (HTTP 412) when the object exists.
ContentType	string		Content type of the object, for example text/csv. Empty: the ContentType of the TextRecord, otherwise from the extension of the key (.csv, .json, .jsonl, .xml, .txt, .pdf, .zip, .gz, .parquet, images, ...), otherwise application/octet-stream.
StorageClass	dropdown: 0 Default, 1 Standard, 2 StandardInfrequentAccess, 3 OneZoneInfrequentAccess, 4 IntelligentTiering, 5 GlacierInstantRetrieval, 6 GlacierFlexibleRetrieval, 7 GlacierDeepArchive, 8 ReducedRedundancy	0	Storage class of the object; 0 sends none (the bucket default, Standard).
Metadata	dictionary		Name/value pairs stored as user metadata (x-amz-meta-<name>).

Behaviour

• The S3 client is created at flow start; missing sign-in fields or an unknown storage class fail the start. Sign-in modes as in S3 enumerator. The identity needs s3:PutObject (plus s3:ListBucket and s3:CreateBucket for CreateBucket).
• Key: for a TextRecord, its Name replaces Key, or is appended when Key ends with /; without a Name, Key is used. For a ViaFileStream, Key is used; when it is empty the stream's Name is used, and when it ends with / the stream's Name is appended. An empty result fails the record.
• With CreateBucket the bucket is created on the first write of the run if it is missing.
• Content up to 16 MB is sent in one PUT. A larger stream is uploaded in parts of 16 MB as it is read (no length or seeking needed); a failed multipart upload is aborted. Content type, metadata and storage class are set on the object.
• With Overwrite false the write is conditional (If-None-Match: *): an existing object fails it with HTTP 412 and stays unchanged.
• Then Output is called with the input record. Other input types fail with a "not supported" error. S3 errors (missing bucket, authentication, existing object with Overwrite false, a storage class the store doesn't offer) raise an error.

Notes

• ViaFileStream input: the content of Stream is uploaded from its current position; the stream stays open (it belongs to the component that opened it).
• S3 objects can't be appended to, so there is no append mode as in Blob writer: every record writes a whole object. For logs or JSON lines write one object per record (Key a folder, the record name unique) or collect the records into one stream first.
• At most 10,000 parts: one object can hold up to about 156 GB.
• Use ASCII for metadata names and values; S3 returns metadata names in lower case.
• Glacier classes (6, 7) take objects offline: they must be restored before they can be read. S3-compatible stores offer fewer classes (MinIO: Standard and Reduced Redundancy).
• Store secrets as expression settings (=Secrets.AwsSecretKey).

Example

{ "Authentication": 1, "Region": "eu-central-1", "BucketName": "exports", "CreateBucket": true, "Key": "daily/", "StorageClass": 2, "Metadata": { "source": "erp" } }

Timer → Sql query → Json serializer → S3 writer → Log (the TextRecord.Name of the serializer becomes the file in daily/)