S3 operation

Deletes, copies, moves, tags or changes the storage class of an object in Amazon S3 (or an S3-compatible store such as MinIO). The object comes from an incoming object file info (S3 enumerator) or from the configuration; the file info of the resulting object is passed on. Use it for housekeeping (archive or delete old objects) or to move objects between buckets.

Pins

Pin	Kind	Types	Description
Input	input	Viaducta.TypeSystem.Metadata.ViaFileInfo	Triggers the operation. A ViaFileInfo whose FilePath is an s3:// URL selects the object; any other input uses BucketName/Key.
Output	output	Viaducta.TypeSystem.Metadata.ViaFileInfo	Called after the operation with the object's file info: the target of a copy or move, otherwise the object itself.
NoData	no-data output	same as input	Called with the input record when the bucket or object does not exist (HTTP 404).

Configuration

Field	Type	Default	Description
Authentication	dropdown: 0 AccessKey, 1 DefaultChain, 2 AssumeRole	0	How to sign in.
AccessKeyId	string		Access key id. Required for 0. For 2 optional: with it the role is assumed with this key, without it with the default chain.
SecretAccessKey	password		Secret of AccessKeyId. Required with AccessKeyId.
SessionToken	password		Session token of temporary credentials; optional.
RoleArn	string		2: ARN of the role to assume. Required for 2.
ExternalId	string		2: external id the trust policy of the role requires; optional.
StsServiceUrl	string		2: STS endpoint. Empty: the regional AWS STS endpoint of Region.
Region	string		AWS region of the bucket, for example eu-central-1. Required when ServiceUrl is empty; with ServiceUrl the signing region (empty: us-east-1).
ServiceUrl	string		Endpoint of an S3-compatible store or an S3 VPC endpoint, http(s) without query. Empty: AWS S3 in Region.
ForcePathStyle	boolean	false	true addresses <endpoint>/<bucket>/<key>; most S3-compatible stores need it.
Operation	dropdown: 0 Delete, 1 Copy, 2 Move, 3 Tag, 4 ChangeStorageClass	0	What to do with the object.
BucketName	string		Bucket of the object, used when the input is not an object ViaFileInfo.
Key	string		Object key including folders, used when the input is not an object ViaFileInfo.
TargetBucket	string		Copy/Move: target bucket (must exist); empty keeps the object's bucket.
TargetNamePattern	string	%SN%	Copy/Move: target key. %SN% source key with folders, %FN% its last part (after the last /), %BN% the last part without extension, %TS% yyyyMMdd-HHmmss, %DATE% yyyyMMdd, %TIME% HHmmss (UTC).
ExistingObjectOption	dropdown: 0 Fail, 1 Overwrite, 2 Rename	0	Copy/Move when the target exists: fail (HTTP 412), replace it, or write to <name>_yyyyMMdd_HHmmss<ext> (plus _1, _2, ... if that exists too).
Tag	string	viaducta-processed	Tag: object tag to add, name or name=value; without a value the UTC time is the value. Other tags are kept.
StorageClass	dropdown: 1 Standard, 2 StandardInfrequentAccess, 3 OneZoneInfrequentAccess, 4 IntelligentTiering, 5 GlacierInstantRetrieval, 6 GlacierFlexibleRetrieval, 7 GlacierDeepArchive, 8 ReducedRedundancy	2	ChangeStorageClass: the new storage class.

Behaviour

• The S3 client is created at flow start; missing sign-in fields, an unknown operation or existing-object option, Tag without a name or ChangeStorageClass without a class fail the start. Sign-in modes as in S3 enumerator. The identity needs the S3 permissions of the operation (s3:DeleteObject; s3:GetObject, s3:PutObject, s3:GetObjectTagging, s3:PutObjectTagging for copies; s3:DeleteObject too for Move).
• Per input the object is chosen (from the ViaFileInfo s3:// URL or BucketName/Key; with neither the input fails) and its metadata is read (HEAD). A 404 calls NoData with the input record.
• 0 Delete deletes the object. 1 Copy copies it server-side to <TargetBucket or the same bucket>/<TargetNamePattern> with content type, metadata, tags and storage class. 2 Move copies, then deletes the source. 3 Tag adds Tag to the object's tags. 4 ChangeStorageClass copies the object onto itself with the new class (content, metadata and tags stay; the last modification time changes); an object that already has the class is left as it is.
• Delete, Copy, Move and ChangeStorageClass only act on the version whose metadata was read: an object changed meanwhile fails with HTTP 412.
• Output is called with a ViaFileInfo (FileName = key, FilePath = s3://<bucket>/<key>, size and last modification): of the target for Copy and Move, of the object otherwise (for Delete the object that was deleted). S3 errors raise an error.

Notes

• Copy and move stay within the endpoint, region and sign-in; the target bucket must exist. Objects over 5 GB are copied in parts of 512 MB (content type, metadata and tags are carried over; other headers such as Cache-Control are not).
• AWS S3 checks ExistingObjectOption 0 and 2 atomically (If-None-Match on the copy) and the version on Delete (If-Match). Some S3-compatible stores (MinIO) ignore these headers; there a HEAD request right before does the check, which leaves a short window.
• Objects in Glacier Flexible Retrieval or Deep Archive must be restored before they can be copied, moved or changed to another class (S3 returns InvalidObjectState); restoring is not part of this plugin. S3 allows at most 10 tags per object.
• In a versioned bucket Delete and Move leave a delete marker, and ChangeStorageClass adds a version.
• Store secrets as expression settings (=Secrets.AwsSecretKey).

Example

{ "Authentication": 1, "Region": "eu-central-1", "Operation": 2, "TargetBucket": "archive", "TargetNamePattern": "%DATE%/%FN%", "ExistingObjectOption": 2 }

Timer → S3 enumerator (OrderBy 1, MinAgeMinutes 1440) → S3 operation → Log