Blob reader

Opens a blob in Azure Storage for reading and emits it as a file stream, for example for a CSV or ASN.1 decoder. Every blob is a batch: once it is processed and committed downstream, the blob can be deleted, moved or tagged; a failed blob can be moved to an error location. The blob comes from an incoming blob file info (Blob enumerator) or from the configuration. If the blob does not exist, the input is passed to NoData.

Pins

Pin	Kind	Types	Description
Input	input	Viaducta.TypeSystem.Metadata.ViaFileInfo	Triggers the read. A ViaFileInfo whose FilePath is a blob URL selects the blob; any other input reads the configured blob.
Output	output	Viaducta.Plugins.Files.Types.ViaFileStream	The opened blob stream with name, size and dates, sent in the batch of the blob.
NoData	no-data output	same as input	Called with the input record when the container or blob does not exist (HTTP 404).

Configuration

Field	Type	Default	Description
Authentication	dropdown: 0 ConnectionString, 1 EntraId, 2 ServicePrincipal, 3 SasToken	0	How to sign in to the storage account.
ConnectionString	string		Storage connection string. Required when Authentication is 0.
AccountUrl	string		Blob endpoint, for example https://myaccount.blob.core.windows.net, no query. Required when Authentication is 1, 2 or 3.
TenantId	string		Entra ID tenant id. Required for 2, optional for 1.
ClientId	string		2: client id of the app registration (required). 1: client id of a user-assigned managed identity (optional).
ClientSecret	password		Client secret of the app registration. Required for 2.
SasToken	password		Account or container SAS token (a leading ? is ignored). Required for 3.
ContainerName	string		Container of the blob, used when the input is not a blob ViaFileInfo.
BlobName	string		Blob name including virtual folders, for example daily/export.csv, used when the input is not a blob ViaFileInfo.
AllowModifications	boolean	true	true lets the read continue if the blob is changed meanwhile; false makes the read fail when the blob changes during reading.
PostProcessing	dropdown: 0 None, 1 Delete, 2 Move, 3 Tag	0	What happens to the blob after its batch committed.
PostProcessContainer	string		2 Move: target container (must exist); empty keeps the blob's container.
PostProcessPrefix	string		2 Move: put in front of the blob name, for example done/. Move needs this or PostProcessContainer.
PostProcessTag	string	viaducta-processed	3 Tag: blob index tag to set, name or name=value; without a value the UTC time (yyyy-MM-ddTHH:mm:ssZ) is the value.
ErrorContainer	string		Container a failed blob is moved to (must exist); empty keeps the blob's container.
ErrorPrefix	string		Put in front of the name of a failed blob, for example error/. With ErrorContainer and ErrorPrefix empty, failed blobs stay where they are.

Behaviour

• The storage client is created at flow start; post-processing settings are checked then (Move without target, Tag without name fail the start). 1 EntraId uses the default Azure credential (managed identity of the host, workload identity, environment variables or Azure CLI). The identity needs Storage Blob Data Contributor (Storage Blob Data Owner for Tag).
• Per input the blob to read is chosen: when the input is a ViaFileInfo whose FilePath is an http(s) blob URL (<account URL>/<container>/<blob>, as emitted by Blob enumerator), container and blob are taken from that URL; otherwise ContainerName/BlobName are used. The account always comes from the sign-in. With neither, the input fails with an error.
• A 404 when opening the blob calls NoData with the input record (no batch); other errors (authentication, network) raise an error.
• Otherwise a batch labelled with the blob name is opened and Output is called with a ViaFileStream in that batch: Stream (a read stream that downloads on demand), FilePath = <container>/<blob>, Name = blob name, Extension from the blob name, Size = content length, CreateDateTime = creation time (UTC), LastAccessTime = last modification time (UTC). When Output returns the stream is closed and the batch completes: downstream sinks (database writers, ...) commit.
• After the commit the blob is post-processed: 1 Delete deletes it with its snapshots; 2 Move copies it within the account to <PostProcessContainer or the same container>/<PostProcessPrefix><blob name> and deletes the original; 3 Tag adds PostProcessTag to the blob's index tags (other tags are kept). Delete and Move only act on the version that was read: if the blob was changed meanwhile they fail with HTTP 412 and the blob is kept. A post-processing error raises an error; the data is already committed and the blob is read again by the next run.
• When Output or the commit fails, the batch fails (sinks roll back) and the blob is not post-processed. With ErrorContainer or ErrorPrefix set it is moved to <ErrorContainer or the same container>/<ErrorPrefix><blob name> (a failed move is logged). The error is raised in both cases.

Notes

• Downstream components must consume the stream before Output returns (decoders do); the stream is closed afterwards.
• Moving within the listed container: give the target a prefix outside the enumerator's Prefix (for example list in/, move to done/), or the next run lists the moved blobs again. Tag + the enumerator's SkipTag keeps the blobs in place.
• With 3 SasToken the token needs read, write, delete (and tag) permissions for post-processing.
• Store secrets as expression settings (=Secrets.AzureStorage).

Example

{ "Authentication": 1, "AccountUrl": "https://myaccount.blob.core.windows.net", "PostProcessing": 2, "PostProcessPrefix": "done/", "ErrorPrefix": "error/" }

Timer → Blob enumerator (Prefix in/) → Blob reader → CSV decoder → Table writer (container and blob from the enumerator)

Timer → Blob reader (ContainerName, BlobName set) → CSV decoder → Table writer; (NoData) Log