Bulk insert

Buffers incoming records and writes them to a database table with the database's bulk load mechanism (minimally logged, much faster than row-by-row inserts). Use it for high-volume ingestion, for example after a file decoder or a mapper that produces many records.

Pins

Pin	Kind	Types	Description
Input	input	generated table type (<flow namespace>.<TypePrefix><Table>Table)	Records to insert. The type is generated from the table when TableName is set.
Continue	continue output	Viaducta.TypeSystem.Metadata.NoInputData	Batch: called once when the batch was written. Without batch: called after every successful flush (see Behaviour).

Configuration

Field	Type	Default	Description
ConnectionString	database link name		Required. Name of a database link from the repository (picked with the link selector). The value is looked up by name exactly; a raw connection string or a #name prefix is not resolved.
TableName	string (database explorer)		Required. Target table, table or schema.table. Without a schema the link's default schema is used. Triggers reconfigure.
TypePrefix	string	DB_	Prefix of the generated input type name.
AutoTruncate	boolean	false	Not used by the plugin (see Notes).
CommandTimeOut	integer (seconds)	20	Command timeout of every flush.
BatchSize	integer (1-10000000)	10000	Number of buffered rows that forces a flush. Values of 10 or less are raised to 10.
AutoFlushTimeout	integer (ms, 50-10000)	100	Idle time after the last added row after which buffered rows are flushed. Values of 10 or less become 100.
BatchScope	batch scope selector	empty	Which batch is inserted together. Empty = the closest batch producer upstream (e.g. one file read by Read file, one Sql query execution). A component id of a batch producer upstream = the batches of that producer. None = ignore batches.
KeepOriginalNames	boolean	false	true keeps the database column names as property names (and the type name unconverted); false converts them to C# names.

Behaviour

• Reconfigure (changing TableName) reads the table definition, generates a locked flow type named TypePrefix + table name + Table suffix (suffix omitted if the table name already ends with Table; converted to a C# name unless KeepOriginalNames), removes the types the component created before and sets the new type on Input. A component still carrying its default name is renamed to BLK_<TableName>.
• Changing TypePrefix or KeepOriginalNames does not regenerate the type: re-select the table afterwards.
• Records in a batch (see BatchScope) are buffered per batch. When the whole batch has been processed the remaining rows are flushed and Continue is called once with the metadata of the last record. When the batch failed the buffered rows are discarded. BatchSize still bounds the memory: a full buffer is flushed before the batch ends, and those rows stay in the table even if the batch fails later. For all-or-nothing per batch set BatchSize above the largest batch. A failed flush fails the batch, the rest of its rows are dropped. Batches are flushed independently of each other; AutoFlushTimeout doesn't apply to them. If the selected producer is not upstream of a record, the record fails.
• Records without a batch (or BatchScope None) are added to an in-memory buffer. A flush (bulk write of the buffered rows) happens when:
  • the buffer holds BatchSize rows,
  • no row was added for AutoFlushTimeout ms (background timer).
• After each flush Continue is called once: with NoInputData for flushes triggered by a record, with the last received input record for timer flushes.
• The non batch buffer is shared by all callers of the component.

Notes

• AutoTruncate is ignored: strings longer than the column cause the database to reject the batch.
• Put a batch producer upstream for exact boundaries.
• The input type comes from the incoming connection. If the component has no typed input connection the bulk provider is not created and the first record fails.
• If a flush fails, the error is raised in the record that triggered it (or in the timer for timer flushes); buffered rows of that batch are not retried.
• Downstream logic on Continue runs once per batch, not once per record.

Example

{ "ConnectionString": "MainDb", "TableName": "dbo.Measurements", "TypePrefix": "DB_", "BatchSize": 5000, "AutoFlushTimeout": 500 }

File reader → CSV decoder → Data mapper → Bulk insert → (Continue) Log