File operation

Housekeeping component that keeps the data and archive folders tidy. Each call scans a file group and moves, copies, compresses, decompresses or deletes every matching file. It is not part of the data path: it emits no file data, only Continue when it's done. Typically connected to a Timer.

Pins

Pin	Kind	Types	Description
Input	input	Viaducta.TypeSystem.Metadata.NoInputData	Starts the scan of SourceFileGroup and processes every matching file.
Continue	continue	same as input	Called once after all files were processed (also when no file matched).

Configuration

Field	Type	Default	Description
Operation	dropdown, see below	1 MoveToArchive	Operation executed over every file.
SourceLocation	dropdown: 0 DataFolder, 1 ArchiveFolder	0	Root folder of SourceFileGroup.
SourceFileGroup	folder (FolderExplorer)		Folder to scan, relative to SourceLocation, e.g. /inbound/orders. Required (the whole data or archive folder can't be used). Created if missing.
FileNamePattern	string	*	File mask with * and ? wildcards (*.csv). Empty = all files.
RecurseSubDirs	checkbox	false	Scan subfolders too. The relative subfolder structure is kept in the target.
OlderThanHours	integer	0	Process only files whose last write time is at least this many hours ago. 0 = all files.
OrderBy	dropdown: 0 CreateTime, 1 FileName	0	Processing order: oldest creation time first, or by path ascending.
TargetFileGroup	folder (FolderExplorer)		Target folder for all ...ToGroup (relative to the data folder) and ...ToArchive (relative to the archive folder) operations. Required for them; created if missing.
TargetNamePattern	string	%SN%	Target file name: %SN% source file name, %BN% source name without extension, %TS% timestamp yyyyMMdd-HHmmss, %DATE% yyyyMMdd, %TIME% HHmmss (UTC).
ExistingFileOption	dropdown: 0 Fail, 1 Overwrite, 2 Rename	0	When the target file exists: Fail stops with an error, Overwrite replaces it, Rename writes <name>_yyyyMMdd_HHmmss<ext> (plus _1, _2, ... if needed).
SourceAfterProcessing	dropdown: 0 Keep, 1 Delete, 2 MoveToArchive	1	What happens to the source file after CompressToGroup, CompressToArchive, DecompressToGroup or DecompressToArchive.
SourceArchiveFileGroup	folder (FolderExplorer)		Archive folder (relative to the archive folder) for SourceAfterProcessing = MoveToArchive. Required for that option.

Operations

Value	Operation	Result
0	MoveToGroup	moved to TargetFileGroup in the data folder
1	MoveToArchive	moved to TargetFileGroup/yyyyMMdd in the archive folder
2	CopyToGroup	copied to TargetFileGroup in the data folder
3	CopyToArchive	copied to TargetFileGroup/yyyyMMdd in the archive folder
4	CompressToGroup	gzipped to TargetFileGroup as <name>.gz, source handled by SourceAfterProcessing
5	CompressToArchive	gzipped to TargetFileGroup/yyyyMMdd in the archive, source handled by SourceAfterProcessing
6	CompressInPlace	gzipped next to the source as <source name>.gz, source deleted
7	DecompressToGroup	extracted to TargetFileGroup, source handled by SourceAfterProcessing
8	DecompressToArchive	extracted to TargetFileGroup/yyyyMMdd in the archive, source handled by SourceAfterProcessing
9	Delete	deleted

Behaviour

• The file list is built once per call: files matching FileNamePattern and OlderThanHours, ordered by OrderBy. Skipped: files and folders whose name starts with ., subfolders named temp or tmp, and partially written files of this component (*.fileop.tmp).
• If a scan is still running when the next call arrives, the call returns immediately and Continue is not called.
• The archive date folder yyyyMMdd is today's local date.
• Copy, compress and decompress write <target>.fileop.tmp first and rename it at the end, so other components never see a partially written file.
• Compress uses gzip. Hidden files and files that are already compressed (.gz, .tgz, .tar.gz, .zip, .7z, .rar) are skipped.
• Decompress detects the format from the extension (case-insensitive): .gz (single file, named by TargetNamePattern applied to the name without .gz), .tar.gz / .tgz, .tar, .zip, .7z, .rar. Multi-file archives keep their entry names and folders; TargetNamePattern is not used for them, ExistingFileOption applies per entry. An entry that would be written outside the target folder fails the file. Other files and hidden files are skipped.
• SourceAfterProcessing = MoveToArchive moves the source under its own name to SourceArchiveFileGroup/yyyyMMdd in the archive folder.
• If SourceFileGroup was removed while the flow is running, it is recreated and the call finds no files.
• Errors: the first failed file stops the scan with FileOperationException (operation, file path, cause). Like every component error, the engine passes it to error handling wrapped in RecoverableFlowException; the FileOperationException is its InnerException. Files already processed stay processed; the remaining files are handled on the next call.

Notes

• Password-protected archives are not supported.
• Write file writes <name>.tmp while writing; use OlderThanHours or a pattern that excludes *.tmp when cleaning a folder Write file writes to.
• With SourceLocation = ArchiveFolder and RecurseSubDirs, a ...ToArchive target inside the source folder is picked up again on the next scan.
• The folder picker browses the data folder; with archive source or targets the path is used relative to the archive folder.

Examples

Archive processed files older than 1 day, compressed:

{ "Operation": 5, "SourceFileGroup": "/inbound/processed", "OlderThanHours": 24, "TargetFileGroup": "/inbound", "SourceAfterProcessing": 1 }

Delete archive files older than 90 days:

{ "Operation": 9, "SourceLocation": 1, "SourceFileGroup": "/inbound", "RecurseSubDirs": true, "OlderThanHours": 2160 }

Timer → File operation → Log