Collect files

Downloads files from a remote server over FTP, FTPS or SFTP into a local folder of the environment data folder and emits a ViaFileInfo for each downloaded file. Optionally deletes or renames the remote file afterwards and remembers collected file names to skip duplicates.

Pins

Pin	Kind	Types	Description
Start	start		Starts one collection run (list, download, post-process). Typically connected to a Timer.
Output	output	Viaducta.TypeSystem.Metadata.ViaFileInfo	Called once per downloaded file, after the file is stored locally.
Continue	continue	NoInputData	Called once after the collection run finished, also when no file was collected.

Configuration

Field	Type	Default	Description
TransferProtocol	dropdown: 1 FTP, 2 SFTP, 3 FTPS	2	Protocol.
RemoteHost	string	localhost	Required. Host name or IP of the server.
Port	integer	0	0 = default port (21 for FTP/FTPS, 22 for SFTP).
UserName	string (max 256)		Login user.
Password	password (max 256)		Login password. For SFTP password authentication is used only when set.
ClientCredentials	string		Optional files on the Viaducta server, separated by ;. SFTP: private key files (without passphrase). FTP/FTPS: client certificate files.
RemotePath	string		Remote directory to collect from, e.g. /outgoing/orders. For FTP/FTPS it is required and must start with /.
RecurseSubDirs	boolean	false	Also collect from subdirectories.
FileNamePattern	string	*	File name mask, * matches any characters (*.csv, ORD_*.xml).
OlderThanHrs	number	0	SFTP only: collect only files last modified more than this many hours ago. Must be 0 for FTP/FTPS.
LocalFileGroup	folder (FolderExplorer)		Required. Local folder relative to the environment data folder, e.g. /inbound/orders. Created if missing.
DeduplicateFiles	boolean	false	true: every file name is collected only once (remembered across runs and restarts).
DuplicateRaiseError	boolean	false	With DeduplicateFiles: log an error when a duplicate is skipped.
DeduplicateHashSalt	string	MyFiles	Prefix of the stored duplicate key. Changing it "forgets" all previously collected names.
RemotePostProc	dropdown: 1 None, 2 Delete, 3 Rename	1	What to do with the remote file after it was processed.
RenameTemplate	string	%FILENAME%.processed	New remote name for 3 Rename. %FILENAME% = original name, %BASENAME% = name without extension.
LogFileInfo	boolean	false	Log name and size of every downloaded file.
BatchMode	dropdown: 1 Auto, 2 Always, 3 Never	1	Whether the files collected in one run form a batch. Auto: only if the input is not already in a batch (Timer). Always: a child batch per run. Never: no batch. Read file opens a child batch per file, so consumers behind it work per file by default; select Collect files in their BatchScope to get one output per run (e.g. one archive of all collected files).

Behaviour

• Every Start connects, lists RemotePath (and subdirectories with RecurseSubDirs; names from subfolders are relative, e.g. 2024/file.csv) and processes the files whose name matches FileNamePattern.
• For each matching file:
  1. With DeduplicateFiles, a file name that was collected before is skipped.
  2. The file is downloaded to a hidden temporary file and then renamed to <data folder>/<LocalFileGroup>/<file name>. An existing local file with the same name is replaced.
  3. Output is called with a ViaFileInfo: FileName (name with extension), FilePath (full local path including the file name), Size (bytes), CreateDateTime, LastAccessDateTime.
  4. After the downstream processing returns, RemotePostProc is applied to the remote file.
• The files of a run form a batch (BatchMode, opened on the first downloaded file) that ends after the last file was processed downstream; an error fails it and its consumers roll back.
• An error while downloading or in the downstream processing stops the run; the remaining files are collected on the next Start.
• After all files were processed Continue is called once with NoInputData (also when no file matched, then Output is not called). It is not called when the run stopped with an error.
• Progress: during a run the component shows done/total files (matching files, duplicates skipped count as done) and N/M files collected at the end.

Notes

• Output carries file info only, not the content. Connect Read file (Input pin) to get a ViaFileStream.
• Remote post-processing (delete / rename) runs per file right after its downstream processing returns, also when the batch of the run fails later: with a consumer bound to the run batch, a failed run loses the remote files of the already processed files. Use Rename instead of Delete if that matters.
• FileNamePattern is turned into a regular expression where only * is translated; . matches any character and other regex characters keep their regex meaning. With RecurseSubDirs the mask is matched against the relative path (*.csv also matches sub/x.csv).
• Files from subdirectories are stored flat in LocalFileGroup; equal names from different folders overwrite each other. Duplicate detection uses only the file name, without folder.
• Port: the field validator requires a value of at least 1, although 0 (default) is what selects the default port.
• FTPS accepts any server certificate (no certificate validation).
• SFTP with an empty RemotePath lists / but downloads relative to the login directory; always set RemotePath.
• %BASENAME% drops the subfolder, so with RecurseSubDirs a rename using it moves the file to RemotePath. Rename fails if the target already exists.
• Server host keys are accepted without verification for SFTP. Use expression settings (=Secrets.SftpPassword) for secrets.

Example

{ "TransferProtocol": 2, "RemoteHost": "sftp.partner.com", "UserName": "viaducta", "Password": "=Secrets.SftpPassword",
  "RemotePath": "/outgoing", "FileNamePattern": "*.csv", "LocalFileGroup": "/inbound/partner",
  "DeduplicateFiles": true, "RemotePostProc": 3, "RenameTemplate": "%FILENAME%.done" }

Timer → Collect files → Read file (Input) → Csv decoder → ...