---
id: file--collect-files
title: "Collect files"
summary: "Downloads files from a remote server over FTP, FTPS or SFTP into a local folder of the environment data folder and emits a `ViaFileInfo` for each downloaded file. Optionally deletes or renames the remote file afterwards and remembers collected file names to skip duplicates."
audience: user
keywords: "start, output, continue, transferprotocol, remotehost, port, username, password, clientcredentials, remotepath, recursesubdirs, filenamepattern, olderthanhrs, localfilegroup, deduplicatefiles, duplicateraiseerror, deduplicatehashsalt, remotepostproc, renametemplate, logfileinfo, batchmode"
---
# Collect files

Downloads files from a remote server over FTP, FTPS or SFTP into a local folder of the environment data folder and emits a `ViaFileInfo` for each downloaded file. Optionally deletes or renames the remote file afterwards and remembers collected file names to skip duplicates.

## Pins

| Pin | Kind | Types | Description |
|---|---|---|---|
| Start | start | | Starts one collection run (list, download, post-process). Typically connected to a Timer. |
| Output | output | `Viaducta.TypeSystem.Metadata.ViaFileInfo` | Called once per downloaded file, after the file is stored locally. |
| Continue | continue | `NoInputData` | Called once after the collection run finished, also when no file was collected. |

## Configuration

| Field | Type | Default | Description |
|---|---|---|---|
| `TransferProtocol` | dropdown: `1` FTP, `2` SFTP, `3` FTPS | `2` | Protocol. |
| `RemoteHost` | string | `localhost` | Required. Host name or IP of the server. |
| `Port` | integer | `0` | `0` = default port (21 for FTP/FTPS, 22 for SFTP). |
| `UserName` | string (max 256) | | Login user. |
| `Password` | password (max 256) | | Login password. For SFTP password authentication is used only when set. |
| `ClientCredentials` | string | | Optional files on the Viaducta server, separated by `;`. SFTP: private key files (without passphrase). FTP/FTPS: client certificate files. |
| `RemotePath` | string | | Remote directory to collect from, e.g. `/outgoing/orders`. For FTP/FTPS it is required and must start with `/`. |
| `RecurseSubDirs` | boolean | `false` | Also collect from subdirectories. |
| `FileNamePattern` | string | `*` | File name mask, `*` matches any characters (`*.csv`, `ORD_*.xml`). |
| `OlderThanHrs` | number | `0` | SFTP only: collect only files last modified more than this many hours ago. Must be `0` for FTP/FTPS. |
| `LocalFileGroup` | folder (FolderExplorer) | | Required. Local folder relative to the environment data folder, e.g. `/inbound/orders`. Created if missing. |
| `DeduplicateFiles` | boolean | `false` | `true`: every file name is collected only once (remembered across runs and restarts). |
| `DuplicateRaiseError` | boolean | `false` | With `DeduplicateFiles`: log an error when a duplicate is skipped. |
| `DeduplicateHashSalt` | string | `MyFiles` | Prefix of the stored duplicate key. Changing it "forgets" all previously collected names. |
| `RemotePostProc` | dropdown: `1` None, `2` Delete, `3` Rename | `1` | What to do with the remote file after it was processed. |
| `RenameTemplate` | string | `%FILENAME%.processed` | New remote name for `3` Rename. `%FILENAME%` = original name, `%BASENAME%` = name without extension. |
| `LogFileInfo` | boolean | `false` | Log name and size of every downloaded file. |
| `BatchMode` | dropdown: `1` Auto, `2` Always, `3` Never | `1` | Whether the files collected in one run form a batch. `Auto`: only if the input is not already in a batch (Timer). `Always`: a child batch per run. `Never`: no batch. Read file opens a child batch per file, so consumers behind it work per file by default; select Collect files in their `BatchScope` to get one output per run (e.g. one archive of all collected files). |

## Behaviour

- Every `Start` connects, lists `RemotePath` (and subdirectories with `RecurseSubDirs`; names from subfolders are relative, e.g. `2024/file.csv`) and processes the files whose name matches `FileNamePattern`.
- For each matching file:
  1. With `DeduplicateFiles`, a file name that was collected before is skipped.
  2. The file is downloaded to a hidden temporary file and then renamed to `<data folder>/<LocalFileGroup>/<file name>`. An existing local file with the same name is replaced.
  3. `Output` is called with a `ViaFileInfo`: `FileName` (name with extension), `FilePath` (full local path including the file name), `Size` (bytes), `CreateDateTime`, `LastAccessDateTime`.
  4. After the downstream processing returns, `RemotePostProc` is applied to the remote file.
- The files of a run form a batch (`BatchMode`, opened on the first downloaded file) that ends after the last file was processed downstream; an error fails it and its consumers roll back.
- An error while downloading or in the downstream processing stops the run; the remaining files are collected on the next `Start`.
- After all files were processed `Continue` is called once with `NoInputData` (also when no file matched, then `Output` is not called). It is not called when the run stopped with an error.
- Progress: during a run the component shows `done/total files` (matching files, duplicates skipped count as done) and `N/M files collected` at the end.

## Notes

- `Output` carries file info only, not the content. Connect **Read file** (`Input` pin) to get a `ViaFileStream`.
- Remote post-processing (delete / rename) runs per file right after its downstream processing returns, also when the batch of the run fails later: with a consumer bound to the run batch, a failed run loses the remote files of the already processed files. Use `Rename` instead of `Delete` if that matters.
- `FileNamePattern` is turned into a regular expression where only `*` is translated; `.` matches any character and other regex characters keep their regex meaning. With `RecurseSubDirs` the mask is matched against the relative path (`*.csv` also matches `sub/x.csv`).
- Files from subdirectories are stored flat in `LocalFileGroup`; equal names from different folders overwrite each other. Duplicate detection uses only the file name, without folder.
- `Port`: the field validator requires a value of at least 1, although `0` (default) is what selects the default port.
- FTPS accepts any server certificate (no certificate validation).
- SFTP with an empty `RemotePath` lists `/` but downloads relative to the login directory; always set `RemotePath`.
- `%BASENAME%` drops the subfolder, so with `RecurseSubDirs` a rename using it moves the file to `RemotePath`. Rename fails if the target already exists.
- Server host keys are accepted without verification for SFTP. Use expression settings (`=Secrets.SftpPassword`) for secrets.

## Example

```json
{ "TransferProtocol": 2, "RemoteHost": "sftp.partner.com", "UserName": "viaducta", "Password": "=Secrets.SftpPassword",
  "RemotePath": "/outgoing", "FileNamePattern": "*.csv", "LocalFileGroup": "/inbound/partner",
  "DeduplicateFiles": true, "RemotePostProc": 3, "RenameTemplate": "%FILENAME%.done" }
```

Timer → **Collect files** → Read file (`Input`) → Csv decoder → ...