---
id: llm--llm-chat-client
title: "LLM chat client"
summary: "Sends the input text to a large language model on an OpenAI-compatible endpoint (vLLM, Ollama, llama.cpp, OpenAI, Gemini's OpenAI endpoint, ...) and emits the model's answer. Tools defined with **AI tool** components can be attached, so the model can call flow logic while answering."
audience: user
keywords: "register, input, output, endpoint, apikey, modelname, modelsettings, prompt, sessionhandling, sessionkey, idlesessiontimeout, key, temperature, top_p, top_k, min_p, max_tokens, presence_penalty, frequency_penalty, repetition_penalty, seed, stop, reasoning_effort, any other key"
---
# LLM chat client

Sends the input text to a large language model on an OpenAI-compatible endpoint (vLLM, Ollama, llama.cpp, OpenAI, Gemini's OpenAI endpoint, ...) and emits the model's answer. Tools defined with **AI tool** components can be attached, so the model can call flow logic while answering.

## Pins

| Pin | Kind | Types | Description |
|---|---|---|---|
| Register | input | `Viaducta.Plugins.Llm.Types.LlmToolRegistrationRequest` | Connect the `Register` pin of AI tool components here. Only the connection matters; no data is sent through it. |
| Input | input | `Viaducta.Plugins.TextPlugin.Types.TextRecord`, `Viaducta.Plugins.Http.Types.WebHttpRequest`, `Viaducta.TypeSystem.Metadata.NoInputData` | The user message: `TextRecord.Value`, or the request `Body` (falls back to `ByteBody` as UTF-8). |
| Output | output | `Viaducta.Plugins.TextPlugin.Types.TextRecord` | The model's answer in `Value` (`Name` and `ContentType` are empty). |

## Configuration

| Field | Type | Default | Description |
|---|---|---|---|
| `Endpoint` | string | `http://localhost:8000/v1` | Base URL of the OpenAI-compatible API, ending in `/v1` (e.g. `https://api.openai.com/v1`, `https://generativelanguage.googleapis.com/v1beta/openai/`). Required. |
| `ApiKey` | password | empty | API key of the endpoint; empty when it needs none (sent as `EMPTY`). Use a setting (`=Secrets.LlmKey`). |
| `ModelName` | string | empty | Model name served by the endpoint (e.g. `gpt-5-mini`, `Qwen/Qwen3-32B`). Required. |
| `ModelSettings` | JSON object | empty | Request settings: chat completions fields, see below. Empty = the endpoint's defaults. |
| `Prompt` | multi-line text | empty | System instructions for the model (role, rules, output format, when to use which tool). |
| `SessionHandling` | dropdown: `1` SESSION_NOT_ALLOWED, `2` SESSION_KEY | `1` | `1`: every message is an independent conversation. `2`: conversation history is kept per `SessionKey`. |
| `SessionKey` | string | empty | Only when `SessionHandling` is `2`: messages with the same key continue the same conversation. |
| `IdleSessionTimeout` | integer seconds, 0-604800 | `3600` | Conversations unused for this long are discarded; `0` keeps them until the flow stops. |

`ModelSettings` keys (all optional; a key that is absent is not sent):

| Key | Type | Description |
|---|---|---|
| `temperature` | number 0-2 | Randomness; 0 = deterministic. |
| `top_p` | number 0-1 | Nucleus sampling. |
| `top_k` | integer | vLLM/Ollama: only the k most likely tokens (`-1` = off). |
| `min_p` | number 0-1 | vLLM/Ollama: drop tokens below this fraction of the top token's probability. |
| `max_tokens` | integer >= 1 | Upper limit of tokens in one answer. |
| `presence_penalty`, `frequency_penalty` | number -2..2 | Discourage repeating topics / words. |
| `repetition_penalty` | number | vLLM/Ollama: 1 = off, above 1 discourages repetition. |
| `seed` | integer | Reproducible sampling where the endpoint supports it. |
| `stop` | string or array of strings | The answer ends before any of them. |
| `reasoning_effort` | `none`, `low`, `medium`, `high`, `xhigh` | Reasoning models only. |
| any other key | any JSON | Sent in the request body unchanged (e.g. `chat_template_kwargs`, `guided_json`, `response_format`). |

`model`, `messages`, `tools`, `tool_choice` and `stream` are set by the plugin and are refused in `ModelSettings`.

## Behaviour

- When the flow starts, the plugin checks that `Endpoint` and `ModelName` are set and that `ModelSettings` is a valid JSON object with valid values; otherwise the component fails to start with the reason.
- For every message on `Input` the plugin sends the text to the model (with `Prompt` as system prompt, the registered tools and `ModelSettings`) and emits the final answer as one `TextRecord`.
- Tools: when the flow starts, every AI tool whose `Register` pin is connected to this component registers itself. If the model calls a tool, the AI tool component runs its branch and the result goes back to the model; this can happen several times before the final answer. A tool that fails or times out returns an error text to the model ("Failed to execute tool ...") and an error message is shown on this component.
- Sessions (`2`): the first message for a key creates the conversation, later messages add to it. Without sessions (`1`) a new connection is made for each message.
- Transient endpoint errors (408, 429, 5xx, network) are retried; other errors (authentication, unknown model, quota) go to the flow's error handling.
- Components saved by older versions with a hosted model (`ModelProvider` 1-19) and no `ModelName` run on OpenAI's or Google's OpenAI-compatible endpoint with the old model name and `ApiKey`; Gemini with a service account no longer works. Set `Endpoint` and `ModelName` to update them.

## Notes

- In the designer, the AI settings dialog (button on `ModelSettings`, or double-click the component) edits `Endpoint`, `ApiKey`, `ModelName` and `ModelSettings` together: it lists the endpoint's models (`GET /models`), sends a test message, and builds the JSON from a form; the JSON tab takes anything the form does not cover.
- `NoInputData` is accepted by the pin but raises "Unsupported input type"; send a `TextRecord` (e.g. from Handlebars).
- With `SessionHandling` = `2` and a fixed `SessionKey`, all messages of the flow share one conversation (and its growing history). Use a per-user value if conversations must be separated.
- OpenAI's own endpoint refuses the vLLM keys (`top_k`, `min_p`, `repetition_penalty`); use them only with endpoints that accept them.

## Example

```json
{ "Endpoint": "http://gpu-host:8000/v1", "ApiKey": "=Secrets.LlmKey", "ModelName": "Qwen/Qwen3-32B", "ModelSettings": "{\"temperature\": 0.2, \"top_k\": 40, \"max_tokens\": 1024}", "Prompt": "You are a support assistant for a retailer. Use the order_status tool to look up orders. Answer in at most 3 sentences.", "SessionHandling": 1, "IdleSessionTimeout": 3600 }
```

Http listener → Handlebars (build question) → **LLM chat client** → Http response; AI tool "order_status" (`Register` → this component's `Register`)