LLM chat client

Sends the input text to a large language model on an OpenAI-compatible endpoint (vLLM, Ollama, llama.cpp, OpenAI, Gemini's OpenAI endpoint, ...) and emits the model's answer. Tools defined with AI tool components can be attached, so the model can call flow logic while answering.

Pins

Pin	Kind	Types	Description
Register	input	Viaducta.Plugins.Llm.Types.LlmToolRegistrationRequest	Connect the Register pin of AI tool components here. Only the connection matters; no data is sent through it.
Input	input	Viaducta.Plugins.TextPlugin.Types.TextRecord, Viaducta.Plugins.Http.Types.WebHttpRequest, Viaducta.TypeSystem.Metadata.NoInputData	The user message: TextRecord.Value, or the request Body (falls back to ByteBody as UTF-8).
Output	output	Viaducta.Plugins.TextPlugin.Types.TextRecord	The model's answer in Value (Name and ContentType are empty).

Configuration

Field	Type	Default	Description
Endpoint	string	http://localhost:8000/v1	Base URL of the OpenAI-compatible API, ending in /v1 (e.g. https://api.openai.com/v1, https://generativelanguage.googleapis.com/v1beta/openai/). Required.
ApiKey	password	empty	API key of the endpoint; empty when it needs none (sent as EMPTY). Use a setting (=Secrets.LlmKey).
ModelName	string	empty	Model name served by the endpoint (e.g. gpt-5-mini, Qwen/Qwen3-32B). Required.
ModelSettings	JSON object	empty	Request settings: chat completions fields, see below. Empty = the endpoint's defaults.
Prompt	multi-line text	empty	System instructions for the model (role, rules, output format, when to use which tool).
SessionHandling	dropdown: 1 SESSION_NOT_ALLOWED, 2 SESSION_KEY	1	1: every message is an independent conversation. 2: conversation history is kept per SessionKey.
SessionKey	string	empty	Only when SessionHandling is 2: messages with the same key continue the same conversation.
IdleSessionTimeout	integer seconds, 0-604800	3600	Conversations unused for this long are discarded; 0 keeps them until the flow stops.

ModelSettings keys (all optional; a key that is absent is not sent):

Key	Type	Description
temperature	number 0-2	Randomness; 0 = deterministic.
top_p	number 0-1	Nucleus sampling.
top_k	integer	vLLM/Ollama: only the k most likely tokens (-1 = off).
min_p	number 0-1	vLLM/Ollama: drop tokens below this fraction of the top token's probability.
max_tokens	integer >= 1	Upper limit of tokens in one answer.
presence_penalty, frequency_penalty	number -2..2	Discourage repeating topics / words.
repetition_penalty	number	vLLM/Ollama: 1 = off, above 1 discourages repetition.
seed	integer	Reproducible sampling where the endpoint supports it.
stop	string or array of strings	The answer ends before any of them.
reasoning_effort	none, low, medium, high, xhigh	Reasoning models only.
any other key	any JSON	Sent in the request body unchanged (e.g. chat_template_kwargs, guided_json, response_format).

model, messages, tools, tool_choice and stream are set by the plugin and are refused in ModelSettings.

Behaviour

• When the flow starts, the plugin checks that Endpoint and ModelName are set and that ModelSettings is a valid JSON object with valid values; otherwise the component fails to start with the reason.
• For every message on Input the plugin sends the text to the model (with Prompt as system prompt, the registered tools and ModelSettings) and emits the final answer as one TextRecord.
• Tools: when the flow starts, every AI tool whose Register pin is connected to this component registers itself. If the model calls a tool, the AI tool component runs its branch and the result goes back to the model; this can happen several times before the final answer. A tool that fails or times out returns an error text to the model ("Failed to execute tool ...") and an error message is shown on this component.
• Sessions (2): the first message for a key creates the conversation, later messages add to it. Without sessions (1) a new connection is made for each message.
• Transient endpoint errors (408, 429, 5xx, network) are retried; other errors (authentication, unknown model, quota) go to the flow's error handling.
• Components saved by older versions with a hosted model (ModelProvider 1-19) and no ModelName run on OpenAI's or Google's OpenAI-compatible endpoint with the old model name and ApiKey; Gemini with a service account no longer works. Set Endpoint and ModelName to update them.

Notes

• In the designer, the AI settings dialog (button on ModelSettings, or double-click the component) edits Endpoint, ApiKey, ModelName and ModelSettings together: it lists the endpoint's models (GET /models), sends a test message, and builds the JSON from a form; the JSON tab takes anything the form does not cover.
• NoInputData is accepted by the pin but raises "Unsupported input type"; send a TextRecord (e.g. from Handlebars).
• With SessionHandling = 2 and a fixed SessionKey, all messages of the flow share one conversation (and its growing history). Use a per-user value if conversations must be separated.
• OpenAI's own endpoint refuses the vLLM keys (top_k, min_p, repetition_penalty); use them only with endpoints that accept them.

Example

{ "Endpoint": "http://gpu-host:8000/v1", "ApiKey": "=Secrets.LlmKey", "ModelName": "Qwen/Qwen3-32B", "ModelSettings": "{\"temperature\": 0.2, \"top_k\": 40, \"max_tokens\": 1024}", "Prompt": "You are a support assistant for a retailer. Use the order_status tool to look up orders. Answer in at most 3 sentences.", "SessionHandling": 1, "IdleSessionTimeout": 3600 }

Http listener → Handlebars (build question) → LLM chat client → Http response; AI tool "order_status" (Register → this component's Register)