gpu inference · signal-1.4b · int8

Perception Engine
for production agent stacks

Transform every conversation turn into calibrated signals — so teams can steer sessions in-flight, tune operations, and manage production risk.

Request access →Read the architecture
1.4B signal core, INT8<100 ms p95 turn budget20-turn / 8k context window10-signal taxonomyPII-isolated inference
Failure Modes

Three missing primitives
in most production agent stacks.

Not tooling gaps — missing primitives. A conversation is a per-turn time series; reading it takes a calibrated inference head, session state, and an in-request decision path. Most generation APIs don’t expose these as first-class outputs — they’re built for producing turns, not measuring them.

No calibrated perception head

Generation and measurement share one forward pass. A decoder tuned for fluency emits tokens, not calibrated signals — its logits were never trained as session measurements, leaving little to threshold, trend, or export downstream.

Stateless analytics

Most sentiment tooling scores one message at a time. A session is a trajectory — attention decays, frustration compounds — and without session state, per-message scores struggle to capture the slopes that matter.

Offline, post-hoc output

Batch dashboards typically land well after the conversation ends — usually without a same-request verdict, a trace id, or deterministic replay. Useful for reading trends; harder to act on, tune, or audit in the loop.

Architecture

Built for real-time inference at scale

Viora runs a fine-tuned open-weight language model paired with a dedicated classifier head, GPU-hosted for low-latency scoring. Text and interaction telemetry are fused on GPU in real time, so perception isn't limited to what's typed. The perception layer is decoupled from the conversational layer, so it stays deterministic, auditable, and fast enough to act inside a live conversation.

board  / request-path.sketchsketched · not to scale
Hand-drawn architecture diagram of the Viora signal path

Inference Stack

Base modelFine-tuned open-weight LLM + classifier head
ModalityText + telemetry, GPU-fused
ServingTriton-compatible GPU inference server
OptimizationTensorRT-LLM–class quantization
HardwareNVIDIA GPU (A100 / H100 class)
Latency target< 100ms p95 end-to-end
Throughput target500+ concurrent sessions / GPU
sample_trace.log · illustrative
> input: "this is the third time I'm asking about this refund"
> context         14/20 turns · 3.8k/8k tok
> signals
  frustration       0.72
  attention         0.31
  resolution_conf   0.28
> trajectory      frustration +0.21/turn (5-turn window)
> policy          cx.v3 · τ=0.65 · pinned
> decision        STEER → supervisor_review
> latency          41ms / 100ms budget

Latency and throughput figures are pre-launch design targets, not independently benchmarked results.

Inference Layer

The model,
specified.

Signal inference runs on a small purpose-trained model, not a prompt wrapped around a general-purpose LLM. That means fixed weights, a fixed label set, and calibrated per-label probabilities — so a threshold moves along a measured precision–recall curve, not a vibe. Here is the target spec and what it emits.

Model card — target spec

ArchitectureEncoder + multi-head classifier
Parameters1.4B signal core · 340M distilled edge
Context window20 turns / 8k tokens
State updatePer-turn posterior update over the window · weights pinned
PrecisionINT8 weight-only, TensorRT-LLM class
ModalitiesText + interaction telemetry
Output10 signals + calibrated posteriors
CalibrationPer-label Platt scaling, held-out adjudicated set
DeterminismT=0, pinned weights · verdict = f(build, policy, window)
Release cadenceMonthly; version pinned per tenant

Signal taxonomy — scored every turn

frustration.acuteT1
escalation_likelihoodT1
churn.signalT1
compliance.exposureT1
attention.decayT2
resolution.stallT2
confusion.persistentT2
trust.erosionT2
intent.driftT3
sentiment.trendT3
T1  route to humanT2  steer in-flowT3  log + trend
Evaluation
PR-AUC per label · ECE
FNR at operating point, temporal holdout
Labelling
Expert adjudicated
dual-rated, arbitrated, label-set versioned
Versioning
Build + policy pinned per tenant
replay: trace_id → (build, policy, window)
Drift
Per-label score distributions
PSI alert against release baseline

Model card fields describe the reporting template design partners receive each release; figures are populated from the evaluation set at that release, not asserted here.

Inference Path

Not a prompt.
A separate inference path.

General-purpose model APIs optimise for generation, and anything they report about a conversation is a by-product of the same weights doing the talking. Viora runs a second, smaller inference path with its own weights, its own state and its own operating point.

Property
Prompted analytics on a general-purpose model
Viora inference layer
Data handling
Shared multi-tenant inference
PII-isolated GPU processing, no training on conversation data
Inference
One forward pass, generation and measurement entangled
Separate pass: encoder + classifier head + rules ensemble
Agent coverage
Scores the user message only
Scores user turns, model output and tool results
Modality
Mostly text-only filters
Text + interaction telemetry, GPU-fused
Determinism
Sampled output, verdict varies run to run
Temperature 0, pinned weights, replayable by trace id

In-line PII Redaction

PII spans are detected and tokenised before GPU inference, then redacted again before storage or export — no shared third-party inference, no training on conversation data.

Multi-Engine Reasoning

Verdicts come from an ensemble — LLM contextual understanding, a dedicated classifier head, and a rules engine — instead of one model's single best guess.

Adjudicated Taxonomy

Labels come from a dual-rated, expert-adjudicated corpus with arbitration on disagreement — not statistical pattern-matching over scraped text.

Temporal Fusion

Interaction telemetry — response latency, turn cadence, tool-call traces — is fused with text on GPU, catching signals a text-only pipeline would miss.

Integration

A perception layer
built into the loop.

Viora sits between your app and your model: one call per turn, a full signal profile back inside the same request, and no change to how your model generates. Run it inline to act on the profile, or in shadow mode to observe traffic without touching the loop.

engine.py  / one call per turnsync verdict · one round-trip
01from viora import PerceptionEngine
02
03engine = PerceptionEngine(api_key=VIORA_KEY, mode="inline")  # "shadow" = observe only
04
05def on_user_turn(session_id, text, telemetry=None):
06    profile = engine.infer(
07        session_id=session_id,
08        text=text,
09        telemetry=telemetry,  # optional; fused when present
10    )
11
12    if profile.action.type == "escalate":
13        return handoff(profile.action.route, profile.trace_id)
14
15    reply = llm.chat(text)   # your model, untouched
16    return engine.postcheck(session_id, reply)  # completion + tool hops
profile  / application-json200 OK
01{
02  "session_id": "s_9f3c1a",
03  "turn": 14,
04  "context_window": {
05    "turns":      "14 / 20",
06    "tokens":     "3847 / 8192",
07    "modalities": ["text", "telemetry"]
08  },
09  "signals": {
10    "attention":             {"p":0.88,"ci":[0.83,0.92],"slope":-0.04},
11    "resolution_confidence": {"p":0.79,"ci":[0.71,0.85],"slope":+0.09},
12    "frustration":           {"p":0.41,"ci":[0.35,0.48],"slope":+0.12},
13    "escalation_likelihood": {"p":0.26,"ci":[0.19,0.31],"slope":+0.03}
14  },
15  "state_update": { "prior": "turn_13",
16    "evidence": ["lexical","semantic","timing"] },
17  "action": {
18    "type": "steer",  "route": "resolution_assist",
19    "trigger": "frustration.slope > 0.10",  "confidence": 0.83
20  },
21  "policy": "cx.v3:tau=0.65",  "model": "viora-signal-1.4b",
22  "latency_ms": 41,  "trace_id": "01JBQ7B4RF"
23}
stageruns onp95 budgetshare of budget
redact + tokenisein-process3 ms
encodegpu11 ms
ensemble ×3gpu19 ms
session state r/wredis4 ms
policy evalin-process2 ms
serialise + returngrpc2 ms
end to endsingle turn41 ms
Transport
gRPC over HTTP/2
REST fallback, keep-alive pooling
Agent loop
turn · completion · tool result
screen any hop, same schema
Auth & isolation
mTLS + scoped keys
per-tenant GPU namespaces
Deployment
SaaS · VPC peer · on-prem
bring your own A100 / H100

Stage timings are design targets for a single screened turn, not independently benchmarked results.

Deployment Profiles

Three integration shapes,
one inference schema.

The API surface is identical across deployments. What changes is session length, traffic shape and which taxonomy tiers you wire to a blocking action.

Task-oriented dialogue agents

Long-lived sessions, high turn counts, tool calls in the loop. Frustration and resolution signals route straight into supervisor queues and review tooling.

inline modefull 20-turn stateT1 → supervisor handofftelemetry fused

Enterprise & workforce agents

Multi-tenant, SSO-bound identities, retention limits

shadow → inlineper-tenant thresholdsVPC peeringaudit export

Platform & marketplace agents

High QPS, short sessions, batch plus streaming

distilled 340M edgebatch scoringcompliance signalswebhook fan-out
Early Access

Get Early Access

Design-partner access to the inference layer: pinned model build, per-tenant thresholds, and direct access to the engineers running it.

Priority API access
Design partner pricing
Direct engineering support

No spam. We'll reach out when your spot is ready.