Building Multi-Tenant AI SaaS with NeuroLink

A multi-tenant AI architecture has to solve a constraint that a single shared provider configuration does not address: Tenant A may require OpenAI with strict data-handling controls, Tenant B may prefer Vertex AI for cost optimization, and both still share the same application code and API surface. Provider credentials, model selection, rate limits, cost attribution, and conversation isolation can all vary per tenant.

A practical design is to use per-tenant NeuroLink instances as the application-level isolation boundary. Each tenant gets its own configured instance, model preferences, and conversation-memory namespace. This pattern favors explicit boundaries over middleware-only routing; the trade-off is the memory cost of caching multiple instances, which lazy initialization and bounded instance pools can control.

This deep dive covers the full architecture: per-tenant provider instantiation, fallback chains, session isolation, server adapters for multi-tenant APIs, and cost attribution.

graph TB
    subgraph "API Layer"
        LB[Load Balancer]
        MW[Auth + Tenant Middleware]
    end

    subgraph "Tenant Isolation"
        T1[Tenant A Config]
        T2[Tenant B Config]
        T3[Tenant C Config]
    end

    subgraph "NeuroLink Instances"
        NL1[NeuroLink Instance A]
        NL2[NeuroLink Instance B]
        NL3[NeuroLink Instance C]
    end

    subgraph "Providers"
        OAI[OpenAI]
        VTX[Vertex AI]
        BDK[Bedrock]
    end

    subgraph "Shared Infrastructure"
        REDIS[Redis - Scoped Keys]
        OBS[Langfuse Observability]
    end

    LB --> MW
    MW --> T1 --> NL1 --> OAI
    MW --> T2 --> NL2 --> VTX
    MW --> T3 --> NL3 --> BDK
    NL1 --> REDIS
    NL2 --> REDIS
    NL3 --> REDIS
    NL1 --> OBS
    NL2 --> OBS
    NL3 --> OBS

Per-Tenant Provider Instantiation

This design gives each tenant its own NeuroLink instance and selects that tenant’s provider, model, and credentials at request time. Pair the per-instance setup with tenant-prefixed conversation memory and explicit request authorization.

Start by defining what a tenant’s AI configuration looks like:

import { NeuroLink } from '@juspay/neurolink';
import type { NeurolinkCredentials } from '@juspay/neurolink';

// Tenant configuration store; resolve secrets from a secret manager.
interface TenantAIConfig {
  tenantId: string;
  provider: string;
  model?: string;
  region?: string;
  fallbackProvider?: string;
  maxTokens?: number;
  credentials?: NeurolinkCredentials;
}

// Create per-tenant NeuroLink instances
function createTenantNeuroLink(config: TenantAIConfig): NeuroLink {
  return new NeuroLink({
    credentials: config.credentials,
    conversationMemory: {
      enabled: true,
      redisConfig: {
        url: process.env.REDIS_URL,
        keyPrefix: `tenant:${config.tenantId}:`
      }
    }
  });
}

// Per-tenant generation with provider routing
async function generateForTenant(
  tenantConfig: TenantAIConfig,
  prompt: string
) {
  const neurolink = getTenantInstance(tenantConfig.tenantId);

  return await neurolink.generate({
    input: { text: prompt },
    provider: tenantConfig.provider,
    model: tenantConfig.model,
    region: tenantConfig.region,
  });
}

The critical detail is the Redis key prefix: tenant:${config.tenantId}:. This scopes all conversation memory to the tenant, preventing any data leakage between tenants even though they share the same Redis cluster.

In production, you want to cache NeuroLink instances rather than creating them on every request. A simple instance pool works well:

const MAX_CACHED_TENANTS = 1000;
const tenantInstances = new Map<string, NeuroLink>();

function getTenantInstance(tenantId: string): NeuroLink {
  if (!tenantInstances.has(tenantId)) {
    // Evict oldest entry if cache is full (simple LRU)
    if (tenantInstances.size >= MAX_CACHED_TENANTS) {
      const oldest = tenantInstances.keys().next().value;
      if (oldest !== undefined) tenantInstances.delete(oldest);
    }
    const config = loadTenantConfig(tenantId); // From your database
    tenantInstances.set(tenantId, createTenantNeuroLink(config));
  }
  return tenantInstances.get(tenantId)!;
}

When you want environment-driven provider selection rather than a tenant-specific policy, NeuroLink’s AIProviderFactory.createBestProvider() selects the requested provider when available or discovers a configured provider from the supported credential environment variables.

Provider Fallback Chains Per Tenant

Each tenant can define its own fallback policy. Tenant A might use OpenAI first and Bedrock second, while Tenant B might use Vertex AI first and OpenAI as a backup. This extends the createTenantNeuroLink helper above with a providerFallback callback; configure it when you construct the tenant’s NeuroLink instance:

function createTenantNeuroLinkWithFallback(config: TenantAIConfig): NeuroLink {
  return new NeuroLink({
    credentials: config.credentials,
    conversationMemory: {
      enabled: true,
      redisConfig: {
        url: process.env.REDIS_URL,
        keyPrefix: `tenant:${config.tenantId}:`,
      },
    },
    providerFallback: async (error: unknown) => {
      reportProviderFailure(config.tenantId, error);
      return config.fallbackProvider
        ? { provider: config.fallbackProvider }
        : null;
    },
  });
}

providerFallback receives the original error and returns the next { provider, model } to try, or null to let the error propagate. Because the callback closes over one tenant’s configuration, routing policy remains tenant-specific. providerFallback is consulted on the first qualifying failure – any error except one caused by the caller’s own AbortSignal firing.

Retry logic via the withRetry() utility adds another layer of resilience. Configure retries per tenant based on their latency tolerance:

import { withRetry } from '@juspay/neurolink';

async function resilientGenerate(tenantId: string, prompt: string) {
  const config = loadTenantConfig(tenantId);
  const neurolink = getTenantInstance(tenantId);

  return await withRetry(
    () => neurolink.generate({
      input: { text: prompt },
      provider: config.provider,
      model: config.model,
    }),
    {
      maxRetries: config.maxRetries || 3,
      baseDelayMs: 1000,
    }
  );
}

Session and Memory Isolation

Conversation memory in a multi-tenant system requires strict isolation at two levels: tenant isolation (Tenant A cannot see Tenant B’s conversations) and session isolation (User 1 within Tenant A cannot see User 2’s sessions).

NeuroLink achieves this through session context scoping under the hood – internally, each provider carries a setSessionContext(sessionId, userId) method that scopes tool state to the caller. At the public API level, you reach the same isolation by passing tenant-prefixed identifiers through context on generate():

// In multi-tenant context
async function handleRequest(
  tenantId: string,
  sessionId: string,
  userId: string,
  prompt: string,
) {
  const neurolink = getTenantInstance(tenantId);
  const tenantConfig = loadTenantConfig(tenantId);

  // Scope session and user identifiers to the tenant
  return await neurolink.generate({
    input: { text: prompt },
    provider: tenantConfig.provider,
    model: tenantConfig.model,
    context: {
      sessionId: `${tenantId}:${sessionId}`,
      userId: `${tenantId}:${userId}`,
    },
  });
}

Prefixing session and user IDs with the tenant ID creates a natural namespace. Treat that namespace as defense in depth: authorize each request before generation and test that session-scoped tool state cannot cross tenant or user boundaries.

For HITL (Human-in-the-Loop) workflows, the HITLManager supports tenant-specific approval flows. A healthcare tenant might require approval for any action that accesses patient data, while a marketing tenant might only require approval for content publication:

const neurolink = new NeuroLink({
  conversationMemory: { enabled: true },
  hitl: {
    enabled: true,
    dangerousActions: getTenantDangerousActions(tenantId),
    // Each tenant defines which actions need human approval
  },
});

Note: Prefix Redis keys and session IDs with the tenant identifier, and pair that namespace with request authorization and tenant-scoped credentials. Prefixes alone are not a security boundary.

Server Adapter for Multi-Tenant APIs

When you need to expose your multi-tenant AI as an API (rather than embedding NeuroLink directly in each tenant’s application), the server adapter layer provides production-grade infrastructure.

The BaseServerAdapter creates a ServerContext for each request that includes the NeuroLink reference, tool registry, and request metadata:

// Simplified from src/lib/server/abstract/baseServerAdapter.ts
protected createContext(options: {
  requestId: string;
  method: string;
  path: string;
  headers: Record<string, string>;
  query?: Record<string, string>;
  params?: Record<string, string>;
  body?: unknown;
}): ServerContext {
  return {
    requestId: options.requestId,
    method: options.method,
    path: options.path,
    headers: options.headers,
    query: options.query ?? {},
    params: options.params ?? {},
    body: options.body,
    neurolink: this.neurolink,
    toolRegistry: this.toolRegistry,
    externalServerManager: this.externalServerManager,
    timestamp: Date.now(),
    metadata: {},
    redaction: this.redactionConfig,
  };
}

For multi-tenant APIs, you add tenant identification middleware that resolves the tenant from the request (via API key, JWT, or header) and attaches the appropriate NeuroLink instance to the context:

import { ServerAdapterFactory } from '@juspay/neurolink';

// Create the server with tenant middleware
const server = await ServerAdapterFactory.create({
  framework: 'hono',
  neurolink: defaultNeuroLink, // Default instance
  config: {
    port: 3000,
    basePath: '/api',
    cors: { enabled: true },
    rateLimit: {
      enabled: true,
      windowMs: 60000,
      maxRequests: 100,
      keyGenerator: (ctx) => extractTenantId(ctx), // Per-tenant rate limits
    },
  },
});

The keyGenerator function in the rate limit configuration is critical. By generating rate limit keys based on tenant ID, each tenant gets their own request quota. Tenant A making 90 requests does not affect Tenant B’s remaining quota.

Per-tenant rate limiting prevents a single tenant from monopolizing your AI infrastructure. Combined with the per-tenant provider instances, you get full resource isolation without separate deployments.

Cost Attribution and Observability

In a multi-tenant SaaS, you need to know exactly how much each tenant costs. NeuroLink’s telemetry system tracks usage per provider per request, giving you the raw data for cost attribution.

Every response includes analytics data with token counts:

const result = await neurolink.generate({
  input: { text: prompt },
  provider: tenantConfig.provider,
  model: tenantConfig.model,
});

// Access usage data for billing
if (result.usage) {
  console.log(
    `Tokens - Input: ${result.usage.input}, Output: ${result.usage.output}, Total: ${result.usage.total}`,
  );
}

For per-tenant observability, the Langfuse integration supports scoped tracing. Each tenant’s AI operations appear in their own trace group, making it straightforward to debug issues and analyze performance per tenant:

// Per-tenant Langfuse tracing
const neurolink = new NeuroLink({
  conversationMemory: { enabled: true },
  observability: {
    langfuse: {
      enabled: true,
      publicKey: process.env.LANGFUSE_PUBLIC_KEY,
      secretKey: process.env.LANGFUSE_SECRET_KEY,
      userId: `tenant:${tenantId}`, // Attach traces to the tenant
    },
  },
});

Build a cost attribution pipeline that aggregates token usage by tenant:

import type { GenerateResult } from '@juspay/neurolink';

interface TenantUsageRecord {
  tenantId: string;
  provider: string;
  model: string;
  inputTokens: number;
  outputTokens: number;
  total: number;
  estimatedCost: number;
  timestamp: Date;
}

async function trackUsage(tenantId: string, result: GenerateResult) {
  if (!result.usage || !result.provider || !result.model) return;

  const costPerToken = getCostPerToken(result.provider, result.model);
  const record: TenantUsageRecord = {
    tenantId,
    provider: result.provider,
    model: result.model,
    inputTokens: result.usage.input,
    outputTokens: result.usage.output,
    total: result.usage.total,
    estimatedCost:
      (result.usage.input * costPerToken.input +
        result.usage.output * costPerToken.output) /
      1_000_000,
    timestamp: new Date(),
  };

  await usageStore.insert(record);
}

This usage data feeds into your billing system, enabling usage-based pricing, cost alerts, and tenant-level budgets.

Note: Token prices vary by provider, model, token direction, and date. Calculate tenant costs from measured usage and your providers’ current pricing rather than embedding a fixed model-to-model ratio.

Putting It All Together

Here is the complete multi-tenant request flow, from incoming API call to AI response:

import { NeuroLink } from '@juspay/neurolink';

// 1. Tenant resolution middleware extracts tenant from request
function tenantMiddleware(req: Request) {
  const apiKey = req.headers.get('x-api-key');
  const tenant = resolveTenant(apiKey);
  return tenant;
}

// 2. Get or create per-tenant NeuroLink instance
function getTenantNeuroLink(tenant: TenantAIConfig): NeuroLink {
  if (!tenantInstances.has(tenant.tenantId)) {
    // Evict oldest entry if cache is full (simple LRU)
    if (tenantInstances.size >= MAX_CACHED_TENANTS) {
      const oldest = tenantInstances.keys().next().value;
      if (oldest !== undefined) tenantInstances.delete(oldest);
    }
    tenantInstances.set(tenant.tenantId, new NeuroLink({
      credentials: tenant.credentials,
      conversationMemory: {
        enabled: true,
        redisConfig: {
          url: process.env.REDIS_URL,
          keyPrefix: `tenant:${tenant.tenantId}:`
        }
      },
    }));
  }
  return tenantInstances.get(tenant.tenantId)!;
}

// 3. Handle the request with tenant-scoped everything
async function handleAIRequest(req: Request) {
  const tenant = tenantMiddleware(req);
  const neurolink = getTenantNeuroLink(tenant);
  const { prompt, sessionId } = await req.json();

  const result = await neurolink.generate({
    input: { text: prompt },
    provider: tenant.provider,
    model: tenant.model,
  });

  // 4. Track usage for billing
  await trackUsage(tenant.tenantId, result);

  return result;
}

Design Decisions and Trade-offs

Per-tenant NeuroLink instances make configuration boundaries explicit, while a shared instance with tenant context uses less memory. The per-tenant pattern does not replace authorization checks, secret isolation, or tests that prove one tenant cannot access another tenant’s conversations.

Redis key prefixing (tenant:{id}:) provides a simple conversation-memory namespace. Treat the prefix as defense in depth rather than a complete security boundary: validate tenant ownership on every request and keep provider credentials in a tenant-scoped secret store.

A server adapter with one HTTP endpoint and tenant resolution middleware simplifies deployment, but it concentrates authorization and rate-limit enforcement in the request path. If stronger infrastructure isolation is required, use separate deployments or accounts for those tenants rather than relying only on application-level namespaces.

---

Related posts:

• Security Best Practices for AI Applications
• NeuroLink vs Portkey vs Helicone: Choosing the Right LLM Gateway
• AI Observability: Monitoring LLM Applications in Production