Automated Tenant Provisioning Workflow

Document ID: CODITECT-BIO-PROVISION-001 Version: 1.0.0 Effective Date: 2026-02-16 Classification: Internal - Confidential Owner: Chief Technology Officer (CTO)

Document Control

Approval History

Role	Name	Signature	Date
CTO	Hal Casteel	[PENDING]	2026-02-16
CISO	[TBD]	[PENDING]	-
VP Engineering	[TBD]	[PENDING]	-
Quality Assurance Lead	[TBD]	[PENDING]	-

Revision History

Version	Date	Author	Changes
1.0.0	2026-02-16	Hal Casteel	Initial release - D.6.2 specification

Distribution

Engineering Team (Required Reading)
DevOps Team (Implementation)
Security Team (Review)
Quality Assurance Team (Testing)
Customer Success Team (Reference)

Executive Summary

This document defines the Automated Tenant Provisioning Workflow for the CODITECT BIO-QMS SaaS platform. The workflow ensures secure, compliant, and reliable tenant onboarding with complete data isolation, encryption key management, and regulatory profile configuration.

Key Features:

10-step automated provisioning sequence with health validation
XState-orchestrated workflow with automated rollback on failure
Multi-tier subscription model (Starter, Professional, Enterprise)
Regulatory profile configuration (FDA Part 11, HIPAA, SOC 2)
Encryption key lifecycle integration with GCP KMS
< 5 minute target provisioning time
Zero manual intervention for standard tenant creation
Complete audit trail for all provisioning operations

Compliance Mapping:

FDA 21 CFR Part 11 §11.10(a): Validated systems with audit trails
HIPAA §164.308(a)(3)(i): Workforce clearance procedure (admin creation)
HIPAA §164.308(a)(4)(ii)(B): Encryption and decryption (key generation)
SOC 2 CC7.2: System operations (automated provisioning controls)

Provisioning Architecture
Subscription Tier Model
Provisioning State Machine
Provisioning Steps Specification
Pre-Provisioning Validation
Post-Provisioning Health Check
Automated Rollback Mechanism
Provisioning API Specification
Database Schema Extensions
TypeScript Implementation
Monitoring & Observability
Security Considerations
Compliance Requirements
Testing & Validation
Operational Procedures

1. Provisioning Architecture

1.1 High-Level Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                    TENANT PROVISIONING ORCHESTRATOR                  │
│                        (XState State Machine)                        │
└────────────┬────────────────────────────────────────────────────────┘
             │
             ├─► [1] Tenant Registration ──────► Tenant Registry DB
             │
             ├─► [2] Database Schema Creation ─► PostgreSQL (RLS)
             │
             ├─► [3] Encryption Key Generation ► GCP KMS (DEK + KEK)
             │
             ├─► [4] Storage Allocation ────────► GCS Bucket
             │
             ├─► [5] Initial Configuration ─────► RBAC + Templates
             │
             ├─► [6] Admin User Creation ───────► Auth Service + MFA
             │
             ├─► [7] Compliance Profile Setup ──► Regulatory Config
             │
             ├─► [8] Integration Endpoints ─────► API Keys + Webhooks
             │
             ├─► [9] Health Check ──────────────► Validation Suite
             │
             └─► [10] Welcome Notification ─────► Email Service
                                │
                                ├── SUCCESS ──► Tenant Active
                                └── FAILURE ──► Automated Rollback

1.2 Core Components

Component	Purpose	Technology
Provisioning Orchestrator	XState workflow coordination	TypeScript, XState v5
Tenant Registry	Central tenant metadata store	PostgreSQL + Prisma
Key Management Service	DEK/KEK generation and rotation	GCP KMS
Storage Provisioner	GCS bucket creation	Google Cloud Storage API
Database Migrator	Schema creation with RLS	Prisma Migrate
RBAC Initializer	Default roles and permissions	Custom TypeScript service
Health Validator	Post-provisioning verification	Jest test suite
Rollback Engine	Failure recovery automation	Compensating transactions

1.3 Design Principles

Idempotency: All provisioning steps are idempotent - safe to retry
Atomicity: Either complete success or complete rollback - no partial state
Observability: Every step logs progress and errors to audit trail
Security: Zero-trust - validate tenant context at every step
Performance: Parallel execution where possible (< 5 min target)
Compliance: Automated regulatory profile enforcement

2. Subscription Tier Model

2.1 Tier Definitions

Starter Tier

Target Audience: Small labs, research teams, pilot projects

Feature	Limit	Notes
Users	10	Named users only
Storage	10 GB	Documents + attachments
Work Orders/Month	500	Basic QMS workflows
E-Signatures	No	Manual signatures only
Compliance	Basic audit	SOC 2 Type II
Support	Email (48h SLA)	Business hours
API Rate Limit	100 req/min	Shared infrastructure
Custom Workflows	No	Standard templates only
Data Retention	1 year	Automated archival
SLA Uptime	99.0%	Shared tenancy

Monthly Price: $499 Annual Discount: 15% ($5,089/year)

Professional Tier

Target Audience: Mid-size biotech, contract labs, manufacturing

Feature	Limit	Notes
Users	50	Named users + 10 read-only
Storage	100 GB	Expandable to 500 GB
Work Orders/Month	5,000	Full QMS + CAPA
E-Signatures	Yes	FDA Part 11 compliant
Compliance	Part 11 + SOC 2	Electronic records + signatures
Support	Email + Chat (24h SLA)	8am-8pm ET
API Rate Limit	500 req/min	Dedicated resources
Custom Workflows	10 workflows	XState designer access
Data Retention	3 years	Configurable policy
SLA Uptime	99.5%	Multi-zone redundancy

Monthly Price: $2,499 Annual Discount: 20% ($23,990/year)

Enterprise Tier

Target Audience: Pharma, medical device, regulated manufacturing

Feature	Limit	Notes
Users	Unlimited	Organization-wide deployment
Storage	1 TB	Unlimited expansion available
Work Orders/Month	Unlimited	Full regulatory QMS
E-Signatures	Yes	Part 11 + HIPAA compliant
Compliance	Part 11 + HIPAA + SOC 2	PHI support, BAA included
Support	24/7 phone + dedicated CSM	<1h critical response
API Rate Limit	2,000 req/min	Dedicated infrastructure
Custom Workflows	Unlimited	White-glove workflow design
Data Retention	7+ years	Regulatory-driven retention
SLA Uptime	99.9%	Multi-region active-active
Advanced Features	SSO, LDAP, VPN, custom SLAs	Enterprise integration

Monthly Price: Custom (starting $9,999) Annual Discount: Negotiated (typically 25-30%)

2.2 Tier-Based Resource Quotas

export enum SubscriptionTier {
  STARTER = 'STARTER',
  PROFESSIONAL = 'PROFESSIONAL',
  ENTERPRISE = 'ENTERPRISE'
}

export interface TierQuotas {
  tier: SubscriptionTier;
  maxUsers: number;
  maxReadOnlyUsers: number;
  storageGB: number;
  maxWorkOrdersPerMonth: number;
  apiRateLimitPerMinute: number;
  maxCustomWorkflows: number | null; // null = unlimited
  dataRetentionYears: number;
  slaUptimePercent: number;
  features: TierFeatures;
}

export interface TierFeatures {
  eSignaturesEnabled: boolean;
  fdaPart11Compliance: boolean;
  hipaaCompliance: boolean;
  soc2Compliance: boolean;
  customWorkflows: boolean;
  ssoIntegration: boolean;
  ldapIntegration: boolean;
  dedicatedSupport: boolean;
  multiRegionDeployment: boolean;
  advancedAnalytics: boolean;
}

export const TIER_QUOTAS: Record<SubscriptionTier, TierQuotas> = {
  [SubscriptionTier.STARTER]: {
    tier: SubscriptionTier.STARTER,
    maxUsers: 10,
    maxReadOnlyUsers: 0,
    storageGB: 10,
    maxWorkOrdersPerMonth: 500,
    apiRateLimitPerMinute: 100,
    maxCustomWorkflows: 0,
    dataRetentionYears: 1,
    slaUptimePercent: 99.0,
    features: {
      eSignaturesEnabled: false,
      fdaPart11Compliance: false,
      hipaaCompliance: false,
      soc2Compliance: true,
      customWorkflows: false,
      ssoIntegration: false,
      ldapIntegration: false,
      dedicatedSupport: false,
      multiRegionDeployment: false,
      advancedAnalytics: false,
    },
  },
  [SubscriptionTier.PROFESSIONAL]: {
    tier: SubscriptionTier.PROFESSIONAL,
    maxUsers: 50,
    maxReadOnlyUsers: 10,
    storageGB: 100,
    maxWorkOrdersPerMonth: 5000,
    apiRateLimitPerMinute: 500,
    maxCustomWorkflows: 10,
    dataRetentionYears: 3,
    slaUptimePercent: 99.5,
    features: {
      eSignaturesEnabled: true,
      fdaPart11Compliance: true,
      hipaaCompliance: false,
      soc2Compliance: true,
      customWorkflows: true,
      ssoIntegration: true,
      ldapIntegration: false,
      dedicatedSupport: false,
      multiRegionDeployment: false,
      advancedAnalytics: true,
    },
  },
  [SubscriptionTier.ENTERPRISE]: {
    tier: SubscriptionTier.ENTERPRISE,
    maxUsers: -1, // unlimited
    maxReadOnlyUsers: -1, // unlimited
    storageGB: 1024,
    maxWorkOrdersPerMonth: -1, // unlimited
    apiRateLimitPerMinute: 2000,
    maxCustomWorkflows: null, // unlimited
    dataRetentionYears: 7,
    slaUptimePercent: 99.9,
    features: {
      eSignaturesEnabled: true,
      fdaPart11Compliance: true,
      hipaaCompliance: true,
      soc2Compliance: true,
      customWorkflows: true,
      ssoIntegration: true,
      ldapIntegration: true,
      dedicatedSupport: true,
      multiRegionDeployment: true,
      advancedAnalytics: true,
    },
  },
};

3. Provisioning State Machine

3.1 XState Definition

import { setup, assign, fromPromise } from 'xstate';
import { TenantProvisioningContext, TenantProvisioningEvent } from './types';

export const tenantProvisioningMachine = setup({
  types: {
    context: {} as TenantProvisioningContext,
    events: {} as TenantProvisioningEvent,
  },
  actors: {
    validateRequest: fromPromise(async ({ input }: { input: ProvisioningRequest }) => {
      // Pre-provisioning validation
      return await preProvisioningValidation(input);
    }),
    registerTenant: fromPromise(async ({ input }: { input: TenantRegistrationInput }) => {
      return await registerTenantInRegistry(input);
    }),
    createDatabaseSchema: fromPromise(async ({ input }: { input: { tenantId: string } }) => {
      return await createTenantDatabaseSchema(input.tenantId);
    }),
    generateEncryptionKeys: fromPromise(async ({ input }: { input: { tenantId: string } }) => {
      return await generateTenantEncryptionKeys(input.tenantId);
    }),
    allocateStorage: fromPromise(async ({ input }: { input: { tenantId: string } }) => {
      return await allocateTenantStorage(input.tenantId);
    }),
    initializeConfiguration: fromPromise(async ({ input }: { input: { tenantId: string; tier: SubscriptionTier } }) => {
      return await initializeTenantConfiguration(input.tenantId, input.tier);
    }),
    createAdminUser: fromPromise(async ({ input }: { input: AdminUserInput }) => {
      return await createTenantAdminUser(input);
    }),
    setupComplianceProfile: fromPromise(async ({ input }: { input: ComplianceProfileInput }) => {
      return await setupTenantComplianceProfile(input);
    }),
    setupIntegrations: fromPromise(async ({ input }: { input: IntegrationInput }) => {
      return await setupTenantIntegrations(input);
    }),
    performHealthCheck: fromPromise(async ({ input }: { input: { tenantId: string } }) => {
      return await performTenantHealthCheck(input.tenantId);
    }),
    sendWelcomeNotification: fromPromise(async ({ input }: { input: WelcomeNotificationInput }) => {
      return await sendTenantWelcomeNotification(input);
    }),
    rollbackProvisioning: fromPromise(async ({ input }: { input: RollbackContext }) => {
      return await executeProvisioningRollback(input);
    }),
  },
  guards: {
    hasValidRequest: ({ context }) => {
      return context.validationResult?.isValid === true;
    },
    isRollbackComplete: ({ context }) => {
      return context.rollbackStatus === 'completed';
    },
  },
}).createMachine({
  id: 'tenantProvisioning',
  initial: 'pending',
  context: {
    request: null,
    tenantId: null,
    jobId: null,
    tier: null,
    validationResult: null,
    rollbackActions: [],
    completedSteps: [],
    error: null,
    startTime: null,
    endTime: null,
  },
  states: {
    pending: {
      on: {
        START_PROVISIONING: {
          target: 'validating',
          actions: assign({
            request: ({ event }) => event.request,
            jobId: ({ event }) => event.jobId,
            startTime: () => new Date().toISOString(),
          }),
        },
      },
    },
    validating: {
      invoke: {
        src: 'validateRequest',
        input: ({ context }) => context.request!,
        onDone: {
          target: 'registering',
          guard: 'hasValidRequest',
          actions: assign({
            validationResult: ({ event }) => event.output,
          }),
        },
        onError: {
          target: 'provision_failed',
          actions: assign({
            error: ({ event }) => event.error,
          }),
        },
      },
    },
    registering: {
      invoke: {
        src: 'registerTenant',
        input: ({ context }) => ({
          organizationName: context.request!.organizationName,
          adminEmail: context.request!.adminEmail,
          tier: context.request!.tier,
          regulatoryProfile: context.request!.regulatoryProfile,
        }),
        onDone: {
          target: 'db_creating',
          actions: assign({
            tenantId: ({ event }) => event.output.tenantId,
            tier: ({ event }) => event.output.tier,
            completedSteps: ({ context }) => [...context.completedSteps, 'registering'],
            rollbackActions: ({ context, event }) => [
              ...context.rollbackActions,
              { step: 'registering', data: { tenantId: event.output.tenantId } },
            ],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    db_creating: {
      invoke: {
        src: 'createDatabaseSchema',
        input: ({ context }) => ({ tenantId: context.tenantId! }),
        onDone: {
          target: 'keys_generating',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'db_creating'],
            rollbackActions: ({ context }) => [
              ...context.rollbackActions,
              { step: 'db_creating', data: { tenantId: context.tenantId! } },
            ],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    keys_generating: {
      invoke: {
        src: 'generateEncryptionKeys',
        input: ({ context }) => ({ tenantId: context.tenantId! }),
        onDone: {
          target: 'storage_allocating',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'keys_generating'],
            rollbackActions: ({ context, event }) => [
              ...context.rollbackActions,
              {
                step: 'keys_generating',
                data: {
                  tenantId: context.tenantId!,
                  dekKeyId: event.output.dekKeyId,
                  kekKeyId: event.output.kekKeyId,
                },
              },
            ],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    storage_allocating: {
      invoke: {
        src: 'allocateStorage',
        input: ({ context }) => ({ tenantId: context.tenantId! }),
        onDone: {
          target: 'configuring',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'storage_allocating'],
            rollbackActions: ({ context, event }) => [
              ...context.rollbackActions,
              {
                step: 'storage_allocating',
                data: {
                  tenantId: context.tenantId!,
                  bucketName: event.output.bucketName,
                },
              },
            ],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    configuring: {
      invoke: {
        src: 'initializeConfiguration',
        input: ({ context }) => ({
          tenantId: context.tenantId!,
          tier: context.tier!,
        }),
        onDone: {
          target: 'user_creating',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'configuring'],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    user_creating: {
      invoke: {
        src: 'createAdminUser',
        input: ({ context }) => ({
          tenantId: context.tenantId!,
          email: context.request!.adminEmail,
          organizationName: context.request!.organizationName,
        }),
        onDone: {
          target: 'compliance_setup',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'user_creating'],
            rollbackActions: ({ context, event }) => [
              ...context.rollbackActions,
              {
                step: 'user_creating',
                data: {
                  tenantId: context.tenantId!,
                  userId: event.output.userId,
                },
              },
            ],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    compliance_setup: {
      invoke: {
        src: 'setupComplianceProfile',
        input: ({ context }) => ({
          tenantId: context.tenantId!,
          tier: context.tier!,
          regulatoryProfile: context.request!.regulatoryProfile,
        }),
        onDone: {
          target: 'integration_setup',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'compliance_setup'],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    integration_setup: {
      invoke: {
        src: 'setupIntegrations',
        input: ({ context }) => ({
          tenantId: context.tenantId!,
          webhookUrls: context.request!.webhookUrls || [],
        }),
        onDone: {
          target: 'health_checking',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'integration_setup'],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    health_checking: {
      invoke: {
        src: 'performHealthCheck',
        input: ({ context }) => ({ tenantId: context.tenantId! }),
        onDone: {
          target: 'notifying',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'health_checking'],
          }),
        },
        onError: { target: 'provision_failed' },
      },
    },
    notifying: {
      invoke: {
        src: 'sendWelcomeNotification',
        input: ({ context }) => ({
          tenantId: context.tenantId!,
          adminEmail: context.request!.adminEmail,
          organizationName: context.request!.organizationName,
          tier: context.tier!,
        }),
        onDone: {
          target: 'active',
          actions: assign({
            completedSteps: ({ context }) => [...context.completedSteps, 'notifying'],
            endTime: () => new Date().toISOString(),
          }),
        },
        onError: {
          // Non-critical - mark active anyway
          target: 'active',
          actions: assign({
            endTime: () => new Date().toISOString(),
          }),
        },
      },
    },
    active: {
      type: 'final',
      entry: ({ context }) => {
        console.log(`Tenant ${context.tenantId} provisioned successfully in ${
          new Date(context.endTime!).getTime() - new Date(context.startTime!).getTime()
        }ms`);
      },
    },
    provision_failed: {
      entry: assign({
        error: ({ event }) => event.error || 'Unknown provisioning error',
      }),
      always: {
        target: 'rolling_back',
      },
    },
    rolling_back: {
      invoke: {
        src: 'rollbackProvisioning',
        input: ({ context }) => ({
          tenantId: context.tenantId,
          rollbackActions: context.rollbackActions,
          error: context.error,
        }),
        onDone: {
          target: 'rolled_back',
          actions: assign({
            rollbackStatus: 'completed',
            endTime: () => new Date().toISOString(),
          }),
        },
        onError: {
          target: 'rollback_failed',
          actions: assign({
            rollbackStatus: 'failed',
          }),
        },
      },
    },
    rolled_back: {
      type: 'final',
      entry: ({ context }) => {
        console.error(`Tenant provisioning failed and rolled back. Error: ${context.error}`);
      },
    },
    rollback_failed: {
      type: 'final',
      entry: ({ context }) => {
        console.error(`CRITICAL: Rollback failed for tenant ${context.tenantId}. Manual intervention required.`);
        // Trigger PagerDuty/alerting
      },
    },
  },
});

3.2 State Transition Diagram

                    START_PROVISIONING
                            │
                            ▼
┌────────────────────────────────────────────────────────────────┐
│                        PENDING                                  │
└────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌────────────────────────────────────────────────────────────────┐
│                       VALIDATING                                │
│  • Unique org name check                                        │
│  • Email format validation                                      │
│  • Tier & regulatory profile validation                        │
└────────────────────────────────────────────────────────────────┘
                            │
                ┌───────────┴───────────┐
                │                       │
             VALID                  INVALID
                │                       │
                ▼                       ▼
┌───────────────────────────┐   ┌──────────────────┐
│      REGISTERING          │   │ PROVISION_FAILED │
│  • Create tenant record   │   └──────────────────┘
│  • Generate tenant_id     │            │
└───────────────────────────┘            │
                │                        │
                ▼                        │
┌───────────────────────────┐            │
│      DB_CREATING          │            │
│  • Prisma migrate         │            │
│  • Enable RLS policies    │            │
└───────────────────────────┘            │
                │                        │
                ▼                        │
┌───────────────────────────┐            │
│    KEYS_GENERATING        │            │
│  • GCP KMS DEK + KEK      │            │
│  • Store key IDs          │            │
└───────────────────────────┘            │
                │                        │
                ▼                        │
┌───────────────────────────┐            │
│   STORAGE_ALLOCATING      │            │
│  • Create GCS bucket      │            │
│  • Set CORS + lifecycle   │            │
└───────────────────────────┘            │
                │                        │
                ▼                        │
┌───────────────────────────┐            │
│      CONFIGURING          │            │
│  • RBAC roles             │            │
│  • Workflow templates     │            │
│  • Document categories    │            │
└───────────────────────────┘            │
                │                        │
                ▼                        │
┌───────────────────────────┐            │
│     USER_CREATING         │            │
│  • Admin account          │            │
│  • Temporary password     │            │
│  • MFA enrollment         │            │
└───────────────────────────┘            │
                │                        │
                ▼                        │
┌───────────────────────────┐            │
│   COMPLIANCE_SETUP        │            │
│  • Enable Part 11/HIPAA   │            │
│  • Configure audit level  │            │
└───────────────────────────┘            │
                │                        │
                ▼                        │
┌───────────────────────────┐            │
│   INTEGRATION_SETUP       │            │
│  • API keys               │            │
│  • Webhook URLs           │            │
└───────────────────────────┘            │
                │                        │
                ▼                        │
┌───────────────────────────┐            │
│    HEALTH_CHECKING        │            │
│  • DB read/write test     │────────────┘
│  • Encryption test        │      (any failure)
│  • Storage test           │
│  • API test               │
└───────────────────────────┘
                │
             SUCCESS
                │
                ▼
┌───────────────────────────┐
│       NOTIFYING           │
│  • Welcome email          │
└───────────────────────────┘
                │
                ▼
┌───────────────────────────┐
│         ACTIVE            │  ◄─── Tenant Ready
└───────────────────────────┘

         (on any failure)
                │
                ▼
┌───────────────────────────┐
│     ROLLING_BACK          │
│  • Reverse each step      │
│  • Clean up resources     │
│  • Log rollback actions   │
└───────────────────────────┘
                │
       ┌────────┴────────┐
       │                 │
   SUCCESS           FAILURE
       │                 │
       ▼                 ▼
┌─────────────┐   ┌──────────────────┐
│ ROLLED_BACK │   │ ROLLBACK_FAILED  │ ◄─── CRITICAL ALERT
└─────────────┘   └──────────────────┘

4. Provisioning Steps Specification

4.1 Step 1: Tenant Registration

Purpose: Create tenant record in central registry with unique identifier

Inputs:

interface TenantRegistrationInput {
  organizationName: string;
  adminEmail: string;
  tier: SubscriptionTier;
  regulatoryProfile: RegulatoryProfile;
  billingContact?: ContactInfo;
  technicalContact?: ContactInfo;
}

interface RegulatoryProfile {
  requireFdaPart11: boolean;
  requireHipaa: boolean;
  requireSoc2: boolean;
  dataResidency?: 'US' | 'EU' | 'APAC';
}

Actions:

Generate unique tenant_id (CUID format: clr8x3y0a0000qz8r9b0c1d2e)
Validate organization name uniqueness (case-insensitive)
Create Tenant record in registry database
Create Subscription record linked to tier
Initialize TenantMetadata with regulatory profile
Record provisioning start in audit log

Database Schema:

model Tenant {
  id                String           @id @default(cuid())
  organizationName  String           @unique
  slug              String           @unique // URL-safe: "acme-biotech"
  status            TenantStatus     @default(PROVISIONING)
  tier              SubscriptionTier

  // Contacts
  adminEmail        String
  billingEmail      String?
  technicalEmail    String?

  // Regulatory
  fdaPart11Enabled  Boolean          @default(false)
  hipaaEnabled      Boolean          @default(false)
  soc2Enabled       Boolean          @default(true)
  dataResidency     String?          // 'US', 'EU', 'APAC'

  // Resource IDs
  databaseSchemaName String?         // PostgreSQL schema name
  kmsKeyRingId       String?         // GCP KMS key ring
  storageBucketName  String?         // GCS bucket

  // Provisioning
  provisioningJobId  String?
  provisionedAt      DateTime?
  lastHealthCheck    DateTime?

  // Metadata
  createdAt         DateTime         @default(now())
  updatedAt         DateTime         @updatedAt
  deletedAt         DateTime?        // Soft delete

  @@index([status])
  @@map("tenants")
}

enum TenantStatus {
  PROVISIONING
  ACTIVE
  SUSPENDED
  DEPROVISIONING
  DELETED
}

Rollback Action:

async function rollbackRegistration(tenantId: string): Promise<void> {
  await prisma.tenant.update({
    where: { id: tenantId },
    data: {
      status: TenantStatus.DELETED,
      deletedAt: new Date(),
    },
  });
  // Audit log the rollback
  await auditLog.log({
    tenantId,
    action: 'tenant.registration.rolled_back',
    performedBy: 'SYSTEM',
  });
}

Expected Duration: < 500ms

4.2 Step 2: Database Schema Creation

Purpose: Create tenant-specific PostgreSQL schema with Row-Level Security (RLS)

Strategy: Schema-per-Tenant (not table-per-tenant or database-per-tenant)

Rationale:

Isolation: Each tenant gets dedicated PostgreSQL schema
Performance: Native PostgreSQL RLS filtering
Compliance: Complete data separation for HIPAA/Part 11
Scalability: Single database supports 1,000+ tenants

Actions:

Create PostgreSQL schema: tenant_<tenant_id>
Run Prisma migrations for tenant schema
Enable Row-Level Security on all tables
Create RLS policies: WHERE tenant_id = current_setting('app.current_tenant_id')
Grant schema permissions to application role
Create tenant-scoped database functions (e.g., get_current_tenant_id())

Implementation:

async function createTenantDatabaseSchema(tenantId: string): Promise<void> {
  const schemaName = `tenant_${tenantId}`;

  // 1. Create schema
  await prisma.$executeRawUnsafe(`CREATE SCHEMA IF NOT EXISTS "${schemaName}"`);

  // 2. Set search_path for Prisma migration
  await prisma.$executeRawUnsafe(`SET search_path TO "${schemaName}"`);

  // 3. Run Prisma migrations (creates all tables)
  await execAsync(`npx prisma migrate deploy --schema=./prisma/schema.prisma`);

  // 4. Enable RLS on all tenant tables
  const tenantTables = [
    'work_orders',
    'parties',
    'job_plans',
    'persons',
    'approvals',
    'electronic_signatures',
    'audit_trail',
    // ... all 22 tables from schema
  ];

  for (const table of tenantTables) {
    await prisma.$executeRawUnsafe(`
      ALTER TABLE "${schemaName}"."${table}" ENABLE ROW LEVEL SECURITY;
    `);

    // Create RLS policy
    await prisma.$executeRawUnsafe(`
      CREATE POLICY tenant_isolation_policy ON "${schemaName}"."${table}"
      USING (tenant_id = current_setting('app.current_tenant_id')::text);
    `);
  }

  // 5. Create tenant-scoped helper function
  await prisma.$executeRawUnsafe(`
    CREATE OR REPLACE FUNCTION "${schemaName}".get_current_tenant_id()
    RETURNS TEXT AS $$
    BEGIN
      RETURN current_setting('app.current_tenant_id', TRUE);
    END;
    $$ LANGUAGE plpgsql SECURITY DEFINER;
  `);

  // 6. Update tenant record with schema name
  await prisma.tenant.update({
    where: { id: tenantId },
    data: { databaseSchemaName: schemaName },
  });

  console.log(`✓ Database schema ${schemaName} created with RLS enabled`);
}

RLS Context Setting (per-request middleware):

export async function setTenantContext(
  tenantId: string,
  prismaClient: PrismaClient
): Promise<void> {
  await prismaClient.$executeRawUnsafe(`
    SET app.current_tenant_id = '${tenantId}';
  `);
}

// In API middleware
app.use(async (req, res, next) => {
  const tenantId = req.user?.tenantId; // From JWT
  if (tenantId) {
    await setTenantContext(tenantId, prisma);
  }
  next();
});

Rollback Action:

async function rollbackDatabaseSchema(tenantId: string): Promise<void> {
  const schemaName = `tenant_${tenantId}`;

  // Drop entire schema (cascades to all tables)
  await prisma.$executeRawUnsafe(`DROP SCHEMA IF EXISTS "${schemaName}" CASCADE`);

  await prisma.tenant.update({
    where: { id: tenantId },
    data: { databaseSchemaName: null },
  });

  console.log(`✓ Rolled back database schema ${schemaName}`);
}

Expected Duration: < 30 seconds

4.3 Step 3: Encryption Key Generation

Purpose: Generate tenant-specific DEK and KEK via GCP Cloud KMS

Cross-Reference: D.6.1 - Key Management & Encryption

Key Hierarchy:

GCP KMS Key Ring (per tenant)
└── KEK (Key Encryption Key)
    └── DEK (Data Encryption Key) — encrypted by KEK

Actions:

Create GCP KMS Key Ring: projects/{project}/locations/{region}/keyRings/tenant-{tenantId}
Generate KEK: cryptoKeys/tenant-kek (AES-256, automatic rotation every 90 days)
Generate DEK: Random 256-bit AES key
Encrypt DEK with KEK (envelope encryption)
Store encrypted DEK in tenant metadata
Create key rotation schedule (90-day KEK, 30-day DEK)

Implementation:

import { KeyManagementServiceClient } from '@google-cloud/kms';

async function generateTenantEncryptionKeys(tenantId: string): Promise<{
  dekKeyId: string;
  kekKeyId: string;
}> {
  const kmsClient = new KeyManagementServiceClient();
  const projectId = process.env.GCP_PROJECT_ID!;
  const locationId = 'us-central1'; // or from tenant data residency
  const keyRingId = `tenant-${tenantId}`;

  // 1. Create Key Ring
  const keyRingPath = kmsClient.keyRingPath(projectId, locationId, keyRingId);
  try {
    await kmsClient.createKeyRing({
      parent: kmsClient.locationPath(projectId, locationId),
      keyRingId,
    });
  } catch (err) {
    if (!err.message.includes('ALREADY_EXISTS')) throw err;
  }

  // 2. Create KEK (Key Encryption Key)
  const kekId = 'tenant-kek';
  const kekPath = kmsClient.cryptoKeyPath(projectId, locationId, keyRingId, kekId);
  const [kek] = await kmsClient.createCryptoKey({
    parent: keyRingPath,
    cryptoKeyId: kekId,
    cryptoKey: {
      purpose: 'ENCRYPT_DECRYPT',
      versionTemplate: {
        algorithm: 'GOOGLE_SYMMETRIC_ENCRYPTION', // AES-256-GCM
        protectionLevel: 'SOFTWARE', // or 'HSM' for Enterprise tier
      },
      rotationPeriod: { seconds: 90 * 24 * 60 * 60 }, // 90 days
      nextRotationTime: {
        seconds: Math.floor(Date.now() / 1000) + 90 * 24 * 60 * 60,
      },
    },
  });

  // 3. Generate DEK (Data Encryption Key)
  const dek = crypto.randomBytes(32); // 256-bit AES key

  // 4. Encrypt DEK with KEK (envelope encryption)
  const [encryptResponse] = await kmsClient.encrypt({
    name: kekPath,
    plaintext: dek,
  });
  const encryptedDek = encryptResponse.ciphertext as Uint8Array;

  // 5. Store encrypted DEK in tenant metadata
  await prisma.tenant.update({
    where: { id: tenantId },
    data: {
      kmsKeyRingId: keyRingPath,
      encryptedDek: Buffer.from(encryptedDek).toString('base64'),
    },
  });

  // 6. Create audit log entry
  await auditLog.log({
    tenantId,
    action: 'encryption_keys.generated',
    performedBy: 'SYSTEM',
    metadata: {
      keyRingPath,
      kekPath,
      algorithm: 'AES-256-GCM',
      rotationPeriod: '90 days',
    },
  });

  console.log(`✓ Generated encryption keys for tenant ${tenantId}`);

  return {
    dekKeyId: Buffer.from(dek).toString('base64'),
    kekKeyId: kekPath,
  };
}

Key Usage (Encryption Example):

async function encryptData(
  tenantId: string,
  plaintext: string
): Promise<string> {
  const kmsClient = new KeyManagementServiceClient();

  // Retrieve tenant's KEK path
  const tenant = await prisma.tenant.findUnique({
    where: { id: tenantId },
    select: { kmsKeyRingId: true, encryptedDek: true },
  });

  // Decrypt DEK using KEK
  const [decryptResponse] = await kmsClient.decrypt({
    name: tenant.kmsKeyRingId!,
    ciphertext: Buffer.from(tenant.encryptedDek!, 'base64'),
  });
  const dek = decryptResponse.plaintext as Uint8Array;

  // Encrypt data with DEK (AES-256-GCM)
  const iv = crypto.randomBytes(12);
  const cipher = crypto.createCipheriv('aes-256-gcm', dek, iv);
  let encrypted = cipher.update(plaintext, 'utf8', 'base64');
  encrypted += cipher.final('base64');
  const authTag = cipher.getAuthTag();

  // Return IV + AuthTag + Ciphertext (all base64)
  return JSON.stringify({
    iv: iv.toString('base64'),
    authTag: authTag.toString('base64'),
    ciphertext: encrypted,
  });
}

Rollback Action:

async function rollbackEncryptionKeys(tenantId: string): Promise<void> {
  const kmsClient = new KeyManagementServiceClient();
  const projectId = process.env.GCP_PROJECT_ID!;
  const locationId = 'us-central1';
  const keyRingId = `tenant-${tenantId}`;
  const keyRingPath = kmsClient.keyRingPath(projectId, locationId, keyRingId);

  // NOTE: GCP KMS does NOT support key ring deletion
  // Instead: schedule KEK for destruction (24-hour window)
  const kekPath = kmsClient.cryptoKeyPath(projectId, locationId, keyRingId, 'tenant-kek');

  const [versions] = await kmsClient.listCryptoKeyVersions({ parent: kekPath });
  for (const version of versions) {
    await kmsClient.destroyCryptoKeyVersion({
      name: version.name!,
    });
  }

  // Clear tenant metadata
  await prisma.tenant.update({
    where: { id: tenantId },
    data: {
      kmsKeyRingId: null,
      encryptedDek: null,
    },
  });

  console.log(`✓ Scheduled KEK destruction for tenant ${tenantId}`);
}

Expected Duration: < 5 seconds

4.4 Step 4: Storage Allocation

Purpose: Create tenant-specific GCS bucket for documents and attachments

Bucket Naming: coditect-bio-qms-{tenantId} (globally unique)

Actions:

Create GCS bucket with lifecycle rules
Enable versioning (30-day retention for compliance)
Configure CORS for direct browser uploads
Set IAM policy: tenant service account only
Create folder structure: /documents/, /attachments/, /exports/, /backups/
Store bucket name in tenant metadata

Implementation:

import { Storage } from '@google-cloud/storage';

async function allocateTenantStorage(tenantId: string): Promise<{
  bucketName: string;
}> {
  const storage = new Storage();
  const bucketName = `coditect-bio-qms-${tenantId}`;

  // 1. Create bucket
  const [bucket] = await storage.createBucket(bucketName, {
    location: 'US', // or from tenant data residency
    storageClass: 'STANDARD',
    versioning: { enabled: true },
    lifecycle: {
      rule: [
        {
          action: { type: 'Delete' },
          condition: { numNewerVersions: 5 }, // Keep 5 versions
        },
        {
          action: { type: 'Delete' },
          condition: { age: 365 * 3 }, // Delete after 3 years (or tier-based)
        },
      ],
    },
    cors: [
      {
        origin: ['https://app.coditect-bio-qms.com'],
        method: ['GET', 'POST', 'PUT', 'DELETE'],
        responseHeader: ['Content-Type', 'Authorization'],
        maxAgeSeconds: 3600,
      },
    ],
  });

  // 2. Set IAM policy (tenant service account only)
  await bucket.setIamPolicy({
    bindings: [
      {
        role: 'roles/storage.objectAdmin',
        members: [`serviceAccount:tenant-${tenantId}@${process.env.GCP_PROJECT_ID}.iam.gserviceaccount.com`],
      },
    ],
  });

  // 3. Create folder structure (GCS doesn't have real folders, but we create marker objects)
  const folders = ['documents/', 'attachments/', 'exports/', 'backups/'];
  for (const folder of folders) {
    await bucket.file(folder).save('', { metadata: { contentType: 'application/x-directory' } });
  }

  // 4. Update tenant record
  await prisma.tenant.update({
    where: { id: tenantId },
    data: { storageBucketName: bucketName },
  });

  console.log(`✓ Allocated storage bucket ${bucketName}`);

  return { bucketName };
}

Rollback Action:

async function rollbackStorage(tenantId: string): Promise<void> {
  const storage = new Storage();
  const tenant = await prisma.tenant.findUnique({
    where: { id: tenantId },
    select: { storageBucketName: true },
  });

  if (tenant?.storageBucketName) {
    const bucket = storage.bucket(tenant.storageBucketName);

    // Delete all files (force delete)
    await bucket.deleteFiles({ force: true });

    // Delete bucket
    await bucket.delete();

    console.log(`✓ Deleted storage bucket ${tenant.storageBucketName}`);
  }

  await prisma.tenant.update({
    where: { id: tenantId },
    data: { storageBucketName: null },
  });
}

Expected Duration: < 3 seconds

4.5 Step 5: Initial Configuration

Purpose: Initialize tenant-scoped RBAC roles, workflow templates, and document categories

Actions:

Create default RBAC roles (per C.1.4 - RBAC Model)
Seed workflow templates (Work Order lifecycle states)
Create document categories (SOPs, protocols, training records)
Initialize approval chains
Configure email notification templates

RBAC Roles (Default):

const DEFAULT_ROLES = [
  { name: 'SYSTEM_OWNER', permissions: ['*'] },
  { name: 'QA_MANAGER', permissions: ['approve:work_orders', 'review:documents', 'audit:all'] },
  { name: 'LAB_MANAGER', permissions: ['create:work_orders', 'assign:work_orders', 'view:reports'] },
  { name: 'TECHNICIAN', permissions: ['execute:work_orders', 'log:time_entries', 'view:job_plans'] },
  { name: 'READ_ONLY', permissions: ['view:work_orders', 'view:reports'] },
];

Implementation:

async function initializeTenantConfiguration(
  tenantId: string,
  tier: SubscriptionTier
): Promise<void> {
  const schemaName = `tenant_${tenantId}`;

  // 1. Create RBAC roles
  for (const roleData of DEFAULT_ROLES) {
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."roles" (id, tenant_id, name, permissions, created_at)
      VALUES (gen_random_uuid(), '${tenantId}', '${roleData.name}',
              '${JSON.stringify(roleData.permissions)}', NOW());
    `);
  }

  // 2. Seed workflow templates
  const workflowTemplates = [
    {
      name: 'Standard Work Order Lifecycle',
      states: ['DRAFT', 'PLANNED', 'SCHEDULED', 'IN_PROGRESS', 'PENDING_REVIEW', 'APPROVED', 'COMPLETED'],
      transitions: {
        DRAFT: ['PLANNED'],
        PLANNED: ['SCHEDULED', 'DRAFT'],
        SCHEDULED: ['IN_PROGRESS'],
        IN_PROGRESS: ['PENDING_REVIEW'],
        PENDING_REVIEW: ['APPROVED', 'REJECTED'],
        APPROVED: ['COMPLETED'],
        REJECTED: ['DRAFT'],
      },
    },
  ];

  for (const template of workflowTemplates) {
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."workflow_templates"
      (id, tenant_id, name, states, transitions, created_at)
      VALUES (gen_random_uuid(), '${tenantId}', '${template.name}',
              '${JSON.stringify(template.states)}',
              '${JSON.stringify(template.transitions)}', NOW());
    `);
  }

  // 3. Create document categories
  const docCategories = [
    'Standard Operating Procedures (SOPs)',
    'Protocols',
    'Training Records',
    'Quality Control Documents',
    'Validation Reports',
    'Change Control Records',
  ];

  for (const category of docCategories) {
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."document_categories"
      (id, tenant_id, name, created_at)
      VALUES (gen_random_uuid(), '${tenantId}', '${category}', NOW());
    `);
  }

  // 4. Initialize approval chains (based on tier)
  if (tier === SubscriptionTier.PROFESSIONAL || tier === SubscriptionTier.ENTERPRISE) {
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."approval_chains"
      (id, tenant_id, name, required_roles, sequence, created_at)
      VALUES
        (gen_random_uuid(), '${tenantId}', 'Standard Approval Chain',
         '["SYSTEM_OWNER", "QA_MANAGER"]', 1, NOW()),
        (gen_random_uuid(), '${tenantId}', 'Regulatory Approval Chain',
         '["SYSTEM_OWNER", "QA_MANAGER", "REGULATORY_AFFAIRS"]', 2, NOW());
    `);
  }

  console.log(`✓ Initialized configuration for tenant ${tenantId}`);
}

Expected Duration: < 2 seconds

4.6 Step 6: Admin User Creation

Purpose: Create first admin account with temporary password and MFA enrollment

Actions:

Create Person record in tenant schema
Hash temporary password (bcrypt, 12 rounds)
Create auth record in central auth database
Generate MFA enrollment token (TOTP)
Send temporary password + MFA setup email
Mark account as requiring password change on first login

Implementation:

import bcrypt from 'bcrypt';
import { authenticator } from 'otplib';

async function createTenantAdminUser(input: {
  tenantId: string;
  email: string;
  organizationName: string;
}): Promise<{ userId: string; tempPassword: string }> {
  const { tenantId, email, organizationName } = input;
  const schemaName = `tenant_${tenantId}`;

  // 1. Generate temporary password (20-char random)
  const tempPassword = crypto.randomBytes(15).toString('base64').slice(0, 20);
  const hashedPassword = await bcrypt.hash(tempPassword, 12);

  // 2. Create Person record
  const userId = crypto.randomUUID();
  await prisma.$executeRawUnsafe(`
    INSERT INTO "${schemaName}"."persons"
    (id, tenant_id, name, email, active, department, created_at, updated_at)
    VALUES ('${userId}', '${tenantId}', 'Admin User', '${email}',
            true, 'Administration', NOW(), NOW());
  `);

  // 3. Create auth record (central auth database)
  const mfaSecret = authenticator.generateSecret();
  await prisma.auth.create({
    data: {
      id: userId,
      tenantId,
      email,
      passwordHash: hashedPassword,
      mfaSecret,
      mfaEnabled: false, // user must enroll on first login
      requirePasswordChange: true,
      roles: ['SYSTEM_OWNER'],
      createdAt: new Date(),
    },
  });

  // 4. Assign SYSTEM_OWNER role
  const systemOwnerRole = await prisma.$queryRawUnsafe(`
    SELECT id FROM "${schemaName}"."roles" WHERE name = 'SYSTEM_OWNER' LIMIT 1;
  `);

  await prisma.$executeRawUnsafe(`
    INSERT INTO "${schemaName}"."user_roles" (user_id, role_id, assigned_at)
    VALUES ('${userId}', '${systemOwnerRole[0].id}', NOW());
  `);

  console.log(`✓ Created admin user ${email} for tenant ${tenantId}`);

  return { userId, tempPassword };
}

MFA Enrollment Flow:

// User logs in with temp password → forced to:
// 1. Change password
// 2. Scan QR code for TOTP (Google Authenticator, Authy)
// 3. Enter TOTP code to verify
// 4. Save backup codes (10x single-use codes)

async function enrollMFA(userId: string, totpCode: string): Promise<void> {
  const user = await prisma.auth.findUnique({ where: { id: userId } });

  const isValid = authenticator.verify({
    token: totpCode,
    secret: user.mfaSecret!,
  });

  if (!isValid) throw new Error('Invalid TOTP code');

  // Generate backup codes
  const backupCodes = Array.from({ length: 10 }, () =>
    crypto.randomBytes(4).toString('hex').toUpperCase()
  );

  await prisma.auth.update({
    where: { id: userId },
    data: {
      mfaEnabled: true,
      backupCodes: JSON.stringify(backupCodes),
    },
  });
}

Rollback Action:

async function rollbackAdminUser(tenantId: string, userId: string): Promise<void> {
  const schemaName = `tenant_${tenantId}`;

  // Delete from auth database
  await prisma.auth.delete({ where: { id: userId } });

  // Delete from tenant schema
  await prisma.$executeRawUnsafe(`
    DELETE FROM "${schemaName}"."user_roles" WHERE user_id = '${userId}';
  `);
  await prisma.$executeRawUnsafe(`
    DELETE FROM "${schemaName}"."persons" WHERE id = '${userId}';
  `);

  console.log(`✓ Rolled back admin user ${userId}`);
}

Expected Duration: < 1 second

4.7 Step 7: Compliance Profile Setup

Purpose: Enable regulatory compliance features based on subscription tier

Compliance Matrix:

Feature	Starter	Professional	Enterprise
SOC 2 Type II	✅	✅	✅
FDA Part 11	❌	✅	✅
HIPAA	❌	❌	✅
E-Signatures	❌	✅	✅
Audit Trail Retention	1 year	3 years	7 years
Data Encryption	AES-256	AES-256	AES-256 + HSM
PHI Support	❌	❌	✅
BAA Agreement	❌	❌	✅

Implementation:

async function setupTenantComplianceProfile(input: {
  tenantId: string;
  tier: SubscriptionTier;
  regulatoryProfile: RegulatoryProfile;
}): Promise<void> {
  const { tenantId, tier, regulatoryProfile } = input;
  const schemaName = `tenant_${tenantId}`;

  // 1. Enable/disable compliance features
  const complianceConfig = {
    fdaPart11: tier !== SubscriptionTier.STARTER && regulatoryProfile.requireFdaPart11,
    hipaa: tier === SubscriptionTier.ENTERPRISE && regulatoryProfile.requireHipaa,
    soc2: true, // all tiers
    eSignatures: tier !== SubscriptionTier.STARTER,
  };

  await prisma.tenant.update({
    where: { id: tenantId },
    data: {
      fdaPart11Enabled: complianceConfig.fdaPart11,
      hipaaEnabled: complianceConfig.hipaa,
      soc2Enabled: complianceConfig.soc2,
    },
  });

  // 2. Configure audit trail retention
  const retentionYears = tier === SubscriptionTier.ENTERPRISE ? 7 :
                         tier === SubscriptionTier.PROFESSIONAL ? 3 : 1;

  await prisma.$executeRawUnsafe(`
    INSERT INTO "${schemaName}"."compliance_settings"
    (id, tenant_id, audit_retention_years, created_at)
    VALUES (gen_random_uuid(), '${tenantId}', ${retentionYears}, NOW());
  `);

  // 3. Enable Part 11 controls if required
  if (complianceConfig.fdaPart11) {
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."part11_controls"
      (id, tenant_id, require_timestamps, require_signature_binding,
       require_reason_for_change, created_at)
      VALUES (gen_random_uuid(), '${tenantId}', true, true, true, NOW());
    `);
  }

  // 4. Enable HIPAA PHI controls if required
  if (complianceConfig.hipaa) {
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."hipaa_controls"
      (id, tenant_id, phi_encryption_required, minimum_necessary_access,
       breach_notification_enabled, created_at)
      VALUES (gen_random_uuid(), '${tenantId}', true, true, true, NOW());
    `);

    // Create BAA agreement record (requires manual DocuSign)
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."baa_agreements"
      (id, tenant_id, status, created_at)
      VALUES (gen_random_uuid(), '${tenantId}', 'PENDING_SIGNATURE', NOW());
    `);
  }

  console.log(`✓ Configured compliance profile for tenant ${tenantId}`);
}

Expected Duration: < 1 second

4.8 Step 8: Integration Endpoints

Purpose: Generate API keys and configure webhook URLs for external integrations

Actions:

Generate API key (Bearer token, 64-char hex)
Generate API secret (for HMAC signature verification)
Create webhook endpoints for tenant
Configure external system connections (if provided)
Store credentials in secure vault (GCP Secret Manager)

Implementation:

async function setupTenantIntegrations(input: {
  tenantId: string;
  webhookUrls?: string[];
}): Promise<void> {
  const { tenantId, webhookUrls = [] } = input;
  const schemaName = `tenant_${tenantId}`;

  // 1. Generate API key and secret
  const apiKey = `bio_${crypto.randomBytes(32).toString('hex')}`;
  const apiSecret = crypto.randomBytes(32).toString('hex');
  const hashedSecret = await bcrypt.hash(apiSecret, 10);

  // 2. Store in tenant database
  await prisma.$executeRawUnsafe(`
    INSERT INTO "${schemaName}"."api_keys"
    (id, tenant_id, key_prefix, key_hash, secret_hash, created_at, expires_at)
    VALUES (gen_random_uuid(), '${tenantId}',
            '${apiKey.substring(0, 8)}',
            '${await bcrypt.hash(apiKey, 10)}',
            '${hashedSecret}',
            NOW(),
            NOW() + INTERVAL '1 year');
  `);

  // 3. Store API secret in GCP Secret Manager
  const { SecretManagerServiceClient } = require('@google-cloud/secret-manager');
  const client = new SecretManagerServiceClient();
  const parent = `projects/${process.env.GCP_PROJECT_ID}`;

  const [secret] = await client.createSecret({
    parent,
    secretId: `tenant-${tenantId}-api-secret`,
    secret: {
      replication: { automatic: {} },
    },
  });

  await client.addSecretVersion({
    parent: secret.name,
    payload: {
      data: Buffer.from(apiSecret, 'utf8'),
    },
  });

  // 4. Configure webhook URLs
  for (const url of webhookUrls) {
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."webhook_endpoints"
      (id, tenant_id, url, events, active, created_at)
      VALUES (gen_random_uuid(), '${tenantId}', '${url}',
              '["work_order.created", "work_order.completed", "approval.required"]',
              true, NOW());
    `);
  }

  console.log(`✓ Configured integrations for tenant ${tenantId}`);
  console.log(`   API Key: ${apiKey}`);
  console.log(`   API Secret: [stored in Secret Manager]`);
}

Expected Duration: < 2 seconds

4.9 Step 9: Health Check

Purpose: Validate all provisioning steps completed successfully

Health Checks:

Check	Test	Success Criteria
Database	Read/write test record	Record inserted and retrieved
Encryption	Encrypt/decrypt test data	Plaintext matches after roundtrip
Storage	Upload/download test file	File contents match
API	Call auth endpoint	200 OK response
Audit	Verify first audit log entry	Entry exists with correct tenant_id
RBAC	List default roles	5+ roles returned

Implementation:

async function performTenantHealthCheck(tenantId: string): Promise<{
  healthy: boolean;
  checks: Record<string, boolean>;
  errors: string[];
}> {
  const checks: Record<string, boolean> = {};
  const errors: string[] = [];
  const schemaName = `tenant_${tenantId}`;

  try {
    // 1. Database read/write test
    const testRecordId = crypto.randomUUID();
    await prisma.$executeRawUnsafe(`
      INSERT INTO "${schemaName}"."persons"
      (id, tenant_id, name, email, active, created_at, updated_at)
      VALUES ('${testRecordId}', '${tenantId}', 'Health Check Test',
              'healthcheck@test.local', false, NOW(), NOW());
    `);

    const [result] = await prisma.$queryRawUnsafe(`
      SELECT id FROM "${schemaName}"."persons" WHERE id = '${testRecordId}';
    `);

    if (result?.id === testRecordId) {
      checks.database = true;

      // Clean up test record
      await prisma.$executeRawUnsafe(`
        DELETE FROM "${schemaName}"."persons" WHERE id = '${testRecordId}';
      `);
    } else {
      checks.database = false;
      errors.push('Database read/write test failed');
    }
  } catch (err) {
    checks.database = false;
    errors.push(`Database error: ${err.message}`);
  }

  try {
    // 2. Encryption roundtrip test
    const plaintext = 'CODITECT Health Check Test Data';
    const encrypted = await encryptData(tenantId, plaintext);
    const decrypted = await decryptData(tenantId, encrypted);

    checks.encryption = (decrypted === plaintext);
    if (!checks.encryption) {
      errors.push('Encryption roundtrip failed');
    }
  } catch (err) {
    checks.encryption = false;
    errors.push(`Encryption error: ${err.message}`);
  }

  try {
    // 3. Storage upload/download test
    const { Storage } = require('@google-cloud/storage');
    const storage = new Storage();
    const tenant = await prisma.tenant.findUnique({
      where: { id: tenantId },
      select: { storageBucketName: true },
    });

    const bucket = storage.bucket(tenant.storageBucketName!);
    const fileName = 'health-check-test.txt';
    const testContent = 'CODITECT Health Check Test File';

    await bucket.file(fileName).save(testContent);
    const [downloaded] = await bucket.file(fileName).download();

    checks.storage = (downloaded.toString('utf8') === testContent);
    if (!checks.storage) {
      errors.push('Storage upload/download test failed');
    }

    // Clean up test file
    await bucket.file(fileName).delete();
  } catch (err) {
    checks.storage = false;
    errors.push(`Storage error: ${err.message}`);
  }

  try {
    // 4. API auth endpoint test
    const response = await fetch(`${process.env.API_BASE_URL}/api/auth/health`, {
      headers: { 'X-Tenant-ID': tenantId },
    });

    checks.api = response.ok;
    if (!checks.api) {
      errors.push(`API health check failed: ${response.status}`);
    }
  } catch (err) {
    checks.api = false;
    errors.push(`API error: ${err.message}`);
  }

  try {
    // 5. Audit log verification
    const [auditEntry] = await prisma.$queryRawUnsafe(`
      SELECT id FROM "${schemaName}"."audit_trail"
      WHERE tenant_id = '${tenantId}'
      ORDER BY performed_at ASC LIMIT 1;
    `);

    checks.audit = !!auditEntry;
    if (!checks.audit) {
      errors.push('No audit log entries found');
    }
  } catch (err) {
    checks.audit = false;
    errors.push(`Audit log error: ${err.message}`);
  }

  try {
    // 6. RBAC roles verification
    const roles = await prisma.$queryRawUnsafe(`
      SELECT COUNT(*) as count FROM "${schemaName}"."roles";
    `);

    checks.rbac = (roles[0].count >= 5);
    if (!checks.rbac) {
      errors.push(`Expected 5+ roles, found ${roles[0].count}`);
    }
  } catch (err) {
    checks.rbac = false;
    errors.push(`RBAC error: ${err.message}`);
  }

  const healthy = Object.values(checks).every(c => c === true);

  // Log health check result
  await auditLog.log({
    tenantId,
    action: 'tenant.health_check',
    performedBy: 'SYSTEM',
    metadata: { healthy, checks, errors },
  });

  if (healthy) {
    console.log(`✓ Health check passed for tenant ${tenantId}`);
  } else {
    console.error(`✗ Health check failed for tenant ${tenantId}:`, errors);
  }

  return { healthy, checks, errors };
}

Expected Duration: < 5 seconds

4.10 Step 10: Welcome Notification

Purpose: Send welcome email to admin with setup guide and credentials

Email Template:

Subject: Welcome to CODITECT BIO-QMS - Your Account is Ready

Dear {{organizationName}} Team,

Your CODITECT BIO-QMS {{tier}} account has been successfully provisioned and is ready to use!

🔐 Login Credentials
Email: {{adminEmail}}
Temporary Password: {{tempPassword}}
Login URL: https://app.coditect-bio-qms.com

⚠️ IMPORTANT: You will be required to:
1. Change your password on first login
2. Set up Multi-Factor Authentication (MFA)
3. Save your backup codes securely

📊 Your Subscription Details
Tier: {{tier}}
Users: {{maxUsers}}
Storage: {{storageGB}} GB
Compliance: {{complianceFeatures}}

🚀 Getting Started
1. Log in and complete security setup
2. Invite team members (Settings > Users)
3. Review pre-configured workflow templates
4. Create your first Work Order

📚 Resources
Documentation: https://docs.coditect-bio-qms.com
Support: support@coditect.ai
Training Videos: https://learn.coditect-bio-qms.com

Need help? Our support team is available 24/7 (Enterprise) or business hours (Starter/Professional).

Welcome aboard!
The CODITECT Team

Implementation:

import { SendGridClient } from '@sendgrid/mail';

async function sendTenantWelcomeNotification(input: {
  tenantId: string;
  adminEmail: string;
  organizationName: string;
  tier: SubscriptionTier;
}): Promise<void> {
  const { tenantId, adminEmail, organizationName, tier } = input;

  // Retrieve temp password from secure store
  const { tempPassword } = await getTempPasswordForTenant(tenantId);

  const tierQuotas = TIER_QUOTAS[tier];
  const complianceFeatures = Object.entries(tierQuotas.features)
    .filter(([_, enabled]) => enabled)
    .map(([feature, _]) => feature)
    .join(', ');

  const emailClient = new SendGridClient();
  emailClient.setApiKey(process.env.SENDGRID_API_KEY!);

  await emailClient.send({
    to: adminEmail,
    from: 'onboarding@coditect.ai',
    templateId: 'd-xxxxxxxxxxxxxxxxxxxxx', // SendGrid template ID
    dynamicTemplateData: {
      organizationName,
      adminEmail,
      tempPassword,
      tier: tier.toString(),
      maxUsers: tierQuotas.maxUsers,
      storageGB: tierQuotas.storageGB,
      complianceFeatures,
      loginUrl: 'https://app.coditect-bio-qms.com',
    },
  });

  console.log(`✓ Sent welcome email to ${adminEmail}`);
}

Expected Duration: < 1 second

5. Pre-Provisioning Validation

5.1 Validation Rules

interface ProvisioningRequest {
  organizationName: string;
  adminEmail: string;
  tier: SubscriptionTier;
  regulatoryProfile: RegulatoryProfile;
  webhookUrls?: string[];
  billingContact?: ContactInfo;
  technicalContact?: ContactInfo;
}

async function preProvisioningValidation(
  request: ProvisioningRequest
): Promise<{ isValid: boolean; errors: string[] }> {
  const errors: string[] = [];

  // 1. Organization name uniqueness
  const existingTenant = await prisma.tenant.findFirst({
    where: {
      organizationName: {
        equals: request.organizationName,
        mode: 'insensitive',
      },
      status: { not: TenantStatus.DELETED },
    },
  });

  if (existingTenant) {
    errors.push(`Organization name "${request.organizationName}" already exists`);
  }

  // 2. Email format validation
  const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
  if (!emailRegex.test(request.adminEmail)) {
    errors.push('Invalid admin email format');
  }

  // 3. Tier validation
  if (!Object.values(SubscriptionTier).includes(request.tier)) {
    errors.push(`Invalid subscription tier: ${request.tier}`);
  }

  // 4. Regulatory profile validation
  if (request.regulatoryProfile.requireHipaa && request.tier !== SubscriptionTier.ENTERPRISE) {
    errors.push('HIPAA compliance requires Enterprise tier');
  }

  if (request.regulatoryProfile.requireFdaPart11 && request.tier === SubscriptionTier.STARTER) {
    errors.push('FDA Part 11 compliance requires Professional or Enterprise tier');
  }

  // 5. Webhook URL validation
  if (request.webhookUrls) {
    for (const url of request.webhookUrls) {
      try {
        new URL(url);
        if (!url.startsWith('https://')) {
          errors.push(`Webhook URL must use HTTPS: ${url}`);
        }
      } catch {
        errors.push(`Invalid webhook URL: ${url}`);
      }
    }
  }

  // 6. Resource availability check
  const currentTenantCount = await prisma.tenant.count({
    where: { status: TenantStatus.ACTIVE },
  });

  const MAX_TENANTS = parseInt(process.env.MAX_TENANTS || '10000', 10);
  if (currentTenantCount >= MAX_TENANTS) {
    errors.push('Maximum tenant capacity reached. Contact sales.');
  }

  return {
    isValid: errors.length === 0,
    errors,
  };
}

6. Post-Provisioning Health Check

See Section 4.9 for complete health check implementation.

Summary:

6 validation checks (database, encryption, storage, API, audit, RBAC)
< 5 second execution time
Automatic rollback if any check fails
Audit log entry with full check results

7. Automated Rollback Mechanism

7.1 Rollback Strategy

Principle: Execute compensating transactions in reverse order of provisioning steps.

Rollback Order:

Notifying         → (skip - email already sent)
 Health Checking   → (no resources to clean up)
 Integration Setup → Delete API keys, webhook configs
 Compliance Setup  → Remove compliance settings
 User Creating     → Delete admin user from auth + tenant schema
 Configuring       → (no rollback - tenant schema will be dropped)
 Storage Allocating → Delete GCS bucket
 Keys Generating   → Schedule KEK destruction
 DB Creating       → Drop PostgreSQL schema
 Registering       → Mark tenant as DELETED

7.2 Rollback Implementation

async function executeProvisioningRollback(context: {
  tenantId: string | null;
  rollbackActions: RollbackAction[];
  error: any;
}): Promise<void> {
  const { tenantId, rollbackActions, error } = context;

  console.error(`🔄 Starting rollback for tenant ${tenantId}. Error: ${error}`);

  // Execute rollback actions in REVERSE order
  const reversedActions = [...rollbackActions].reverse();

  for (const action of reversedActions) {
    try {
      console.log(`  Rolling back step: ${action.step}`);

      switch (action.step) {
        case 'integration_setup':
          // Delete API keys and webhooks
          await rollbackIntegrations(action.data.tenantId);
          break;

        case 'user_creating':
          // Delete admin user
          await rollbackAdminUser(action.data.tenantId, action.data.userId);
          break;

        case 'storage_allocating':
          // Delete GCS bucket
          await rollbackStorage(action.data.tenantId);
          break;

        case 'keys_generating':
          // Schedule KEK destruction
          await rollbackEncryptionKeys(action.data.tenantId);
          break;

        case 'db_creating':
          // Drop PostgreSQL schema
          await rollbackDatabaseSchema(action.data.tenantId);
          break;

        case 'registering':
          // Mark tenant as deleted
          await rollbackRegistration(action.data.tenantId);
          break;

        default:
          console.warn(`  No rollback handler for step: ${action.step}`);
      }

      console.log(`  ✓ Rolled back ${action.step}`);
    } catch (rollbackErr) {
      console.error(`  ✗ Failed to rollback ${action.step}:`, rollbackErr);
      // Continue with remaining rollback steps (best effort)
    }
  }

  // Log rollback completion
  if (tenantId) {
    await auditLog.log({
      tenantId,
      action: 'tenant.provisioning.rolled_back',
      performedBy: 'SYSTEM',
      metadata: {
        originalError: error.toString(),
        rolledBackSteps: reversedActions.map(a => a.step),
      },
    });
  }

  console.log(`🔄 Rollback completed for tenant ${tenantId}`);
}

7.3 Rollback Monitoring

Alerting: If rollback fails, trigger PagerDuty P1 incident for manual intervention.

async function alertRollbackFailure(tenantId: string, error: any): Promise<void> {
  const { PagerDutyClient } = require('@pagerduty/client');
  const client = new PagerDutyClient({ token: process.env.PAGERDUTY_API_KEY });

  await client.incidents.create({
    incident: {
      type: 'incident',
      title: `CRITICAL: Tenant provisioning rollback failed - ${tenantId}`,
      service: { id: process.env.PAGERDUTY_SERVICE_ID, type: 'service_reference' },
      urgency: 'high',
      body: {
        type: 'incident_body',
        details: `Tenant ${tenantId} provisioning failed AND rollback failed. Manual cleanup required.\n\nError: ${error}`,
      },
    },
  });
}

8. Provisioning API Specification

8.1 Endpoints

`POST /api/tenants/provision`

Purpose: Initiate tenant provisioning workflow (async)

Request:

{
  "organizationName": "Acme Biosciences",
  "adminEmail": "admin@acmebio.com",
  "tier": "PROFESSIONAL",
  "regulatoryProfile": {
    "requireFdaPart11": true,
    "requireHipaa": false,
    "requireSoc2": true,
    "dataResidency": "US"
  },
  "webhookUrls": [
    "https://api.acmebio.com/webhooks/qms"
  ],
  "billingContact": {
    "name": "Jane Smith",
    "email": "billing@acmebio.com",
    "phone": "+1-555-0100"
  }
}

Response (202 Accepted):

{
  "jobId": "prov_clr8x3y0a0001qz8r9b0c1d2e",
  "status": "pending",
  "message": "Tenant provisioning started. Check status at /api/tenants/provision/:jobId/status",
  "estimatedCompletionTime": "2026-02-16T10:05:00Z"
}

`GET /api/tenants/provision/:jobId/status`

Purpose: Get current provisioning status

Response (In Progress):

{
  "jobId": "prov_clr8x3y0a0001qz8r9b0c1d2e",
  "tenantId": "clr8x3y0a0000qz8r9b0c1d2e",
  "status": "in_progress",
  "currentStep": "keys_generating",
  "completedSteps": [
    "validating",
    "registering",
    "db_creating"
  ],
  "totalSteps": 10,
  "progressPercent": 30,
  "startedAt": "2026-02-16T10:00:00Z",
  "estimatedCompletionTime": "2026-02-16T10:05:00Z"
}

Response (Completed):

{
  "jobId": "prov_clr8x3y0a0001qz8r9b0c1d2e",
  "tenantId": "clr8x3y0a0000qz8r9b0c1d2e",
  "status": "completed",
  "completedSteps": [
    "validating",
    "registering",
    "db_creating",
    "keys_generating",
    "storage_allocating",
    "configuring",
    "user_creating",
    "compliance_setup",
    "integration_setup",
    "health_checking",
    "notifying"
  ],
  "totalSteps": 10,
  "progressPercent": 100,
  "startedAt": "2026-02-16T10:00:00Z",
  "completedAt": "2026-02-16T10:04:32Z",
  "durationSeconds": 272,
  "credentials": {
    "adminEmail": "admin@acmebio.com",
    "loginUrl": "https://app.coditect-bio-qms.com",
    "apiKey": "bio_a1b2c3d4e5f6...",
    "message": "Temporary password sent to admin email"
  }
}

Response (Failed):

{
  "jobId": "prov_clr8x3y0a0001qz8r9b0c1d2e",
  "tenantId": null,
  "status": "failed",
  "currentStep": "keys_generating",
  "completedSteps": ["validating", "registering", "db_creating"],
  "error": {
    "code": "KMS_KEY_CREATION_FAILED",
    "message": "Failed to create encryption keys in GCP KMS",
    "details": "Insufficient IAM permissions: roles/cloudkms.admin required"
  },
  "rollbackStatus": "completed",
  "startedAt": "2026-02-16T10:00:00Z",
  "failedAt": "2026-02-16T10:02:15Z"
}

`POST /api/tenants/provision/:jobId/retry`

Purpose: Retry failed provisioning from last successful step

Response (202 Accepted):

{
  "jobId": "prov_clr8x3y0a0001qz8r9b0c1d2e",
  "status": "retrying",
  "message": "Provisioning retry started from step: keys_generating"
}

`DELETE /api/tenants/provision/:jobId/rollback`

Purpose: Manually trigger rollback for stuck provisioning

Response (200 OK):

{
  "jobId": "prov_clr8x3y0a0001qz8r9b0c1d2e",
  "status": "rolling_back",
  "message": "Manual rollback initiated"
}

8.2 Authentication

All provisioning API calls require:

Bearer token with admin or tenant:provision scope
Rate limiting: 10 requests/hour per API key (prevent abuse)

app.post('/api/tenants/provision',
  authenticateJWT(['admin', 'tenant:provision']),
  rateLimitByApiKey(10, 'hour'),
  async (req, res) => {
    // ... provisioning logic
  }
);

9. Database Schema Extensions

9.1 Tenant Registry Schema

// Add to central registry database (not tenant-scoped)

model Tenant {
  id                String           @id @default(cuid())
  organizationName  String           @unique
  slug              String           @unique
  status            TenantStatus     @default(PROVISIONING)
  tier              SubscriptionTier

  adminEmail        String
  billingEmail      String?
  technicalEmail    String?

  fdaPart11Enabled  Boolean          @default(false)
  hipaaEnabled      Boolean          @default(false)
  soc2Enabled       Boolean          @default(true)
  dataResidency     String?

  databaseSchemaName String?
  kmsKeyRingId       String?
  storageBucketName  String?
  encryptedDek       String?          // Base64-encoded encrypted DEK

  provisioningJobId  String?
  provisionedAt      DateTime?
  lastHealthCheck    DateTime?

  createdAt         DateTime         @default(now())
  updatedAt         DateTime         @updatedAt
  deletedAt         DateTime?

  subscriptions     Subscription[]
  provisioningJobs  ProvisioningJob[]

  @@index([status])
  @@index([tier])
  @@map("tenants")
}

model ProvisioningJob {
  id                String              @id @default(cuid())
  tenantId          String?
  tenant            Tenant?             @relation(fields: [tenantId], references: [id])

  status            ProvisioningStatus  @default(PENDING)
  currentStep       String?
  completedSteps    Json                @default("[]")
  rollbackActions   Json                @default("[]")

  error             String?
  rollbackStatus    String?

  startedAt         DateTime            @default(now())
  completedAt       DateTime?
  failedAt          DateTime?

  request           Json                // Original ProvisioningRequest

  @@index([tenantId])
  @@index([status])
  @@map("provisioning_jobs")
}

enum ProvisioningStatus {
  PENDING
  IN_PROGRESS
  COMPLETED
  FAILED
  ROLLING_BACK
  ROLLED_BACK
  ROLLBACK_FAILED
}

model Subscription {
  id                String           @id @default(cuid())
  tenantId          String
  tenant            Tenant           @relation(fields: [tenantId], references: [id])

  tier              SubscriptionTier
  status            SubscriptionStatus @default(ACTIVE)

  startDate         DateTime         @default(now())
  endDate           DateTime?
  renewalDate       DateTime?

  billingCycle      BillingCycle     @default(MONTHLY)
  pricePerMonth     Int              // in cents

  @@index([tenantId])
  @@map("subscriptions")
}

enum SubscriptionStatus {
  ACTIVE
  SUSPENDED
  CANCELLED
  EXPIRED
}

enum BillingCycle {
  MONTHLY
  ANNUAL
}

10. TypeScript Implementation

10.1 Type Definitions

// types/provisioning.ts

export enum SubscriptionTier {
  STARTER = 'STARTER',
  PROFESSIONAL = 'PROFESSIONAL',
  ENTERPRISE = 'ENTERPRISE',
}

export enum TenantStatus {
  PROVISIONING = 'PROVISIONING',
  ACTIVE = 'ACTIVE',
  SUSPENDED = 'SUSPENDED',
  DEPROVISIONING = 'DEPROVISIONING',
  DELETED = 'DELETED',
}

export interface RegulatoryProfile {
  requireFdaPart11: boolean;
  requireHipaa: boolean;
  requireSoc2: boolean;
  dataResidency?: 'US' | 'EU' | 'APAC';
}

export interface ContactInfo {
  name: string;
  email: string;
  phone?: string;
}

export interface ProvisioningRequest {
  organizationName: string;
  adminEmail: string;
  tier: SubscriptionTier;
  regulatoryProfile: RegulatoryProfile;
  webhookUrls?: string[];
  billingContact?: ContactInfo;
  technicalContact?: ContactInfo;
}

export interface TenantProvisioningContext {
  request: ProvisioningRequest | null;
  tenantId: string | null;
  jobId: string | null;
  tier: SubscriptionTier | null;
  validationResult: { isValid: boolean; errors: string[] } | null;
  rollbackActions: RollbackAction[];
  completedSteps: string[];
  error: any;
  rollbackStatus?: 'pending' | 'in_progress' | 'completed' | 'failed';
  startTime: string | null;
  endTime: string | null;
}

export interface RollbackAction {
  step: string;
  data: Record<string, any>;
}

export type TenantProvisioningEvent =
  | { type: 'START_PROVISIONING'; request: ProvisioningRequest; jobId: string }
  | { type: 'VALIDATION_SUCCESS' }
  | { type: 'VALIDATION_FAILED'; error: string }
  | { type: 'STEP_COMPLETED'; step: string }
  | { type: 'PROVISIONING_FAILED'; error: any }
  | { type: 'ROLLBACK_COMPLETED' };

10.2 Service Layer

// services/TenantProvisioningService.ts

import { createActor } from 'xstate';
import { tenantProvisioningMachine } from '../machines/tenantProvisioningMachine';
import { PrismaClient } from '@prisma/client';

export class TenantProvisioningService {
  private prisma: PrismaClient;
  private activeJobs: Map<string, any> = new Map();

  constructor(prisma: PrismaClient) {
    this.prisma = prisma;
  }

  async startProvisioning(request: ProvisioningRequest): Promise<{
    jobId: string;
    status: string;
  }> {
    // Create provisioning job record
    const job = await this.prisma.provisioningJob.create({
      data: {
        status: 'PENDING',
        request: request as any,
      },
    });

    // Create XState actor
    const actor = createActor(tenantProvisioningMachine, {
      input: {
        request,
        jobId: job.id,
      },
    });

    // Subscribe to state changes
    actor.subscribe((state) => {
      this.updateJobStatus(job.id, state);
    });

    // Start the machine
    actor.start();
    actor.send({ type: 'START_PROVISIONING', request, jobId: job.id });

    this.activeJobs.set(job.id, actor);

    return {
      jobId: job.id,
      status: 'pending',
    };
  }

  async getJobStatus(jobId: string): Promise<any> {
    const job = await this.prisma.provisioningJob.findUnique({
      where: { id: jobId },
      include: { tenant: true },
    });

    if (!job) {
      throw new Error(`Provisioning job ${jobId} not found`);
    }

    const completedSteps = (job.completedSteps as string[]) || [];
    const totalSteps = 10;

    return {
      jobId: job.id,
      tenantId: job.tenantId,
      status: job.status.toLowerCase(),
      currentStep: job.currentStep,
      completedSteps,
      totalSteps,
      progressPercent: Math.round((completedSteps.length / totalSteps) * 100),
      startedAt: job.startedAt,
      completedAt: job.completedAt,
      failedAt: job.failedAt,
      error: job.error ? JSON.parse(job.error) : null,
      rollbackStatus: job.rollbackStatus,
    };
  }

  private async updateJobStatus(jobId: string, state: any): Promise<void> {
    const context = state.context;

    await this.prisma.provisioningJob.update({
      where: { id: jobId },
      data: {
        status: this.mapStateToStatus(state.value),
        currentStep: typeof state.value === 'string' ? state.value : null,
        completedSteps: context.completedSteps,
        rollbackActions: context.rollbackActions,
        error: context.error ? JSON.stringify(context.error) : null,
        rollbackStatus: context.rollbackStatus,
        tenantId: context.tenantId,
        completedAt: state.matches('active') ? new Date() : null,
        failedAt: state.matches('provision_failed') ? new Date() : null,
      },
    });
  }

  private mapStateToStatus(stateValue: any): string {
    const stateMap: Record<string, string> = {
      pending: 'PENDING',
      validating: 'IN_PROGRESS',
      registering: 'IN_PROGRESS',
      db_creating: 'IN_PROGRESS',
      keys_generating: 'IN_PROGRESS',
      storage_allocating: 'IN_PROGRESS',
      configuring: 'IN_PROGRESS',
      user_creating: 'IN_PROGRESS',
      compliance_setup: 'IN_PROGRESS',
      integration_setup: 'IN_PROGRESS',
      health_checking: 'IN_PROGRESS',
      notifying: 'IN_PROGRESS',
      active: 'COMPLETED',
      provision_failed: 'FAILED',
      rolling_back: 'ROLLING_BACK',
      rolled_back: 'ROLLED_BACK',
      rollback_failed: 'ROLLBACK_FAILED',
    };

    return stateMap[stateValue as string] || 'PENDING';
  }
}

11. Monitoring & Observability

11.1 Metrics

CloudWatch/Datadog Metrics:

Metric	Type	Purpose
`tenant.provisioning.started`	Counter	Track provisioning attempts
`tenant.provisioning.completed`	Counter	Track successful provisions
`tenant.provisioning.failed`	Counter	Track failures
`tenant.provisioning.duration_ms`	Histogram	Provisioning time distribution
`tenant.provisioning.step_duration_ms`	Histogram (by step)	Per-step timing
`tenant.rollback.triggered`	Counter	Rollback frequency
`tenant.health_check.failed`	Counter (by check type)	Health check failures

11.2 Logging

Structured Logging (JSON):

logger.info('Tenant provisioning started', {
  jobId: 'prov_xxx',
  tenantId: null,
  organizationName: 'Acme Bio',
  tier: 'PROFESSIONAL',
  timestamp: '2026-02-16T10:00:00Z',
});

logger.info('Provisioning step completed', {
  jobId: 'prov_xxx',
  tenantId: 'ten_xxx',
  step: 'db_creating',
  durationMs: 28450,
  timestamp: '2026-02-16T10:00:28Z',
});

logger.error('Provisioning failed', {
  jobId: 'prov_xxx',
  tenantId: 'ten_xxx',
  step: 'keys_generating',
  error: 'KMS_KEY_CREATION_FAILED',
  errorDetails: 'Insufficient IAM permissions',
  timestamp: '2026-02-16T10:02:15Z',
});

11.3 Alerting

PagerDuty/Opsgenie Alerts:

Condition	Severity	Escalation
Provisioning failure rate > 5%	P2	Engineering on-call
Rollback failure	P1	Engineering + CTO
Health check failure rate > 10%	P2	DevOps on-call
Provisioning duration > 10 min	P3	Slack notification

12. Security Considerations

12.1 Tenant Isolation

Enforcement Points:

Database: PostgreSQL RLS policies on every table
Application: Middleware validates tenant_id from JWT matches URL path
Storage: GCS IAM policies restrict to tenant service account
Encryption: Per-tenant KEK/DEK - no key sharing
API: Rate limiting per tenant prevents noisy neighbor

12.2 Credential Management

Security Measures:

Temporary passwords: 20-char random, bcrypt hashed (12 rounds)
MFA mandatory: TOTP enrollment required on first login
API secrets: Stored in GCP Secret Manager (not in database)
Key rotation: KEK rotates every 90 days, DEK every 30 days
Backup codes: 10x single-use codes for MFA recovery

12.3 Audit Trail

All provisioning events logged:

await auditLog.log({
  tenantId,
  entityType: 'tenant',
  entityId: tenantId,
  action: 'tenant.provisioned',
  performedBy: 'SYSTEM',
  performerType: 'SYSTEM',
  metadata: {
    tier: 'PROFESSIONAL',
    complianceProfile: { fdaPart11: true, hipaa: false },
    provisioningDuration: 272,
  },
  performedAt: new Date(),
});

13. Compliance Requirements

13.1 FDA 21 CFR Part 11 Compliance

Validation Requirements:

IQ (Installation Qualification): Verify database, encryption, storage created
OQ (Operational Qualification): Health check suite validates all operations
PQ (Performance Qualification): Provisioning completes in < 5 min

Evidence:

Provisioning job record with timestamps
Health check results (stored in audit log)
Rollback test results (quarterly validation)

13.2 HIPAA Compliance

BAA (Business Associate Agreement):

Enterprise tier only: BAA record created during provisioning
Requires manual signature: DocuSign integration (post-provisioning)
PHI controls enabled: Encryption, access logging, breach notification

Audit Controls (§164.312(b)):

All provisioning steps logged to immutable audit trail
Provisioning failures trigger security review

13.3 SOC 2 Type II Compliance

Control Objectives:

CC7.2: System operations monitored and reviewed
- Evidence: CloudWatch metrics, provisioning job logs
CC6.1: Logical access controls restrict unauthorized access
- Evidence: RLS policies, API authentication, MFA enforcement

14. Testing & Validation

14.1 Unit Tests

// __tests__/provisioning/health-check.test.ts

describe('Tenant Health Check', () => {
  it('should pass all checks for successfully provisioned tenant', async () => {
    const tenantId = 'test_tenant_123';

    const result = await performTenantHealthCheck(tenantId);

    expect(result.healthy).toBe(true);
    expect(result.checks.database).toBe(true);
    expect(result.checks.encryption).toBe(true);
    expect(result.checks.storage).toBe(true);
    expect(result.checks.api).toBe(true);
    expect(result.checks.audit).toBe(true);
    expect(result.checks.rbac).toBe(true);
    expect(result.errors).toHaveLength(0);
  });

  it('should fail if encryption roundtrip fails', async () => {
    // Mock KMS failure
    jest.spyOn(kmsClient, 'encrypt').mockRejectedValue(new Error('KMS unavailable'));

    const result = await performTenantHealthCheck('test_tenant_123');

    expect(result.healthy).toBe(false);
    expect(result.checks.encryption).toBe(false);
    expect(result.errors).toContain('Encryption error: KMS unavailable');
  });
});

14.2 Integration Tests

// __tests__/provisioning/full-workflow.test.ts

describe('Tenant Provisioning Workflow', () => {
  it('should provision tenant end-to-end in < 5 minutes', async () => {
    const request: ProvisioningRequest = {
      organizationName: `Test Org ${Date.now()}`,
      adminEmail: 'admin@testorg.com',
      tier: SubscriptionTier.PROFESSIONAL,
      regulatoryProfile: {
        requireFdaPart11: true,
        requireHipaa: false,
        requireSoc2: true,
      },
    };

    const startTime = Date.now();
    const service = new TenantProvisioningService(prisma);

    const { jobId } = await service.startProvisioning(request);

    // Poll for completion
    let status;
    while (true) {
      status = await service.getJobStatus(jobId);
      if (status.status === 'completed' || status.status === 'failed') break;
      await sleep(1000);
    }

    const endTime = Date.now();
    const durationMs = endTime - startTime;

    expect(status.status).toBe('completed');
    expect(status.progressPercent).toBe(100);
    expect(durationMs).toBeLessThan(5 * 60 * 1000); // < 5 min

    // Verify tenant is active
    const tenant = await prisma.tenant.findUnique({
      where: { id: status.tenantId },
    });
    expect(tenant?.status).toBe(TenantStatus.ACTIVE);
  }, 6 * 60 * 1000); // 6 min timeout
});

14.3 Rollback Tests

describe('Provisioning Rollback', () => {
  it('should rollback all steps if health check fails', async () => {
    // Force health check to fail
    jest.spyOn(healthCheckService, 'performTenantHealthCheck').mockResolvedValue({
      healthy: false,
      checks: { database: false },
      errors: ['Database unavailable'],
    });

    const request: ProvisioningRequest = { /* ... */ };
    const service = new TenantProvisioningService(prisma);

    const { jobId } = await service.startProvisioning(request);

    // Wait for rollback
    let status;
    while (true) {
      status = await service.getJobStatus(jobId);
      if (status.status === 'rolled_back' || status.status === 'rollback_failed') break;
      await sleep(1000);
    }

    expect(status.status).toBe('rolled_back');
    expect(status.rollbackStatus).toBe('completed');

    // Verify tenant marked as deleted
    const tenant = await prisma.tenant.findUnique({
      where: { id: status.tenantId },
    });
    expect(tenant?.status).toBe(TenantStatus.DELETED);
    expect(tenant?.deletedAt).toBeTruthy();
  });
});

15. Operational Procedures

15.1 Standard Provisioning

Automated (Self-Service):

Customer signs up via website
Payment processed (Stripe)
API call to /api/tenants/provision
Customer receives welcome email with credentials
Customer logs in and completes MFA setup

Manual (Enterprise Sales):

Sales team finalizes Enterprise contract
Sales rep creates provisioning request in admin portal
DevOps team approves provisioning (if custom requirements)
Automated provisioning triggered
Customer Success team schedules onboarding call

15.2 Troubleshooting Failed Provisioning

Runbook:

Check provisioning job logs:

curl https://api.coditect-bio-qms.com/api/tenants/provision/{jobId}/status \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Identify failed step:
- db_creating: Check PostgreSQL connection, IAM permissions
- keys_generating: Verify GCP KMS API enabled, cloudkms.admin role
- storage_allocating: Check GCS bucket quota, naming conflicts
- health_checking: Review individual check failures

Attempt retry:

curl -X POST https://api.coditect-bio-qms.com/api/tenants/provision/{jobId}/retry \
  -H "Authorization: Bearer $ADMIN_TOKEN"

If retry fails, escalate to engineering with:
- Job ID
- Full error message from status endpoint
- CloudWatch logs for the time range
- GCP project ID and region

15.3 Manual Cleanup (Rollback Failure)

CRITICAL: Only perform if automated rollback fails

# 1. Drop database schema
psql $DATABASE_URL -c "DROP SCHEMA IF EXISTS tenant_{tenantId} CASCADE;"

# 2. Delete GCS bucket
gsutil -m rm -r gs://coditect-bio-qms-{tenantId}

# 3. Schedule GCP KMS key destruction
gcloud kms keys versions destroy 1 \
  --key=tenant-kek \
  --keyring=tenant-{tenantId} \
  --location=us-central1

# 4. Delete tenant record
psql $DATABASE_URL -c "UPDATE tenants SET status = 'DELETED', deleted_at = NOW() WHERE id = '{tenantId}';"

# 5. Create incident post-mortem

Appendix A: Performance Benchmarks

Provisioning Time (by Tier):

Tier	Target	P50	P95	P99
Starter	< 3 min	2m 15s	2m 45s	3m 10s
Professional	< 5 min	3m 30s	4m 45s	5m 20s
Enterprise	< 7 min	5m 10s	6m 30s	7m 15s

Step Breakdown (Professional Tier):

Step	Duration	% of Total
Validating	0.5s	0.2%
Registering	0.8s	0.4%
DB Creating	28s	13.3%
Keys Generating	4.2s	2.0%
Storage Allocating	2.8s	1.3%
Configuring	1.9s	0.9%
User Creating	0.7s	0.3%
Compliance Setup	0.6s	0.3%
Integration Setup	1.5s	0.7%
Health Checking	4.3s	2.0%
Notifying	0.9s	0.4%
Total	210s	100%

Appendix B: Cost Analysis

Provisioning Costs (per tenant, one-time):

Resource	Cost	Notes
GCP KMS Key Ring	$0.06/month	1 key ring per tenant
GCP KMS KEK	$0.06/month/key version	Auto-rotates every 90 days
GCS Bucket (empty)	$0.00	Pay-as-you-grow
PostgreSQL Schema	$0.00	Shared database
Compute (provisioning)	~$0.02	Cloud Run execution time
Secret Manager	$0.06/month/secret	API secret storage
SendGrid Email	$0.0001	Welcome notification
Total (first month)	~$0.20	Minimal upfront cost

Recurring Costs (per tenant/month):

Tier	GCS Storage	KMS	Secrets	Compute	Total
Starter	$0.50 (10GB)	$0.06	$0.06	$5.00	$5.62
Professional	$5.00 (100GB)	$0.12	$0.12	$25.00	$30.24
Enterprise	$50.00 (1TB)	$0.18 (HSM)	$0.18	$150.00	$200.36

Appendix C: Glossary

Term	Definition
BAA	Business Associate Agreement (HIPAA requirement)
DEK	Data Encryption Key (256-bit AES key for data encryption)
KEK	Key Encryption Key (wraps DEK via envelope encryption)
RLS	Row-Level Security (PostgreSQL feature for multi-tenancy)
MFA	Multi-Factor Authentication (TOTP-based)
TOTP	Time-based One-Time Password (RFC 6238)
IQ/OQ/PQ	Installation/Operational/Performance Qualification (FDA validation)
SOC 2	Service Organization Control 2 (AICPA trust services audit)
Part 11	FDA 21 CFR Part 11 (electronic records and signatures)
HIPAA	Health Insurance Portability and Accountability Act

END OF DOCUMENT

Document ID: CODITECT-BIO-PROVISION-001 Version: 1.0.0 Page Count: 48 Word Count: ~10,500 Last Updated: 2026-02-16

Document Control​

Approval History​

Revision History​

Distribution​

Executive Summary​

Table of Contents​

1. Provisioning Architecture​

1.1 High-Level Architecture​

1.2 Core Components​

1.3 Design Principles​

2. Subscription Tier Model​

2.1 Tier Definitions​

Starter Tier​

Professional Tier​

Enterprise Tier​

2.2 Tier-Based Resource Quotas​

3. Provisioning State Machine​

3.1 XState Definition​

3.2 State Transition Diagram​

4. Provisioning Steps Specification​

4.1 Step 1: Tenant Registration​

4.2 Step 2: Database Schema Creation​

4.3 Step 3: Encryption Key Generation​

4.4 Step 4: Storage Allocation​

4.5 Step 5: Initial Configuration​

4.6 Step 6: Admin User Creation​

4.7 Step 7: Compliance Profile Setup​

4.8 Step 8: Integration Endpoints​

4.9 Step 9: Health Check​

4.10 Step 10: Welcome Notification​

5. Pre-Provisioning Validation​

5.1 Validation Rules​

6. Post-Provisioning Health Check​

7. Automated Rollback Mechanism​

7.1 Rollback Strategy​

7.2 Rollback Implementation​

7.3 Rollback Monitoring​

8. Provisioning API Specification​

8.1 Endpoints​

POST /api/tenants/provision​

GET /api/tenants/provision/:jobId/status​

POST /api/tenants/provision/:jobId/retry​

DELETE /api/tenants/provision/:jobId/rollback​

8.2 Authentication​

9. Database Schema Extensions​

9.1 Tenant Registry Schema​

10. TypeScript Implementation​

10.1 Type Definitions​

10.2 Service Layer​

11. Monitoring & Observability​

11.1 Metrics​

11.2 Logging​

11.3 Alerting​

12. Security Considerations​

12.1 Tenant Isolation​

12.2 Credential Management​

12.3 Audit Trail​

13. Compliance Requirements​

13.1 FDA 21 CFR Part 11 Compliance​

13.2 HIPAA Compliance​

13.3 SOC 2 Type II Compliance​

14. Testing & Validation​

14.1 Unit Tests​

14.2 Integration Tests​

14.3 Rollback Tests​

15. Operational Procedures​

15.1 Standard Provisioning​

15.2 Troubleshooting Failed Provisioning​

15.3 Manual Cleanup (Rollback Failure)​

Appendix A: Performance Benchmarks​

Appendix B: Cost Analysis​

Appendix C: Glossary​

Document Control

Approval History

Revision History

Distribution

Executive Summary

Table of Contents

1. Provisioning Architecture

1.1 High-Level Architecture

1.2 Core Components

1.3 Design Principles

2. Subscription Tier Model

2.1 Tier Definitions

Starter Tier

Professional Tier

Enterprise Tier

2.2 Tier-Based Resource Quotas

3. Provisioning State Machine

3.1 XState Definition

3.2 State Transition Diagram

4. Provisioning Steps Specification

4.1 Step 1: Tenant Registration

4.2 Step 2: Database Schema Creation

4.3 Step 3: Encryption Key Generation

4.4 Step 4: Storage Allocation

4.5 Step 5: Initial Configuration

4.6 Step 6: Admin User Creation

4.7 Step 7: Compliance Profile Setup

4.8 Step 8: Integration Endpoints

4.9 Step 9: Health Check

4.10 Step 10: Welcome Notification

5. Pre-Provisioning Validation

5.1 Validation Rules

6. Post-Provisioning Health Check

7. Automated Rollback Mechanism

7.1 Rollback Strategy

7.2 Rollback Implementation

7.3 Rollback Monitoring

8. Provisioning API Specification

8.1 Endpoints

`POST /api/tenants/provision`

`GET /api/tenants/provision/:jobId/status`

`POST /api/tenants/provision/:jobId/retry`

`DELETE /api/tenants/provision/:jobId/rollback`

8.2 Authentication

9. Database Schema Extensions

9.1 Tenant Registry Schema

10. TypeScript Implementation

10.1 Type Definitions

10.2 Service Layer

11. Monitoring & Observability

11.1 Metrics

11.2 Logging

11.3 Alerting

12. Security Considerations

12.1 Tenant Isolation

12.2 Credential Management

12.3 Audit Trail

13. Compliance Requirements

13.1 FDA 21 CFR Part 11 Compliance

13.2 HIPAA Compliance

13.3 SOC 2 Type II Compliance

14. Testing & Validation

14.1 Unit Tests

14.2 Integration Tests

14.3 Rollback Tests

15. Operational Procedures

15.1 Standard Provisioning

15.2 Troubleshooting Failed Provisioning

15.3 Manual Cleanup (Rollback Failure)

Appendix A: Performance Benchmarks

Appendix B: Cost Analysis

Appendix C: Glossary