Skip to content

🗃️ Collection Manager

🎯 Overview

The Collection Manager is a powerful component of the API Gateway that enables dynamic creation and management of data structures (collections). Collections are equivalent to database tables, but can be created, modified, and deleted programmatically through the API, eliminating the need for manual database migrations or schema changes.

📋 Purpose

The Collection Manager provides: - Dynamic Schema Creation: Create database tables on-the-fly via API - Automatic REST API Generation: Each collection gets full CRUD endpoints automatically - Field Management: Add, modify, and remove fields from collections - Relationship Management: Define complex relationships between collections - Schema Migration: Automatic database schema synchronization - Type Safety: Strong typing and validation for all fields

🏗️ Architecture

graph TB
    API[API Request]

    subgraph "Collection Manager"
        CollectionAPI[Collection API]
        FieldManager[Field Manager]
        SchemaValidator[Schema Validator]
        MigrationEngine[Migration Engine]
        ResourceGenerator[Resource Generator]
    end

    subgraph "Generated Resources"
        CRUD[CRUD Operations]
        Endpoints[REST Endpoints]
        Swagger[Swagger Docs]
    end

    subgraph "Data Layer"
        Metadata[(Schema Metadata)]
        AppDB[(Application Database)]
    end

    API --> CollectionAPI
    CollectionAPI --> SchemaValidator
    SchemaValidator --> FieldManager
    FieldManager --> MigrationEngine

    MigrationEngine --> Metadata
    MigrationEngine --> AppDB

    ResourceGenerator --> CRUD
    ResourceGenerator --> Endpoints
    ResourceGenerator --> Swagger

    CollectionAPI --> ResourceGenerator

    classDef manager fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    classDef resource fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    classDef data fill:#e8f5e8,stroke:#2e7d32,stroke-width:2px

    class CollectionAPI,FieldManager,SchemaValidator,MigrationEngine,ResourceGenerator manager
    class CRUD,Endpoints,Swagger resource
    class Metadata,AppDB data

🔑 Key Concepts

📦 Collections

Collections are dynamic data structures that: - Define a table structure in the database - Automatically generate REST API endpoints - Support relationships with other collections - Include validation and constraints - Can be created, modified, and deleted via API

Properties: - name: Unique identifier (e.g., "articles", "users") - title: Human-readable display name - fields: Array of field definitions - options: Additional configuration (timestamps, soft deletes, etc.)

🏷️ Fields

Fields define the columns/properties of a collection: - Each field has a type (string, integer, date, etc.) - Fields can be required, unique, or have default values - Support for relationships (belongsTo, hasMany, etc.) - Custom validation rules - UI hints for form generation

📡 Collection Management API

📋 List All Collections

GET /api/collections:list
X-App: main
Authorization: Bearer {api-key}

Response: 

{
  "data": [
    {
      "name": "articles",
      "title": "Articles",
      "fields": [
        {
          "name": "id",
          "type": "integer",
          "primaryKey": true,
          "autoIncrement": true
        },
        {
          "name": "title",
          "type": "string",
          "required": true
        },
        {
          "name": "content",
          "type": "text"
        }
      ],
      "options": {
        "timestamps": true,
        "paranoid": false
      }
    }
  ]
}

➕ Create a Collection

POST /api/collections:create
X-App: main
Authorization: Bearer {api-key}
Content-Type: application/json

{
  "name": "blog_posts",
  "title": "Blog Posts",
  "description": "Collection for blog articles",
  "fields": [
    {
      "name": "title",
      "type": "string",
      "required": true,
      "unique": false
    },
    {
      "name": "slug",
      "type": "string",
      "required": true,
      "unique": true
    },
    {
      "name": "content",
      "type": "text",
      "required": false
    },
    {
      "name": "excerpt",
      "type": "text"
    },
    {
      "name": "status",
      "type": "string",
      "defaultValue": "draft"
    },
    {
      "name": "publishedAt",
      "type": "datetime"
    },
    {
      "name": "viewCount",
      "type": "integer",
      "defaultValue": 0
    },
    {
      "name": "author",
      "type": "belongsTo",
      "target": "users",
      "foreignKey": "authorId"
    },
    {
      "name": "tags",
      "type": "belongsToMany",
      "target": "tags",
      "through": "blog_posts_tags"
    }
  ],
  "options": {
    "timestamps": true,
    "paranoid": true
  }
}

What happens automatically: 1. ✅ Database table is created 2. ✅ REST endpoints are generated (/api/blog_posts:list, :create, etc.) 3. ✅ Swagger documentation is updated 4. ✅ Validation rules are configured 5. ✅ Relationship joins are set up

🔄 Update a Collection

POST /api/collections:update?filterByTk=blog_posts
X-App: main
Authorization: Bearer {api-key}
Content-Type: application/json

{
  "title": "Blog Articles",
  "description": "Published blog content and drafts"
}

Note: Updating a collection modifies metadata only. To change fields, use the Field Management API.

🗑️ Delete a Collection

POST /api/collections:destroy?filterByTk=blog_posts
X-App: main
Authorization: Bearer {api-key}

⚠️ Warning: This will: - Drop the database table - Delete all data in the collection - Remove all REST endpoints - Remove Swagger documentation

Safe deletion with cascade: 

POST /api/collections:destroy?filterByTk=blog_posts&cascade=true
X-App: main

This also removes relationships and dependent collections.

🏷️ Field Types

📝 Basic Field Types

Type Description Database Type Example
string Short text VARCHAR(255) "Hello World"
text Long text TEXT "Long article content..."
integer Whole number INTEGER 42
bigInt Large integer BIGINT 9007199254740991
float Decimal number FLOAT 3.14159
double Double precision DOUBLE 3.141592653589793
boolean True/False BOOLEAN true
date Date only DATE "2025-01-15"
time Time only TIME "14:30:00"
datetime Date and time DATETIME/TIMESTAMP "2025-01-15T14:30:00Z"
json JSON data JSON/JSONB {"key": "value"}
array Array/List JSON/ARRAY [1, 2, 3]

🔗 Relationship Field Types

Type Description Use Case Example
belongsTo Many-to-One Each post has one author Post → User
hasOne One-to-One User has one profile User → Profile
hasMany One-to-Many Post has many comments Post → Comments
belongsToMany Many-to-Many Post has many tags Post ↔ Tags

🔐 Special Field Types

Type Description Behavior
uid Unique Identifier Auto-generated UUID
password Password field Auto-hashed before storage
email Email address Validated format
url URL field Validated format
phone Phone number Validated format
color Color code Hex color validation

🛠️ Field Management API

➕ Add Field to Collection

POST /api/fields:create
X-App: main
Authorization: Bearer {api-key}
Content-Type: application/json

{
  "collectionName": "blog_posts",
  "name": "featuredImage",
  "type": "string",
  "description": "URL to featured image",
  "required": false,
  "defaultValue": null
}

🔄 Update Field

POST /api/fields:update
X-App: main
Authorization: Bearer {api-key}
Content-Type: application/json

{
  "collectionName": "blog_posts",
  "name": "featuredImage",
  "description": "Featured image URL or path",
  "uiSchema": {
    "x-component": "Upload",
    "x-component-props": {
      "accept": "image/*"
    }
  }
}

🗑️ Remove Field

POST /api/fields:destroy
X-App: main
Authorization: Bearer {api-key}
Content-Type: application/json

{
  "collectionName": "blog_posts",
  "name": "featuredImage"
}

⚠️ Warning: Removes the column from the database and deletes all data in that field.

🔗 Relationship Management

🔗 One-to-Many Relationship (hasMany)

Scenario: Posts have many comments

# Add relationship field to posts collection
POST /api/fields:create
X-App: main
Content-Type: application/json

{
  "collectionName": "blog_posts",
  "name": "comments",
  "type": "hasMany",
  "target": "comments",
  "foreignKey": "postId",
  "sourceKey": "id"
}

Usage: 

# Get post with comments
GET /api/blog_posts:get?filterByTk=1&appends=comments
X-App: main

# List comments of a post
GET /api/blog_posts/1/comments:list
X-App: main

# Create comment on a post
POST /api/blog_posts/1/comments:create
X-App: main
{
  "content": "Great article!",
  "authorId": 5
}

🔗 Many-to-One Relationship (belongsTo)

Scenario: Each comment belongs to one user (author)

POST /api/fields:create
X-App: main
Content-Type: application/json

{
  "collectionName": "comments",
  "name": "author",
  "type": "belongsTo",
  "target": "users",
  "foreignKey": "authorId",
  "targetKey": "id"
}

Usage: 

# Get comment with author info
GET /api/comments:get?filterByTk=1&appends=author
X-App: main

# Set comment author
POST /api/comments/1/author:set?filterByTk=5
X-App: main

🔗 Many-to-Many Relationship (belongsToMany)

Scenario: Posts can have multiple tags, tags can belong to multiple posts

# Add relationship to posts
POST /api/fields:create
X-App: main
Content-Type: application/json

{
  "collectionName": "blog_posts",
  "name": "tags",
  "type": "belongsToMany",
  "target": "tags",
  "through": "blog_posts_tags",
  "foreignKey": "postId",
  "otherKey": "tagId"
}

# Add reverse relationship to tags
POST /api/fields:create
X-App: main
Content-Type: application/json

{
  "collectionName": "tags",
  "name": "posts",
  "type": "belongsToMany",
  "target": "blog_posts",
  "through": "blog_posts_tags",
  "foreignKey": "tagId",
  "otherKey": "postId"
}

Usage: 

# Add tags to post
POST /api/blog_posts/1/tags:add
X-App: main
{
  "values": [2, 5, 8]  # Tag IDs
}

# Set tags (replace all)
POST /api/blog_posts/1/tags:set
X-App: main
{
  "values": [2, 5, 8]
}

# Remove tags from post
POST /api/blog_posts/1/tags:remove
X-App: main
{
  "values": [5]
}

# List tags of a post
GET /api/blog_posts/1/tags:list
X-App: main

🔗 One-to-One Relationship (hasOne)

Scenario: User has one profile

POST /api/fields:create
X-App: main
Content-Type: application/json

{
  "collectionName": "users",
  "name": "profile",
  "type": "hasOne",
  "target": "user_profiles",
  "foreignKey": "userId",
  "sourceKey": "id"
}

Usage: 

# Get user with profile
GET /api/users:get?filterByTk=1&appends=profile
X-App: main

# Update user profile
POST /api/users/1/profile:update
X-App: main
{
  "bio": "Full Stack Developer",
  "website": "https://example.com"
}

⚙️ Field Options

Common Field Options

{
  "name": "title",
  "type": "string",

  // Validation
  "required": true,
  "unique": true,

  // Default value
  "defaultValue": "Untitled",

  // Database constraints
  "allowNull": false,
  "primaryKey": false,
  "autoIncrement": false,

  // Display
  "description": "Post title",
  "uiSchema": {
    "title": "Title",
    "x-component": "Input",
    "x-component-props": {
      "placeholder": "Enter title..."
    }
  }
}

String Field Options

{
  "name": "email",
  "type": "string",
  "required": true,
  "unique": true,

  // String-specific
  "maxLength": 255,
  "pattern": "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
}

Number Field Options

{
  "name": "price",
  "type": "float",

  // Number-specific
  "min": 0,
  "max": 999999.99,
  "precision": 2,
  "scale": 2
}

Date/Time Field Options

{
  "name": "publishedAt",
  "type": "datetime",

  // Date-specific
  "dateOnly": false,
  "autoNow": false,      // Set to current time on create
  "autoNowAdd": false    // Set to current time on every update
}

🎯 Built-in Collection Features

⏰ Timestamps

Automatically add createdAt and updatedAt fields:

{
  "name": "articles",
  "title": "Articles",
  "fields": [...],
  "options": {
    "timestamps": true
  }
}

🗑️ Soft Deletes (Paranoid)

Add deletedAt field for soft deletion:

{
  "name": "articles",
  "title": "Articles",
  "fields": [...],
  "options": {
    "timestamps": true,
    "paranoid": true
  }
}

Records are marked as deleted but not actually removed from database.

📝 Logging

Track all changes to records:

{
  "name": "articles",
  "title": "Articles",
  "fields": [...],
  "options": {
    "logging": true
  }
}

💡 Best Practices

🎯 Collection Design

DO: - ✅ Use clear, descriptive names (snake_case or camelCase) - ✅ Add descriptions to collections and fields - ✅ Define required: true for mandatory fields - ✅ Use unique: true for fields that must be unique - ✅ Set appropriate default values - ✅ Enable timestamps for audit trails - ✅ Plan relationships before creation

DON'T: - ❌ Use reserved keywords as names - ❌ Create overly complex nested structures - ❌ Skip field validation - ❌ Forget to document field purposes - ❌ Create circular dependencies in relationships

🔗 Relationship Design

DO: - ✅ Choose the correct relationship type - ✅ Name relationship fields intuitively - ✅ Consider query performance implications - ✅ Use through tables for many-to-many - ✅ Set up bidirectional relationships when needed

DON'T: - ❌ Create unnecessary relationships - ❌ Use many-to-many for one-to-many scenarios - ❌ Forget to index foreign keys - ❌ Create deeply nested relationship chains

⚡ Performance

DO: - ✅ Add indexes to frequently queried fields - ✅ Use appropriate field types - ✅ Normalize data structures - ✅ Consider query patterns when designing

DON'T: - ❌ Over-index (slows down writes) - ❌ Use text fields for short strings - ❌ Create too many relationships per collection - ❌ Skip database optimization

📚 Complete Example: Blog System

Step 1: Create Users Collection

POST /api/collections:create
X-App: main
{
  "name": "users",
  "title": "Users",
  "fields": [
    {"name": "email", "type": "email", "required": true, "unique": true},
    {"name": "username", "type": "string", "required": true, "unique": true},
    {"name": "password", "type": "password", "required": true},
    {"name": "name", "type": "string", "required": true},
    {"name": "bio", "type": "text"}
  ],
  "options": {"timestamps": true}
}

Step 2: Create Tags Collection

POST /api/collections:create
X-App: main
{
  "name": "tags",
  "title": "Tags",
  "fields": [
    {"name": "name", "type": "string", "required": true, "unique": true},
    {"name": "slug", "type": "string", "required": true, "unique": true},
    {"name": "description", "type": "text"}
  ],
  "options": {"timestamps": true}
}

Step 3: Create Blog Posts Collection

POST /api/collections:create
X-App: main
{
  "name": "blog_posts",
  "title": "Blog Posts",
  "fields": [
    {"name": "title", "type": "string", "required": true},
    {"name": "slug", "type": "string", "required": true, "unique": true},
    {"name": "content", "type": "text", "required": true},
    {"name": "excerpt", "type": "text"},
    {"name": "status", "type": "string", "defaultValue": "draft"},
    {"name": "publishedAt", "type": "datetime"},
    {"name": "viewCount", "type": "integer", "defaultValue": 0},
    {
      "name": "author",
      "type": "belongsTo",
      "target": "users",
      "foreignKey": "authorId"
    },
    {
      "name": "tags",
      "type": "belongsToMany",
      "target": "tags",
      "through": "blog_posts_tags"
    }
  ],
  "options": {
    "timestamps": true,
    "paranoid": true
  }
}

Step 4: Create Comments Collection

POST /api/collections:create
X-App: main
{
  "name": "comments",
  "title": "Comments",
  "fields": [
    {"name": "content", "type": "text", "required": true},
    {
      "name": "post",
      "type": "belongsTo",
      "target": "blog_posts",
      "foreignKey": "postId"
    },
    {
      "name": "author",
      "type": "belongsTo",
      "target": "users",
      "foreignKey": "authorId"
    }
  ],
  "options": {
    "timestamps": true,
    "paranoid": true
  }
}

Step 5: Add Reverse Relationships

# Add comments relationship to blog_posts
POST /api/fields:create
X-App: main
{
  "collectionName": "blog_posts",
  "name": "comments",
  "type": "hasMany",
  "target": "comments",
  "foreignKey": "postId"
}

# Add posts relationship to users
POST /api/fields:create
X-App: main
{
  "collectionName": "users",
  "name": "posts",
  "type": "hasMany",
  "target": "blog_posts",
  "foreignKey": "authorId"
}

# Add reverse relationship for tags
POST /api/fields:create
X-App: main
{
  "collectionName": "tags",
  "name": "posts",
  "type": "belongsToMany",
  "target": "blog_posts",
  "through": "blog_posts_tags",
  "foreignKey": "tagId",
  "otherKey": "postId"
}

Now you have a complete blog system with automatic REST APIs for all CRUD operations!

  • API Gateway: Overall API architecture
  • CRUD Operations: Using the generated APIs
  • Authentication: Securing collection access
  • Permissions: Fine-grained access control

Note: The Collection Manager provides the foundation for Continuum's flexible, API-first data management approach, enabling rapid development without traditional database migrations.