FHIR Group Creation with ClickHouse Indexing

System Design · sequence diagram · MIT

Illustrates the process of creating a FHIR Group resource, including stripping members for primary storage and asynchronously indexing group members in Cli

Source: https://github.com/icanbwell/fhir-server/blob/97ba98f5a1285179af54cbd4a17fb9334a728d0c/readme/clickhouse.md
Curated by icanbwell
FHIR ClickHouse MongoDB Group Management Data Indexing Sequence Diagram API

Mermaid source

sequenceDiagram
    participant Client
    participant API as FHIR API
    participant BulkInserter as DatabaseBulkInserter
    participant Context as httpContext
    participant Mongo as MongoDB
    participant PostSave as PostSaveProcessor
    participant Handler as ClickHouseGroupHandler
    participant Builder as GroupMemberEventBuilder
    participant Repo as GroupMemberRepository
    participant CH as ClickHouse

    Client->>API: POST /4_0_0/Group<br/>{member: [Patient/1, Patient/2]}
    API->>BulkInserter: insertOneAsync(groupResource)

    Note over BulkInserter: _handleGroupMemberStripping()
    BulkInserter->>Context: Store original member[]
    BulkInserter->>BulkInserter: Strip member array<br/>(set to [])

    BulkInserter->>Mongo: insertOne({...group, member: []})
    Mongo-->>BulkInserter: Success

    BulkInserter->>PostSave: afterSaveAsync(CREATE)
    PostSave->>Handler: afterSaveAsync(groupId, CREATE)

    Handler->>Context: Get original member[]
    Handler->>Builder: buildAddedEvents(members)
    Builder-->>Handler: events[]

    Handler->>Repo: appendEvents(events)
    Repo->>CH: INSERT INTO Group_4_0_0_MemberEvents
    CH->>CH: Materialized view auto-updates<br/>(Group_4_0_0_MemberCurrent)
    CH-->>Repo: Success

    Repo-->>Handler: Success
    Handler-->>PostSave: Success
    PostSave-->>API: Success
    API-->>Client: 201 Created<br/>{id, meta, member: []}

What this diagram shows

This diagram details the sequence for creating a FHIR Group resource. A client sends a POST request to the FHIR API with group members. The API uses a BulkInserter which first strips the member array from the group resource, storing the original members in the HTTP context. The group is then inserted into MongoDB without members. Post-save, a processor retrieves the original members, builds "added" events, and appends them to a GroupMemberRepository. This repository inserts the events into ClickHouse, which then updates a materialized view for current group members. Finally, the API responds to the client with the created group, showing an empty member array.

When to use it

Use this pattern when designing a FHIR server that requires a separate, highly performant index for querying group memberships, especially for large groups or frequent lookups. It's suitable for scenarios where the primary database (e.g., MongoDB) is not optimized for complex analytical queries on nested arrays.

How to adapt it for your project

This pattern can be adapted for indexing other FHIR resource relationships or attributes into a specialized data store. The ClickHouse component could be replaced with other analytical databases or search engines like Elasticsearch. The event-driven approach can be extended to handle updates and deletions of group members, ensuring eventual consistency across the primary and indexed stores.

Key concepts