FHIR Group Creation with ClickHouse Member Indexing

System Design · sequence diagram · MIT

Details the FHIR Group creation process, showing how members are stripped for initial MongoDB storage and then asynchronously indexed in ClickHouse for fas

Source: https://github.com/icanbwell/fhir-server/blob/1bb6a360bf7d199ee31787c65b42a2e5ffc75594/readme/clickhouse.md
Curated by icanbwell
FHIR REST API MongoDB ClickHouse Data Indexing System Design Asynchronous

Mermaid source

sequenceDiagram
    participant Client
    participant API as FHIR API
    participant BulkInserter as DatabaseBulkInserter
    participant Context as httpContext
    participant Mongo as MongoDB
    participant PostSave as PostSaveProcessor
    participant Handler as ClickHouseGroupHandler
    participant Builder as GroupMemberEventBuilder
    participant Repo as GroupMemberRepository
    participant CH as ClickHouse

    Client->>API: POST /4_0_0/Group<br/>{member: [Patient/1, Patient/2]}
    API->>BulkInserter: insertOneAsync(groupResource)

    Note over BulkInserter: _handleGroupMemberStripping()
    BulkInserter->>Context: Store original member[]
    BulkInserter->>BulkInserter: Strip member array<br/>(set to [])

    BulkInserter->>Mongo: insertOne({...group, member: []})
    Mongo-->>BulkInserter: Success

    BulkInserter->>PostSave: afterSaveAsync(CREATE)
    PostSave->>Handler: afterSaveAsync(groupId, CREATE)

    Handler->>Context: Get original member[]
    Handler->>Builder: buildAddedEvents(members)
    Builder-->>Handler: events[]

    Handler->>Repo: appendEvents(events)
    Repo->>CH: INSERT INTO Group_4_0_0_MemberEvents
    CH->>CH: Materialized view auto-updates<br/>(Group_4_0_0_MemberCurrent)
    CH-->>Repo: Success

    Repo-->>Handler: Success
    Handler-->>PostSave: Success
    PostSave-->>API: Success
    API-->>Client: 201 Created<br/>{id, meta, member: []}

What this diagram shows

This diagram illustrates the end-to-end flow for creating a FHIR Group resource. It details how a client initiates a POST request, the API handles the resource, members are temporarily stripped before being saved to MongoDB, and then an asynchronous post-save process indexes these members into ClickHouse, leveraging materialized views for efficient querying.

When to use it

Use this pattern when designing systems that require fast, denormalized lookups for complex relationships (like group memberships) that are stored in a primary transactional database. It's ideal for scenarios where the primary database isn't optimized for analytical queries or large-scale indexing.

How to adapt it for your project

This flow can be adapted for any resource creation where specific attributes need to be indexed or processed asynchronously in a separate data store. Replace FHIR Group with your resource, MongoDB with your primary database, and ClickHouse with your chosen analytical/indexing database. The _handleGroupMemberStripping() and PostSaveProcessor are key extension points.

Key concepts