FHIR Group Member PATCH Flow with ClickHouse and MongoDB

System Design · sequence diagram · MIT

Details the FHIR Group member PATCH operation, showing how new member events are written to ClickHouse for efficient storage and materialized views, while

Source: https://github.com/icanbwell/fhir-server/blob/1bb6a360bf7d199ee31787c65b42a2e5ffc75594/readme/clickhouse.md
Curated by icanbwell
FHIR API PATCH ClickHouse MongoDB System Design Data Pipeline

Mermaid source

sequenceDiagram
    participant Client
    participant API as FHIR API
    participant PatchOp as PATCH Handler
    participant Handler as ClickHouseGroupHandler
    participant Repo as GroupMemberRepository
    participant CH as ClickHouse
    participant Context as httpContext
    participant Mongo as MongoDB

    Client->>API: PATCH /4_0_0/Group/{id}<br/>[{op: "add", path: "/member/-", value: {...}}]
    API->>PatchOp: handle PATCH

    Note over PatchOp: Detect member operations
    PatchOp->>PatchOp: Parse member ops<br/>eventsToAdd[]<br/>eventsToRemove[]

    PatchOp->>Handler: writeEventsAsync(groupId, added, removed)
    Handler->>Repo: appendEvents(events)
    Repo->>CH: INSERT INTO Group_4_0_0_MemberEvents
    CH->>CH: Materialized view updates
    CH-->>Repo: Success
    Repo-->>Handler: Success
    Handler-->>PatchOp: Success

    PatchOp->>Context: Set GROUP_MEMBER_EVENTS_WRITTEN flag
    PatchOp->>Mongo: Update metadata only<br/>(versionId, lastUpdated)

    Note over PatchOp: PostSave handler sees flag,<br/>skips member processing

    PatchOp-->>API: Success
    API-->>Client: 200 OK

What this diagram shows

This diagram illustrates the sequence of operations for incrementally adding members to a FHIR Group resource using a PATCH request. It details the interaction between the client, FHIR API, a dedicated PATCH handler, a ClickHouse-based event handler and repository, ClickHouse for event storage and materialized views, and MongoDB for metadata updates. A key aspect is the optimization where a flag is set in the HTTP context to prevent redundant member processing in a post-save handler.

When to use it

Use this pattern when designing APIs for complex resource updates, especially for large collections like group members, where event-driven updates, performance, and eventual consistency are critical. It's suitable for systems integrating specialized data stores (e.g., analytical/event store like ClickHouse) with traditional databases (e.g., MongoDB for core resource data).

How to adapt it for your project

This flow can be adapted for other FHIR resources requiring incremental updates or for similar event-driven architectures in non-FHIR contexts. The ClickHouse component could be replaced with other event stores (e.g., Kafka, Cassandra), and MongoDB with any document or relational database. The 'GROUP_MEMBER_EVENTS_WRITTEN' flag mechanism can be generalized to skip post-processing steps based on specific operation types.

Key concepts