Je Auto Control Multi-Platform Automation System Architecture

System Design · flowchart diagram · MIT

A comprehensive system architecture for 'je_auto_control', detailing client interfaces, transport layers, server-side logic, core automation functionalitie

Source: https://github.com/Integration-Automation/AutoControlGUI/blob/ecfdb0354a64519036ab53ad5a8e3c9b2eea324f/README/README_zh-CN.md
Curated by Integration-Automation
Automation RPA System Architecture Cross-platform Remote Control Mermaid Flowchart

Mermaid source

flowchart LR
    subgraph Clients["客户端接口"]
        direction TB
        Claude[["Claude Desktop /<br/>Claude Code"]]
        APIUser[["自定义 Anthropic /<br/>OpenAI tool-use 循环"]]
        HTTPClient[["HTTP / SSE clients"]]
        TCPClient[["Socket / REST clients"]]
        Browser[["浏览器<br/>(/dashboard · /docs)"]]
        GUIUser[["PySide6 GUI"]]
        CLIUser[["python -m<br/>je_auto_control[.cli]"]]
        Library[["Library 使用者<br/>(import je_auto_control)"]]
    end

    subgraph Transports["传输与服务器"]
        direction TB
        Stdio["MCP stdio<br/>JSON-RPC 2.0"]
        HTTPMCP["MCP HTTP /<br/>SSE + auth + TLS"]
        REST["REST 服务器 :9939<br/>bearer auth · rate-limit ·<br/>OpenAPI · /metrics · /dashboard"]
        Socket["Socket 服务器<br/>:9938"]
        WebRTC["WebRTC sessions<br/>(远程桌面 ·<br/>文件 · 音频 · USB)"]
    end

    subgraph MCP["mcp_server/"]
        direction TB
        Dispatcher["MCPServer<br/>(JSON-RPC dispatcher)"]
        Tools["tools/<br/>~90 ac_* + 别名"]
        Resources["resources/<br/>files · history ·<br/>commands · screen-live"]
        Prompts["prompts/<br/>内置模板"]
        Context["context · audit ·<br/>rate-limit · log-bridge"]
        FakeBE["fake_backend<br/>(CI 烟雾测试)"]
    end

    subgraph Core["无头核心 (wrapper/ + utils/)"]
        direction TB
        Wrapper["wrapper/<br/>鼠标 · 键盘 · 屏幕 ·<br/>图像 · 录制 · 窗口"]
        Executor["executor/<br/>AC_* JSON 动作引擎"]
        Vision["vision/ · ocr/ ·<br/>accessibility/"]
        Recorder["scheduler/ · triggers/ ·<br/>hotkey/ · plugin_loader/<br/>run_history/"]
        IOUtils["clipboard/ · cv2_utils/ ·<br/>shell_process/ · json/"]
    end

    subgraph Ops["运维层 (utils/)"]
        direction TB
        Admin["admin/<br/>多主机轮询 +<br/>广播"]
        Audit["remote_desktop/<br/>audit_log<br/>(SHA-256 链)"]
        Inspector["remote_desktop/<br/>webrtc_inspector"]
        Diag["diagnostics/<br/>自我诊断"]
        ConfigB["config_bundle/<br/>导出/导入"]
    end

    subgraph USB["USB"]
        direction TB
        UsbEnum["usb/<br/>列举 + hotplug"]
        UsbPass["usb/passthrough/<br/>session · client · ACL ·<br/>libusb · WinUSB · IOKit"]
    end

    subgraph Remote["远程桌面 (utils/remote_desktop/)"]
        direction TB
        RDHost["host · webrtc_host ·<br/>signaling · multi_viewer"]
        RDFiles["webrtc_files · file_sync ·<br/>clipboard_sync · audio"]
        RDTrust["trust_list · fingerprint ·<br/>turn_config · lan_discovery"]
    end

    subgraph Backends["操作系统后端"]
        direction TB
        Win["windows/<br/>Win32 ctypes"]
        Mac["osx/<br/>pyobjc · Quartz"]
        X11["linux_with_x11/<br/>python-Xlib"]
    end

    Claude --> Stdio
    APIUser --> Stdio
    HTTPClient --> HTTPMCP
    TCPClient --> Socket
    TCPClient --> REST
    Browser --> REST

    Stdio --> Dispatcher
    HTTPMCP --> Dispatcher
    Dispatcher --> Tools
    Dispatcher --> Resources
    Dispatcher --> Prompts
    Dispatcher -.- Context
    Tools -.可选.-> FakeBE

    Tools --> Wrapper
    Tools --> Executor
    Tools --> Vision
    Tools --> Recorder
    Tools --> IOUtils
    Resources --> Recorder
    Resources --> Wrapper

    REST --> Executor
    REST --> Ops
    REST --> USB
    Socket --> Executor
    WebRTC --> Remote
    WebRTC --> UsbPass

    GUIUser --> Wrapper
    GUIUser --> Recorder
    GUIUser --> Ops
    GUIUser --> USB
    GUIUser --> Remote
    CLIUser --> Executor
    Library --> Wrapper
    Library --> Executor
    Library --> Ops

    Admin --> REST
    Inspector -.- WebRTC
    Audit -.- REST
    Audit -.- USB
    UsbPass --> Backends

    Wrapper --> Backends
    Vision -.- Wrapper
    Recorder -.- Executor

What this diagram shows

This diagram illustrates the layered architecture of the 'je_auto_control' system. It shows how various client interfaces (e.g., GUI, CLI, API, browser) connect through different transport mechanisms (e.g., HTTP, TCP, WebRTC) to a central MCP server. The server dispatches requests to a headless core responsible for platform-agnostic automation tasks like mouse, keyboard, and screen control, vision, and scheduling. It also highlights operational layers, USB management, remote desktop capabilities, and the underlying OS-specific backends (Windows, macOS, Linux X11) that provide the actual control mechanisms.

When to use it

This architecture is ideal for designing robust, multi-platform automation tools that require diverse client access, remote control capabilities, and extensibility. It's suitable for projects involving UI automation, robotic process automation (RPA), remote desktop solutions, and systems needing a unified API over various operating systems.

How to adapt it for your project

This architecture can be adapted by adding new client interfaces or transport protocols to the 'Clients' and 'Transports' subgraphs. New automation actions or vision capabilities can be integrated into the 'Core' and 'Vision' components. Extending to new operating systems would involve creating new 'Backends'. The 'Ops' layer can be enhanced with more sophisticated monitoring or administration tools, and the 'MCP' server can be customized with additional tools or resource types.

Key concepts