Pre-training Data Preparation Flow for MicroLM

ML & AI · flowchart diagram · MIT

Illustrates the complete data preparation pipeline for pre-training a MicroLM, from raw Chinese corpus to cleaned, split, and EOS-encoded datasets.

Source: https://github.com/jiaran-king/MicroLM/blob/782ae02f10c14b484a317f22115a066b3b10b91d/Readme/%E9%A1%B9%E7%9B%AE%E5%85%A8%E6%99%AF%E5%9B%BE/00-%E5%85%A8%E6%B5%81%E7%A8%8B%E5%88%86%E6%9E%90%EF%BC%88%E8%AE%AD%E7%BB%83%E3%80%81%E6%8E%A8%E7%90%86%E3%80%81%E8%AF%84%E6%B5%8B%E4%B8%8E%E9%83%A8%E7%BD%B2%EF%BC%89.md
Curated by jiaran-king
LLM Pre-training Data Preparation NLP BPE Corpus Data Pipeline

Mermaid source

%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff", "primaryColor": "#eef6ff", "primaryBorderColor": "#60a5fa", "primaryTextColor": "#0f172a", "lineColor": "#64748b"}}}%%
flowchart TB
    D0["原始中文语料<br>~141 万条"] --> D1["清洗与规整<br>去重 · HTML 清理 · 空白压缩 · 长度过滤"]
    D1 --> D2["确定性 train-valid 切分<br>SHA1 哈希"]
    D2 --> D3["插入 EOS 文档分隔符"]
    D3 --> D4["产物<br>train.txt · valid.txt · tokenizer_corpus.txt"]

    classDef data fill:#eff6ff,stroke:#60a5fa,color:#0f172a;
    classDef step fill:#f8fafc,stroke:#94a3b8,color:#0f172a;
    classDef out fill:#f0fdf4,stroke:#22c55e,color:#0f172a;
    class D0 data;
    class D1,D2,D3 step;
    class D4 out;

What this diagram shows

This diagram details the sequential steps involved in preparing pre-training data for a language model. It starts with a raw Chinese corpus, proceeds through cleaning and normalization (deduplication, HTML cleanup, whitespace compression, length filtering), a deterministic train-valid split using SHA1 hashing, and finally, the insertion of EOS (End-Of-Sentence/Document) delimiters. The output consists of train.txt, valid.txt, and tokenizer_corpus.txt files.

When to use it

Use this flow when preparing large text corpora for pre-training language models, especially when deterministic data processing, explicit document boundaries, and robust cleaning are critical for model performance and reproducibility.

How to adapt it for your project

This flow can be adapted by modifying cleaning rules for different data sources or languages, implementing alternative deterministic splitting methods, or incorporating different tokenization corpus generation strategies. The EOS delimiter can be customized based on the specific model architecture's requirements.

Key concepts