Why Deduplicate JSONL Files?
Duplicate records in JSONL training data can skew model behaviour during fine-tuning — the model sees certain examples more often and may overfit to them. Deduplication is also essential when merging multiple event-log exports that may contain overlapping time windows, or when a pipeline crashes and writes the same records twice.
Exact Match vs Key-Field Deduplication
Exact match treats two lines as duplicates only if every character is identical. This is fast and deterministic but will treat {"a":1,"b":2} and {"b":2,"a":1} as different records even though they represent the same object (key order differs).
Key-field mode parses each line as JSON and extracts the value at a specified path (e.g. id or messages[0].content). Two lines with the same value at that path are treated as duplicates regardless of key order or other field values. This is the right choice for deduplicating by a natural identifier.
Keep First vs Keep Last
By default the tool keeps the first occurrence of a duplicate and discards later ones. Toggle Keep last occurrence to reverse this — useful when your JSONL is time-ordered and newer records contain corrections or updates to earlier ones.
Deduplicating OpenAI Fine-Tuning Datasets
For chat-format JSONL ({"messages":[...]}), use key-field mode with the path messages[0].content (the system prompt) or messages[1].content (the user turn) to remove repeated prompt-response pairs. This gives you unique examples without discarding records that share a system prompt but have different user questions.