Shared-payload TOC pattern

The pattern

A RESO TOC entry is a (length, data_offset) pair (variant=1, 8 bytes per entry). The format allows two TOC entries to carry the same data_offset and length, in which case they share a single physical payload. The engine reads the payload identically regardless of which TOC entry index requested it.

Empirical scan across data/message/**/*.ofs:

Example: msg51016.ofs

Original file: 3,472 bytes. RESO header variant=1, file_count=100. Of the 100 TOC entries, only 27 distinct data_offset values.

Unique payload bytes: 2,526. Phantom bytes saved by sharing: 11,412 (3.5× the payload section if expanded out).

Why this matters for re-emission

Without dedup, a fresh payload is written per TOC entry, so a 100-entry-but-27-payload original becomes a 90+ distinct-payload rebuilt file. msg51016.ofs ballooned from 3,472 B → 19,728 B before the dedup fix. Across the corpus the unfixed re-emit added ≈3.54 MB of pure layout waste.

Dedup logic in re-emit

From src/translator/reemit.py::rewrite_reso_container (variant-1 header, 8-byte entries):

header_entry_size = 16 if h.variant == 0 else 8
payload_start     = 0x10 + h.file_count * header_entry_size

new_header        = bytearray()
payload_section   = bytearray()
payload_offsets   : dict[bytes, int] = {}   # payload-bytes -> offset

for ei in range(h.file_count):
    payload = new_payloads.get(ei, b"")
    if len(payload) == 0:
        new_header += struct.pack("<II", 0, 0)
    elif payload in payload_offsets:                     # ← REUSE OFFSET
        offset = payload_offsets[payload]
        new_header += struct.pack("<II", len(payload), offset)
    else:                                                # ← NEW PAYLOAD
        offset = payload_start + len(payload_section)
        payload_offsets[payload] = offset
        payload_section += payload
        pad = (16 - (len(payload) % 16)) % 16            # ← 16-B align
        payload_section += b"\x00" * pad
        new_header += struct.pack("<II", len(payload), offset)
    if header_entry_size == 16:                          # variant-0: copy type_tag + unk
        new_header += raw[0x10 + ei*16 + 8 : 0x10 + ei*16 + 16]

Canonicalisation

Dedup only helps when the per-entry EN payloads happen to be byte-identical. Because each entry was translated independently, two entries that shared a JP payload often produced different EN paraphrases. Canonicalisation walks each original shared group and forces all members to the same EN string before re-emit.

Selection rule per group (from shared_payload_canonicalization.md):

1. If any row in the group is marked manual, that row wins.
2. Else: longest en_text wins (most likely to fit budget).
3. Tie-break: lowest entry_id.

Top growers (before dedup + canonicalisation)

All ten files are msg<N>016.ofs or msg<N>043.ofs in the per-chapter career directories. The shared-group pattern is the engine's "pick one variant by (npc_id mod N)" table.

Source

• src/translator/reemit.py — dedup + emit logic
• notes/msg51016_investigation.md — root-cause + numbers
• notes/shared_payload_canonicalization.md — canon pass stats