MSG wire format · Tongari Boushi Archive

A message file is a RESO container whose payloads are dialog streams. Each stream is a sequence of little-endian 16-bit words. Bit 15 of every word distinguishes control tokens (bit 15 clear) from data words (bit 15 set: opcodes, arguments, and the UTF-16-LE characters of Japanese text — all of which sit above U+8000 or are encoded as data above that line).

Control tokens

The parser at ARM9 0x02066c58 recognises four control words:

Name	u16	ASCII	Role
`OPEN`	`0x0024`	`$`	Start of CMD
`ARG_SEP`	`0x0027`	`'`	Argument separator inside a CMD
`CLOSE`	`0x003b`	`;`	End of CMD arg list
`END`	`0x000c`	`\f`	End of dialog payload

Grammar

PAYLOAD     := (HEADER_CMD)? CONTENT 0x000C
CONTENT     := (TEXT_RUN | INLINE_CMD | NUL)*
TEXT_RUN    := utf16_word+                       (all words have bit 15 set or are kana/kanji code units)
NUL         := 0x0000                             (phrase boundary; "§" in en_text)
INLINE_CMD  := 0x0024 OPCODE_W (0x0027 ARG_W)* 0x003B (DATA_W* 0x0000)?
HEADER_CMD  := 0x0024 0x8037 (0x0027 ARG_W){1,2} 0x003B 0x0000
OPCODE_W    := u16 with bit 15 set; opcode = (word & 0x7FFF)
ARG_W       := u16 with bit 15 set; arg    = sign_extend15(word & 0x7FFF)
DATA_W      := raw u16 emitted after CLOSE for opcodes that carry a data tail

sign_extend15(x) is ((int16)(x << 17)) >> 17 — that is, the 15-bit value treated as signed.

Re-emission rules

From src/translator/reemit.py::cmd_bytes_for_tag — the encoder that rebuilds CMD bytes from [TAG] notation:

bytes  ← <0x0024> <opcode | 0x8000>
for arg in args:
    bytes ← bytes + <0x0027> <(arg & 0x7FFF) | 0x8000>
bytes  ← bytes + <0x003B>
if data_words is not empty:
    for dw in data_words: bytes ← bytes + <dw>
    bytes ← bytes + <0x0000>

The trailing 0x0000 after CLOSE belongs to the CMD only when the opcode emits a data tail (e.g. headers, certain script calls). A bare CMD with no data words ends at 0x003B.

Body text encoding

Japanese text: UTF-16-LE. All kana/kanji code units fall in 0x3000+, so they read as data words even though bit 15 is clear on some of them — the parser uses the four explicit control tokens (above) as the only escape sequence.
§ (U+00A7) in the translation database encodes a phrase boundary; on the wire it becomes 0x0000.
▼ (U+25BC) is a "press to advance" page break — emitted verbatim into the UTF-16 stream as 0x25BC.
Furigana ruby annotations use literal {kanji|reading} — three ASCII bytes encoded as UTF-16 code units 0x007B, 0x007C, 0x007D. The renderer special-cases these.
U+3000 (the full-width space) is the in-game word separator.

Annotated hex dump

First non-empty entry of data/message/msg00/00/msg00000.ofs (excerpted). Words shown LE, two bytes per word, address relative to start of the payload:

offs  bytes        decoded
----  -----------  ---------------------------------------------------
0000  24 00        OPEN                          ┐
0002  18 80        opcode 0x0018  (SPEAKER)      │
0004  27 00        ARG_SEP                       │ inline CMD
0006  72 9b        arg = 0x1B72                  │ opcode=0x18
0008  27 00        ARG_SEP                       │ args=[0x1B72, 0x00]
000A  00 80        arg = 0x0000                  │
000C  3b 00        CLOSE                         ┘
000E  24 00        OPEN                          ┐
0010  17 80        opcode 0x0017  (TEXTBOX)      │
0012  27 00 72 9b  ARG_SEP, arg 0x1B72           │ inline CMD
0016  27 00 b2 80  ARG_SEP, arg 0x0032 (=0xB2&7FFF)│ opcode=0x17
001A  27 00 0a 80  ARG_SEP, arg 0x000A           │ args=[0x1B72,0x32,0x0A,0x01]
001E  27 00 01 80  ARG_SEP, arg 0x0001           │
0022  3b 00        CLOSE                         ┘
0024  24 00 16 80 ... CMD opcode 0x0016 (EXPR), 2 args ...
...
003E  4a 30 84 30   utf-16: お や              (TEXT_RUN starts here)
0042  20 00         space                       (U+0020 — in this file, half-width)
0044  42 30 6a 30 5f 30 ... utf-16: あなた
... (more body text) ...
00A0  bc 25         '▼' page break
... more text + CMDs ...
0180  00 00         NUL (phrase boundary, §)
0182  0c 00         END

How the parser walks the stream

Paraphrased from FUN_02066c58 (decompiled from arm9_full_plaintext.bin):

short *p = msg->text;          // payload pointer (struct field +0x24)
ushort max = msg->len;         // word count       (struct field +0x30)

// 1. optional header — only if first CMD is opcode 0x37
if (*p == 0x0024 && (p[1] & 0x7FFF) == 0x0037) {
    p += 2;
    while (*p != 0x003B && p < end) {
        if (*p == 0x0027) header_arg[i++] = sign_extend15(*++p & 0x7FFF);
        p++;
    }
    if (*p == 0x003B && p[1] == 0) p += 2;     // skip "3B 00"
}

// 2. content loop
for (int line = 0; p < end; ) {
    while (*p != 0x0024 && *p != 0x0000 && *p != 0x000C) p++;
    if (*p == 0x000C) break;                   // END
    if (*p == 0x0000) { p++; continue; }       // phrase boundary

    *p++ = 0;                                  // null-terminate preceding text
    ushort op = *p & 0x7FFF;
    switch (op) {
      case 0x10: line[i].flags = (flags & 0xF0) | 3; break;
      case 0x11: line[i].flags = (flags & 0xF0) | 2; break;
      case 0x12: line[i].flags = (flags & 0xF0) | 1; break;
      case 0x13: line[i].flags &= 0xF0;             break;
      case 0x4C: p += FUN_0206726C(...); continue;  // sub-handler: 2 typed args
      default:   break;                              // renderer handles it
    }
    p++;                                       // advance past opcode
    while (*p == 0x0027) {                     // collect args
        p++; args[k++] = sign_extend15(*p & 0x7FFF); p++;
    }
    if (*p == 0x003B) p++;                     // CLOSE
}

The parser only directly interprets six opcodes (0x10, 0x11, 0x12, 0x13, 0x37, 0x4C) — everything else is forwarded to the renderer func_0x021c0ef8 as part of the per-line record. See opcodes for the full catalog.

Empirical census

Across 1,328 message files and 50,196 non-empty entries:

88 distinct opcodes appear.
2,526 entries open with the 0x37 HEADER CMD.
703 of 1,328 files (52.9%) use payload sharing — see shared payloads.
Top 10 opcodes account for >80% of all CMDs.

Source

src/translator/reemit.py — encoder (CMD bytes, tags, NUL handling)
src/archive/reso.py — TOC parser used to slice payloads from .ofs
arm9_full_plaintext.bin @ 0x02066c58 — parser FUN_02066c58
arm9_full_plaintext.bin @ 0x021c0ef8 — renderer (external symbol)