open https://gitlab.synchro.net/main/sbbs/-/issues/1195

## Summary

On a clean link, the Synchronet `sexyz` **ZMODEM sender is ~30–36× slower than
`lrzsz`'s `lsz`** (≈6 MB/s vs ≈200 MB/s on localhost). The bottleneck is `sexyz.c`'s transmit path, **not** the shared `zmodem.c` protocol engine, so **SyncTERM is not affected** (its sender is a simple buffered path).

## Measurements (32 MB random file, localhost, `-8`, unlimited window)

| Sender → Receiver | Goodput |
|---|--:|
| lsz → lrz (lrzsz baseline) | 203.8 MB/s |
| lsz → **sexyz** (sexyz receives) | 106.5 MB/s |
| **sexyz** → lrz (sexyz sends) | **5.7 MB/s** |
| sexyz → sexyz | 6.2 MB/s |

Wire overhead is identical (+2.81%) in every case — protocol efficiency is the
same; the gap is purely the sender implementation. sexyz's **receiver** is fine (~106 MB/s); only its **transmitter** is slow.

## Root cause (syscall profile, 32 MB)

| Sender | `write()` calls | avg write | `futex` calls |
|---|--:|--:|--:|
| lsz | 12,302 | ~2,803 B | 0 (single-threaded) |
| sexyz | 409,188 | **~84 B** | **2,268,600** (872 K erroring) |

`sexyz.c` feeds protocol output one byte at a time into a `RingBuf` (`send_byte()` → `RingBufWrite`, with per-write event signaling), drained by a
separate `output_thread`. Producer and consumer ping-pong: the consumer wakes on
each trickle, drains ~84 bytes, `write()`s, and waits again — a futex storm plus
badly fragmented output. Disabling the `OutbufHighwaterMark`/`OutbufDrainTimeout`
batching via `sexyz.ini` changes nothing (6.13 → 6.18 MB/s): that batching only
engages when the ring hits *empty*, which steady production avoids.

By contrast, SyncTERM's `send_byte` (`src/syncterm/term.c`) accumulates into an 8 KB `transfer_buffer` and `flush_send()`→`conn_send()` writes it in bulk (one
send per subpacket) — single-threaded, no ring buffer, no futex. Its write pattern resembles `lsz`'s, so SyncTERM does not have this problem.

## Suggested fix

Batch escaped output into the ring in spans (or bypass the ring for the bulk data path) so `write()`s are subpacket-sized rather than ~84 B and the producer/consumer stop ping-ponging on a futex. This is a `sexyz.c`-only change.

## Related enhancement (separate)

`zmodem.c` grows/shrinks the subpacket length with a blind ×2-up / ÷2-down ramp,
whereas `lrzsz` continuously tunes block length to the measured error rate (`calc_blklen`, a cost model over bytes-on-wire). Adding an adaptive block-length
model to `zmodem.c` would help both sexyz and SyncTERM on lossy/variable links.

## Method

Benchmarked with a userspace relay harness wiring two stdio ZMODEM endpoints (relay ceiling measured at ~5 GB/s, so not the bottleneck); random data on tmpfs; SHA-256 integrity-checked every run; `sexyz` = `v3.3 master/074785210`, `lrzsz` 0.12.21rc.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9660

### Refinement: it's two stacked overheads, and SyncTERM shares one of them

Follow-up measurement refines the root cause above (and corrects the
"does not affect SyncTERM" claim). By linking the **real, unmodified `zmodem.o`** behind two different `send_byte` implementations and sending
32 MB to `lrz` (identical wire bytes in every case), the ~25x sexyz-vs-lrzsz gap splits into **two independent layers**:

| Sender | Send path | Goodput |
|---|---|--:|
| `lsz` (lrzsz's own inlined `zm.c`) | buffered | **204 MB/s** |
| test sender: real `zmodem.c` + **buffered** `send_byte` (a faithful SyncTERM model) | buffered | **~26 MB/s** |
| `sexyz`: real `zmodem.c` + **ring-buffer + output thread** | ring/thread | **8 MB/s** |

- **204 → 26 (~8x): `zmodem.c`'s per-byte send design.** `strace` of the
buffered sender shows *no futex* and only ~5 ms in syscalls for a 1.3 s run
— it is CPU-bound *inside `zmodem.c`* at ~38 ns/byte, on the
`send_byte`-callback-per-byte + ZDLE-escape + CRC-32 pipeline (vs `lsz`'s
~5 ns/byte inlined). Block size barely moves it. **This layer is shared, so
SyncTERM inherits it.**
- **26 → 8 (~3x): `sexyz.c`'s ring-buffer + output thread** (the futex storm /
~84-byte writes described above). **This layer is sexyz-only; SyncTERM's
buffered `term.c` send path avoids it.**

So SyncTERM's ZMODEM upload lands at **~26 MB/s** — ~3x faster than sexyz, but
still ~8x slower than lrzsz. The correction: SyncTERM escapes sexyz's worst layer but is **not** immune to the throughput problem; both Synchronet tools inherit `zmodem.c`'s per-byte send cost.

**Two fixes, both worthwhile:**
1. (`sexyz.c`) De-fragment the ring/thread TX path — ~3x, sexyz-only, the easy win.
2. (`zmodem.c`) Batch the ZDLE-escape/CRC inner loop and hand `send_byte` a span
instead of a byte — the larger ~8x lever, benefits sexyz **and SyncTERM**.

Full analysis, conditions matrix, and the 2 GiB-window issue are written up in `docs/zmodem_comparison.md` on master.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9661

### Note for whoever fixes this: the `SINGLE_THREADED` build is not the fix

`sexyz.c:84` still has `#define SINGLE_THREADED FALSE` (source-level only — there's no makefile/`.vcxproj` toggle). It's tempting to flip it to `TRUE` to kill the two-thread futex storm, but that path is **worse**: its single-threaded
`send_byte` (`sexyz.c:682`) calls `sendbuf()` with `len == 1`, i.e. an **unbuffered `write()`/`send()` per byte** (~34 M syscalls per 32 MB). It swaps the futex storm for a write-syscall storm.

The three send architectures:

| sexyz `send_byte` | threads | batching | per-32 MB kernel cost | |---|---|---|---|
| `SINGLE_THREADED FALSE` (default) | 2 (ring + drain) | ~84 B fragmented writes | 2.3 M `futex` |
| `SINGLE_THREADED TRUE` | 1 | none — 1 byte / `write()` | ~34 M `write()` | | buffered (SyncTERM `term.c`) | 1 | flush per subpacket | ~8 K `write()`, 0 futex |

So the win is **output buffering**, not thread count: accumulate escaped bytes and flush per subpacket (single-threaded *and* buffered, exactly as SyncTERM's `term.c` already does). That's the ~26 MB/s tier measured above.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9662

### Correction: steady-state numbers, and Fix A is landed

My two earlier notes quoted throughput from **32 MB** transfers, which were short enough that per-transfer startup dominated and **distorted the magnitudes**. Re-measured at **256 MB steady-state** (same harness, same localhost socket), the picture is:

| Sender | 256 MB steady-state |
|---|--:|
| `lsz` (lrzsz) | 204 MB/s |
| SyncTERM model (`zmodem.c` + buffered) | **~85 MB/s** (not ~26) |
| sexyz **after fix** (batched ring) | **~66 MB/s** |
| sexyz **original** (ring per-byte) | **~11 MB/s** |

So the corrected split of the original ~18× gap (not ~25×):
- **~7.5× `sexyz.c`** (ring per-byte, 85→11) — **now fixed**: `send_byte`
accumulates a subpacket span and does one `RingBufWrite` per flush → ~66 MB/s.
The output thread is **kept** (a single-threaded buffered sexyz *hung* under
injected errors — the thread's async socket drain is what preserves error
recovery).
- **~2.4× `zmodem.c`** (per-byte `send_byte`-callback + escape + CRC, 204→85) —
still open. This is the shared ceiling, so **SyncTERM runs at ~85 MB/s**, not
~26. Addressed next by a `send_buf` (span) callback.

The **mechanism** in the notes above is unchanged and correct (2.3 M→12 K futex,
409 K→4 K writes with the fix); only the throughput magnitudes are corrected. Full write-up: `docs/zmodem_comparison.md` (§3.2), tagged to sexyz.c 3.4 / zmodem.c rev 2.3.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9663

### Update: batching fix attempted and REVERTED — it regressed error recovery

A fix for the sender throughput was prototyped two ways and **both were reverted** after multi-run error-recovery testing:

- **Fix A** — batch a subpacket span into the ring per flush (`sexyz.c` only).
Clean throughput ~11 → ~66 MB/s.
- **Fix B** — a `send_buf` (span) callback in `zmodem.c` so the escape/CRC loop
hands whole spans to the transport. ~66 → ~88 MB/s; benefits SyncTERM too.

Under a heavy injected-error test (forward bit-flips, ~3e-6/byte, 8 MB), the **shipped per-byte sender passed 3/3** (~50 s, slow but correct), while **Fix A timed out 0/3** and **Fix B hard-failed 1/3**.

**Root cause:** batching a whole subpacket before one *blocking* flush leaves the
sender stuck inside the flush (ring full while the receiver is back-pressured mid-retransmit-storm), so it never services the incoming ZRPOS → a bidirectional
stall. The original per-byte-to-ring keeps the output thread draining continuously, so the protocol thread reaches the back-channel check promptly and
recovers.

**Takeaway for the eventual fix:** batched-and-robust *is* achievable (lrzsz does
it), but the design must **service the back-channel while sending** — interleave,
or flush in back-channel-yielding chunks — rather than block on a single whole-subpacket flush. The prototypes here don't, so they're reverted; patches saved out-of-tree for a future redo. Issue stays open.

The only shipped change from this investigation is the separate 2 GB fix (#1196).

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9665

### Redesign attempt: bottleneck confirmed, but batching keeps breaking error recovery

Worked the throughput fix further. Findings:

- **The bottleneck is the producer's per-byte `RingBufWrite`** (mutex-locked) on
the protocol thread: ~11 MB/s. `ztx_buf` (the same `zmodem.o`, buffered
`send_byte`, no ring) does the identical per-byte escape/CRC at ~85 MB/s, so it
is the ring write, not zmodem's escaping. **Consumer-side coalescing of the
output thread was tried and changed nothing** (11.4→11.5 MB/s) — the data
already arrives per byte, so only *producer* batching can help.

- **Producer batching keeps breaking error recovery.** Three distinct prototypes
each hit ~66–88 MB/s and each **failed the 3× error-recovery gate** that the
per-byte sender passes 3/3:
1. accumulate a subpacket, flush via the void `flush()` callback → a failed
flush is silently dropped → receiver desync;
2. a `send_buf` span callback in `zmodem.c` → same class, plus a mid-subpacket
blocking flush that starves the back-channel;
3. flush on an 8 KB threshold *inside* `send_byte` (so the error propagates) +
a non-dropping boundary flush → still 0/3; it sent **ZFIN at 4.9 MB of an
8 MB file** — the sender's position accounting desynced from the wire.

- **Root cause:** batching decouples *what zmodem believes it has sent*
(`send_byte` returned success) from *what is actually on the wire* (still
buffered). ZMODEM error recovery (ZRPOS → seek → resend) assumes those track;
on a reposition the buffered-but-unsent bytes are stale and are not reconciled.
lrzsz batches *and* recovers because its buffering is integrated with the
protocol and it purges/repositions its output buffer on ZRPOS.

- **What a correct fix needs:** an **abort/reposition-aware transport** — a hook
from `zmodem.c` to purge the pending output buffer on ZRPOS/abort — a
coordinated `zmodem.c` + `sexyz.c`/`term.c` change, not a transport-only tweak.

No code ships from this; the robust per-byte sender is retained. The bench and a
full write-up of these dead-ends are in `src/bench/zmodem/` (README) for whoever
takes the coordinated approach. Issue stays open.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9666

### Abort-aware transport purge: implemented, helps, still not enough

Implemented the abort-aware fix from the previous note: a `purge_output` callback in `zmodem.c` (called from `zmodem_handle_zrpos`) that, in `sexyz.c`, clears the send accumulation buffer, re-inits the output ring (`RingBufReInit`), and drops the output thread's linear buffer on every reposition — plus a bounded (2 KB) output-thread read so the in-flight chunk a
purge cannot recall stays small.

Result: the purge **fires correctly** (confirmed via the ZRPOS log) and the transfer gets noticeably further under injected errors (~1.3 MB → ~2.7 MB before
failing), but it **still fails the multi-run error gate**. Under heavy corruption the block size collapses to 128 B, retransmit storms fill the ring, the receiver eventually gives up, and the sender stalls (`waiting for output buffer to flush, 16384 bytes`). That's now **five** distinct batched-transport variants failing the gate while the per-byte sender passes 3/3.

**Conclusion / recommended path:** the two-thread ring + `output_thread` architecture is the wrong substrate for a batched, error-robust sender — a second thread inevitably holds stale bytes it cannot recall in step with the protocol. The reference that is *both* fast and robust is lrzsz: single-threaded,
the protocol thread owns its output buffer and flushes/repositions it synchronously with protocol state. Re-architecting sexyz's **sender** that way (a protocol-integrated buffered send, not a ring+thread) is the change most likely to be fast *and* survive error recovery. It is a rewrite, not a patch, so
it is left for a focused effort.

No source ships from this; the robust per-byte sender is retained. Full trail and
the harness are in `src/bench/zmodem/`.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9666

@/tmp/claude-1000/-home-rswindell-sbbs-src-sbbs3/4beb0350-f06e-4a25-a96e-701d9c55a483/scratchpad/gl/n9666.txt
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9666

### Abort-aware transport purge: implemented, helps, still not enough

Implemented the abort-aware fix from the previous note: a `purge_output` callback in `zmodem.c` (called from `zmodem_handle_zrpos`) that, in `sexyz.c`, clears the send accumulation buffer, re-inits the output ring (`RingBufReInit`), and drops the output thread's linear buffer on every reposition — plus a bounded (2 KB) output-thread read so the in-flight chunk a
purge cannot recall stays small.

Result: the purge **fires correctly** (confirmed via the ZRPOS log) and the transfer gets noticeably further under injected errors (≈1.3 MB → ≈2.7 MB before
failing), but it **still fails the multi-run error gate**. Under heavy corruption the block size collapses to 128 B, retransmit storms fill the ring, the receiver eventually gives up, and the sender stalls (`waiting for output buffer to flush, 16384 bytes`). That's now **five** distinct batched-transport variants failing the gate while the per-byte sender passes 3/3.

**Conclusion / recommended path:** the two-thread ring + `output_thread` architecture is the wrong substrate for a batched, error-robust sender — a second thread inevitably holds stale bytes it cannot recall in step with the protocol. The reference that is *both* fast and robust is lrzsz: single-threaded,
the protocol thread owns its output buffer and flushes/repositions it synchronously with protocol state. Re-architecting sexyz's **sender** that way (a protocol-integrated buffered send, not a ring+thread) is the change most likely to be fast *and* survive error recovery. It is a rewrite, not a patch, so
it is left for a focused effort.

No source ships from this; the robust per-byte sender is retained. Full trail and
the harness are in `src/bench/zmodem/`.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9660

### Refinement: it's two stacked overheads, and SyncTERM shares one of them

Follow-up measurement refines the root cause above (and corrects the
"does not affect SyncTERM" claim). By linking the **real, unmodified `zmodem.o`** behind two different `send_byte` implementations and sending
32 MB to `lrz` (identical wire bytes in every case), the ≈25x sexyz-vs-lrzsz gap splits into **two independent layers**:

| Sender | Send path | Goodput |
|---|---|--:|
| `lsz` (lrzsz's own inlined `zm.c`) | buffered | **204 MB/s** |
| test sender: real `zmodem.c` + **buffered** `send_byte` (a faithful SyncTERM model) | buffered | **≈26 MB/s** |
| `sexyz`: real `zmodem.c` + **ring-buffer + output thread** | ring/thread | **8 MB/s** |

- **204 → 26 (≈8x): `zmodem.c`'s per-byte send design.** `strace` of the
buffered sender shows *no futex* and only ≈5 ms in syscalls for a 1.3 s run
— it is CPU-bound *inside `zmodem.c`* at ≈38 ns/byte, on the
`send_byte`-callback-per-byte + ZDLE-escape + CRC-32 pipeline (vs `lsz`'s
≈5 ns/byte inlined). Block size barely moves it. **This layer is shared, so
SyncTERM inherits it.**
- **26 → 8 (≈3x): `sexyz.c`'s ring-buffer + output thread** (the futex storm /
≈84-byte writes described above). **This layer is sexyz-only; SyncTERM's
buffered `term.c` send path avoids it.**

So SyncTERM's ZMODEM upload lands at **≈26 MB/s** — ≈3x faster than sexyz, but
still ≈8x slower than lrzsz. The correction: SyncTERM escapes sexyz's worst layer but is **not** immune to the throughput problem; both Synchronet tools inherit `zmodem.c`'s per-byte send cost.

**Two fixes, both worthwhile:**
1. (`sexyz.c`) De-fragment the ring/thread TX path — ≈3x, sexyz-only, the easy win.
2. (`zmodem.c`) Batch the ZDLE-escape/CRC inner loop and hand `send_byte` a span
instead of a byte — the larger ≈8x lever, benefits sexyz **and SyncTERM**.

Full analysis, conditions matrix, and the 2 GiB-window issue are written up in `docs/zmodem_comparison.md` on master.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9661

### Note for whoever fixes this: the `SINGLE_THREADED` build is not the fix

`sexyz.c:84` still has `#define SINGLE_THREADED FALSE` (source-level only — there's no makefile/`.vcxproj` toggle). It's tempting to flip it to `TRUE` to kill the two-thread futex storm, but that path is **worse**: its single-threaded
`send_byte` (`sexyz.c:682`) calls `sendbuf()` with `len == 1`, i.e. an **unbuffered `write()`/`send()` per byte** (≈34 M syscalls per 32 MB). It swaps
the futex storm for a write-syscall storm.

The three send architectures:

| sexyz `send_byte` | threads | batching | per-32 MB kernel cost | |---|---|---|---|
| `SINGLE_THREADED FALSE` (default) | 2 (ring + drain) | ≈84 B fragmented writes | 2.3 M `futex` |
| `SINGLE_THREADED TRUE` | 1 | none — 1 byte / `write()` | ≈34 M `write()` |
| buffered (SyncTERM `term.c`) | 1 | flush per subpacket | ≈8 K `write()`, 0 futex |

So the win is **output buffering**, not thread count: accumulate escaped bytes and flush per subpacket (single-threaded *and* buffered, exactly as SyncTERM's `term.c` already does). That's the ≈26 MB/s tier measured above.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9662

### Correction: steady-state numbers, and Fix A is landed

My two earlier notes quoted throughput from **32 MB** transfers, which were short enough that per-transfer startup dominated and **distorted the magnitudes**. Re-measured at **256 MB steady-state** (same harness, same localhost socket), the picture is:

| Sender | 256 MB steady-state |
|---|--:|
| `lsz` (lrzsz) | 204 MB/s |
| SyncTERM model (`zmodem.c` + buffered) | **≈85 MB/s** (not ≈26) |
| sexyz **after fix** (batched ring) | **≈66 MB/s** |
| sexyz **original** (ring per-byte) | **≈11 MB/s** |

So the corrected split of the original ≈18× gap (not ≈25×):
- **≈7.5× `sexyz.c`** (ring per-byte, 85→11) — **now fixed**: `send_byte`
accumulates a subpacket span and does one `RingBufWrite` per flush → ≈66 MB/s.
The output thread is **kept** (a single-threaded buffered sexyz *hung* under
injected errors — the thread's async socket drain is what preserves error
recovery).
- **≈2.4× `zmodem.c`** (per-byte `send_byte`-callback + escape + CRC, 204→85) —
still open. This is the shared ceiling, so **SyncTERM runs at ≈85 MB/s**, not
≈26. Addressed next by a `send_buf` (span) callback.

The **mechanism** in the notes above is unchanged and correct (2.3 M→12 K futex,
409 K→4 K writes with the fix); only the throughput magnitudes are corrected. Full write-up: `docs/zmodem_comparison.md` (§3.2), tagged to sexyz.c 3.4 / zmodem.c rev 2.3.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9663

### Update: batching fix attempted and REVERTED — it regressed error recovery

A fix for the sender throughput was prototyped two ways and **both were reverted** after multi-run error-recovery testing:

- **Fix A** — batch a subpacket span into the ring per flush (`sexyz.c` only).
Clean throughput ≈11 → ≈66 MB/s.
- **Fix B** — a `send_buf` (span) callback in `zmodem.c` so the escape/CRC loop
hands whole spans to the transport. ≈66 → ≈88 MB/s; benefits SyncTERM too.

Under a heavy injected-error test (forward bit-flips, ≈3e-6/byte, 8 MB), the **shipped per-byte sender passed 3/3** (≈50 s, slow but correct), while **Fix A
timed out 0/3** and **Fix B hard-failed 1/3**.

**Root cause:** batching a whole subpacket before one *blocking* flush leaves the
sender stuck inside the flush (ring full while the receiver is back-pressured mid-retransmit-storm), so it never services the incoming ZRPOS → a bidirectional
stall. The original per-byte-to-ring keeps the output thread draining continuously, so the protocol thread reaches the back-channel check promptly and
recovers.

**Takeaway for the eventual fix:** batched-and-robust *is* achievable (lrzsz does
it), but the design must **service the back-channel while sending** — interleave,
or flush in back-channel-yielding chunks — rather than block on a single whole-subpacket flush. The prototypes here don't, so they're reverted; patches saved out-of-tree for a future redo. Issue stays open.

The only shipped change from this investigation is the separate 2 GB fix (#1196).

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9665

### Redesign attempt: bottleneck confirmed, but batching keeps breaking error recovery

Worked the throughput fix further. Findings:

- **The bottleneck is the producer's per-byte `RingBufWrite`** (mutex-locked) on
the protocol thread: ≈11 MB/s. `ztx_buf` (the same `zmodem.o`, buffered
`send_byte`, no ring) does the identical per-byte escape/CRC at ≈85 MB/s, so it
is the ring write, not zmodem's escaping. **Consumer-side coalescing of the
output thread was tried and changed nothing** (11.4→11.5 MB/s) — the data
already arrives per byte, so only *producer* batching can help.

- **Producer batching keeps breaking error recovery.** Three distinct prototypes
each hit ≈66–88 MB/s and each **failed the 3× error-recovery gate** that the
per-byte sender passes 3/3:
1. accumulate a subpacket, flush via the void `flush()` callback → a failed
flush is silently dropped → receiver desync;
2. a `send_buf` span callback in `zmodem.c` → same class, plus a mid-subpacket
blocking flush that starves the back-channel;
3. flush on an 8 KB threshold *inside* `send_byte` (so the error propagates) +
a non-dropping boundary flush → still 0/3; it sent **ZFIN at 4.9 MB of an
8 MB file** — the sender's position accounting desynced from the wire.

- **Root cause:** batching decouples *what zmodem believes it has sent*
(`send_byte` returned success) from *what is actually on the wire* (still
buffered). ZMODEM error recovery (ZRPOS → seek → resend) assumes those track;
on a reposition the buffered-but-unsent bytes are stale and are not reconciled.
lrzsz batches *and* recovers because its buffering is integrated with the
protocol and it purges/repositions its output buffer on ZRPOS.

- **What a correct fix needs:** an **abort/reposition-aware transport** — a hook
from `zmodem.c` to purge the pending output buffer on ZRPOS/abort — a
coordinated `zmodem.c` + `sexyz.c`/`term.c` change, not a transport-only tweak.

No code ships from this; the robust per-byte sender is retained. The bench and a
full write-up of these dead-ends are in `src/bench/zmodem/` (README) for whoever
takes the coordinated approach. Issue stays open.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9668

### Correction (per Deuce, SyncTERM's author) — three earlier claims retracted

Deuce reviewed this and corrected framings in my later notes. The corrected write-up is in `docs/zmodem_comparison.md` (see its correction banner) and `src/bench/zmodem/README.md`.

1. **"SyncTERM ≈85 MB/s (faithful `ztx_buf` model)" is wrong.** `ztx_buf` models
the *shape* of SyncTERM's send path but under-measures its throughput by ≈3×
(Python-relay harness + poll-per-subpacket). SyncTERM's speed is a deliberate
BDP / socket-buffer choice (tuned to ≈75% of a 1 Gb LAN; bump the buffer for
10 Gb), not an emergent send-path fact. Read ≈85 as a localhost floor, not
SyncTERM's throughput.

2. **The "recall stale bytes / abort-aware purge on ZRPOS" root-cause framing was
a wrong garden path.** You cannot recall bytes once they hit the socket/network
buffers — lrz can't either, yet it recovers. The five batched-sender failures
were *implementation bugs* (silent drops, blocking-flush back-channel
starvation, position accounting), not something inherent to batching. lrz
proves a batched sender can be both fast and error-robust; the fix is the
single-threaded, protocol-integrated rewrite (the lrz model), not a purge hook.

3. **Optimizing for localhost is largely beside the point.** Below ≈8 ms RTT the
socket buffer / bandwidth-delay product dominates the send loop; real transfers
live over the network. sexyz's send-loop deficit only bites on fast LANs.

Net: the 2 GB fix (#1196) stands; this throughput item is worth pursuing only if
fast-LAN send-loop throughput matters — otherwise the real lever is socket-buffer
sizing, not the send loop. Thanks to Deuce for the corrections.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9680

It's worth noting that the windowed and streaming protocols both assume that unacked data is in-flight (that's the entire point of these). If error recovery doesn't work when data is in flight, that points strongly to a problem with the error recovery paths, *not* a problem with the buffering behaviour.

Using broken error recovery as an excuse to subvert the entire purpose of windowed and streaming modes is inexcusable.
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1195#note_9704

## Fixed — buffered streaming send path

Resolved in **`34af5f34b5`** (stamps-40-wave), sexyz.c 3.4.

The two output threads and the ring were never the bottleneck; **feeding
the ring one byte at a time** was. `send_byte()` took the ring mutex twice
per byte while `output_thread` hot-looped on it, so a 256 MB localhost send cost ~44 CPU-seconds and 1.46 million voluntary context switches and topped
out near 11 MB/s.

The fix accumulates bytes into a 4 KB buffer and hands whole spans to the
ring. The two-thread architecture, flow control and error handling are otherwise unchanged — the producer just stops trickling:

| | before | after |
|---|--:|--:|
| streaming 256 MB (localhost → lrz) | 11.5 MB/s | **~115 MB/s** |
| sender CPU / 256 MB | 44 s | **0.85 s** |
| voluntary context switches | 1,464,130 | **64,061** |
| error-recovery gate | passes | still passes (verified 5×) |

This is the same throughput a single-threaded rewrite would give, for a ~105-line diff instead of a rewrite, and with better-proven error recovery
(the single-threaded prototype only reached 2/3 on the gate).

### Deliberately scoped to streaming

Buffering is applied **only for full ZMODEM streaming**
(`max_window_size == 0 && !no_streaming`). A transmit window (`-w`) or segmented mode (`-s`) depends on the receiver's ACKs flowing back as data
is sent; bursty buffered output fills the window before any ACK returns and stalls the transfer. Those modes fall through to the original
byte-at-a-time path and are byte-for-byte unchanged.

### Error recovery preserved

A fast streaming sender fills the socket's auto-tuned (multi-megabyte) send buffer, so a single line error would cost a ZRPOS retransmission of
everything in flight. `SO_SNDBUF` is bounded to the ring size
(`OutbufSize`) for streaming to cap the in-flight backlog; `[sockopts]` can override for a high bandwidth-delay link.

### Not in scope

- **`zmodem.c` is unchanged**, so SyncTERM (which already has a buffered
send path) is untouched.
- **Windowed throughput** is still far below lrzsz — it stays on the
byte-at-a-time path by design. The related `-w` correctness fixes shipped
separately: the `window < 4×block` SIGFPE and the `window ≤ block` stall
(block clamped to window/4 like lsz/sz, **`aaf394a0d8`** hello-27-tabs),
both tracked in #1197. Deuce's shared-engine `zmodem.c` optimizations
(byte-classification table, slicing-by-4 CRC-32, buffered `fcrc32`) also
landed under this issue.

Full analysis and the benchmark harness are in `docs/zmodem_comparison.md`
and `src/bench/zmodem/`.

Closing.

*Note by Claude (Anthropic).*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

close https://gitlab.synchro.net/main/sbbs/-/issues/1195
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)