# Slow receivers and API latency: experiment data

Measured on 2026-11-02. Article statistics use only the main measurements.

## Conditions

Linux 6.12 / aarch64 / glibc 2.36, CPython 3.11.2, Gunicorn 23.0.0, NGINX 1.22.1 (Debian 1.22.1-9+deb12u10).

- Main: two Gunicorn sync workers; direct access, NGINX buffering off, or buffering on; zero, one, or two slow connections; three runs each.
- Supplementary: one Gunicorn gthread worker with four threads, direct access, two slow connections, three runs. Concurrent capacity differs, so this does not rank worker models.
- `/fast`: 12-byte JSON, eight requests per second at 125ms intervals for nine seconds; 72 requests per run. The next request does not wait for the previous response.
- `/large`: one 64KiB block repeated 256 times, totaling 16MiB. Each slow client reads at about 2MiB/s and completes in about eight seconds. Fast requests start about 250ms after slow connections start.
- Slow socket SO_RCVBUF is set to 8KiB; Linux reports 16KiB. The asyncio StreamReader limit is 16KiB. This controls application read pacing, not the bandwidth of a real network link.
- All traffic uses 127.0.0.1 on the same host, HTTP/1.1, plaintext, and connections closed after responses. No TLS, compression, or cache. No measurement traffic went to existing applications or public URLs.
- CPU caps: application 0.25 core, NGINX 0.20 core, controller/load generator 0.20 core; quota period 10ms. Memory caps: 128MiB, 96MiB, and 128MiB respectively.
- NGINX settings are identical except buffering on/off: proxy_buffer_size=16k, proxy_buffers=4×16k, proxy_busy_buffers_size=32k, proxy_max_temp_file_size=32m per request, proxy_temp_file_write_size=64k. One NGINX process, master_process off.

## Files and aggregation

- `conditions.csv`: main conditions. p50/p95 are nearest-rank percentiles within each run, followed by the median of three runs. p95_min/max give the range of the three run values. app_send_s and client_receive_s first take each run's median of two responses, then the median of three runs. temp_peak_mib is the maximum total logical size observed across three runs.
- `runs.csv`: 30 completed runs. CPU core = cgroup CPU-time increment / elapsed time; 1.0 means one core. throttled_ms is the increment of cgroup throttled_usec in the same interval. memory_peak_bytes is the maximum sampled memory.current at about 200ms intervals, not necessarily the true instantaneous peak. rss_peak_bytes is the maximum summed process RSS within the cgroup and may double-count shared pages.
- `requests.csv`: all 2,160 `/fast` requests. Times are relative to the beginning of each run. latency_ms spans actual start to receipt of the body; schedule_lag_ms spans scheduled to actual start; connect_ms measures TCP establishment. Every status is 200.
- `large-responses.csv`: 33 `/large` responses. receive_s spans request start after TCP establishment to completion of consuming the final chunk at the paced read rate. app_send_s spans response-generator start to the point after the server writes the final chunk; it is not arrival at the user.
- `resource-samples.csv`: time series at about 200ms intervals. temp_logical_bytes totals logical sizes of NGINX's open proxy_temp files; temp_allocated_bytes totals st_blocks×512. Open files whose names were removed are included through /proc/PID/fd, deduplicated by inode. These are not physical I/O totals. memory_bytes is cgroup memory.current, not RSS.
- `app.py`, `experiment.py`: application and measurement code. SSH launchers containing execution hostnames or credentials and original internal logs are excluded.
- `dependencies.json`: distribution URLs, downloaded byte counts, SHA-256 hashes, and the NGINX package version.

Latency-chart points are medians of three runs; bars show min–max. Write/receive bars are medians of six responses with two slow connections; dots show individual app write times. Time-series plots select the run with median p95 for each route. Each figure states its aggregation unit.

## Reproduction

The standard-library measurement code needs Linux systemd user services and cgroup v2. `prepare` downloads and extracts Debian bookworm-security arm64 indexes and distributions into a dedicated directory; it does not install host packages. As it uses the current distribution index, a rerun may fetch a different NGINX package version. For exact versions, check URLs and SHA-256 hashes in `dependencies.json`.

This example assumes a Linux host matching the original conditions and sufficient spare resources. It is not a procedure for changing an existing service or cluster.

```sh
lab_root=$(mktemp -d /tmp/kamui-slow-client-XXXXXXXX)
cp app.py experiment.py "$lab_root/"
python3 "$lab_root/experiment.py" --root "$lab_root" --stage prepare
lab_unit="$(basename "$lab_root")-controller.service"
systemd-run --user --wait --pipe --collect --unit "$lab_unit" \
  -p CPUQuota=20% -p CPUQuotaPeriodSec=10ms -p MemoryMax=128M \
  -p MemorySwapMax=0 -p TasksMax=24 -p RuntimeMaxSec=720 -p TimeoutStopSec=3 \
  /usr/bin/python3 "$lab_root/experiment.py" --root "$lab_root" \
  --stage main --controller-unit "$lab_unit"
```

The code stops each run's servers in finally blocks. Collect results and verify that no dedicated experiment services or processes remain before removing only the dedicated temporary directory you created.

## Exclusions and limitations

Exploratory pilots at 8MiB/1MiB/s and 16MiB are excluded from the main statistics. Moving from 8MiB/1MiB/s to 16MiB/2MiB/s changes both response size and read rate; the difference cannot be attributed to size alone. A host-wide CPU threshold stopped the main measurements once. The incomplete run was excluded; after 17 completed runs, the remaining 13 resumed without changing safety thresholds or application conditions.

receive_s ends after read-pacing waits, including the wait after the final socket read. Approximately eight seconds therefore includes pacing and is not pure real-network transfer performance. The gthread supplement changes worker model, process count, and concurrent capacity together; it does not isolate model effects or determine an optimal worker count.

The application is handwritten WSGI without a framework. There is no comparison of Flask, Django, FastAPI, asynchronous execution, or actual PaaS serving configurations. Individual TCP-buffer occupancy and worker stacks were not collected; observed write times, API latency, and temporary files must be distinguished from specification-based explanations.

This is not a maximum-throughput experiment. Individual OS TCP-buffer capacities, TCP_INFO, real Internet paths, TLS, HTTP/2/3, SSE, WebSockets, slow uploads, long durations, many connections, and disk exhaustion were not measured. Results do not guarantee performance for other web servers or PaaS platforms.