Slow receivers and API latency: experiment data

Measured on 2026-11-02. Article statistics use only the main measurements.

Conditions

Linux 6.12 / aarch64 / glibc 2.36, CPython 3.11.2, Gunicorn 23.0.0, NGINX 1.22.1 (Debian 1.22.1-9+deb12u10).

Main: two Gunicorn sync workers; direct access, NGINX buffering off, or buffering on; zero, one, or two slow connections; three runs each.
Supplementary: one Gunicorn gthread worker with four threads, direct access, two slow connections, three runs. Concurrent capacity differs, so this does not rank worker models.
/fast: 12-byte JSON, eight requests per second at 125ms intervals for nine seconds; 72 requests per run. The next request does not wait for the previous response.
/large: one 64KiB block repeated 256 times, totaling 16MiB. Each slow client reads at about 2MiB/s and completes in about eight seconds. Fast requests start about 250ms after slow connections start.
Slow socket SO_RCVBUF is set to 8KiB; Linux reports 16KiB. The asyncio StreamReader limit is 16KiB. This controls application read pacing, not the bandwidth of a real network link.
All traffic uses 127.0.0.1 on the same host, HTTP/1.1, plaintext, and connections closed after responses. No TLS, compression, or cache. No measurement traffic went to existing applications or public URLs.
CPU caps: application 0.25 core, NGINX 0.20 core, controller/load generator 0.20 core; quota period 10ms. Memory caps: 128MiB, 96MiB, and 128MiB respectively.
NGINX settings are identical except buffering on/off: proxy_buffer_size=16k, proxy_buffers=4×16k, proxy_busy_buffers_size=32k, proxy_max_temp_file_size=32m per request, proxy_temp_file_write_size=64k. One NGINX process, master_process off.

Files and aggregation

conditions.csv: main conditions. p50/p95 are nearest-rank percentiles within each run, followed by the median of three runs. p95_min/max give the range of the three run values. app_send_s and client_receive_s first take each run's median of two responses, then the median of three runs. temp_peak_mib is the maximum total logical size observed across three runs.
runs.csv: 30 completed runs. CPU core = cgroup CPU-time increment / elapsed time; 1.0 means one core. throttled_ms is the increment of cgroup throttled_usec in the same interval. memory_peak_bytes is the maximum sampled memory.current at about 200ms intervals, not necessarily the true instantaneous peak. rss_peak_bytes is the maximum summed process RSS within the cgroup and may double-count shared pages.
requests.csv: all 2,160 /fast requests. Times are relative to the beginning of each run. latency_ms spans actual start to receipt of the body; schedule_lag_ms spans scheduled to actual start; connect_ms measures TCP establishment. Every status is 200.
large-responses.csv: 33 /large responses. receive_s spans request start after TCP establishment to completion of consuming the final chunk at the paced read rate. app_send_s spans response-generator start to the point after the server writes the final chunk; it is not arrival at the user.
resource-samples.csv: time series at about 200ms intervals. temp_logical_bytes totals logical sizes of NGINX's open proxy_temp files; temp_allocated_bytes totals st_blocks×512. Open files whose names were removed are included through /proc/PID/fd, deduplicated by inode. These are not physical I/O totals. memory_bytes is cgroup memory.current, not RSS.
app.py, experiment.py: application and measurement code. SSH launchers containing execution hostnames or credentials and original internal logs are excluded.
dependencies.json: distribution URLs, downloaded byte counts, SHA-256 hashes, and the NGINX package version.

Latency-chart points are medians of three runs; bars show min–max. Write/receive bars are medians of six responses with two slow connections; dots show individual app write times. Time-series plots select the run with median p95 for each route. Each figure states its aggregation unit.

Reproduction

The standard-library measurement code needs Linux systemd user services and cgroup v2. prepare downloads and extracts Debian bookworm-security arm64 indexes and distributions into a dedicated directory; it does not install host packages. As it uses the current distribution index, a rerun may fetch a different NGINX package version. For exact versions, check URLs and SHA-256 hashes in dependencies.json.

This example assumes a Linux host matching the original conditions and sufficient spare resources. It is not a procedure for changing an existing service or cluster.

lab_root=$(mktemp -d /tmp/kamui-slow-client-XXXXXXXX)
cp app.py experiment.py "$lab_root/"
python3 "$lab_root/experiment.py" --root "$lab_root" --stage prepare
lab_unit="$(basename "$lab_root")-controller.service"
systemd-run --user --wait --pipe --collect --unit "$lab_unit" \
  -p CPUQuota=20% -p CPUQuotaPeriodSec=10ms -p MemoryMax=128M \
  -p MemorySwapMax=0 -p TasksMax=24 -p RuntimeMaxSec=720 -p TimeoutStopSec=3 \
  /usr/bin/python3 "$lab_root/experiment.py" --root "$lab_root" \
  --stage main --controller-unit "$lab_unit"

The code stops each run's servers in finally blocks. Collect results and verify that no dedicated experiment services or processes remain before removing only the dedicated temporary directory you created.

Exclusions and limitations

Exploratory pilots at 8MiB/1MiB/s and 16MiB are excluded from the main statistics. Moving from 8MiB/1MiB/s to 16MiB/2MiB/s changes both response size and read rate; the difference cannot be attributed to size alone. A host-wide CPU threshold stopped the main measurements once. The incomplete run was excluded; after 17 completed runs, the remaining 13 resumed without changing safety thresholds or application conditions.

receive_s ends after read-pacing waits, including the wait after the final socket read. Approximately eight seconds therefore includes pacing and is not pure real-network transfer performance. The gthread supplement changes worker model, process count, and concurrent capacity together; it does not isolate model effects or determine an optimal worker count.

The application is handwritten WSGI without a framework. There is no comparison of Flask, Django, FastAPI, asynchronous execution, or actual PaaS serving configurations. Individual TCP-buffer occupancy and worker stacks were not collected; observed write times, API latency, and temporary files must be distinguished from specification-based explanations.

This is not a maximum-throughput experiment. Individual OS TCP-buffer capacities, TCP_INFO, real Internet paths, TLS, HTTP/2/3, SSE, WebSockets, slow uploads, long durations, many connections, and disk exhaustion were not measured. Results do not guarantee performance for other web servers or PaaS platforms.

Markdown