Compare

How the optimum is computed

Each AXI read is split into MRds. Every MRd reserves its worst-case completion space at issue (the completer may split on every RCB boundary), because a root port must advertise infinite completion credits. The read path is then a closed queueing network: a delay stage (pipeline, switches, endpoint) and the upstream link.

$BW_{\max} \;=\; \frac{E[L]}{D} \qquad\qquad N^{*} \;=\; 1 + \frac{Z}{D}$

$N_{\eta} \;=\; \min\left\{\, N \;:\; X(N)\,E[L] \;\ge\; \eta\,BW_{\max} \,\right\}$

$B_{\mathrm{opt}} \;=\; N_{\eta}\,\frac{E[R\,T]}{E[T]} \;+\; \frac{E[R]}{2} \;+\; \text{staging} \;+\; \text{stall headroom}$

$H_{\mathrm{opt}} \;=\; N_{\eta}\,\frac{E[H\,T]}{E[T]} \;+\; \frac{E[H]}{2} \qquad\qquad \text{tags} \;\ge\; N_{\eta}$

where D is the mean completion wire time at the bottleneck link, Z the mean non-link hold time, and X(N) the throughput from approximate MVA with link service variability c_s². R and H are the completion data and header footprint of one read, T its wire time, and L its payload.

Only the mean of the endpoint and switch latency affects saturated throughput; the delay stage is insensitive to the distribution. What costs extra buffer is reservation rounding (misaligned addresses, coarse slots), the head-of-line gap of in-order admission, and variation in completion size on the link. The simulation check runs a discrete-event model of the same closed loop with random latencies.

The theory behind it, and further reading

This is a textbook capacity-planning model from operations research — the kind used for computer systems for decades — applied to a hardware buffer. Each part of the calculator rests on a named result:

Little’s law (L = λW) is the foundation, used three times: the latency-bound line BW = N·E[L]/(Z+D); the time-weighted reservation E[R·T]/E[T], which samples reads in flight in proportion to how long each is held; and the latency curve, N/X(N). It holds for any distribution.
Asymptotic bound analysis (the operational laws of Denning and Buzen) gives the two straight lines, N·E[L]/(Z+D) and E[L]/D. They cross at the knee, N* = 1 + Z/D — the bandwidth-delay product counted in reads.
A closed queueing network: one infinite-server delay station (pipeline, switches, endpoint) and one FIFO server (the link). That only the mean latency matters for saturated throughput is not Little’s law but product-form theory: an infinite-server station is insensitive to its service-time distribution (the BCMP theorem).
Mean Value Analysis (Reiser and Lavenberg, 1980) computes the curve X(N) around the knee. The c_s² term is a standard residual-service correction for non-exponential link service; it is what rounds the corner.
Pollaczek–Khinchine (M/G/1) gives the link waiting time below saturation, and an M/G/∞ compound-Poisson occupancy sized with a Gaussian tail sets the buffer for a target backpressure probability — the square-root staffing idea from call-centre OR (Halfin–Whitt). Both live in the Python reference’s open regime, not on this page.
Monte Carlo sampling estimates the per-read moments (sizes, reservations, link times), and discrete-event simulation validates the analytic approximation — the way OR checks any such model.

Queueing and performance modelling

Lazowska, Zahorjan, Graham and Sevcik, Quantitative System Performance (Prentice Hall, 1984) — the classic on exactly this method: operational laws, asymptotic and balanced bounds, MVA for closed networks. Freely available online from the authors.
Mor Harchol-Balter, Performance Modeling and Design of Computer Systems: Queueing Theory in Action (Cambridge, 2013) — the best modern intuitive treatment: Little’s law, closed versus open systems, and why the knee looks the way it does.
Raj Jain, The Art of Computer Systems Performance Analysis (Wiley, 1991) — a practical engineer’s reference: operational laws, MVA, simulation design and validation.
Bolch, Greiner, de Meer and Trivedi, Queueing Networks and Markov Chains, 2nd ed. (Wiley, 2006) — the rigorous reference for MVA with non-exponential service and for product-form theory.
Leonard Kleinrock, Queueing Systems, Vol. 1: Theory and Vol. 2: Computer Applications (Wiley, 1975–76) — the foundation for M/G/1 and Pollaczek–Khinchine.

Interconnect and PCIe

Dally and Towles, Principles and Practices of Interconnection Networks (Morgan Kaufmann, 2004) — credit-based flow control and sizing buffers to the credit round trip: the same problem from the network-on-chip side.
Jackson and Budruk (MindShare), PCI Express Technology 3.0 (2012) — flow-control credits, tags, completion splitting and RCB, and why a root complex advertises infinite completion credits. It stops at Gen 3; for flit mode, 14-bit tags and Gen 6 onward the PCI-SIG Base Specification is the only authoritative source.
Jacob, Ng and Wang, Memory Systems: Cache, DRAM, Disk (Morgan Kaufmann, 2007) — the endpoint-latency side: memory-level parallelism and outstanding-request limits, the same Little’s-law trade-off seen from the memory controller.

To read only one: Harchol-Balter for intuition, then Lazowska as the method reference. Between them they cover every equation here except the PCIe-specific overhead accounting.

Config for pcie_cpl_opt.py

Save as a .json file and run python pcie_cpl_opt.py --config file.json --sim to reproduce this point.

Download pcie_cpl_opt.py — the Python reference model (31 KB, no dependencies beyond numpy/matplotlib). It also covers the below-saturation open regime, which this page does not.

Symbol	What it is	Why it moves
L	Payload of one MRd, in bytes — what the requester actually asked for	Your burst-size distribution, capped by MRRS
R	Completion buffer bytes one read reserves. Always ≥ L	RCB fragmentation, address misalignment and rounding up to the allocation granularity
H	CPLH entries one read reserves — one per completion fragment	How many RCB boundaries the request spans
S, D	Completion wire time at the bottleneck link; D = E[S]	Link speed and width, MPS, ECRC, and the endpoint link if it is slower
Z	Non-link hold time: everything a tag is held for that is not completion wire time	Root-port pipeline, link flight, endpoint latency, switch hops
T	Total time one read occupies a tag, Z + S	Both of the above
c_s²	Squared coefficient of variation of completion service time	Spread in completion sizes. Zero for fixed bursts; large for a wide uniform mix
N*	The knee, 1 + Z/D — the bandwidth-delay product expressed in reads	Rises with latency, falls with link speed
X(N)	Throughput with N reads in flight, from approximate MVA	Saturates at 1/D
η, N_η	Your target share of link bandwidth, and the reads in flight that reach it	η near 1 climbs the flat part of the curve, so N_η rises steeply

Control	What it means, and what it changes
Generation, Width	Link speed and lane count. Together they set the raw byte rate, and with it D. Encoding changes with generation — 8b/10b through Gen 2, 128b/130b for Gen 3–5, flit mode from Gen 6 — which is why efficiency jumps at Gen 6 rather than scaling smoothly.
Max payload size	Largest data payload in one TLP. Bigger means fewer headers per byte carried, so completion wire efficiency rises. This is the single biggest lever on write bandwidth.
Max read request size	Largest amount one MRd may ask for. Bigger means fewer requests for the same bandwidth, so fewer tags in flight — often the difference between needing 10-bit tags and not.
Read completion boundary	The address boundary a completer is allowed to split completions on. A smaller RCB means more fragments per read, which costs CPLH entries and adds rounding waste in CPLD.
Tag budget	How many reads may be outstanding at once. Left at no limit the tool answers “how many tags does this target need”; set it to 8, 10 or 14 bits and it answers the other question — what you actually get with the tags you have. 10-bit tags need the requester enabled in Device Control 2 and a completer that supports them; 14-bit needs flit mode at both ends. When the budget binds, extra buffer buys nothing, because there are no tags left to fill it.
ECRC on completions	Adds a 4-byte digest to every TLP. Pure wire-efficiency cost; it buys end-to-end data protection the LCRC cannot give you across a switch.
Flit TLP efficiency	Gen 6 and later only: the fraction of each flit that carries TLP payload rather than flit overhead and FEC.
Direction, Read share	Reads need completion buffer; posted writes need none. In a mix the two use opposite directions of a full-duplex link, and the read share decides which direction runs out first. In a mix the buffer is sized for the read rate the mix actually allows, which at a low read share is far less than the read-only optimum.
Burst size and its sliders	The AXI request size distribution. Its spread matters as well as its mean: variance in completion size feeds c_s² and costs a little extra buffer.
Start address granularity	How aligned request start addresses are. Coarse alignment means fewer partial RCB fragments and less reservation waste; byte-aligned traffic is the worst case and the tool shows it.
Split MRds at MRRS-aligned addresses	Whether the requester splits a long burst on MRRS boundaries, or simply chops every MRRS bytes from wherever it started. The aligned form produces fewer fragments.
DMA buffers may straddle 4 KiB	An AXI burst may not cross a 4 KiB boundary, so a DMA transfer whose buffer is even slightly off 4 KiB becomes two bursts — and each pays its own MRds, tags, completion headers and RCB rounding. Off, every transfer is placed inside one page. On, the start offset is drawn over the whole page. The straddle costs tags and headers far more than buffer: completion buffer holds bytes in flight, and splitting a transfer adds requests, not bytes.
Architectural max (readout)	The traditional sizing: every tag the requester can issue, each owning a full MRRS of buffer — 256, 768 or 15,360 tags, whichever width the point needs, or the tag budget if one is set. It is an upper bound, not a requirement: the link can only have bandwidth × round-trip time in flight, which is the CPLD figure above, and it is typically several times smaller.
Fixed MRRS-sized slot per tag	How the root port carves up its completion buffer. Off, each MRd reserves the bytes its completions can actually occupy, rounded to the allocation granularity. On, the buffer is indexed by tag and every outstanding MRd owns a whole MRRS-sized slot, whatever it asked for — simpler hardware, but a short MRd wastes most of its slot, and a DMA transfer split at 4 KiB takes two. With 4 KiB transfers at MRRS 4096, turning on the straddle nearly doubles the buffer in this mode, while in byte mode it barely moves.
Return each transfer in order	AXI returns a burst’s read data in order, so an MRd whose completions arrive early must wait in the buffer until every earlier MRd of the same transfer has drained. Each MRd then holds its space for the slowest of its predecessors’ round trips, not its own — with a random endpoint latency that is markedly longer, so the buffer grows by roughly a quarter at the defaults. Tags are still freed when each MRd’s own completions land. This is where a 4 KiB straddle costs buffer even in byte mode: one more MRd in the ordered group. Ordering is modelled per transfer; a DMA engine that uses one AXI ID for everything orders across transfers too, which costs more again.
Endpoint latency	How long the completer takes to answer. Only the mean affects saturated throughput — the delay stage is insensitive to the distribution — but the 99th percentile drives the simulation check and the tail you would see in latency mode.
Root-port pipeline	Fixed contributions to Z: AXI-to-TLP, transmit and receive pipelines, one-way link flight, and drain to AXI R. Every nanosecond here is buffer, because it is time a tag is held with no data moving.
AXI R stall to absorb	Extra buffer bought deliberately, to ride out a downstream stall on the AXI read-data channel without back-pressuring the link.
Switch hops and per-hop settings	Each hop adds latency in both directions, so it adds to Z and therefore to the buffer. Store-and-forward switches add one completion TLP’s serialization per hop on top of the cut-through delay — one TLP, not the whole read, because completions pipeline through the switch.
Endpoint link generation / width	If the far link is slower than the root port's, it becomes the bottleneck and sets D.
Allocation granularity	The slot size the completion buffer allocates in. Every fragment rounds up to it, so coarse slots waste buffer on small fragments.
Stage one completion for LCRC check	Store-and-forward at the root port: hold a whole completion to check its LCRC before releasing it. Costs one MPS plus a header of buffer, flat.
Target share of link bandwidth	How close to the ceiling you are sizing for. The curve is flat past the knee, so 99.5% can cost substantially more buffer than 98% for very little extra bandwidth.
Monte Carlo size	How many synthetic bursts the moment estimates average over. More samples, steadier numbers, slower updates.

PCIe root-port completion buffer sizing

What it is sizing

Where the buffer goes

Reading the result

The symbols in those equations

What each control does

How far to trust it

More free tools

New tools and notes, once a month at most