Move data between machines with curl: pipes, not file servers
You have a file on a box in one place and you want it on a box somewhere else. The reflex is to reach for scp, spin up an S3 bucket, or paste it into a file host. For a one-shot transfer, all three are more setup than the job needs. Often what you actually want is a pipe: one side reads, one side writes, the bytes flow straight through, and nothing is left behind.
That is what a streaming HTTP pipe gives you — the old netcat trick, but over plain HTTP so it survives NAT, firewalls, and proxies. We built one at exl.ink/pipe. This post is about why that shape works, where it beats a file server, and where it does not.
netcat, then HTTP
The classic move is netcat: one machine listens on a port, the other connects, and stdin on one end comes out of stdout on the other. It is a raw TCP wire between two processes, and it is perfect — until the two machines are not on the same network. A listening port behind home NAT or a cloud firewall is unreachable, and now you are forwarding ports or running a VPN to move a tarball once.
HTTP fixes the reachability problem. Both ends make outbound connections to a public rendezvous point, so neither needs an open inbound port. The relay just hands the writer's bytes to the reader. You keep the netcat ergonomics — pipe in, pipe out — and lose the network plumbing.
How the bytes actually move: chunked transfer
To stream over HTTP you have to send a body whose length you do not know yet. HTTP/1.1's answer is chunked transfer encoding. Instead of a Content-Length up front, the sender sets Transfer-Encoding: chunked and emits a sequence of length-prefixed pieces: a hex byte count, a CRLF, that many bytes, a CRLF, repeat — ending with a zero-length chunk. The recipient reads pieces as they arrive and knows it is done when it sees the terminating zero. The exact framing is specified in RFC 9112 §7.1.
This is the difference between streaming and buffering. A buffered transfer has to know the size first, which usually means writing the whole payload somewhere before a single byte leaves. Chunked lets you start sending immediately and forward a stream of unknown length — a database dump in progress, the live output of a command, a 4 GB image you would rather not stage to disk first.
One caveat worth knowing: HTTP/2 disallows the Transfer-Encoding header entirely, and using it can trigger a protocol error. HTTP/2 has its own framing and streams natively, so chunked is an HTTP/1.1 mechanism — the streaming idea carries over, the specific header does not.
curl is both ends
You do not need a special client. curl uploads a stream with -T (upload-file), and a single dash means "read from stdin" — so anything you can pipe, you can send. When the source is stdin, curl cannot know the length in advance, so it uploads with Transfer-Encoding: chunked automatically. The curl uploads guide and the manpage cover the flags. Receiving is just a plain GET piped wherever you want it.
# receiver: waits for the sender, writes to disk
curl https://exl.ink/p/secret7 > out.tar.gz
# sender: pipe anything into the same path
cat in.tar.gz | curl -T - https://exl.ink/p/secret7Because both ends stream, there is no staging step. pg_dump mydb | curl -T - .../p/db on one host and curl .../p/db | psql otherdb on the other moves a database between machines without a dump file ever touching either disk. (For one-off form posts of small bodies you would use --data-binary @file, which reads the whole thing and sets a Content-Length; for streaming a body of unknown size, -T - is the right tool.)
Backpressure and the 1:1 rendezvous
A naive relay would accept bytes from the writer as fast as it can and pile them up while the reader catches up. That turns the relay into a buffer, and a buffer is the thing you were trying to avoid — it costs memory or disk and undoes the "nothing is stored" property. The honest design honors backpressure: when the reader is slow, the relay stops reading from the writer, and TCP flow control pauses the sender. Bytes move at the speed of the slower end and almost nothing accumulates in the middle.
The other half is the rendezvous. A pipe is strictly 1:1 — exactly one reader and one writer share a channel, matched by a path you choose. Whoever arrives first waits a short window for the other; a second reader or writer on the same path is rejected, not fanned out. That keeps the semantics simple — it is a wire, not a pub/sub topic — and means a transfer either has both ends or it does not exist.
Why the path is the only secret
There is no account and no access list. The path you pick is the credential — a capability URL. Anyone who knows it can be the other end; anyone who does not, cannot find it. The W3C TAG's guidance on these is the rule to follow: the identifier must be unguessable (a long random string, not a counter), it should travel over HTTPS so it is not exposed in transit, and it should be short-lived.
A pipe satisfies the last point by construction. The channel exists only while both ends are connected; there is no object sitting at the URL afterward to leak. The discipline that remains is on you: pick a random path, not /p/test, and treat the URL like the password it is.
A pipe with no listener is just a string. The secret is real only for the seconds both ends are live.
When a pipe is the wrong tool
A pipe needs both ends present at the same time. If the file is finished and the recipient is asleep, you do not want a live rendezvous — you want a parked object they can fetch later. That is a one-time download link: upload once, get a URL, hand it over, it serves the file (once, or until it expires) and then it is gone. You can make one at exl.ink, and shorten the link with /short if it has to fit in a chat message.
Be clear-eyed about the limits, too. Our pipe is ephemeral and best-effort: there are size and duration caps, per-IP limits, and a rendezvous window that times out if the other side never shows. A dropped connection mid-stream is a failed transfer with no resume — for a multi-gigabyte critical sync over a flaky link, a tool that checkpoints and resumes is the better call. The pipe is for the common case: get these bytes from here to there, now, with nothing to clean up after.