Skip to main content

What Actually Happens When You Set https_proxy (and Why Some Tools Ignore It)

Updated 6 min readMarkdown

Almost every developer I meet has set an environment variable like https_proxy at some point, and almost none of them can describe precisely what it does. That gap is harmless until the day your Python script happily tunnels through your proxy while your Node script ignores it completely, or your curl works and your git does not. The behavior is logical once you see the mechanics, so this post walks through what actually happens on the wire, with commands you can run to watch it.

The first confusion: https_proxy does not name the proxy protocol

The variable names describe the traffic, not the tunnel. https_proxy means "the proxy to use for HTTPS URLs". http_proxy means "the proxy to use for plain HTTP URLs". Both proxies are usually reachable over plain HTTP on their local hop - a typical setup is:

export http_proxy="http://gw.example.com:8080"
export https_proxy="http://gw.example.com:8080"
export no_proxy="localhost,127.0.0.1,.internal.corp"

Yes, https_proxy points at an http:// URL. The HTTPS protection happens end-to-end between you and the target server; the local hop to the proxy is plain HTTP by default. Tools that honor these variables include curl, wget, git (mostly), pip, and requests. Tools that famously do not include, depending on version and code path, Node's built-in fetch and many GUI apps. That inconsistency is the source of most "it works on my machine" proxy bugs.

Watching a CONNECT tunnel with curl

For plain HTTP through a proxy, the client simply sends the full URL to the proxy and the proxy fetches it. HTTPS is different, because the client refuses to let the proxy see the TLS handshake. So it uses the CONNECT method: it asks the proxy to open a raw TCP tunnel to host:port, then performs the TLS handshake through that tunnel as if the proxy were a wire.

You can watch every phase with:

curl -v -x http://USER:PASS@gw.example.com:8080 https://api.ipify.org

In the output you will see, in order:

  1. CONNECT api.ipify.org:443 HTTP/1.1 sent to the proxy, and the proxy's 200 Connection established - at this point you have a TCP tunnel, but no TLS yet.

  2. The TLS Client hello and the server certificate - this handshake is between you and the target, and a normal proxy cannot read it.

  3. Your real GET / request, encrypted, invisible to the proxy except for timing and byte counts.

Two consequences follow that matter in practice.

The proxy sees the destination host, not the path. CONNECT requests contain api.ipify.org:443 and nothing else. Privacy-wise this is exactly what you want. Debugging-wise it means your proxy logs will never show query strings, so when you are troubleshooting "which endpoint is slow", proxy-side logs alone cannot tell you. Correlate them with client-side timing.

Authentication failure shows up as a CONNECT response, not an HTTP status from the target. If your credentials are wrong, the proxy replies 407 Proxy Authentication Required to the CONNECT, and curl surfaces it as a proxy-level error. I have watched teams lose an afternoon to this because their retry logic only classified target-level status codes (403, 429) and treated the 407-class tunnel failure as "connection refused", retrying at full speed against a proxy that was rejecting every tunnel. Classify tunnel-establishment failures separately from target responses; they have different causes and different backoff curves.

When the tunnel is up, TLS is still yours

Because the handshake is end-to-end, certificate pinning, custom CAs, and SNI-based routing all work through a well-behaved proxy without special configuration. The flip side: if something in your chain does intercept TLS, you will get certificate errors naming the interceptor's CA. That is a feature. If you ever see a valid-looking certificate for a public site issued by your proxy vendor's CA, you now know your traffic is being decrypted in the middle, which some "smart proxy" products do deliberately and which you should know about before sending anything sensitive.

One honest limitation of CONNECT through a gateway: the proxy always learns which hostnames you connect to, when, and how many bytes flow. There is no configuration that hides destination metadata from the tunnel operator. Choose operators you are willing to reveal that metadata to.

The NO_PROXY traps

no_proxy is where the subtle bugs live. Three I have hit personally:

  1. Matching is suffix-based and inconsistent across tools. curl treats .internal.corp as matching any subdomain; some libraries only match exact hostnames. If a job works in curl and fails in your Python worker against the same env, diff the no_proxy semantics first.

  2. IP literals rarely match CIDR ranges. no_proxy=10.0.0.0/8 is ignored by most implementations. If your internal services are on RFC1918 addresses, list the hosts or use a library that supports ranges.

  3. Uppercase vs lowercase. HTTP_PROXY is conventionally ignored in CGI contexts for security reasons, while HTTPS_PROXY is fine. Scripts that export only the uppercase variants can behave differently under different runtimes for genuinely historical reasons.

Making Node honor the environment (the common case)

Node's global fetch (undici) does not read proxy environment variables by default. If your container has https_proxy set and your Node service ignores it, that is not a bug in your shell. The explicit fix is to install a dispatcher:

// npm i undici
import { ProxyAgent, setGlobalDispatcher } from "undici";

const proxyUrl = process.env.https_proxy || process.env.HTTPS_PROXY;
if (proxyUrl) {
  setGlobalDispatcher(new ProxyAgent(proxyUrl));
}

// every fetch() now tunnels through CONNECT like curl does
const res = await fetch("https://api.ipify.org?format=json");
console.log(await res.json());

The equivalent trap exists in Go (honor HTTP_PROXY automatically, but per-transport Proxy functions can silently disable it) and in various HTTP client libraries that expose a proxies config that overrides the environment entirely. The general rule: the environment is a default, not a guarantee, and every language's most popular client has its own opinion about it.

A five-minute acceptance test for any proxy setup

Before pointing production at a new proxy configuration, I run this sequence and check four things:

# 1. Plain HTTP through the proxy: expect your proxy exit IP, not yours
curl -s -x "$HTTP_PROXY" http://api.ipify.org; echo

# 2. HTTPS through the tunnel: expect the same exit IP
curl -s -x "$HTTPS_PROXY" https://api.ipify.org; echo

# 3. Tunnel integrity: certificate issuer should be the real CA, not your proxy
curl -vs -x "$HTTPS_PROXY" https://api.ipify.org -o /dev/null 2>&1 | grep -i "issuer"

# 4. no_proxy works: this should bypass and print your real IP
no_proxy="api.ipify.org" curl -s https://api.ipify.org; echo

If all four behave as the comments predict, your stack understands the environment the way you think it does. If any of them surprises you, better to learn it here than in an incident.

For the tunnel tests above, any standard residential proxy gateway will do - the one I use day to day is at https://www.thordata.com/products/residential-proxies, and the same sequence works against any gateway that speaks plain HTTP proxy protocol with CONNECT support.

Disclosure: I work with Thordata, which is how the gateway above ended up in my test suite; the mechanics described here are standard HTTP proxy protocol and apply to any compliant provider.

No comments yet. Be the first to comment.