# A docker pull That Never Finished, and the Swap Behind It
Published 2026-09-17.
Canonical URL https://log.delaa.dev/blog/2026-09-17-docker-pull-never-finished-swap
> A docker pull stuck indefinitely on 'Pulling fs layer' with no error, no timeout, and one cryptic containerd log line. Here is the full elimination chain that found the real cause, going from DNS to registry reachability to an MTU theory and finally to memory. The answer was swap exhaustion during layer extraction, not a transfer problem.
---
> **Key Takeaways**
> - A `docker pull` can hang completely silently, with no error, no timeout, and no percentage progress, while the only clue in the logs is a cryptic containerd message about a missing extraction snapshot.
> - Systematic elimination, going from DNS to registry and CDN reachability to MTU/PMTU and finally to memory, is more reliable than jumping to the most plausible-sounding hypothesis first.
> - The MTU/PMTU black-hole theory was a legitimate, well-reasoned hypothesis, tested properly and discarded on clean negative evidence, including a regression it caused on unrelated traffic.
> - The eventual root cause was memory, not network. A specific 844MB layer failed during *extraction*, not download, on an instance already low on swap headroom. Adding swap fixed it immediately.
> - The layer-by-layer isolation was what finally pointed at size-dependent resource exhaustion instead of a categorical network fault. Small images pulled fine, medium images pulled fine, and only this one large layer hung.
A hung `docker pull` is one of the more frustrating failure modes in container operations, because it fails by doing nothing. No stack trace appears. No non-zero exit code comes back. No timeout fires. There is just a progress line that says "Pulling fs layer" and then never updates again. This is a full account of the elimination process that turned that non-event into a specific, fixable cause.
## The Symptom
The command was an ordinary `docker pull` for an image whose largest layer was 844MB. It reached "Pulling fs layer" and stopped. No error. No timeout after any reasonable wait. `docker logs` and `journalctl -u docker` had nothing useful. The only lead, buried in containerd's own logs, was a single cryptic line.
```
failed to cleanup "extract-...": snapshot does not exist
```
That message describes a symptom of the failure, not a cause. An interrupted extraction left a cleanup step with nothing to clean up. It confirmed that *something* was going wrong around image extraction, but not what. Everything else useful had to come from active elimination, not from the logs.
## Ruling Out DNS
The first, cheapest thing to check on any silent network-shaped hang is name resolution. If the registry hostname were not resolving, or resolving to something unexpected, that alone would explain a hang with no useful error, since many tools handle DNS failure by retrying silently rather than failing loudly.
```bash
dig registry-1.docker.io
```
Resolution came back clean and consistent across repeated queries. DNS was not the cause, and this eliminated an entire category of hypothesis in about ten seconds. Misconfigured resolver, intermittent DNS server issues, split-horizon weirdness. All of it gone.
## Ruling Out the Registry and CDN
Next was whether the registry or its backing CDN was actually reachable and serving data. Docker's pull process wraps authentication, manifest resolution, and blob download behind one opaque progress indicator, so a raw, direct request against the same blob URL is a cleaner test than trusting the wrapped one.
```bash
curl -v -o /dev/null
```
This was run against blob URLs at several different sizes, including the specific large layer that was hanging in the actual pull. Every one of them downloaded successfully and quickly through `curl`. That is a meaningfully strong result. It rules out registry-side throttling, CDN edge issues, TLS negotiation problems, and general path reachability all at once, because a raw HTTP client using the same network path succeeded where Docker's own pull was stalling. Whatever was wrong, it was not "the registry is unreachable" or "the CDN is slow." The bytes were available and transferred fine outside of Docker's pull machinery.
## Testing the MTU Theory
With DNS and raw reachability both clean, the next hypothesis was a path MTU discovery black hole. The mechanism is a router somewhere along the path silently dropping the ICMP messages that would normally tell a sender to fragment large packets, so any transfer requiring a packet larger than the constrained link's MTU vanishes without a trace instead of failing visibly. This fits the symptom profile well. It would explain small requests succeeding, since small packets need no fragmentation, while a specific large transfer inside Docker's pull stalls indefinitely with no error surfaced to the application layer.
It is a real, well-documented failure mode, and it deserved an actual test rather than being waved off. The test was to lower the client-side MTU and retry the stuck pull, on the reasoning that if the sender never emits a packet large enough to need fragmentation, a black hole further down the path cannot silently eat it.
The result was a clean negative. Lowering the MTU did not unstick the pull. It also caused a brief, separate regression. A transfer that had been working fine before the MTU change started failing after it. That was itself useful evidence, because it showed the MTU adjustment was affecting real traffic behavior, just not in the direction that would explain the original hang. A hypothesis that is actually correct should produce a positive result when you act on it. This one produced no fix and a new problem, which is about as unambiguous a "wrong theory" signal as you can get from a live-system test.
MTU/PMTU issues are a common and well-known cause of exactly this symptom shape in other contexts. VPNs, certain cloud network overlays, misconfigured jumbo frames. Recognizing that a hypothesis fits the pattern is good instinct. Treating a good pattern-fit as confirmation without testing it is the mistake, and testing it here took maybe ten minutes and definitively closed the door on an entire plausible narrative.
## Isolating by Image Size
With network-layer causes exhausted, the next move was re-examining what actually differed between pulls that worked and the one that did not. A small image pulled without any issue. A medium-sized image also pulled without any issue. Only this specific 844MB layer hung. This was not "large images in general." It was this particular large layer, reliably, every time it was retried.
That is a different shape of evidence than anything gathered so far. It is not "the network sometimes fails." Network-layer problems had already been tested and cleared at multiple sizes via the raw `curl` checks above, which included this exact layer's blob URL and succeeded. It means this exact transfer fails at this exact size, consistently, inside Docker's pull-and-extract pipeline. Combined with the containerd log line pointing at extraction rather than download, the evidence had shifted from "something about moving bytes over the network" toward "something about what happens to those bytes once Docker starts unpacking them locally."
## Finding the Cause in Memory
Checking memory state on the instance confirmed it directly. The box was already under swap pressure, with only about 14MB free out of a 512MB swap allocation. Extracting a compressed image layer is a memory-intensive operation, because decompression buffers, the filesystem writes for the unpacked content, and containerd's own snapshot bookkeeping all draw on RAM concurrently. An 844MB layer's extraction was right at the edge of what the instance's combined RAM and swap could sustain. Smaller layers stayed under that threshold and extracted fine. This one did not, and the process extracting it appears to have been starved mid-operation, which lines up exactly with the "snapshot does not exist" cleanup message. The extraction never completed, so the cleanup step had nothing left to find.
The fix was to add 4GB of additional swap to the instance. Retried immediately after, the same pull that had hung indefinitely completed normally, at normal speed, with no further intervention.
## Why This Particular Elimination Order Worked
Looking back at the full chain, the value was not any single step. It was refusing to stop at a plausible-sounding answer before testing it, twice. The raw `curl` check could have been skipped in favor of assuming a slow network, and the MTU theory could have been treated as sufficient explanation without the retry-and-observe step that actually disproved it. Both of those shortcuts would have led to the wrong fix, either waiting longer or permanently lowering the MTU on the box, and left the actual cause, a swap-starved extraction, undiagnosed and free to recur on the next large layer.
The general method here is not specific to Docker.
1. **Start with the cheapest, most falsifiable checks** and actually run them rather than assuming their outcome. DNS and raw connectivity are the cheap ones.
2. **When a wrapped, opaque process hangs, isolate a minimal direct reproduction of one of its stages.** Here that meant a raw `curl` against the same URL Docker was using internally, which separates "is the data reachable" from "is the tool handling it correctly."
3. **Test plausible hypotheses to a clear pass/fail result, and accept the result even when the hypothesis was a good one.** MTU/PMTU was a reasonable theory that happened to be wrong. The ten minutes spent testing it were what made the "wrong" conclusion trustworthy enough to move past.
4. **When symptom conditions differ, that difference is data.** Works at size A and B, fails at size C. That reframed the entire investigation from "network problem" to "resource-threshold problem" once taken seriously.
5. **Match the resource being checked to where the log evidence points.** The containerd cleanup message specifically named extraction, not transfer, which is what made memory the right resource to check once the earlier layers of the stack were cleared.
None of the individual checks here were exotic. `dig`, `curl -v`, an MTU change, `free -m`. What made the debugging session work was doing them in an order that eliminated broad categories before narrow ones, and being willing to fully discard a good hypothesis the moment it failed its own test.