The bug was DNS: fixing fx on Termux

I wanted to run fx, Vercel Labs’ Zig coding agent, inside Termux on my phone.

The binary started. The interface rendered. Then anything involving the network failed.

$ fx login vercel
fx login: failed to sign in

An API key did not help. Login failed, key validation failed, and the model and credit requests failed. The error looked like an authentication problem, so that was where we started looking.

It was not an authentication problem.

It was DNS.

The device

All of this happened on a Pixel 10 with a Tensor G5 and 12 GB of RAM, running Android 17 and Termux. Zig saw seven available cores.

That matters because we debugged and compiled fx on the phone itself. There was no quick edit-build-test loop hiding behind a desktop machine.

A cold aarch64-linux-musl build took about three minutes. The successful builds I recorded took 2 minutes 46 seconds and 3 minutes 17 seconds; one larger recompilation took 5 minutes 3 seconds. Once the cache was hot, an unchanged build dropped to 13.6 seconds.

Three minutes is not terrible for a project this size on a phone. It is long enough to stop speculative debugging. Every build should answer a question.

First, make the failure less vague

fx login: failed to sign in did not say which part had failed. The login flow has several candidates: opening the browser, OAuth discovery, requesting the device code, polling for the token, and writing credentials to disk.

We checked the filesystem path first. Termux runs under Android’s application sandbox, so permissions and atomic file operations are plausible suspects. Credential persistence in fx uses a temporary file followed by rename, and rename worked. That ruled out the final write.

Opening a browser could also fail because Termux uses termux-open-url rather than the usual Linux xdg-open. But that would only make login less convenient. It would not explain API-key validation, model listing, and credit requests all failing too.

The shared failure had to be earlier and lower in the stack.

We wrote the smallest possible Zig HTTP program and requested Vercel’s OpenID configuration endpoint. The Termux Zig package returned HTTP 200:

STATUS: 200, BODY LEN: 2083

That was confusing. Zig networking worked when the program was compiled and run inside Termux, but the released fx binary could not reach the same host.

The difference turned out to be the toolchain. Termux ships a patched Zig package. fx releases are built with upstream Zig.

strace found it

At this point, reading more auth code was not going to help. We traced the released binary instead:

openat(..., "/etc/resolv.conf", ...) = -1 ENOENT
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_IP) = 3
sendmsg(... sin_port=htons(53), sin_addr=inet_addr("127.0.0.1") ...)

That was the whole bug in three lines.

On ordinary Linux, DNS configuration lives at /etc/resolv.conf. In Termux, /etc points into Android’s read-only /system/etc, and that file does not exist:

$ cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory

$ cat $PREFIX/etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4

Upstream Zig’s resolver tried /etc/resolv.conf. When the file was missing, it fell back to 127.0.0.1:53. Android was not running a DNS resolver there, so every hostname lookup waited for an answer that would never arrive.

OAuth discovery was fine. The token exchange was fine. The API key was fine. fx could not find vercel.com.

The reason our tiny Zig program worked was that Termux already patches its Zig standard library with a fallback to $PREFIX/etc/resolv.conf. The released fx binary had been compiled elsewhere with an unpatched toolchain, so it carried upstream Zig’s behavior with it.

The one-line fix we did not choose

The patch in Termux’s Zig package is essentially this:

Io.Dir.openFileAbsolute(io, "/etc/resolv.conf", .{})
    catch Io.Dir.openFileAbsolute(io, "/data/data/com.termux/files/usr/etc/resolv.conf", .{})

It is the smallest correct change. If fx releases were built with a patched Zig toolchain, that would be enough.

But it would make the release pipeline responsible for modifying its compiler’s standard library. The patch could silently stop applying when Zig reorganized the resolver, and a binary built by a contributor with upstream Zig would still be broken.

We wanted the fix to travel with fx.

The fix in fx

Zig’s new std.Io is a table of operations. fx already wraps that table on macOS to customize process spawning. We followed the same pattern for Linux and replaced only netLookup when both of these conditions are true:

1. /etc/resolv.conf does not exist.
2. Termux’s /data/data/com.termux/files/usr/etc/resolv.conf does exist.

pub fn wrap(original: Io) Io {
    if (builtin.os.tag != .linux) return original;
    if (!termuxDnsNeeded(original)) return original;

    wrapped_vtable = original.vtable.*;
    wrapped_vtable.netLookup = termuxNetLookup;
    return .{ .userdata = original.userdata, .vtable = &wrapped_vtable };
}

Ordinary Linux keeps the original resolver. Other operating systems never enter this path. On Termux, the replacement reads the actual nameservers and sends A and AAAA queries to them. IP literals and localhost keep their normal behavior.

This is more code than the one-line toolchain patch: a 399-line resolver plus a small hook in io.zig. Most of those lines are the unglamorous parts of DNS: building queries, matching response IDs, parsing compressed names, handling A, AAAA and CNAME records, retries, timeouts, and bounded buffers.

I would normally reject 399 lines in favor of one. Here the one line lived in the wrong repository. Shipping the behavior in fx means its release binary works independently of the compiler used to build it.

Building on Android was a second bug hunt

Before we could test the DNS fix, Zig itself refused to build fx in this Termux environment:

error: failed to link temporary file into '.zig-cache/.../options.zig': AccessDenied

Zig’s build cache publishes temporary files with hard links and O_TMPFILE. Android’s SELinux policy denied that operation inside the app data directory. A direct test confirmed it:

$ ln .zig-cache/linktest/a .zig-cache/linktest/b
ln: failed to create hard link: Permission denied

A normal rename worked. We temporarily patched the locally installed Zig’s cache publication code to use the available path, then built fx as aarch64-linux-musl. Those toolchain edits stayed on the phone; they are not part of the fx change or the pull request.

This detour was useful because it kept two unrelated failures separate. The hard-link problem prevented us from compiling. The DNS problem prevented the compiled release from networking. Combining them into one “Termux support” patch would have made both harder to review.

Proving the fix

The focused DNS tests covered A and AAAA query construction:

1/2 termux_dns.test.DNS query builder emits a well-formed A question...OK
2/2 termux_dns.test.DNS query builder tolerates a trailing dot and AAAA records...OK
All 2 tests passed.

Then we tested the real binary rather than stopping at unit tests.

fx models returned 243 models. fx login vercel reached the device endpoint and printed the login URL and code. fx credits reached the gateway and returned a real HTTP 401 for the deliberately invalid saved credential. A 401 was success at this layer: DNS, TCP and TLS had all completed, and the server had answered.

Finally, another trace showed where the DNS packets went:

sin_addr=inet_addr("8.8.4.4")
sin_addr=inet_addr("8.8.8.8")

Before the patch: 127.0.0.1. After the patch: the nameservers configured by Termux.

That is the kind of before-and-after evidence I trust.

The pull request

We searched the fx repository for existing Termux, Android, DNS and resolv.conf reports. There were none, so we opened PR #786: Resolve hostnames on Termux.

The PR contains two files and 408 added lines. Its description also records the one-line CI/toolchain alternative and why we did not choose it. I first put that explanation in a source comment, which was the wrong place; it explains a review decision, not the code’s behavior. We moved it into the PR body.

From the initial report to a working fix took about 90 minutes. The draft PR was open a little over two hours after we started. A meaningful part of that time was spent waiting for the phone to compile.

The first full CI run later failed in two pre-existing, timing-sensitive MCP concurrency tests: a different test on each Linux architecture, while every E2E shard and both macOS native jobs passed. The Termux wrapper is inert on those Linux runners because they have /etc/resolv.conf. We reran the failed jobs and left the upstream workflows for a maintainer to approve, as GitHub requires for a fork PR.

The PR is still open as I write this.

What I would do next time

Start below the feature sooner.

When login, API-key validation, model listing and credit lookup all fail, the common layer is not OAuth. It is the network. A single strace showed more than reading the entire authentication flow did.

The debugging path was:

1. Reproduce the vague failure.
2. List the stages that could produce it.
3. Eliminate browser handling and credential persistence.
4. Prove networking works in a tiny locally compiled program.
5. Trace the released binary.
6. Find the first meaningful divergence: missing resolv.conf, then DNS to localhost.
7. Change one I/O operation at the shared boundary.
8. Verify both the result and the packets on the wire.

The error said authentication. The syscall trace said DNS. The syscall trace was right.