Elixir and the BEAM for AI systems

eMudhra Distrust: Why Bumping castore Won't Save Req

Eight eMudhra CA keys were guessable. Browsers blocked them in hours. Req reads CAs from the OS first, so bumping castore usually does nothing. Here's the fix.

In this post
  1. What the browsers did that your app didn’t get
  2. Which bundle does your app actually use?
  3. What I tested
  4. The hackney exception
  5. What to do
  6. Who should care, and how much
  7. Related reading

On October 17, 2026, a researcher reported that the private keys behind eight eMudhra intermediate CA certificates could be recovered almost instantly. Mozilla’s Dennis Jackson wrote in the Bugzilla thread that P-384 private keys should be random integers between 1 and the group order, but “the 8 impacted keys all used integers less than 256.” Chrome and Mozilla blocked the intermediates within hours. On October 20, Mozilla went further and removed trust in the root they chain to, emSign ECC Root CA - G3.

Most of the coverage is about browsers. I run Elixir services, so my question was what this does to a BEAM app making outbound HTTPS calls. The obvious Elixir fix is to bump castore. I tested that, and for an app on Req, Finch or Mint with default options and an OS bundle, it changes nothing.

What the browsers did that your app didn’t get

Reported at 18:05 UTC on October 17. Per the Bugzilla thread, Mozilla’s OneCRL block was complete at 22:59 UTC the same day, and eMudhra marked the eight certificates revoked on October 18 at 09:38 UTC. Mozilla’s write-up covers the Firefox side, and Google’s post says that because the CRLSet update blocks by public key hash, “Chrome users were immediately protected”.

Browsers have a central switch: they pull a blocklist, and every user has it within hours. Your Elixir release has no such switch. Erlang’s :ssl doesn’t check revocation by default, and Mint, which sits under Finch and Req, has a literal # TODO: Add revocation check in its default SSL options. So eMudhra’s October 18 revocations, and Sectigo’s October 19 revocation of the cross-signs (comment 13), protect none of your outbound HTTPS calls. The only protection is which root certificates are in the bundle your app trusts.

This is why Mozilla removed the root. In the same Bugzilla thread (comment 18) they say the removal “will protect clients which do not use revocation information or lack a mechanism to withdraw trust in specific intermediates.” That sentence describes most server-side HTTP clients.

Which bundle does your app actually use?

The castore maintainers saw the same problem. PR #90 says “Erlang :ssl doesn’t check revocation by default, so the bundle is the only protection castore users get,” and notes that through the cross-sign “they chain to emSign ECC Root CA - G3.” castore 1.0.22 was published on Hex on October 20 at 16:24 UTC with the root removed.

So you bump castore. For a Req app that does nothing, because Req doesn’t use castore unless Mint can’t find anything else. This is the relevant part of Mint 1.11.0 (the version my repo locks, abridged to fit):

defp add_cacerts(opts) do
  if Keyword.has_key?(opts, :cacertfile) or
       Keyword.has_key?(opts, :cacerts) do
    opts
  else
    try do
      Keyword.put(opts, :cacerts,
        :public_key.cacerts_get())
    rescue
      _ ->
        raise_on_missing_castore!()
        Keyword.put(opts, :cacertfile,
          CAStore.file_path())
    end
  end
end

If you pass no CA options, Mint asks OTP for the operating system’s trusted certificates with :public_key.cacerts_get(). It touches castore only when that call raises. On a normal Debian or Alpine image with ca-certificates installed, it never raises. The castore package can sit in your mix.lock at any version and not matter. (The Mint source is in deps/mint/lib/mint/core/transport/ssl.ex if you want to check yours.)

What I tested

On October 21 I ran plain :ssl.connect with verify: :verify_peer against dv-g3b-emudhra.we-have-a-problem.com, a test host set up by the researcher that chains to the affected G3 path. This was on Arch with OTP 29 and ca-certificates-mozilla 3.131-1, which is older than the NSS release that dropped the root.

Trust source Result for the eMudhra test host
castore 1.0.21 accepted
castore 1.0.22 rejected (unknown_ca)
certifi 2.17.0 accepted
OS store via cacerts_get() accepted

example.com was accepted by all four, so the setup itself works. A second test host the researcher runs, dv-g3b-aaa-emudhra, was rejected by all four here because Arch’s bundle has no AAA root, so I left it out of the table. Debian’s pre-fix bundle carried one (the DSA removes it), so an unpatched Debian image may accept that chain too. I haven’t tested that. Only castore 1.0.22 refuses the chain, and the OS store, which is what Req actually reads, accepts it until the OS package updates.

The OS fix exists. Debian published DSA-6572-1 on October 20. It updates ca-certificates to 20250419+deb13u1 on trixie to “remove the ‘emSign ECC Root CA - G3’ and ‘Comodo AAA Services root’ certificates.” I built a fresh debian:trixie-20260803-slim image with apt-get install ca-certificates to check, and the bundle it got has no emSign ECC Root CA - G3. An August base image still gets the fix if apt-get runs again. The emSign C3, C1 and G1 roots are still in Debian’s bundle.

The hackney exception

HTTPoison and anything else on hackney is in a worse spot. hackney 4.8.5 defaults to certifi’s bundle, and certifi 2.17.0 still contains the root. The fix is an open pull request on erlang-certifi, titled “Remove compromised eMudhra roots.” The pull request’s author wrote on October 21: “This library pulls from mkcert.org, which still contains all eMudhra certs.” As of October 21 that pull request is still open, and 2.17.0 (June 5) is the latest certifi on Hex.

If you use hackney, you have to pass a different bundle yourself until certifi ships. The workaround is in the checklist below. PHP’s composer already shipped its version of this fix: composer/ca-bundle 1.6.0 and 1.4.16 went out on October 20, and the changelog lists “Distrust compromised eMudhra intermediates.”

What to do

  1. No OS bundle in your image? Distroless and scratch images often have none. Then Mint is already on castore, and bumping it to 1.0.22 is the fix.

  2. Rebuild the image with no layer cache. The OS fix only reaches your app if the apt-get install ca-certificates (or apk add) step actually runs again. If your Dockerfile was last built before October 20 and your builder reuses that layer, a plain redeploy ships the old bundle. Use --no-cache, or bump the base image tag so the layer changes. Other distros’ images need the same update from their own packages, since OTP doesn’t carry a store of its own.

  3. Check the running release, not the Dockerfile. Open a remote console on a deployed node and run this against the researcher’s test host:

    h = ~c"dv-g3b-emudhra.we-have-a-problem.com"
    :ssl.connect(h, 443, [
      verify: :verify_peer,
      cacerts: :public_key.cacerts_get(),
      server_name_indication: h
    ])

    You want an {:error, ...} with unknown_ca in it. If you get {:ok, socket}, the node still trusts the root. Do this on a node started from the new image.

  4. hackney or HTTPoison users: add castore, run mix deps.update castore so you’re on 1.0.22, and pass ssl: [cacertfile: CAStore.file_path()] to HTTPoison. (HTTPoison’s key is ssl; plain hackney calls take ssl_options.) hackney merges your options over its defaults, so verify_peer and hostname checking stay on. Don’t reach for cacerts_get() here unless you’ve already confirmed the OS store is fixed, and never set verify: :verify_none to make an error go away.

  5. Optional: pin castore for Req and Finch. If you would rather not depend on whatever the base image shipped, add {:castore, "~> 1.0"} to your deps, then pass connect_options: [transport_opts: [cacertfile: CAStore.file_path()]] to Req. Then your trust list changes when you run mix deps.update, and you can see it in a diff. The cost is that you now own that update.

I’d do 1 and 2 on every service and 4 only where the app talks to payment or AI-provider endpoints and you want the bundle under version control.

Who should care, and how much

The attack needs two things. Someone has to hold the keys, and they have to sit on the network path between your app and the server it’s calling, to present a certificate that chains to the bad intermediate. If your bundle still trusts the G3 root, your client accepts it. Anything outbound that carries secrets fits that picture: payment APIs, LLM provider calls with your key in the header, webhook deliveries.

I have seen no report of misuse. Chrome says “no fraudulent certificates issued by these CAs could have been used against CT-enforcing clients,” and eMudhra said in the Bugzilla thread that no end-entity certificates were issued from these intermediates. That covers browsers, which enforce Certificate Transparency. Most server-side HTTP clients don’t enforce CT, so the public record can’t tell us whether anyone used the keys against them between April 2025, when the keys were generated, and now. The keys are trivial to recover, and Mozilla’s root removal note says “multiple individuals have independently recovered the private keys,” so I’d treat the window as open until your bundle is fixed.

One more date. Per Google’s post, the removal of emSign ECC Root CA - G3 from the Chrome Root Store is expected to ship on Tuesday, October 27. eMudhra’s initial estimate in the Bugzilla thread (comment 19) is “about 30,000 certs that require migration” off the removed root, so some servers you call may change chains around now. If an outbound call starts failing with unknown_ca after your rebuild, that’s the cause; don’t turn off verification.

If you’re building a security review for an AI-native company, how I’d run security at an AI-native company is where I’d start, and my Elixir library list covers where Req fits.

If you run Elixir services for an AI startup and want someone to go through your outbound trust, image build and dependency pins, that’s what a Technical Assessment on my consulting page covers.