Elixir and the BEAM for AI systems
eMudhra Distrust: Why Bumping castore Won't Save Req
Eight eMudhra CA keys were guessable. Browsers blocked them in hours. Req reads CAs from the OS first, so bumping castore usually does nothing. Here's the fix.
In this post
On October 17, 2026, a researcher reported that the private keys behind eight eMudhra intermediate CA certificates could be recovered almost instantly. Mozilla’s Dennis Jackson wrote in the Bugzilla thread that P-384 private keys should be random integers between 1 and the group order, but “the 8 impacted keys all used integers less than 256.” Chrome and Mozilla blocked the intermediates within hours. On October 20, Mozilla went further and removed trust in the root they chain to, emSign ECC Root CA - G3.
Most of the coverage is about browsers. I run Elixir services, so my question was what this does to a BEAM app making outbound HTTPS calls. The obvious Elixir fix is to bump castore. I tested that, and for an app on Req, Finch or Mint with default options and an OS bundle, it changes nothing.
What the browsers did that your app didn’t get
Reported at 18:05 UTC on October 17. Per the Bugzilla thread, Mozilla’s OneCRL block was complete at 22:59 UTC the same day, and eMudhra marked the eight certificates revoked on October 18 at 09:38 UTC. Mozilla’s write-up covers the Firefox side, and Google’s post says that because the CRLSet update blocks by public key hash, “Chrome users were immediately protected”.
Browsers have a central switch: they pull a blocklist, and every user
has it within hours. Your Elixir release has no such switch. Erlang’s
:ssl doesn’t check revocation by default, and Mint, which
sits under Finch and Req, has a literal # TODO: Add revocation check
in its default SSL options. So eMudhra’s October 18 revocations, and
Sectigo’s October 19 revocation of the cross-signs (comment
13), protect none of your outbound HTTPS calls. The only protection
is which root certificates are in the bundle your app trusts.
This is why Mozilla removed the root. In the same Bugzilla thread (comment 18) they say the removal “will protect clients which do not use revocation information or lack a mechanism to withdraw trust in specific intermediates.” That sentence describes most server-side HTTP clients.
Which bundle does your app actually use?
The castore maintainers saw the same problem. PR #90 says “Erlang :ssl doesn’t check revocation by default, so the bundle is the only protection castore users get,” and notes that through the cross-sign “they chain to emSign ECC Root CA - G3.” castore 1.0.22 was published on Hex on October 20 at 16:24 UTC with the root removed.
So you bump castore. For a Req app that does nothing, because Req doesn’t use castore unless Mint can’t find anything else. This is the relevant part of Mint 1.11.0 (the version my repo locks, abridged to fit):
defp add_cacerts(opts) do
if Keyword.has_key?(opts, :cacertfile) or
Keyword.has_key?(opts, :cacerts) do
opts
else
try do
Keyword.put(opts, :cacerts,
:public_key.cacerts_get())
rescue
_ ->
raise_on_missing_castore!()
Keyword.put(opts, :cacertfile,
CAStore.file_path())
end
end
endIf you pass no CA options, Mint asks OTP for the operating system’s
trusted certificates with :public_key.cacerts_get(). It
touches castore only when that call raises. On a normal Debian or Alpine
image with ca-certificates installed, it never raises. The
castore package can sit in your mix.lock at any version and
not matter. (The Mint source is in
deps/mint/lib/mint/core/transport/ssl.ex if you want to
check yours.)
What I tested
On October 21 I ran plain :ssl.connect with
verify: :verify_peer against
dv-g3b-emudhra.we-have-a-problem.com, a test host set up by
the researcher that chains to the affected G3 path. This was on Arch
with OTP 29 and ca-certificates-mozilla 3.131-1, which is
older than the NSS release that dropped the root.
| Trust source | Result for the eMudhra test host |
|---|---|
| castore 1.0.21 | accepted |
| castore 1.0.22 | rejected (unknown_ca) |
| certifi 2.17.0 | accepted |
OS store via cacerts_get() |
accepted |
example.com was accepted by all four, so the setup
itself works. A second test host the researcher runs,
dv-g3b-aaa-emudhra, was rejected by all four here because
Arch’s bundle has no AAA root, so I left it out of the table. Debian’s
pre-fix bundle carried one (the DSA removes it), so an unpatched Debian
image may accept that chain too. I haven’t tested that. Only castore
1.0.22 refuses the chain, and the OS store, which is what Req actually
reads, accepts it until the OS package updates.
The OS fix exists. Debian published DSA-6572-1
on October 20. It updates ca-certificates to
20250419+deb13u1 on trixie to “remove the ‘emSign ECC Root
CA - G3’ and ‘Comodo AAA Services root’ certificates.” I built a fresh
debian:trixie-20260803-slim image with
apt-get install ca-certificates to check, and the bundle it
got has no emSign ECC Root CA - G3. An August base image still gets the
fix if apt-get runs again. The emSign C3, C1 and G1 roots
are still in Debian’s bundle.
The hackney exception
HTTPoison and anything else on hackney is in a worse spot. hackney 4.8.5 defaults to certifi’s bundle, and certifi 2.17.0 still contains the root. The fix is an open pull request on erlang-certifi, titled “Remove compromised eMudhra roots.” The pull request’s author wrote on October 21: “This library pulls from mkcert.org, which still contains all eMudhra certs.” As of October 21 that pull request is still open, and 2.17.0 (June 5) is the latest certifi on Hex.
If you use hackney, you have to pass a different bundle yourself until certifi ships. The workaround is in the checklist below. PHP’s composer already shipped its version of this fix: composer/ca-bundle 1.6.0 and 1.4.16 went out on October 20, and the changelog lists “Distrust compromised eMudhra intermediates.”
What to do
No OS bundle in your image? Distroless and scratch images often have none. Then Mint is already on castore, and bumping it to 1.0.22 is the fix.
Rebuild the image with no layer cache. The OS fix only reaches your app if the
apt-get install ca-certificates(orapk add) step actually runs again. If your Dockerfile was last built before October 20 and your builder reuses that layer, a plain redeploy ships the old bundle. Use--no-cache, or bump the base image tag so the layer changes. Other distros’ images need the same update from their own packages, since OTP doesn’t carry a store of its own.Check the running release, not the Dockerfile. Open a remote console on a deployed node and run this against the researcher’s test host:
h = ~c"dv-g3b-emudhra.we-have-a-problem.com" :ssl.connect(h, 443, [ verify: :verify_peer, cacerts: :public_key.cacerts_get(), server_name_indication: h ])You want an
{:error, ...}withunknown_cain it. If you get{:ok, socket}, the node still trusts the root. Do this on a node started from the new image.hackney or HTTPoison users: add
castore, runmix deps.update castoreso you’re on 1.0.22, and passssl: [cacertfile: CAStore.file_path()]to HTTPoison. (HTTPoison’s key isssl; plain hackney calls takessl_options.) hackney merges your options over its defaults, soverify_peerand hostname checking stay on. Don’t reach forcacerts_get()here unless you’ve already confirmed the OS store is fixed, and never setverify: :verify_noneto make an error go away.Optional: pin castore for Req and Finch. If you would rather not depend on whatever the base image shipped, add
{:castore, "~> 1.0"}to your deps, then passconnect_options: [transport_opts: [cacertfile: CAStore.file_path()]]to Req. Then your trust list changes when you runmix deps.update, and you can see it in a diff. The cost is that you now own that update.
I’d do 1 and 2 on every service and 4 only where the app talks to payment or AI-provider endpoints and you want the bundle under version control.
Who should care, and how much
The attack needs two things. Someone has to hold the keys, and they have to sit on the network path between your app and the server it’s calling, to present a certificate that chains to the bad intermediate. If your bundle still trusts the G3 root, your client accepts it. Anything outbound that carries secrets fits that picture: payment APIs, LLM provider calls with your key in the header, webhook deliveries.
I have seen no report of misuse. Chrome says “no fraudulent certificates issued by these CAs could have been used against CT-enforcing clients,” and eMudhra said in the Bugzilla thread that no end-entity certificates were issued from these intermediates. That covers browsers, which enforce Certificate Transparency. Most server-side HTTP clients don’t enforce CT, so the public record can’t tell us whether anyone used the keys against them between April 2025, when the keys were generated, and now. The keys are trivial to recover, and Mozilla’s root removal note says “multiple individuals have independently recovered the private keys,” so I’d treat the window as open until your bundle is fixed.
One more date. Per Google’s post, the removal of emSign ECC Root CA -
G3 from the Chrome Root Store is expected to ship on Tuesday, October
27. eMudhra’s initial estimate in the Bugzilla thread (comment
19) is “about 30,000 certs that require migration” off the removed
root, so some servers you call may change chains around now. If an
outbound call starts failing with unknown_ca after your
rebuild, that’s the cause; don’t turn off verification.
Related reading
If you’re building a security review for an AI-native company, how I’d run security at an AI-native company is where I’d start, and my Elixir library list covers where Req fits.
If you run Elixir services for an AI startup and want someone to go through your outbound trust, image build and dependency pins, that’s what a Technical Assessment on my consulting page covers.