Operations | Monitoring | ITSM | DevOps | Cloud

Last week, dozens of schools found out their certificate authority was being distrusted and they needed to replace all of their certificates in a few days. We wanted to know how we’d handle it if our own CA went away, and how long it would take. So we tried it with one of our other products. It has 6 certificates from Let’s Encrypt, deploying to 18 targets across Windows IIS, Linux nginx, Elasticsearch, and a mail server. Then we wargamed it: “Let’s Encrypt is being distrusted.

Starting Tuesday, October 27, a lot of university websites will open in Chrome to this. Chrome is distrusting a root certificate, emSign ECC Root CA - G3, after eMudhra generated CA keys as small as 4. Every InCommon certificate issued from eMudhra’s ECC chain since April chains to that root.

On Tuesday, October 27, Chrome will distrust an eMudhra root certificate, emSign ECC Root CA - G3. Mozilla already did. Most of the people who have to deal with it run IT at American universities. They get their certificates from InCommon, the certificate service Internet2 runs for higher education. The certificates say Internet2 on them, but since April, eMudhra, an Indian certificate authority, has issued them.

There are lots of ways to get a certificate. Getting it onto your Cisco ISE, IBM i, or WatchGuard VPN is the hard part. We keep making that easier. Since July, CertKit has added 51 new deployment templates, more than doubling the library to 86. Each platform has a built-in deployment template, and most have a deployment guide for the setup details. Setup is just picking a certificate and setting a few variables.

Your site loads in Chrome fine. Then a deploy script, a webhook, or integration calls the same URL and gets this: Python says certificate verify failed: unable to get local issuer certificate. Node says unable to get local issuer certificate, or unable to verify the first certificate. It’s all the same problem. It’s a misconfiguration between your certificate and a trusted root. Either the server isn’t sending it, or the client doesn’t have it.

Renewing a certificate is the easy part. Getting it onto the server is where automation breaks. Somebody rotates the API token on the firewall. A service account password expires. A VM gets rebuilt and the agent never comes back. The certificate renews right on schedule, the deployment fails, and the server keeps serving the old certificate until it expires. Renewed and deployed are not the same thing.

In October 2021 the NSA published an advisory titled “Avoid Dangers of Wildcard TLS Certificates.” It reiterated what security consultants had been saying for years: don’t use wildcard certificates. Single-name certificates per host are safer. Five years later, we counted the certificates that are actually used. One in three domains has a wildcard on it. For.bank domains, where you need a security review just to register, it’s four in five. Operators aren’t careless.

Most customers start CertKit with SSL host monitoring. On day one, you point CertKit at the systems you already run and get every certificate and when it expires. You start with confidence that you have everything under control. Then, you automate, letting CertKit take over the certificates as they need to be renewed. Each renewal is the last time you ever have to worry about that certificate.

In February, Let’s Encrypt announced that DNS-PERSIST-01 was coming, “some time in Q2 2026.” It’s October, and it’s nowhere close to ready. We’ve been waiting for DNS-PERSIST-01 since January, along with every other ACME client and lots of organizations. DNS-PERSIST-01 promised to simplify domain validation and make automation easier, but we had to wait on Let’s Encrypt. Let’s Encrypt is waiting on a redesign, a standards body, and maybe one more vote.