21:41 <mike_sw294503> Hi. It's entirely possible I've done something wacky, but my Sandstorm server just stopped working because my sandcats certificate renewal failed.

21:44 <mike_sw294503> I think this is the important bit of the error: Failed to renew certificate (will try again in 6 hours): Error: queryTxt ENODATA _acme-challenge.swierczek.sandcats.io

21:44 <mike_sw294503> at QueryReqWrap.onresolve [as oncomplete] (dns.js:203:19)

21:44 <mike_sw294503> sandstorm/gateway.c++:951: info: Loading TLS key into Gateway

21:44 <mike_sw294503> Obviously 'swierczek' is my sandcats subdomain. To my knowledge, I haven't messed up the server config.

21:46 <kentonv> mike_sw294503, hmm, the log looks like it actually did complete the exchange. The error probably came from a previous attempt, but then it was retried later. Or do you see several of those errors?

21:46 <kentonv> the "Leading TLS key into Gateway" message indicates that a final key/cert was obtained successfully

21:47 <mike_sw294503> I see several copies.

21:47 <kentonv> well, hmm, I do see that you have an expired cert

21:47 <mike_sw294503> Thanks for the quick response. I restarted the server, that didn't help.

21:48 <kentonv> ok, the "Loading TLS key" message probably happened when restarting the server then

21:48 <kentonv> which is normal

21:48 <kentonv> ok, so, the error could be because the machine is having trouble with DNS queries

21:49 <kentonv> What sandstorm does is, it talks to the sandcats API to ask that the _acme-challenge TXT record be set, and then it tries to do a regular DNS query to check that the record is present

21:49 <kentonv> the error indicates that the check is showing no record

21:50 <kentonv> is it possible that outgoing DNS queries are being blocked somehow on your server?

21:51 <isd> kentonv: fwiw, I just ran dig -t TXT _acme-challenge.swierczek.sandcats.io and got no results, so it seems there in fact is no record -- though I don't know if sandcats clears them at a certain point.

21:52 <mike_sw294503> Thanks for the explanation. I'll start digging into it. This server is running on my Linux workstation (logs going back to July 2016, by the way - thanks for making something wonderful). Maybe something Comcast is doing is messing with my DNS?

21:53 <kentonv> isd, I think code in Sandstorm may actually request they be removed when it's done (even in the error case)

21:57 <kentonv> mike_sw294503, I just tried a renewal to verify the server is working. Seems to be. So yeah I guess I'd try to figure out if Comcast is blocking TXT queries or something. Maybe try using 1.1.1.1 as your DNS. :)

21:57 <mike_sw294503> I switched my DNS to 1.1.1.1,1.0.0.1 and restarted Sandstorm, still no dice.

21:58 <mike_sw294503> Yeah, I hear Cloudflare employs some pretty smart people :)

21:58 <mike_sw294503> Damn, I didn't think of this earlier - could IPv6 screw it up? I only enabled that earlier this year.

21:59 <kentonv> I don't think ipv6 would affect this.

21:59 <kentonv> the log is still showing the same error each time you restart?

22:00 <mike_sw294503> Yes. I'm going to restart my workstation (which will drop me from IRC) just in case I've got something wacky going on that way. Be back in a bit.

22:10 <mike_sw294503> No dice. `nmcli device show bridge0 | grep IP4.DNS` shows that I'm using Cloudflare DNS, but my Sandstorm log has the error I printed and then a string of: sandstorm/gateway.c++:1057: error: exception = kj/compat/tls.c++:63: failed: OpenSSL error; message = error:10000415:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED

22:10 <mike_sw294503> stack: 6edfa4 6f3527 6f3b0c 530301 614040 6efd40 488460

22:11 <kentonv> I think that error is a consequence of the certificate fetch failing, but not a cause of it.

22:12 <kentonv> what does your /etc/resolv.conf say you're using for dNS?

22:13 <mike_sw294503> It refers me to `systemd-resolve --status` ...and the output of that isn't what I expected.

22:13 <kentonv> err

22:14 <kentonv> if your resolv.conf doesn't have a `nameserver 1.1.1.1` line in it then Sandstorm is not going to be able to make DNS queries

22:14 <kentonv> what exactly is the contents of your resolv.conf?

22:14 <mike_sw294503> ...and my wife just walked up and informed me we have plans tonight. I'm going to dig into this later, I'll have to learn how to read systemd-resolve output and get 1.1.1.1 into the list of things it's using.

22:14 <mike_sw294503> Uploaded file: https://uploads.kiwiirc.com/files/a4665e272f9844179460fde54bf4c8e6/pasted.txt

22:15 <mike_sw294503> Output of systemd-resolve:

22:15 <kentonv> ah. Hmm well I think that should be OK, assuming the local resolver does actually respond to queries correctly

22:17 <mike_sw294503> I'll come back to this later, thanks for your help! I do see 1.1.1.1 in the systemd-resolve. The output that confused me is related to the mini-Kubernetes cluster I was playing with. When I get back to this later, I'll uninstall that and disable IPv6 just to remove them as factors.