Is the name of the vulnerability literally *sshole?
* ssh olé!
Yes, it’s actually a very creative name. It’s caused by the character * allowing anyone to log into an open SSH port. So it’s an amalgam of {*}{SSH}{port (hole)}.
The long story short is that if you don’t have a password set for an account, Alpine saves the user’s password hash as “*” because no password hash will ever match it. Should be safe, right? Any password will always automatically fail, because the hash algorithm cannot return “*” as a result. So the only way to log in as that user is via something besides a password.
And when musl (the underlying system that Alpine uses for SSH) gets a Null response from its password hash, it uses “*” to represent that null value. Because lots of programs throw a fit if they get a true null value. So instead of returning a null value, it just returns a *. Which shouldn’t match anything, right?
So an attacker forces the password hash to return a null value. Simple to do, because the password hash is capped at 256 characters. So just enter something with more than 256 characters. Musl takes that null value, turns it into a *, and passes that onto Alpine. Alpine takes this *, and compares it with the list of saved usernames and password hashes. And it finds that the user (who has no password set, and shouldn’t be allowed to log in with a password) has a password hash of “*”. This matches the result musl just provided, so the attacker is able to log into the system.
Cyberpunk as fuck.
tldr: disable password auth in ssh.
Also name misses the point I think, it’s a musl-hole.
which you should have been doing anyway for anything exposed to the internet
i mean assholes are muscle holes.
Yeah, but they are very specific kind of muscle hole. 🙃
Yeah, any application checking passwords would be vulnerable.
The only time I ever exposed my SSH port to the WAN, I got about 750 connection attempts from China and 250 attemps from Russia per hour.
One reason that I have my router ignore all connection attempts from Russia and China
Fail2ban with increasing jail time for repeat offenders usually takes care of the worst of it after a day or two
How do you know this? What tool or utility tells you this fact? I have a port exposed for a game server but realising I don’t know how many attempts are being made.
I was kinda new to linux so I used https://www.papertrail.com/ and hooked it up to my syslogs, and then went “oh… oh no… oh god no…” and immediately turned off SSH.
Thank you kindly, I am reasonably new to Linux myself. Ibuse it but don’t have a great understanding
For ssh it’ll log out to the auth log at /var/log/auth.log
Among other things, change the default ssh port to something way higher.
Did it help you find hot milfs within a mile of your location?
Surprised me too, it’s even worse if you use a cloud host for a VPS since it’ll be a more known IP
Yet some people will still say with a straight face “tHeRe iS nO hYbRiD wAr!1!1”
Not in Ba Sing Se at least…
This was about 20 years ago btw. I know it’s gotten worse. But it started earlier than most think.
Yeah, BBC has a whole documentary about it. Plus the russian state department basically published it all in a propaganda manual. They don’t exactly try to hide this stuff…
99.999% of that is over IPv4. I leave IPv6 open to WAN for my own use and it’s not a problem.
Thanks for the intel
New friends from all over the world!
yeah, while reading this i was going “What maniac exposes SSH to WAN anyway…?” the whole time. I fail to see any reason or point that can be justified exposing SSH to WAN.
I used to have a synology NAS that would sleep of death itself. Synology support asked me to port forward SSH to the open internet and set no password so they could pull logs over the next few weeks.
Yes, it was legit synology support not a scammer.
No, I haven’t bought synology ever again.
What do you use now? Qnap?
I build my own stuff now. I tried qnap after synology but it just sucked in different ways.
Any VPS provider just gives you an SSH IP and port. So that’s the default state, until you do something else, which I imagine many don’t, since key based authentication was always considered secure.
So I don’t know the reason, but that’s the standard practice with most providers, like Digital Ocean or Linode or whatever. You can call them maniacs but you shouldn’t be surprised at something so ubiquitous.
Wait that means I have to expose a port. But I have zero exposed ports.
This post was created by the pangolin-gang.
I’ll expose my port to you.
Privileged port?
Meaning router port, not firewall port? I struggle to know the difference.
Some people seem to think that by not opening a port in their router, and instead opening a reverse tunnel to a VPS which has an open port, it somehow makes them more secure
I’m not sure what a reverse tunnel or vps is so I’m good. Why wouldnt they just use tail scale.?
behold!
the average selfhost admin!
practically need to rotate logs every hour if you foolishly leave ssh open on a open network. usually a bastion with ssh listening to port 34000 or such is a good first step.
that said the conditions for this exploit are fascinating. no pam, shadow password set to *, using a alternative to glibc. The sacrifices to be more efficient feel on par with replacing a tank’s armor with paper mache and hope.
The best armor is to not get hit!
I got into it with another sys admin on shifting port numbers. “Security through obscurity is not security!” to which I responded “Obscurity cuts out most of the bots” (therefore easier to spot real threats).
Eventually we just went the secret squirrel bastion route.
Given this is a vuln when entering the password, just turn off password auth and use ssh keys only. It’s what you should be doing anyway.
The vuln is specifically for key-based accounts:
Second, the target system account must be configured to be passwordless. Passwordless accounts are commonly used for accounts that use certificate-based authentication.
not true. regardless if all your user accounts are using key based auth or not, some system accounts will always be passwordless, because they are not supposed to be logged into.
But passwords need to be enabled for SSH. You must submit a faulty password of a specific kind, and the SSH server misinterprets it for passwordless accounts, which means the server must be configured to allow you to attempt a password.
Easier said than done.
If you have to support the sort of user that barely managed to download a copy of putty a few years ago and kept using this, then walking them through key generation and use is a nightmare.
One could fairly argue that maybe such people shouldn’t have access to a system, but it is a reality.
Of course the likelihood that a system using musl is something that class of user would be accessing is lower.
Maybe now that ssh is generally “just there” for Windows users, you can more easily work with them to generate a key, or give them a zip file to make them have a key in a standard way. But the interaction of ssh and windows file security can be… Interesting, though mostly an issue on the ash server side.
For added security, there is a pam module for 6-digit OTP
Further, OpenSSH using musl (pronounced muscle) must not use PAM (pluggable authentication modules),
which Alpine doesn’t ship by default. Maybe one or the other remote administration suite depends on it, but those are often suspectible to exploits anyway. But you don’t expose either to WAN, right?
Fuck. OpenWRT might fit that bill.
Edit: it does, kind of:
https://forum.gl-inet.com/t/sshole-exploit-does-not-impact-gl-oem-firmware/72657
The person who discovered *sshole said OpenWRT is affected, but by default, that OS uses the Dropbear package, not OpenSSH, for SSH connections (though maintainers provide instructions for installing the latter).
What an aSSHole!
Most ssh ports should be accessed through a VPN anyway, not exposed directly to the internet
Are you sure the VPN server has the smaller attack surface than the SSH server?
OpenVPN, no. Wireguard, yes.
Well that sucks. IIRC a lot of my docker images are alpine based, but they’d be safe due to never having their SSH ports exposed I assume.
Why would you even have an SSH server running in a Docker container (unless the purpose of the container is being a jumpbox or similar)?
I can’t emphasise it enough how much more secure you get when you use rootless, read-only containers that are limited to truly just the software you want to run and its dependencies.
Alpine is great for dev containers, but the final container should be:
- immutable on the app layer (any executable path needs to be read only)
- have only the absolutely necessary GNU tooling, service management, etc. included (for most software you don’t even need these tools, and why would you give an attacker who gain access to your container, an easy way to e.g. scan your network?)
- only the data mounted, rw only if absolutely necessary (e.g. if you host a Jellyfin server, just an example, you should have the mounted media be RO, and only the config and caches be writable, mounted noexec of course)
While I have not done it, I could see it being appealing as it’s quicker than SSHing into the parent box first for a commonly used tool. I had a direct SSH into a systemd container for a while before deciding the extra config wasn’t worth it.
Could that not be configured with a proxyjump through the host machine still, if one needed to have this?
unless the purpose of the container is being a jumpbox or similar
What’s wrong with a rootless jumpbox?
Nothing, my point was that SSH in a container is unnecessary unless it’s for the specific purpose of the container.
Alpine? The one that doesn’t use glibc? Wasn’t this why they chose a different C runtime? For security. Ooops.
I feel like alpine is more well-known for being lightweight than secure
“Security” and getting rid of that pesky copyleft licensing in order to make it so corporate users have to give back as little as possible.
Dude. One issue in the time of how many in glibc…
deleted by creator
