SSH single-sign on

I was looking for a way to deal with ssh/sudo authentication between multiple clients and multiple devices. I wanted something with a balance between convenience and security.

For my clients, I have an Android phone, an Android tablet, a Windows desktop, and a few laptops with various OSes.

For my devices, I have CentOS, Fedora, Ubuntu, and Debian machines. Some of them are SBCs that I use to test out time synchronization code.

The things I explored:

• ssh keys in hardware: yubikey, GNUK, solo/somu, and TPMs
• openssh u2f support
• Cloud ssh agent with the key on my phone's hardware key storage
• Kerberos

I may go into the other solutions in more detail in other blog posts, but this will focus on Kerberos.

Kerberos

Kerberos Version 5 (krb5) has been around since the 90's, and is still in active development. The short version of the protocol: your password generates a secret key that you and the KDC share. See "Explain like I'm 5: Kerberos" for a longer explanation, as there are more details beyond that short explanation.

Benefits/Drawbacks

Everything has benefits and drawbacks, especially when it comes to convenience and security.

Benefits of Kerberos:

• Pretty widely supported by ssh clients (wsl2, Linux, Android termux, MacOS)
• Very widely supported by ssh servers, even older OSes like CentOS7 and Debian 9/stretch
• New ssh clients don't need a lot of configuration to start working
• AD includes a kerberos server and runs in many companies already
• No hardware needed, just software - nothing to buy, get lost, pass through to a VM, or break
• It's very convenient once everything is setup - login once per day to access any ssh server in the realm
• Very scalable design, and it's easy to change passwords
• Mature, well documented, well studied protocol
• All open source, not depending on a single vendor
• Can be used to authenticate other things like sudo and http
• Mutual authentication of both the client and the server
• ssh servers don't get your actual password, just a token

Drawbacks of Kerberos:

• Uses passwords, so that limits security (it can be configured to use 2FA, but that's optional)
• Debugging messages are cryptic to non-existent
• It brings additional services to manage and protect
• Passwords can be brute forced (you'll probably want a fail2ban or some other way to manage that)
• An attacker getting the master key and password file can spoof being any user or service, and also run an offline brute force against the password hashes (modern Kerberos uses PBKDF2 hash type)
• Exposing the KDC to the world makes it possible for attackers to figure out which user accounts exist
• Without SPAKE preauth, offline password brute force attacks are possible from passive network sniffing
• Weak encryption types (RC4, DES, 3DES) in Kerberos need to be disabled by configuration

Setup of Kerberos servers

First, the Kerberos servers. I setup one primary with kadmin+KDC, and one secondary with just a KDC. The official installation guide is useful for this.

After setting up the primary, the secondary can stay in sync with either incremental transfers or copying the entire database with kdb5_util+kprop in a cron job. There's a section in the install guide for setting the secondary KDC up. kpropd is setup to run out of systemd on modern Linux, so you don't need to use inetd.

I'm using the kdc.conf below. Spake preauth is enabled for all users and is configured for the edwards25519 curve. Only AES and Camellia crypto types are supported. SHA1 being used as an HMAC is probably still ok, and disabling it breaks older clients. Incremental transfers to the secondary (iprop) is setup to run on port 755. kpropd on the secondary polls the primary for changes.

[kdcdefaults]
    kdc_ports = 88
    kdc_tcp_ports = 88
    spake_preauth_kdc_challenge = edwards25519

[realms]
 LAN = {
     acl_file = /var/kerberos/krb5kdc/kadm5.acl
     dict_file = /usr/share/dict/words
     admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
     master_key_type = aes256-cts-hmac-sha384-192
     supported_enctypes = aes256-cts-hmac-sha384-192:normal aes128-cts-hmac-sha256-128:normal aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal camellia256-cts-cmac:normal camellia128-cts-cmac:normal
     max_life = 10h 0m 0s
     max_renewable_life = 7d 0h 0m 0s
     default_principal_flags = +preauth
     iprop_enable = true
     iprop_port = 755
 }

Setup of ssh servers

To setup the ssh servers, each needs its hostname in DNS. I'm using dnsmasq to automatically create hostnames based on dhcp client names. The hostname reported by /usr/bin/hostname needs to match the hostname in DNS, so sshd knows what krb host key to use. If you change your system's hostname, you'll probably have to restart sshd for it to pick up the change.

For each ssh server, you'll need to create a krb5 host principal and save it in the file /etc/krb5.keytab:

test:~$ sudo kadmin -p myuser/admin
Authenticating as principal myuser/admin with password.
Password for myuser/admin@LAN:
kadmin:  addprinc -randkey host/test.lan
WARNING: no policy specified for host/test.lan@LAN; defaulting to no policy
Principal "host/test.lan@LAN" created.
kadmin:  ktadd host/test.lan
Entry for principal host/test.lan with kvno 2, encryption type aes256-cts-hmac-sha384-192 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/test.lan with kvno 2, encryption type aes128-cts-hmac-sha256-128 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/test.lan with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/test.lan with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/test.lan with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/test.lan with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.
kadmin:

You'll also need to verify that GSSAPIAuthentication yes is enabled in /etc/ssh/sshd_config

If you try to access the ssh server via IP, or a hostname that isn't in kerberos, you'll get this in the "ssh -v $server" debugging output:

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Server host/192.168.1.6@LAN not found in Kerberos database

Some failure messages are not reported to the client, the ssh client just sees a silent failure:

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

If you change your sshd_config to have "LogLevel DEBUG" you'll get some more detail:

Jan 04 02:06:25 test sshd[876]: debug1: Unspecified GSS failure.  Minor code may provide more information\nNo key table entry found matching host/test@\n\n

The hostname of this vm was set to "test", changing it to "test.lan" to match the krb principal wasn't enough.  dnsmasq was also returning the reverse dns hostname as "test".  So I added "rdns = false" to krb5.conf.

Finally:

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Authentication succeeded (gssapi-with-mic).
Authenticated to test.lan ([192.168.122.116]:22).

My full krb5.conf (needed on both the ssh clients and the ssh servers):

[libdefaults]
        default_realm = LAN

# The following krb5.conf variables are only for MIT Kerberos.
        kdc_timesync = 1
        ccache_type = 4
        forwardable = true
        proxiable = true

# The following libdefaults parameters are only for Heimdal Kerberos.
        fcc-mit-ticketflags = true

    # don't trust reverse dns
    rdns = false
    # use the edwards 25519 curve for spake preauth
    spake_preauth_groups = edwards25519
    # don't lookup the KDCs in DNS, use the [realms] section
    dns_lookup_kdc = false
    
    # set the permitted crypto to aes+camellia
    permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
    default_tgs_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
    default_tkt_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac

[realms]
        LAN = {
                # KDCs are used for authentication
                kdc = kerberos1.lan
                kdc = kerberos2.lan
                # admin server is used for changing passwords
                admin_server = kerberos1.lan
                # use spake for preauth, and don't fall back to encrypted timestamp
                disable_encrypted_timestamp = true
        }

[domain_realm]
        # hostnames ending in .lan are in the LAN realm
        .lan = LAN

Setup of ssh clients

On Debian/Ubuntu, ssh clients need the packages krb5-user and krb5-config.  On Fedora/CentOS, ssh clients need the package krb5-workstation.

I'm using the krb5.conf from "Setup of ssh servers" above on my ssh clients.  I start a session with the command "kinit", and then I can login to other servers:

[client ~]$ kinit
Password for myuser@LAN:
[client ~]$ ssh test.lan
Last login: Mon Jan  4 02:13:04 2021 from 192.168.122.1
test:~$

When you login to a ssh server, your ssh client asks the KDC to sign a message confirming your identity with the ssh server's krb5 host key.  The KDC also generates a private key and encrypts it with both my user's key and the ssh server's krb5 host key.  That way, my ssh client can be sure it is talking to the actual server with the krb5 host key and not a man in the middle.  You can see what host keys are in your cache with the klist command:

[client ~]$ klist -e
Ticket cache: KEYRING:persistent:1000:1000
Default principal: myuser@LAN

Valid starting     Expires            Service principal
01/03/21 20:17:58  01/04/21 06:16:28  host/test.lan@LAN
        renew until 01/03/21 20:16:28, Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192
01/03/21 20:16:28  01/04/21 06:16:28  krbtgt/LAN@LAN
        renew until 01/03/21 20:16:28, Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192

For Android, I'm using termux as my ssh client which has Kerberos support. Its default KDC port is 1088, so I had to specify port 88 in its krb5.conf file. I also had to add "GSSAPIAuthentication true" to its ~/.ssh/config file.

Setup of PAM

sudo is using PAM to authenticate.  The Debian/Ubuntu package krb5-config automatically sets that up after asking a few questions.  On the Fedora/CentOS side, that's setup by "authconfig --enablekrb5"

Kerberos does not automatically create users or home directories, so you'll need another solution for that.

Cross-realm Authentication

Next, I wanted a way to authenticate my user when it accessed my cloud servers. I didn't want to have the local password database copied to the cloud, and I also didn't want to rely on my home network being up to grant access to the cloud servers.

So I setup a second Kerberos realm in the cloud, following the "Setup of Kerberos servers" section above.  I gave it a new realm name.

The relevant documentation is: cross-realm authentication.

I needed to generate a principal for "krbtgt/CLOUD@LAN" and add it to both KDCs. Normally you'd add a second principal "krbtgt/LAN@CLOUD", but I don't need or want authentication to work in that direction.

I had to change my krb5.conf files on my clients to let them know about the new realm as well as the domain realm mapping:

[realms]
CLOUD = {
  kdc = kerberos1.drown.org
  kdc = kerberos2.drown.org
  admin_server = kerberos1.drown.org
}

[domain_realm]
.drown.org = CLOUD

Next, I needed a way to list which kerberos identities were authorized for my user. There's a few ways to do that, but I went with creating the file .k5login in my home directory. It takes one line per principal name including the realm, such as "myuser@LAN".

I allowed my home network access the cloud KDC servers via firewall rules.

Then I was able to login to my cloud servers via my home identity:

[client ~]$ ssh cloud-server
Last login: Sun Jan  3 20:29:01 2021 from home.lan
[cloud-server ~]$

And you can see all the principals that loaded:

[client ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: myuser@LAN

Valid starting     Expires            Service principal
01/03/21 15:10:51  01/04/21 15:10:47  krbtgt/LAN@LAN
01/03/21 20:29:33  01/04/21 06:29:33  krbtgt/CLOUD@LAN
01/03/21 20:29:33  01/04/21 06:29:33  host/cloud-server.drown.org@CLOUD