Back

Centralize SSH Key Management

Lets assume we have 3 colleges (CollegeA, CollegeB, CollegeC) and we write a grant proposal and each institution will do something unique science wise. Grant gets funded and specialized hardware or software gets deployed at each college (for maybe brain scan analyses, deep learning, and engineering).

The grant mentioned that all members of participating colleges can request access at any college. How would one do that without a mess developing? Nobody, not even admins, should have access to any user level passwords. Accounts should be able to be revoked. CPU usage should be accountable. In short, some sort of “federated SSH access”. Command line access so we can rule out InCommon, it appears not to be ready for this. One option might be GSI-OpenSSH but it looks very complicated. Consult

I asked the OpenHPC community and got some great suggestions back. One involved using Kerberos without tickets and tying into Active Directory. I quickly realized not all AD installations are publicly exposed. I'm providing Alan Sill's comments below for future reference. Another suggestion was to centrally manage SSH keys for password less login functionality, by Derek Simmel.

SSH public key authentication

On a selected collge, behind the firewall, set up a management server. In the /etc/ssh/sshd_config file, specify the path to user SSH public key files using a description like:

For this example, the /etc, /etc/ssh, and /etc/ssh/authorized_keys are directories owned and only writable by root. The sshd service interprets %u as the authenticated user's username on the system. So in this approach, every user has a file named after their username in the /etc/ssh/authorized_keys directory, root-owned.

 drwxr-xr-x. 135 root root 12288 Mar 23 16:59 /etc
 drwxr-xr-x.   3 root root  4096 Nov  9 14:02 /etc/ssh
 drwxr-xr-x    2 root root  4096 Nov  9 14:01 /etc/ssh/authorized_keys
 -r--r--r--    2 root root  1024 Mar 29 11:01 /etc/ssh/authorized_keys/weshmeij

We need to collect all root public keys into a single file called root. So that each college can retrieve the guest accounts and add them to their local passwd/shadow/group files. UID/GID ranges and usernames need to be unique. So we can assign ranges 15001-20000, 20001-25000, 25001-30000 for College[A|B|C]. Usernames prefixed with 3 character (wesleyan with wes, lafayette with laf, etc).

Workflow

That would work. Nobody knows the passwords for these guest accounts.

Kerberos & AD

1. install pam by run commands "yum install pam", "yum install pam_krb5", and "yum install pam_ssh". 
The last package is needed on a log-in node, it's not required on compute nodes.

 2. Modify /etc/krb5.conf as below

(Note for historical reasons we use both upper case and lower case for our kerberos domains. 
Both resolve to the same thing. You may not need this - try it and see.)


 [logging]
  default = FILE:/var/log/krb5libs.log
  kdc = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log

 [libdefaults]
  default_realm = YOUR-UNIV-KERB-DOMAIN (e.g., MYUNIVERSITY.EDU)
  dns_lookup_realm = true
  dns_lookup_kdc = true
  ticket_lifetime = 24h
  renew_lifetime = 7d
  forwardable = true

 [realms]
 YOUR-UNIV-KERB-DOMAIN (e.g., MYUNIVERSITY.EDU) = {
    kdc = your-univ-domain (e.g., my university.edu):88
    admin_server = your-univ-domain (e.g., my university.edu):749
  }
  your-univ-kerb-domain (e.g., ttu.edu) = {
    kdc = your-univ-domain (e.g., my university.edu):88
    admin_server = your-univ-domain (e.g., my university.edu):749
  }
  
 [kdc]
  profile = /var/kerberos/krb5kdc/kdc.conf

 [appdefaults]
  pam = {
    debug = false
    ticket_lifetime = 36000
    renew_lifetime = 36000
    forwardable = true
    krb4_convert = false
  }

 3. Replace /etc/ntp.conf using your donain’s preferences; example below,


 server your-univ-ntp-server (e.g., ntp.myuniversiry.edu) iburst
 server  127.127.1.1 iburst
 fudge   127.127.1.1 stratum 10
 driftfile /var/lib/ntp/drift

 4. Replace /etc/ntp/step-tickers


 your-univ-ntp-server (e.g., ntp.myuniversiry.edu)

 
 5. Setup banner announcement in /etc/banner (optional, depending on yoru organizational policies)


 6. Replace /etc/ssh/ssh_config            
 Host *
            CheckHostIP                         no
            ForwardX11                          yes
            ForwardAgent                       yes
            StrictHostKeyChecking          no
            UsePrivilegedPort                  no
            Protocol                               2

 7. Modify /etc/ssh/sshd_config


 Protocol 2
 KerberosAuthentication no
 KerberosOrLocalPasswd yes
 KerberosTicketCleanup yes
 Banner /etc/banner
  
 8. Modify /etc/pam.d/system-auth-ac and /etc/pam.d/password-auth-ac as follows, 
making any changes needed to fit this into your setup.


 auth        required      pam_env.so
 auth        sufficient    pam_unix.so nullok try_first_pass
 auth        sufficient    pam_krb5.so use_first_pass
 auth        requisite     pam_succeed_if.so uid >= 500 quiet
 auth        required      pam_deny.so
 account     required      pam_unix.so
 account     sufficient    pam_localuser.so
 account     sufficient    pam_succeed_if.so uid < 500 quiet
 account     [default=bad success=ok user_unknown=ignore] pam_krb5.so
 account     required      pam_permit.so
 password    requisite     pam_cracklib.so try_first_pass retry=3
 password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
 password    required      pam_deny.so
 session     optional      pam_keyinit.so revoke
 session     required      pam_limits.so
 session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
 session     required      pam_unix.so
 session     optional      pam_krb5.so

 or as the following:

 # User changes will be destroyed the next time authconfig is run.

auth        required      pam_env.so
 auth        sufficient    pam_unix.so nullok try_first_pass
 auth        [default=ignore success=3] pam_succeed_if.so user ingroup ttuhsc debug
 auth        requisite     pam_succeed_if.so uid >= 500 quiet
 auth        sufficient    pam_krb5.so use_first_pass realm=YOUR-UNIV-KERB-DOMAIN (e.g., MYUNIVERSITY.EDU)
 auth        [default=ignore success=1] pam_succeed_if.so user notingroup ttuhsc debug
 auth        sufficient    pam_krb5.so use_first_pass realm=TTUHSC.EDU
 auth        required      pam_deny.so

 account     required      pam_unix.so
 account     sufficient    pam_localuser.so
 account     sufficient    pam_succeed_if.so uid < 500 quiet
 account     [default=bad success=ok user_unknown=ignore] pam_krb5.so
 account     required      pam_permit.so

 password    requisite     pam_cracklib.so try_first_pass retry=3
 password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
 password    sufficient    pam_krb5.so use_authtok
 password    required      pam_deny.so

 session     optional      pam_keyinit.so revoke
 session     required      pam_limits.so
 session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
 session     required      pam_unix.so
 session     optional      pam_krb5.so


 9. Reload and restart sshd and ntpd


 service sshd reload
 service ntpd restart
 ntpdate your-univ-ntp-server (e.g., ntp.myuniversiry.edu) 

(optional, if the difference of the system time to the ntp server time is too long, 
this command can synchronize the system time to the ntp server)

 10. If all the above steps are done but it still doesn't work, try running some 
or all of the following


 authconfig --test    //If the output shows "pam_krb5 is disabled", 
run "authconfig --enablekrb5 --update" to enable it. If the output shows 
"pam_krb5 is enabled" and "krb5 kdc via dns is disabled", 
modify /etc/krb5.conf to make sure it has "dns_lookup_realm = true", 
and run "service sshd restart" to load the settings then check again.


Back