Centralize SSH Key Management
Lets assume we have 3 colleges (CollegeA, CollegeB, CollegeC) and we write a grant proposal and each institution will do something unique science wise. Grant gets funded and specialized hardware or software gets deployed at each college (for maybe brain scan analyses, deep learning, and engineering).
The grant mentioned that all members of participating colleges can request access at any college. How would one do that without a mess developing? Nobody, not even admins, should have access to any user level passwords. Accounts should be able to be revoked. CPU usage should be accountable. In short, some sort of “federated SSH access”. Command line access so we can rule out InCommon, it appears not to be ready for this. One option might be GSI-OpenSSH but it looks very complicated. Consult
- http://toolkit.globus.org/toolkit/about.html for GSI-OpenSSH
I asked the OpenHPC community and got some great suggestions back. One involved using Kerberos without tickets and tying into Active Directory. I quickly realized not all AD installations are publicly exposed. I'm providing Alan Sill's comments below for future reference. Another suggestion was to centrally manage SSH keys for password less login functionality, by Derek Simmel.
SSH public key authentication
On a selected collge, behind the firewall, set up a management server. In the /etc/ssh/sshd_config file, specify the path to user SSH public key files using a description like:
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
For this example, the /etc, /etc/ssh, and /etc/ssh/authorized_keys are directories owned and only writable by root. The sshd service interprets %u as the authenticated user's username on the system. So in this approach, every user has a file named after their username in the /etc/ssh/authorized_keys directory, root-owned.
drwxr-xr-x. 135 root root 12288 Mar 23 16:59 /etc drwxr-xr-x. 3 root root 4096 Nov 9 14:02 /etc/ssh drwxr-xr-x 2 root root 4096 Nov 9 14:01 /etc/ssh/authorized_keys -r--r--r-- 2 root root 1024 Mar 29 11:01 /etc/ssh/authorized_keys/weshmeij
We need to collect all root public keys into a single file called root. So that each college can retrieve the guest accounts and add them to their local passwd/shadow/group files. UID/GID ranges and usernames need to be unique. So we can assign ranges 15001-20000, 20001-25000, 25001-30000 for College[A|B|C]. Usernames prefixed with 3 character (wesleyan with wes, lafayette with laf, etc).
Workflow
- User hmeij of CollegeA (Wesleyan) requests access to College[B|C], goes to a web site and type in 'hmeij'
- Script does the following steps, figures out next UID/GID, referer ip yields prefix
- echo “weshmeij:x:15001” » /etc/group
- useradd -u 15001 -g 15001 weshmeij
- echo `date | md5sum | awk '{print $1}'` | passwd weshmeij –stdin
- su - weshmeij -c “ssh-keygen -b 2048 -t rsa -f /home/weshmeij/.ssh/weshmeij -q -N
” # 4 single quotes before closing double quote - mv /home/weshmeij/.ssh/weshmeij.pub /etc/ssh/authorized_keys/weshmeij
- chown root:root /etc/ssh/authorized_keys/weshmeij
- cat /home/weshmeij/.ssh/weshmeij # present in browser
- CollegeA user hmeij saves private key to $HOME/.ssh/weshmeij.priv; alters permissions chmod go-rwx
- script finishes; rm -f /home/weshmeij/.ssh/weshmeij
- that night college[A|B|C] root retrieves all lines in the range 15001-30000
- makes home dirs if they do not exist (parse lines build useradd, or via pam.d/sshd?)
- download public keys, updates in /etc/ssh/authorized_keys (rsync with –delete)
- replaces local passwd/shadow/group with retrieved lines
- user hmeij@wes: ssh weshmeij@openhpc.lafayette.edu -i /home/hmeij/.ssh/weshmeij.priv
That would work. Nobody knows the passwords for these guest accounts.
Kerberos & AD
1. install pam by run commands "yum install pam", "yum install pam_krb5", and "yum install pam_ssh".
The last package is needed on a log-in node, it's not required on compute nodes.
2. Modify /etc/krb5.conf as below
(Note for historical reasons we use both upper case and lower case for our kerberos domains.
Both resolve to the same thing. You may not need this - try it and see.)
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = YOUR-UNIV-KERB-DOMAIN (e.g., MYUNIVERSITY.EDU)
dns_lookup_realm = true
dns_lookup_kdc = true
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
YOUR-UNIV-KERB-DOMAIN (e.g., MYUNIVERSITY.EDU) = {
kdc = your-univ-domain (e.g., my university.edu):88
admin_server = your-univ-domain (e.g., my university.edu):749
}
your-univ-kerb-domain (e.g., ttu.edu) = {
kdc = your-univ-domain (e.g., my university.edu):88
admin_server = your-univ-domain (e.g., my university.edu):749
}
[kdc]
profile = /var/kerberos/krb5kdc/kdc.conf
[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}
3. Replace /etc/ntp.conf using your donain’s preferences; example below,
server your-univ-ntp-server (e.g., ntp.myuniversiry.edu) iburst
server 127.127.1.1 iburst
fudge 127.127.1.1 stratum 10
driftfile /var/lib/ntp/drift
4. Replace /etc/ntp/step-tickers
your-univ-ntp-server (e.g., ntp.myuniversiry.edu)
5. Setup banner announcement in /etc/banner (optional, depending on yoru organizational policies)
6. Replace /etc/ssh/ssh_config
Host *
CheckHostIP no
ForwardX11 yes
ForwardAgent yes
StrictHostKeyChecking no
UsePrivilegedPort no
Protocol 2
7. Modify /etc/ssh/sshd_config
Protocol 2
KerberosAuthentication no
KerberosOrLocalPasswd yes
KerberosTicketCleanup yes
Banner /etc/banner
8. Modify /etc/pam.d/system-auth-ac and /etc/pam.d/password-auth-ac as follows,
making any changes needed to fit this into your setup.
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth sufficient pam_krb5.so use_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth required pam_deny.so
account required pam_unix.so
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_krb5.so
account required pam_permit.so
password requisite pam_cracklib.so try_first_pass retry=3
password sufficient pam_unix.so md5 shadow nullok try_first_pass use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
session optional pam_krb5.so
or as the following:
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth [default=ignore success=3] pam_succeed_if.so user ingroup ttuhsc debug
auth requisite pam_succeed_if.so uid >= 500 quiet
auth sufficient pam_krb5.so use_first_pass realm=YOUR-UNIV-KERB-DOMAIN (e.g., MYUNIVERSITY.EDU)
auth [default=ignore success=1] pam_succeed_if.so user notingroup ttuhsc debug
auth sufficient pam_krb5.so use_first_pass realm=TTUHSC.EDU
auth required pam_deny.so
account required pam_unix.so
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_krb5.so
account required pam_permit.so
password requisite pam_cracklib.so try_first_pass retry=3
password sufficient pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password sufficient pam_krb5.so use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
session optional pam_krb5.so
9. Reload and restart sshd and ntpd
service sshd reload
service ntpd restart
ntpdate your-univ-ntp-server (e.g., ntp.myuniversiry.edu)
(optional, if the difference of the system time to the ntp server time is too long,
this command can synchronize the system time to the ntp server)
10. If all the above steps are done but it still doesn't work, try running some
or all of the following
authconfig --test //If the output shows "pam_krb5 is disabled",
run "authconfig --enablekrb5 --update" to enable it. If the output shows
"pam_krb5 is enabled" and "krb5 kdc via dns is disabled",
modify /etc/krb5.conf to make sure it has "dns_lookup_realm = true",
and run "service sshd restart" to load the settings then check again.
