Securing Remote Operations feat. SSH-Agent
If you are someone who needs to configure or deploy code to different remote machines, you probably use SSH for the majority of your tasks. Furthermore, if you follow security best practices, you must have disabled password authentication and only authenticate to remote machines and repositories using Key-Based authentication.
Configuring Passwordless SSH Authentication
On your local machine:
# Create your keys
> ssh-keygen -t rsa# Copy your public key to the remote server
> ssh-copy-id -i $HOME/.ssh/id_rsa.pub bob@remote-server.com
On the remote machine as root:
# Disable ChallengeResponseAuthentication
> sed -i ‘s/^.\?ChallengeResponse.*/ChallengeResponseAuthentication no/g’ /etc/ssh/sshd_config# Disable PasswordAuthentication
> sed -i 's/^.\?PasswordAuthentication.*/PasswordAuthentication no/g' /etc/ssh/sshd_config# Disable UsePAM
> sed -i 's/^.\?UsePAM.*/UsePAM no/g' /etc/ssh/sshd_config# Reload SSH server
> systemctl reload ssh
Now you should have remote access to the server, via SSH, without needing to input any credentials!
> ssh bob@remote-server
bob@remote-server:~$However, now that you have access to the remote machine using key-based authentication, if you would like to clone a target repository for our configuration, this would happen:
bob@remote-server:~# git clone git@gitlab.com:Bob/hello-world.git
Cloning into 'hello-world'...
git@gitlab.com: Permission denied (publickey,keyboard-interactive).
fatal: Could not read from remote repository.Please make sure you have the correct access rights
and the repository exists.
Since we are on a different host, where our configured keys are not present, there is no way we can get access, so how do you get authorised access to your tools and repositories from this remote machine without:
- Generating new ssh keys on the remote server and adding them to your repositories or servers
- Copying your key pair to your remote machine
- Credential prompting
SSH-Agent to the rescue
Wikipedia: For added security (…) it is common to store the private key in an encrypted form, where the encryption key is computed from a passphrase that the user has memorised. Because typing the passphrase can be tedious, many users would prefer to enter it just once per local login session. The most secure place to store the unencrypted key is in program memory (…). A normal SSH client process cannot be used to store the unencrypted key because SSH client processes only last the duration of a remote login session. Therefore, users run a program called ssh-agent that runs beyond the duration of a local login session, stores unencrypted keys in memory, and communicates with SSH clients using a Unix domain socket.
So basically, SSH-Agent provides means to save our keys, in memory, and reuse them in future connections. But for our scenario, what we are interested in is a feature called SSH-Agent Forwarding.
SSH-Agent Forwarding makes it possible to forward access to our saved keys, in the SSH-Agent, to be reused in future connections from our remote machine.
The keys themselves are not exposed though, only a communication channel, to grant access when requested, is created.
Configuring SSH-Agent:
To configure and start using SSH-Agent you first need to check if it is already running in the background and if not, start the process:
# Check if SSH-Agent is running
> echo $SSH_AUTH_SOCKThe command ssh-add is used to interact with the agent. Another way to check if the agent is running is to list the current keys stored in memory. The following error would mean it is not running:
bob@remote-server:~$ ssh-add -l
Could not open a connection to your authentication agent.To start the agent you should issue:
# Start SSH-Agent
eval $(ssh-agent)# Now the environment variable should be set
> echo $SSH_AUTH_SOCK
After setting up your agent, you need to add the keys to be stored. Issuing the following command, will by default add the predefined key ~/.ssh/id_rsa to the agent:
> ssh-add ~/.ssh/id_rsa
Identity added: ~/.ssh/id_rsa# You can now list the keys being managed/stored by the agent
> ssh-add -l
Using SSH-Agent Forwarding:
Now that we have our agent running with keys stored in memory, ideally we would like to make use of this keys on our remote servers. For that, we need to forward our agent through the ssh tunnel, so that remotely we are able to access it and use its managed keys for further authentications.
Lets look at two ways on how to forward the agent:
ssh -A bob@remote-server:
Using -A flag will forward the agent and current in-memory stored keys, making it accessible in the remote server:
> ssh -A bob@remote-server# Verifying the agent is forwarded
bob@remote-server:~$ echo $SSH_AUTH_SOCK
/tmp/ssh-4bz3zWANtS/agent.3721
Setting up .ssh/config for the remote host:
Using the ssh config file, we can define that on connection to the remote server it will forward the agent:
# ~/.ssh/config
Host remote-server
ForwardAgent yesFinally, lets try to clone our repository using our forwarded ssh-agent keys:
bob@remote-server:~$ git clone git@gitlab.com:Bob/hello-world.git
Cloning into 'hello-world'...
The authenticity of host 'gitlab.com (172.65.251.78)' can't be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gitlab.com,172.65.251.78' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 10 (delta 2), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (10/10), done.
Resolving deltas: 100% (2/2), done.bob@remote-server:~$ ls -l
total 4
drwxr-xr-x 3 bob bob 4096 May 1 22:23 hello-world
Congratulations! You are now able to securely authenticate, from your remote server, to the systems you need without any password prompting or by storing your private keys remotely.
Automating SSH-Agent Forwarding
The previous configuration steps need to be reproduced each time you logout of your host or the ssh-agent is not running, which is not ideal. To facilitate, you can automate it so that:
- The SSH-Agent is run on startup
- When you connect to a remote host, ssh adds the correspondent keys and forwards the agent automatically
Run ssh-agent on login:
To make sure ssh-agent is running when you login to your machine you can use the ~/.profile file to run a script for you:
# Ensure no agent is running by checking SSH_AUTH_SOCK environment variable and run agentif [ -z "$SSH_AUTH_SOCK" ] ; then
eval `ssh-agent -s`
fi
If you want to ensure no agent is running on the machine when you logout you can add the following command to ~/.bash_logout: ssh-agent -k
Automatically add and forward the right keys to the remote server:
Every time we connect to the target remote server, we can automatically add the correct private key to our ssh-agent memory and forward it, by using the ~/.ssh/config file and AddKeysToAgent keyword:
AddKeysToAgent: Specifies whether keys should be automatically added to a running ssh-agent. If this option is set to
yesand a key is loaded from a file, the key and its passphrase are added to the agent with the default lifetime. (…)
The following configuration will automatically, on connect, add the loaded identity file to the agent running on background, authenticate to the remote-server with it and forward the agent:
Host remote-server
ForwardAgent yes
User bob
IdentityFile ~/.ssh/id_rsa
AddKeysToAgent yesExpanding to DevOps
Another example, are DevOps tasks on remote machines using the ssh-agent. For instance, if you are using Ansible, which relies its connections on ssh, you can make it so that a deployment on the remote-server uses the forwarded agent for all its authentications needed in the process, just by configuring the ~/.ssh/config file described above!
A Word on Security of SSH-Agent
Although the described usefulness of the ssh-agent, care must be taken to not forward agents to untrusted remote machines. Forwarding an agent does not give away access to private keys, but gives access to the communication channel providing further authentications, an UNIX socket:
> ssh -A bob@remote-server
bob@remote-server:~$ file $SSH_AUTH_SOCK
/tmp/ssh-Xp0hdaiMnt/agent.4038: socket# Checking file permissions
bob@remote-server:~$ stat /tmp/ssh-Xp0hdaiMnt/agent.4038 | grep Access
Access: (0755/srwxr-xr-x) Uid: ( 1000/ bob) Gid: ( 1000/ bob)
Despite the socket having very restrictive permissions, when forward an agent to untrusted servers or, for instance, vulnerable servers, an attacker who is able to escalate its privileges to Bob or the root user, will be able to reuse this channel and authenticate itself to other remote systems on behalf of Bob:
root@remote-server:~# SSH_AUTH_SOCK=/tmp/ssh-Xp0hdaiMnt/agent.4038 ssh bob@my-precious.com
bob@my-precious.com:~$_
