# SSH Troubleshooting Guide

I won't go into specific cases in this blog post. This is a general guide on how to gather the necessary information that will help you to get your problem fixed.

In this post, I'll use a **Linux** client and server as a reference. 


## Logging <a href="#logging" id="logging">#</a>

**Client**

Get the verbose logging with the `-v` flag. This normally is enough, but if you need even more information, use `-vv` and `-vvv`.

**Server**

You can find the logs for your SSH Server here `/var/log/auth.log` or `/var/log/secure`.

For troubleshooting sessions, it is recommended to increase the log level from the default `LogLevel INFO` to `LogLevel DEBUG1` in your SSH server configuration `sshd_config`. This will gives you all the necessary information. The following log levels are available: `QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG, DEBUG1, DEBUG2, and DEBUG3`. Remember to **restart the SSH server** after changing this setting.

Another method is to check `journalctl` if you use systemd. The logs should be available via `sudo journalctl -r -u ssh -u sshd`.

Often enough, restarting the server is not an option. You simply can add anoher process with the same options, but inceased debug level and another port. This allows you to monitor the logs for a specific client without interupting the main SSH server.

`sudo /usr/sbin/sshd -dDp 2222`

**Side note**: make sure to use the absolute path or you will be greeted by the following error message `sshd re-exec requires execution with an absolute path`.

Thanks to [youRFate on Lobste.rs](https://lobste.rs/s/wombsw/ssh_troubleshooting_guide#c_fia3jk) for the tip!

## Common errors

As mentioned, there are many more, but the following list will give you a great starting point.

#### Hostname resolution <a href="#hostname" id="hostname">#</a>

```markdown
error output
ssh: Could not resolve hostname example.com: Name or service not known
```

This error message implies a problem with the DNS.

- check that the hostname is correct
- use the IP instead to test general connectivity
- check hostname resolution with `nslookup` or other tools


#### Connection timeout <a href="#timeout" id="timeout">#</a>

```markdown
Error output
ssh: connect to host 10.10.10.10 port 22: connection timed out
```

This error tells you that you can't reach the server at all.

Wrong destination IP:
: verify that the destination IP is correct

Routing:
: can the client reach the destination? Check the routing table and use ICMP to double-check (ping and traceroute). Consider that ICMP sometimes is blocked by network firewalls!

Firewalls:
: check the firewalls on the client, server, and network firewalls and make sure that the connection is allowed.

#### Connection refused <a href="#refused" id="refused">#</a>

```markdown
Error output
ssh: connect to host 10.10.10.10 port 22: connection refused
```

You can reach the server, but the server refuses the connection

Wrong destination IP:
: verify that the destination IP is correct

Listening SSH server port:
: is the default SSH port `22` used? You can check it with the `Port 22` in the `/etc/ssh/sshd_conf` file on the server.
: is the server listening on the communicated port? Check on the server with `ss -tulpen | grep -i :22` (use `netstat` on older Linux versions) or use tools like `nmap` to find the listening port (disclaimer: do not scan server you do not have the permission for)

SSH server running:
: make sure that the SSH server is running, e.x. with `systemctl status sshd`


#### Permission denied <a href="#permission" id="permission">#</a>

`Permission denied (publickey,password)` 

Most likely a problem with the authentication.

Wrong user credentials:
: make sure that you use the correct username and password or private key.
: as a side note: the login as `root` is often forbidden by common security measures. 

Missing permissions on the server:
: make sure that the user is allowed to log in via SSH.
: `/etc/ssh/sshd_config` > `AllowUsers` or `AllowGroups`

Wrong authentication method:
: most commonly, you'd log in via password or public key authentication.
: use the `-v` on the client to look for the following entry: `debug1: Authentications that can continue: password,publickey`. This gives you information on what the server accepts.
: to force an authentication option on the client, you could use the `-o` flag with SSH options. To force the login via password you could use something like this: `ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@10.10.10.10`.
: if the desired option is unavailable, it must be configured on the server. `/etc/ssh/sshd_config`: `PubkeyAuthentication yes` and `PasswordAuthentication yes`. [It is recommended to use public key authentication only](https://ittavern.com/ssh-how-to-use-public-key-authentication-on-linux/).

Wrong permission and/or ownership of SSH-related files:
: most SSH servers check how permissive e.x. the SSH keys are, and can deny access if they are too permissive. 

```markdown
sudo chmod 700 ~/.ssh
sudo chmod 644 ~/.ssh/authorized_keys
sudo chmod 644 ~/.ssh/known_hosts
sudo chmod 644 ~/.ssh/config
sudo chmod 600 ~/.ssh/nameofthekey        # private key
sudo chmod 644 ~/.ssh/nameofthekey.pub    # public key
```

Public key is missing in the `~/.ssh/authorized_keys` file:
: the public key must be added to the a.m. file. A how-to can be found in [this post](https://ittavern.com/ssh-how-to-use-public-key-authentication-on-linux/).

Private key no longer accepted on the server:
: some private keys are no longer considered secure, so the server could refuse the login with those keys.
: the best solution would be to update the SSH applications and generate new keys.
: a workaround would be to add the insecure key algorithm to the SSH server config to the accepted keys `PubkeyAcceptedKeyTypes`.


#### SSH protocol version <a href="#ssh-version" id="ssh-version">#</a>

`Protocol major versions differ: 1 vs. 2`

The client and server do not work with the same protocol version. That said that you should only use SSHv2 and disable SSHv1.

**Client**

With the `-v` flag you can see what the server offers:
: `debug1: Remote protocol version 2.0 [...]`

With the flags `-1` and `-2` you can decide whether the client should use SSH protocol version 1 or 2, respectivly. 

**Server**

On the server, you can check the provided SSH protocol version in the configuration file:
: `grep Protocol /etc/ssh/sshd_config`
: `Protocol 1` *# SSHv1*
: `Protocol 2` *# SSHv2*
: `Protocol 1,2` *# SSHv1 + SSHv2*

If this option is missing, the mordern SSH server will use SSHv2 by default. It is worth adding it just to be sure and have it documented.

#### Failed host key verification <a href="#hostkey" id="hostkey">#</a>

```markdown
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
```

Clearing the host key from `~/.ssh/known_hosts` our use `ssh-keygen -R <ip-of-destination`. You should be able to connect normally.

If you were not informed about any changes, please contact the SSH server administrator to verify that everything is still secure.

#### Unable to negotiate ciphers, MACs, or KexAlgorithms <a href="#ciphers" id="ciphers">#</a>

```
Unable to negotiate with 10.10.10.10: no matching key exchange method found.
Their offer: diffie-hellman-group1-sha1
```

Use the `-vv` flag on the client to output the necessary information. On the server, you can see the information with the `LogLevel DEBUG2` and check with the following commands what is accepted by the server.

[Ciphers](https://man.openbsd.org/ssh_config#Ciphers):
: `ssh -Q cipher`

[MACs](https://man.openbsd.org/ssh_config#MACs): 
: `ssh -Q mac`

[KexAlgorithms](https://man.openbsd.org/ssh_config#KexAlgorithms): 
: `ssh -Q kex`

Most commonly old SSH software is the reason for those errors. They still support old and insecure methods, which are no longer supported by modern applications.

There are workarounds with the `-o` flag to set temporary options, but I am not too familiar with it.

`ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 user@10.10.10.10`


#### Connect without startup file <a href="#startup-file" id="startup-file">#</a>

This is not that common but there are ways to lock you out after changes to the startup files like `.bashrc`, `.profile`, and so on. You simply can avoid loading those profile files with the following command.

`ssh -t user@host bash --norc --noprofile`


#### Handling SSH sessions with escape sequences <a href="#escape-sequence" id="escape-sequence">#</a>

SSH provides some **escape sequences** with which you can kill the session on the client. 

```markdown
Supported escape sequences:
 ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~C   - open a command line
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)
```

**Side note**: Start with a `RETURN` and keep `SHIFT` pressed while typing `~` and e.x. `?` to get this message. This depends on your keyboard layout. 

You can send the sequence through one or more **SSH tunnel** by adding `~` in front of the sequence. 

---