Upterm rejecting all ssh connections

[larsks]

I was trying to spin up my first upterm session; after running upterm host -- bash I am presented with the expected ssh session information, but when I attempt to connect to that session from another system, I get a "permission denied" error:

[lars@madhatter ~]$ ssh hGjlbiMJqi0M9Jjo4SSo:...@uptermd.upterm.dev
hGjlbiMJqi0M9Jjo4SSo:...@uptermd.upterm.dev: Permission denied (publickey).

I haven't included any --authorized-key argument in my upterm host command so I expected any ssh client to be able to connect.

[Meister1593]

Same issue, me and my friend added authorised keys from eachother and it still didn't work

[owenthereal]

Can you try again? I found some issue with the uptermd community server and it's now fixed.

[larsks]

@owenthereal with the new 0.6.8 release I still see the same behavior. In one window:

$ upterm version
Upterm version v0.6.8
$ upterm host
=== F00BAAKRQQ5KSKK7O60NF
Command:                /bin/bash
Force Command:          n/a
Host:                   ssh://uptermd.upterm.dev:22
SSH Session:            ssh F00BAAkRqq5KSkk7O60nF:MITMONO0LjAuNjM6MjI=@uptermd.upterm.dev
Press <q> or <ctrl-c> to continue...

And in another window:

$ ssh F00BAAkRqq5KSkk7O60nF:MITMONO0LjAuNjM6MjI=@uptermd.upterm.dev
F00BAAkRqq5KSkk7O60nF:MITMONO0LjAuNjM6MjI=@uptermd.upterm.dev: Permission denied (publickey).

(And I see the same behavior when I assign an explicit authorized keys file with --authorized-key option)

[owenthereal]

Hmm...I could not reproduce this. Can you try the ssh command with the -vvv flag, e.g. ssh F00BAAkRqq5KSkk7O60nF:MITMONO0LjAuNjM6MjI=@uptermd.upterm.dev -vvv and paste the result here (note that you may want to obfusticate info that is confidential). I suspect this is related to your local ssh config.

[Meister1593]

I personally can't connect, it just stays at infinite connection (0.6.8 version, 0.6.7 has the same thing rn)

debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50

last lines from the ssh connection output, and it just stays like that, never connects, then errors on

Connection closed by 157.230.199.75 port 22

Also, how can i disable pubkey auth for upterm? it asks my key pass to authorise with it on host (almost like if i was connecting to myself), and i don't want pubkey auth, i just want to share it (or at least share it with password at max)

[larsks]

Can you try the ssh command with the -vvv flag, e.g. ssh
F00BAAkRqq5KSkk7O60nF:MITMONO0LjAuNjM6MjI=@uptermd.upterm.dev -vvv
and paste the result here (note that you may want to obfusticate
info that is confidential). I suspect this is related to your local
ssh config

Connecting to uptermd.upterm.dev with ssh -vv:

$ ssh W3JPuwujNLwdjp9fm8PO:MTAuMjQ0LjAuMjI4OjIy@uptermd.upterm.dev
OpenSSH_8.7p1, OpenSSL 1.1.1l  FIPS 24 Aug 2021
debug1: Reading configuration data /home/lars/.ssh/config
debug1: Reading configuration data /home/lars/.ssh/config.d/abc_config
debug1: Reading configuration data /home/lars/.ssh/config.d/house_config
debug1: Reading configuration data /home/lars/.ssh/config.d/def_config
debug1: Reading configuration data /home/lars/.ssh/config.d/mycompany_config
debug1: /home/lars/.ssh/config line 54: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-mycompany.conf
debug2: checking match for 'final all' host uptermd.upterm.dev originally uptermd.upterm.dev
debug2: match not found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug2: resolve_addr: could not resolve name uptermd.upterm.dev as address: Name or service not known
debug1: re-parsing configuration
debug1: Reading configuration data /home/lars/.ssh/config
debug1: Reading configuration data /home/lars/.ssh/config.d/abc_config
debug1: Reading configuration data /home/lars/.ssh/config.d/house_config
debug1: Reading configuration data /home/lars/.ssh/config.d/def_config
debug1: Reading configuration data /home/lars/.ssh/config.d/mycompany_config
debug1: /home/lars/.ssh/config line 54: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-mycompany.conf
debug2: checking match for 'final all' host uptermd.upterm.dev originally uptermd.upterm.dev
debug2: match found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug2: resolving "uptermd.upterm.dev" port 22
debug1: Connecting to uptermd.upterm.dev [157.230.199.75] port 22.
debug1: Connection established.
debug1: identity file /home/lars/.ssh/id_rsa type 0
debug1: identity file /home/lars/.ssh/id_rsa-cert type -1
debug1: identity file /home/lars/.ssh/id_dsa type -1
debug1: identity file /home/lars/.ssh/id_dsa-cert type -1
debug1: identity file /home/lars/.ssh/id_ecdsa type -1
debug1: identity file /home/lars/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/lars/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/lars/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/lars/.ssh/id_ed25519 type -1
debug1: identity file /home/lars/.ssh/id_ed25519-cert type -1
debug1: identity file /home/lars/.ssh/id_ed25519_sk type -1
debug1: identity file /home/lars/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/lars/.ssh/id_xmss type -1
debug1: identity file /home/lars/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version uptermd
debug1: compat_banner: no match: uptermd
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to uptermd.upterm.dev:22 as 'W3JPuwujNLwdjp9fm8PO:MTAuMjQ0LjAuMjI4OjIy'
debug1: load_hostkeys: fopen /home/lars/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com
debug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:...
debug2: Server host certificate hostname: uptermd.upterm.dev
debug1: load_hostkeys: fopen /home/lars/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'uptermd.upterm.dev' is known and matches the ED25519-CERT host certificate.
debug1: Found CA key in /home/lars/.ssh/known_hosts:390
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/lars/.ssh/id_rsa_myhobby RSA SHA256:...
debug1: Will attempt key: /home/lars/.ssh/id_rsa_mycompany RSA SHA256:...
debug1: Will attempt key: /home/lars/.ssh/id_rsa RSA SHA256:...
debug1: Will attempt key: /home/lars/.ssh/id_dsa
debug1: Will attempt key: /home/lars/.ssh/id_ecdsa
debug1: Will attempt key: /home/lars/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/lars/.ssh/id_ed25519
debug1: Will attempt key: /home/lars/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/lars/.ssh/id_xmss
debug2: pubkey_prepare: done
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/lars/.ssh/id_rsa_myhobby RSA SHA256:...
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: /home/lars/.ssh/id_rsa_mycompany RSA SHA256:...
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: /home/lars/.ssh/id_rsa RSA SHA256:...
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Trying private key: /home/lars/.ssh/id_dsa
debug1: Trying private key: /home/lars/.ssh/id_ecdsa
debug1: Trying private key: /home/lars/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/lars/.ssh/id_ed25519
debug1: Trying private key: /home/lars/.ssh/id_ed25519_sk
debug1: Trying private key: /home/lars/.ssh/id_xmss
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
W3JPuwujNLwdjp9fm8PO:MTAuMjQ0LjAuMjI4OjIy@uptermd.upterm.dev: Permission denied (publickey).

Compare that to this connection to a tmate server:

$ ssh XC4DwPpRN6mKS6PQn9bfUcktr@tmate.example.com
OpenSSH_8.7p1, OpenSSL 1.1.1l  FIPS 24 Aug 2021
debug1: Reading configuration data /home/lars/.ssh/config
debug1: Reading configuration data /home/lars/.ssh/config.d/abc_config
debug1: Reading configuration data /home/lars/.ssh/config.d/house_config
debug1: Reading configuration data /home/lars/.ssh/config.d/def_config
debug1: Reading configuration data /home/lars/.ssh/config.d/mycompany_config
debug1: /home/lars/.ssh/config line 54: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-mycompany.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/lars/.ssh/config
debug1: Reading configuration data /home/lars/.ssh/config.d/abc_config
debug1: Reading configuration data /home/lars/.ssh/config.d/house_config
debug1: Reading configuration data /home/lars/.ssh/config.d/def_config
debug1: Reading configuration data /home/lars/.ssh/config.d/mycompany_config
debug1: /home/lars/.ssh/config line 54: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-mycompany.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to tmate.example.com [123.45.67.89] port 22.
debug1: Connection established.
debug1: identity file /home/lars/.ssh/id_rsa type 0
debug1: identity file /home/lars/.ssh/id_rsa-cert type -1
debug1: identity file /home/lars/.ssh/id_dsa type -1
debug1: identity file /home/lars/.ssh/id_dsa-cert type -1
debug1: identity file /home/lars/.ssh/id_ecdsa type -1
debug1: identity file /home/lars/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/lars/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/lars/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/lars/.ssh/id_ed25519 type -1
debug1: identity file /home/lars/.ssh/id_ed25519-cert type -1
debug1: identity file /home/lars/.ssh/id_ed25519_sk type -1
debug1: identity file /home/lars/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/lars/.ssh/id_xmss type -1
debug1: identity file /home/lars/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version tmate
debug1: compat_banner: no match: tmate
debug1: Authenticating to tmate.example.com:22 as 'XC4DwPpRN6mKS6PQn9bfUcktr'
debug1: load_hostkeys: fopen /home/lars/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:...
debug1: load_hostkeys: fopen /home/lars/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'tmate.example.com' is known and matches the RSA host key.
debug1: Found key in /home/lars/.ssh/known_hosts:629
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/lars/.ssh/id_rsa_myhobby RSA SHA256:...
debug1: Will attempt key: /home/lars/.ssh/id_rsa_mycompany RSA SHA256:...
debug1: Will attempt key: /home/lars/.ssh/id_rsa RSA SHA256:...
debug1: Will attempt key: /home/lars/.ssh/id_dsa 
debug1: Will attempt key: /home/lars/.ssh/id_ecdsa 
debug1: Will attempt key: /home/lars/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/lars/.ssh/id_ed25519 
debug1: Will attempt key: /home/lars/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/lars/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Enabling compression at level 6.
Authenticated to tmate.example.com ([123.45.67.89]:22) using "none".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: filesystem full
debug1: Requesting authentication agent forwarding.
debug1: Sending environment.
debug1: channel 0: setting env XMODIFIERS = "@im=ibus"
debug1: channel 0: setting env LANG = "en_US.UTF-8"

[owenthereal]

Did you press q or ctrl-c to accept connection in your host? Notice the hint:

Press or to continue...

[larsks]

I did, yes. If you don't press q or ctrl-c, the connection blocks. It's not rejected until after you activate the terminal.

[owenthereal]

I finally can reproduce it. It happens when I don't have the key in the ssh-agent. I will see where the problem is, fix it and report back. It's likely a recent change that broke it

[owenthereal]

Okay, the root of the issue is that since openssh 8.8 (2021-09-26), the host algorithm type ssh-rsa was retired in favor of rsa-sha2-256 & rsa-sha2-512 (release note). This breaks a lot of ssh servers, including upterm. The Go issue that tracks the fixes of the Go ssh library is here. I will closely monitor it and pull in changes. Can you do a quick check of your openssh version with:

$ ssh -V

What happens is that you only have a rsa key in your ~/.ssh, and upterm is not ready to handle the openssh 8.8+ client that deprecates the ssh-rsa host algorithm type. However, openssh 8.8+ using other key types work with upterm, e.g. Ed25519. For now, you can workaround by adding the ssh-rsa type back:

ssh UPTERM_URL -o "PubkeyAcceptedKeyTypes +ssh-rsa" -o "HostKeyAlgorithms +ssh-rsa"

You can also put in your ~/.ssh/config with the following to save you from typing the -o flags for every Upterm ssh command:

Host uptermd.upterm.dev
    PubkeyAcceptedAlgorithms +ssh-rsa
    HostkeyAlgorithms +ssh-rsa

I would strongly recommend you to upgrade all your ssh keys to Ed25519 because that's the most secured ssh algorithm that is recommended to use over others: https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54.

I will leave this issue open until upterm with the fixed Go ssh library. Hope this helps

[larsks]

Thanks, that was it. Seems to be working fine now!