Hermes 0.3.2 issues after testing with Gaia manager

[adizere]

Crate

mostly ibc-relayer and ibc-relayer-cli

Summary of Bug

This is a collection of issues uncovered with @brapse while getting to understand gm (ref: informalsystems/ibc-rs#928).

• start-multi panics if all chains are unreachable
• BUG: client worker has incorrect (reversed) parameters

Version

0.3.2

Steps to Reproduce

start-multi panics if all chains are unreachable

hermes start-multi dies if no chain is running, with the following panic:

May 21 11:56:37.874 ERROR ibc_relayer::supervisor: failed to spawn chain runtime for network1: RPC error to endpoint http://localhost:27000/: error trying to connect: tcp connect error: Connection refused (os error 61) (code: 0)
...
May 21 11:56:37.896 ERROR ibc_relayer::supervisor: skipping workers for chain id network1. reason: failed to spawn chain runtime with error: RPC error to endpoint http://localhost:27000/: error trying to connect: tcp connect error: Connection refused (os error 61) (code: 0)
...
The application panicked (crashed).
Message:  no operations have been added to `Select`
Location: /Users/adi/.cargo/registry/src/github.yoyoweb123.workers.dev-1ecc6299db9ec823/crossbeam-channel-0.5.1/src/select.rs:466

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

A more helpful way would be to exit is to signal the error:

...
Error: supervisor was not able to connect to any chain

BUG: client worker has incorrect (reversed) parameters

The client worker that Hermes creates upon observing the Channel Open Ack or Confirm event has the wrong parameters.

To reproduce:

• start three chains with no prior state; assume chain identifiers are network1, network2, and network3.
• create three clients hosted on network1
  • hermes create client network1 network3
  • hermes create client network1 network3
  • hermes create client network1 network3
• now create two clients hosted on network2
  • hermes create client network2 network3
  • hermes create client network2 network3

This finishes the setup phase. Setting up these multiple clients will help clarify which client belongs to which network in the following steps.

• now run hermes start-multi from one terminal
• from a separate terminal, we will create a channel (with new clients and connection) linking network1 <> network2
  • hermes create channel network1 network2 --port-a transfer --port-b transfer

At this point, we go back to the first terminal (where start-multi is running) and observe the output. Hermes should eventually pick up the OpenAckChannelEv event, which network1 emits, as follows:

May 21 15:09:32.433 DEBUG ibc_relayer::supervisor: chain network1 sent events:
OpenAckChannelEv(OpenAck(Attributes { height: Height { revision: 0, height: 39 }, port_id: PortId("transfer"), channel_id: Some(ChannelId("channel-0")), connection_id: ConnectionId("connection-0"), counterparty_port_id: PortId("transfer"), counterparty_channel_id: Some(ChannelId("channel-0")) }))
for object Client(Client { dst_chain_id: ChainId { id: "network1", version: 0 }, dst_client_id: ClientId("07-tendermint-3"), src_chain_id: ChainId { id: "network2", version: 0 } })

This log line says that a client worker will be spawned to handle updating the client with id 07-tendermint-3 hosted on chain network1 with headers from network2. We should be able to observe the bug soon after this, which manifests in the following weird log output:

May 21 15:09:32.658 WARN worker loop{worker=network2->network1:07-tendermint-3}: ibc_relayer::foreign_client: [network1 -> network2:07-tendermint-3] misbehaviour checking result Misbehaviour("failed querying client state on dst chain 07-tendermint-3 with error: Query error occurred (failed to query for client state): error converting message type into domain type: the client state was not found")

The network1 -> network2:07-tendermint-3 bit suggests that the client worker has been instantiated with a ForeignClient that is hosted on chain network2 (this is called the destination chain) and has identifier 07-tendermint-3 and is targeting the source chain network1. But client 07-tendermint-3 has reverse parameters: it is hosted on chain network1 and targets network2.

The same bug manifests for the client worker in the direction network2 -> network1. In this case, the wrong client is being used again.

May 21 15:09:39.399 DEBUG worker loop{worker=network1->network2:07-tendermint-2}: ibc_relayer::foreign_client: [network2 -> network1:07-tendermint-2] checking misbehaviour at 0-0, number of consensus states 1

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ancazamfir]

uhhh! nice catch with the client worker! The issue is that we restore the ForeignClient with the chains flipped.

[ancazamfir]

start-multi panics if all chains are unreachable

could you give more detail? In my case when I start hermes without any chain running I see the error messages but no panic. Not sure how to reproduce this (trying to review/ test #974)

[adizere]

start-multi panics if all chains are unreachable

could you give more detail? In my case when I start hermes without any chain running I see the error messages but no panic. Not sure how to reproduce this (trying to review/ test #974)

This is weird. I am actually no longer able to reproduce the panic. If I do hermes start with current master (1f498e3), then the executable hangs, with the last error message being:

May 25 10:47:55.820 ERROR ibc_relayer::supervisor: skipping workers for chain id network5. reason: failed to spawn chain runtime with error: RPC error to endpoint http://localhost:27040/: error trying to connect: tcp connect error: Connection refused (os error 61) (code: 0)

It's important that no gaia is running prior to invoking hermes start.

I think this should still be fixed and the executable should quit (instead of hanging on forever) when no chain is reachable.