[bug] internal server error on publish

[hetao29]

Describe the bug
on Publish message, some time error: internal server error ,and code is 100, and it recover after 1-3 minis later.
I have searched the issue, Like:
#643

I think the reason mayby is:

1. centrifugo config allow history
2. redis drop some key by allkey-lfu config

maxmemory 500M
maxmemory-policy allkeys-lfu

3. and publish the message and add it to history happen error.

thanks.

Versions

Centrifugo version is 5.4.5
Operating system is ubuntu 22.04 , docker centrifugo/centrifugo:v5.4.5

** config below:

{
	"api_key": "xx",
	"admin_password": "xx",
	"admin_secret": "xx",
	"engine": "redis",
	"redis_cluster_address":[
		"xxx"
	],
	"token_hmac_secret_key": "xx",
	"port":8000,
	"ping_interval":"20s",
	"pong_timeout":"5s",
	"redis_prefix": "centrifugov5.1",
	"uni_http_stream":true,
	"namespaces": [
		{
			"name": "aa",
			"presence": true,
			"join_leave":true,
			"history_size": 50,
			"history_ttl": "300s",
			"force_push_join_leave": true,
			"allow_subscribe_for_anonymous":true,
			"allow_subscribe_for_client":true
		}
	],
	"allowed_origins": [
		"*"
	]
}

[FZambia]

Hello @hetao29

I think logs should contain more specific error returned from Redis, not just internal error exposed to the outside.

Generally you can't use Centrifugo history with Redis LRU policies, because it may bring unexpected side effects to message offset logic upon key eviction.

Though I'd like to look at exact error that is happening in this particular case.

[hetao29]

Hello @hetao29

I think logs should contain more specific error returned from Redis, not just internal error exposed to the outside.

Generally you can't use Centrifugo history with Redis LRU policies, because it may bring unexpected side effects to message offset logic upon key eviction.

Though I'd like to look at exact error that is happening in this particular case.

It was a problem in the production environment. I took a rough look at the centrifugo and redis logs and found no obvious error logs; the initial estimate is that it is related to history;
Now the configuration file has deleted the history-related configuration;
But what I think is that if it is combined with redis lru, it may be better to have better fault tolerance;

[FZambia]

Would be nice to see the exact error. Will try to reproduce this myself at some point, and think whether it may be improved somehow. There is an idea to add an option to only use one key for the history instead of two. If possible, it will provide a bit worse characteristics for the recovery in some edge cases, but should be more friendly to cache eviction.

[FZambia]

I reproduced the scenario manually deleting history meta key from Redis, the error on publish after that is:

The ID specified in XADD is equal or smaller than the target stream top item

It happens because Centrifugo tries to add a message to a stream with an offset number lower than the latest offset of message in the stream. I.e. because meta key was evicted we use offset 1 for a new message assuming it's a new stream, but stream key can contain messages with larger offsets at this point.

The error lasts for the time until history stream key expires (up to history_ttl). Then stream disappears, and the error goes away.

I tried to evaluate the solution of only having one key (STREAM key) – unfortunately it seems not viable, we still need meta key because it allows us to have stream meta information when we have subscriber but we do not have messages in a stream (in channel) yet.

But! It seems totally possible to handle this scenario inside LUA, already opened PR with one possible approach, but will try to experiment with a couple of alternative fixes selecting the most performant one. With the fix – there will be no publish error for the time of history TTL, instead - we will delete existing stream emulating its eviction. But it will also mean that clients which rely on Centrifugo message recovery will be disconnected with an insufficient state reason (because a stream is evicted) and will have a chance to recover state from the main application database.

This means that as soon as your Redis setup reaches memory limit the system will work but degrade regarding continuity of stream positions for older streams. From Centrifugo behaviour perspective it seems expected consequence, but generally it will be still recommended to avoid LRU/LFU evictions because it results in more resubscribes and state recovery from the application database.

[FZambia]

The fix described above is released in https://github.com/centrifugal/centrifugo/releases/tag/v5.4.6