Roundcube "Connection to storage server failed." login error under v63

Maintenance
1

Symptoms: When trying to log into RoundCube mail client, I get the error message: Connection to storage server failed. I also can’t log into the MiaB admin pages (get Something went wrong, sorry. message). And, email clients are not showing new mail.

The server reports in /var/log/roundcubemail/errors.log:

[19-Aug-2023 18:02:56 America/New_York] PHP Warning:  stream_socket_client(): Unable to connect to ssl://localhost:993 (Connection refused) in /usr/local/lib/roundcubemail/program/lib/Roundcube/rcube_imap_generic.php on line 1059
[19-Aug-2023 18:02:56 -0400]: <f4qogtan> IMAP Error: Login failed for USER@DOMAIN.com against localhost from REMOTE_IP. Could not connect to ssl://localhost:993: Connection refused in /usr/local/lib/roundcubemail/program/lib/Roundcube/rcube_imap.php on line 211 (POST /mail/?_task=login&_action=login)

The server reports in /var/log/roundcubemail/carddav_http.log:

cURL error 60: SSL certificate problem: self-signed certificate (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://DOMAIN.com/cloud/remote.php/dav/addressbooks/users/USER@DOMAIN.com/contacts/

The output of openssl s_client -connect localhost:993 is:

40F7386A317F0000:error:8000006F:system library:BIO_connect:Connection refused:../crypto/bio/bio_sock2.c:125:calling connect()
40F7386A317F0000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:127:
40F7386A317F0000:error:8000006F:system library:BIO_connect:Connection refused:../crypto/bio/bio_sock2.c:125:calling connect()
40F7386A317F0000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:127:
connect:errno=111

I executed ~/mailinabox/management/status_checks.py and discovered this:

✖  Dovecot LMTP LDA is not running (port 10026).
✖  Mail-in-a-Box Management Daemon is not running (port 10222).
✖  IMAPS (dovecot) is not running (port 993).
✖  Mail Filters (Sieve/dovecot) is not running (port 4190).

This is the second time that I find the system in this state, all since I upgraded to v63. In all my many years of running MiaB, I’ve never seen this before. Any ideas of what’s going on? The initial message is confusing, as there are no “storage server” (i.e., disk) issues at all.

After rebooting the server everything looks fine, including the “openssl s_client -connect localhost:993” command suggestion from @KiekerJan in the Can't Access Box: Connection to Storage Server Failed After Upgrade - #22 by jtwooton thread.

Noteworthy is that the server appears under hacking attacks via both ssh and IMAP connections.

2

It looks like for some reason the Dovecot and the MIAB daemon process failed. (Roundcube uses the IMAP server dovecot as its storage server, which might be confusingly named :wink: )
You might get some additional info by looking at:

sudo journalctl -u dovecot
sudo journalctl -u mailinabox

Not sure what you mean by hacking attempts, but I note that my server is also always being scanned, portmapped or some bot is trying to login. I guess that’s the “internet weather” nowadays. If you have SSH properly secured (no password, login with key file) and have a decent password on IMAP, you should be fine. There is a process called fail2ban running on the system. This looks at failed login attempts and the like, and blocks the IP address if it happens too often in a short period of time.

3

journalctl -u dovecot

Aug 19 03:54:54 DOMAIN.com systemd[1]: dovecot.service: A process of this unit has been killed by the OOM killer.
Aug 19 03:55:00 DOMAIN.com systemd[1]: dovecot.service: Failed with result 'oom-kill'.
Aug 19 03:55:00 DOMAIN.com systemd[1]: dovecot.service: Consumed 20min 37.978s CPU time.

journalctl -u mailinabox

Aug 19 03:48:16 DOMAIN.com systemd[1]: mailinabox.service: A process of this unit has been killed by the OOM killer.
Aug 19 03:48:16 DOMAIN.com start[2055]: [2023-08-19 03:48:16 -0400] [2055] [ERROR] Worker (pid:2076) was sent SIGKILL! Perhaps out of memory?
Aug 19 03:48:16 DOMAIN.com start[1357962]: [2023-08-19 03:48:16 -0400] [1357962] [INFO] Booting worker with pid: 1357962
Aug 19 03:48:16 DOMAIN.com start[2055]: [2023-08-19 03:48:16 -0400] [2055] [INFO] Handling signal: term
Aug 19 03:48:18 DOMAIN.com start[1357962]: [2023-08-19 03:48:18 -0400] [1357962] [INFO] Worker exiting (pid: 1357962)
Aug 19 03:48:20 DOMAIN.com start[2055]: [2023-08-19 03:48:20 -0400] [2055] [INFO] Shutting down: Master
Aug 19 03:48:20 DOMAIN.com systemd[1]: mailinabox.service: Failed with result 'oom-kill'.
Aug 19 03:48:20 DOMAIN.com systemd[1]: mailinabox.service: Consumed 2min 26.537s CPU time.

So, WHAT caused the problem appears pretty straightforward. The question is WHY did it run into this problem? The system typically runs with over 50% available memory. I did see a huge amount of IMAP and SSH connections at the time - definitely looked like a hacking / break-in attempt (trying numerous non-existent accounts in both). Maybe those are related? Regardless, pre-v63 MiaB never exhibited this problem.

I’ve added a watchdog to restart the processes, and the watchdog triggered again this morning. I see a definite before-and-after behavior with v63 - never, ever saw this pre-v63.

Yes, I’ve been leveraging fail2ban information - very insightful.

4

Of course those attempts might be connected. You could perhaps monitor the memory size of the ssh server or the dovecot process, but perhaps it’s better to simply monitor all process memory size for a while?
I also remember that there was an issue here somewhere with munin (not sure) that caused runaway memory usage, but I can’t find it. :frowning:
Finally, maybe the upgrade to v63 caused more memory use, and you just crossed the limit. How much memory do you have in the box?

5

Thanks. I’m now monitoring for and restart if stopped: mailinabox (port 10222), and dovecot (ports 993, 4190, and 10026). Am I missing any others to include in the monitor? The script already caught another event and was able to recover by restarting the failed service.

The server has 2GB and does not run anything other than MiaB. Typically, the server runs at a very steady 50% memory capacity and, under normal conditions, doesn’t exceed 60% memory utilization (which seems to happen every 30 minutes). Similarly, CPU is typically 9% with a max of 15% (which also seems to happen every 30 minutes).

6

I got additional insight form the syslog:

Aug 22 06:46:10 box kernel: [205660.101516] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=do-agent.service,mems_allowed=0,global_oom,task_memcg=/system.slice/dovecot.service,task=imap,pid=294681,uid=8
Aug 22 06:46:11 box systemd[1]: dovecot.service: A process of this unit has been killed by the OOM killer.
Aug 22 06:46:13 box systemd[1]: dovecot.service: Failed with result 'oom-kill'.
Aug 22 06:46:13 box systemd[1]: dovecot.service: Consumed 8min 29.737s CPU time.
Aug 22 06:47:01 box dovecot[514723]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 22 06:47:01 box dovecot[514723]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 22 06:47:01 box dovecot[514722]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 22 06:47:01 box dovecot[514722]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 22 06:47:01 box dovecot[514722]: doveconf: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 22 06:47:01 box dovecot[514722]: doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 22 06:47:01 box dovecot[514722]: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 22 06:47:01 box dovecot[514722]: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 22 09:29:52 box dovecot[833]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 22 09:29:52 box dovecot[833]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 22 09:29:52 box dovecot[743]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 22 09:29:52 box dovecot[743]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 22 09:29:52 box dovecot[743]: doveconf: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 22 09:29:52 box dovecot[743]: doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 22 09:29:52 box dovecot[743]: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 22 09:29:52 box dovecot[743]: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 22 10:18:16 box dovecot[829]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 22 10:18:16 box dovecot[829]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 22 10:18:16 box dovecot[741]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 22 10:18:16 box dovecot[741]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 22 10:18:16 box dovecot[741]: doveconf: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 22 10:18:16 box dovecot[741]: doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 22 10:18:16 box dovecot[741]: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 22 10:18:16 box dovecot[741]: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 22 10:23:28 box dovecot[833]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 22 10:23:28 box dovecot[833]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 22 10:23:29 box dovecot[743]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 22 10:23:29 box dovecot[743]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 22 10:23:29 box dovecot[743]: doveconf: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 22 10:23:29 box dovecot[743]: doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 22 10:23:29 box dovecot[743]: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 22 10:23:29 box dovecot[743]: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 23 09:19:10 box kernel: [82561.521542] [    743]     0   743     2071      854    57344        0             0 dovecot
Aug 23 09:19:10 box kernel: [82561.522154] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/dovecot.service,task=imap,pid=1217,uid=8
Aug 23 09:19:10 box systemd[1]: dovecot.service: A process of this unit has been killed by the OOM killer.
Aug 23 09:19:15 box systemd[1]: dovecot.service: Failed with result 'oom-kill'.
Aug 23 09:19:15 box systemd[1]: dovecot.service: Consumed 3min 46.943s CPU time.
Aug 23 09:20:01 box dovecot[213551]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 23 09:20:01 box dovecot[213551]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 23 09:20:01 box dovecot[213550]: doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn > dovecot-new.conf
Aug 23 09:20:01 box dovecot[213550]: doveconf: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed
Aug 23 09:20:01 box dovecot[213550]: doveconf: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 23 09:20:01 box dovecot[213550]: doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
Aug 23 09:20:01 box dovecot[213550]: Warning: service auth { client_limit=1000 } is lower than required under max. load (1250). Counted for protocol services with service_count != 1: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service lmtp { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 }
Aug 23 09:20:01 box dovecot[213550]: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1003). Counted with: service managesieve-login { process_limit=250 } + service pop3-login { process_limit=250 } + service imap-urlauth-login { process_limit=250 } + service imap-login { process_limit=250 } + service auth { process_limit=1 }
  • The configured service limits for auth and anvil are too low for the protocol services and loads on the server. Any thoughts on (a) whether increasing these values would make the problem better or worse, and (b) how this manual change could impact MiaB?
  • There is an obsolete SSL setting that should be removed. MiaB should not be distributed with obsolete configuration parameters …

Definitely not happy that I upgraded to v63. Life was fine before v63.

7

I’m reading up on client_limit, but it’s not very straightforward to me (yet). At first glance it does have to do with number of processes and number of connections. So if you’re experiencing a lot of attacks, changing these might help you. Whether they should go up or down I don’t know yet.

You can remove the obsolete parameter if you’d like. I don’t think it hurts, but it’s not needed anymore. If you feel up to it you might even create a github pull request to fix that.

What version of MiaB did you use before? I don’t think this will be different for V62, but compared to the Ubuntu 18.04 version, a lot has changed.

8

My concern around modifying the client_limit (I asume in /etc/dovecot/conf.d/10-master.conf) is that by doing this maybe I’d be making a memory problem MUCH worse. Given an apparent memory issue, the client_limit may be serving as a throttle, whereas if I increase the client_limit maybe I’ll see the problem much, much earlier and frequently. If, indeed, this is related to a condition triggered by abuse which exposes a memory leak type of issue, there is no configuration that will resolve this.

I’m not 100% sure if I was running v61.1 (another server that I haven’t upgraded yet is running v61.1, so I’m inclined to think it was v61.1) or v62. I was trying to not let myself go without an upgrade for too long (the post-Ubuntu 18.04 upgrade was painful), but now I’m regretting going into v63. I had NONE of these issues pre-v63; the box was rock solid.

9

That is a valid concern, which is why one would first need to read up on it before changing it.

10

This topic was automatically closed 40 days after the last reply. New replies are no longer allowed.