504 Gateway Timeout error

My mail server works fine most of the time. However, sometimes, it throws 504 Gateway Timeout error in the browser when my users try to open their mailbox via Roundcube. In the nginx logs, I get the following upstream error.
"upstream timed out (110: Connection timed out) while reading response header from upstream"What can be the possible solution for this? I think I get this error when the number of connections increase. Can this be the reason behind this error?

Is this happening when backups are running, if so then that’s normal.