Gzip: stdout: No space left on device

Maintenance
1

I’ve been getting this error message for the past few days:

cron for user root /usr/bin/run-parts /etc/cron.daily 1> /dev/null

error: Compressing program wrote following message to stderr when compressing log /var/log/nginx/error.log.1:

gzip: stdout: No space left on device
error: failed to compress log /var/log/nginx/error.log.1

However, I can’t figure out what’s causing the error, since there’s still enough free memory available:

df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              794M  1,2M  793M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  154G   25G  123G  17% /
tmpfs                              3,9G     0  3,9G   0% /dev/shm
tmpfs                              5,0M     0  5,0M   0% /run/lock
/dev/vda2                          2,0G  260M  1,6G  15% /boot
/dev/vda1                          1,1G  6,1M  1,1G   1% /boot/efi
tmpfs                              794M  4,0K  794M   1% /run/user/1000

What can I do here?

I’m running the current version V76.

2

Hello
isn’t it just the HDD system in read-only mode

3

This does not seem to be a Mail-in-a-Box specific issue.
Can you create new files manually in the /log folder?
What is the content of /etc/logrotate.d/nginx?
What is the output of df -i?

2 Likes
4

Can you create new files manually in the /log folder?

yes

What is the content of /etc/logrotate.d/nginx?

/var/log/nginx/*.log {
	daily
	missingok
	rotate 14
	compress
	delaycompress
	notifempty
	create 0640 www-data adm
	sharedscripts
	prerotate
		if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
			run-parts /etc/logrotate.d/httpd-prerotate; \
		fi \
	endscript
	postrotate
		invoke-rc.d nginx rotate >/dev/null 2>&1
	endscript
}

What is the output of df -i?

df -i
Filesystem                          Inodes  IUsed   IFree IUse% Mounted on
tmpfs                              1015461    900 1014561    1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 10289152 383294 9905858    4% /
tmpfs                              1015461      1 1015460    1% /dev/shm
tmpfs                              1015461      3 1015458    1% /run/lock
/dev/vda2                           131072    310  130762    1% /boot
/dev/vda1                                0      0       0     - /boot/efi
tmpfs                               203092     25  203067    1% /run/user/1000

isn’t it just the HDD system in read-only mode

Why would the permissions have changed?
Wouldn’t other errors occur in that case, or would it be impossible to create files?

5

“read-only” some OSes switch to “read-only” mode on disk error

mount | grep “ro,” ??

6 
mount | grep "ro,"
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/canonical-livepatch_406.snap on /snap/canonical-livepatch/406 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/canonical-livepatch_393.snap on /snap/canonical-livepatch/393 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core22_2339.snap on /snap/core22/2339 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_2769.snap on /snap/core20/2769 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core22_2411.snap on /snap/core22/2411 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core24_1587.snap on /snap/core24/1587 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_2866.snap on /snap/core20/2866 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core24_1643.snap on /snap/core24/1643 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/duplicity_811.snap on /snap/duplicity/811 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/lxd_40115.snap on /snap/lxd/40115 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/duplicity_865.snap on /snap/duplicity/865 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_27406.snap on /snap/snapd/27406 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/lxd_38800.snap on /snap/lxd/38800 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_27591.snap on /snap/snapd/27591 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)

What does this mean?

7

do you have local access? after rebooting it doesn’t ask for disk check? on other ubuntu it happened to me that it mounted a faulty disk for read only.
sometimes sudo fsck -y /dev/sdX1 helped me

8

now I tried findmnt -n -o OPTIONS /path/to/mount/point

is there RO or RW?

9

yes

I just rebooted the system and didn’t see any such notification.

What mount point should I specify here?

10

physical connection HDD /dev/sda1 for example

11 
findmnt -n -o OPTIONS /dev/dm-0
rw,relatime

That’s what I got.

12

okay

well that’s a different problem :frowning:

13

Did the reboot fix the issue? Otherwise, check access rights of /var/log/nginx On my system it’s drwxr-xr-x 2 root adm
Check the contents of /var/log/nginx maybe there’s a very big file there?

14

Unfortunately no.

ls -ld /var/log/nginx
drwxr-xr-x 2 root adm 4096 Jul 27 00:00 /var/log/nginx

/var/log/nginx$ du -h --max-depth=0 * | sort -hr
1,4M    access.log.1
1,3M    error.log.1
588K    error.log
548K    access.log
164K    access.log.14.gz
128K    access.log.7.gz
120K    access.log.2.gz
112K    error.log.14.gz
100K    error.log.2.gz
96K     access.log.5.gz
96K     access.log.11.gz
88K     error.log.7.gz
88K     access.log.4.gz
84K     access.log.13.gz
80K     access.log.12.gz
76K     access.log.9.gz
72K     access.log.8.gz
72K     access.log.3.gz
64K     error.log.5.gz
64K     error.log.11.gz
64K     access.log.6.gz
64K     access.log.10.gz
60K     error.log.4.gz
52K     error.log.12.gz
48K     error.log.9.gz
48K     error.log.8.gz
48K     error.log.3.gz
40K     error.log.13.gz
36K     error.log.6.gz
36K     error.log.10.gz

As far as I can see, there are no errors here.

15

So I’m not sure what to check next. Did a bit of googling and a check might be: are you sure /dev/mapper/ubuntu--vg-ubuntu--lv has enough space, even if it reports it has. Also, no quota on that device?

16

The physical hdd is using 22 of 160 GB.
There is no quota.
All other functions also appear to be unrestricted, and as far as I can tell, the mail server is working without any issues.
Nevertheless, I receive this error message via email every morning at 4:40 a.m.

17

The daily error message has changed since Monday. It now reads:

error: stat of /var/log/nginx/*.log failed: No such file or directory
error: stat of /var/log/syslog failed: No such file or directory

But the files are there.
Could this be a permissions issue after all?

Currently, the permissions are as follows:

ls -ld /var/log/nginx/error.log.1
-rw-r----- 1 www-data adm 971477 Jul 29 23:49 /var/log/nginx/error.log.1

ls -ld /var/log/syslog
-rw-r----- 1 syslog adm 8872352 Jul 30 11:02 /var/log/syslog

Are the permissions set correctly?

18

MY
-rw-r----- 1 syslog adm 12138091 čec 30 13:20 /var/log/syslog
-rw-r----- 1 www-data adm 5421549 čec 30 12:07 /var/log/nginx/error.log.1

19

I would try to see if it is possible to manually create or copy a test file in /var/log/syslog?

20

Even though I don’t fully understand the details, the error no longer seems to be occurring