[Bug 254210] 13.0-RC1: jail: nullfs: deleted files does not free up space

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

Kyle Evans <***@freebsd.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org,
| |***@freebsd.org

--- Comment #1 from Kyle Evans <***@freebsd.org> ---
Let's loop in fs as well, since this is at the intersection of jails and
filesystems.
--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
freebsd-***@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-***@freebsd.org"

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

Konstantin Belousov <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

--- Comment #2 from Konstantin Belousov <***@FreeBSD.org> ---
Add "nocache" option to both nullfs mounts.
--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
freebsd-***@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-***@freebsd.org"

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

--- Comment #3 from ***@superbox.pl ---
(In reply to Konstantin Belousov from comment #2)

It is not documented and mount -t nullfs does not indicate that the option is
used but it work.

One of the jails is working as file server and there is a lot of read
operations.

Should I worry about hard disk head contention?
--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
freebsd-***@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-***@freebsd.org"

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

--- Comment #7 from Christos Chatzaras <***@cretaforce.gr> ---
FreeBSD 13.2:

The same thing happens when I use a program to optimise images (the images size
shrinks but ZFS shows more usage than before). I have to restart jails to get
the correct usage.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

Mark Linimon <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

--- Comment #8 from Mark Linimon <***@FreeBSD.org> ---
^Triage: is this still a problem on supported versions of FreeBSD?

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

--- Comment #9 from ***@superbox.pl ---
Yes, on 14.1-RELEASE at least.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

Danilo G. Baio <***@freebsd.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@freebsd.org
Summary|13.0-RC1: jail: nullfs: |jail: nullfs: deleted files
|deleted files does not free |does not free up space
|up space |
Version|13.0-STABLE |14.2-STABLE

--- Comment #10 from Danilo G. Baio <***@freebsd.org> ---
Hi,

I can confirm this is still happening on a recent 14.2-STABLE.

Regards,

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

Ronald Klop <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

--- Comment #11 from Ronald Klop <***@FreeBSD.org> ---
The jails are not needed. It looks purely nullfs.

# zfs create zroot/data/pr-test
# cd /data/pr-test
# mkdir storage test1 test2
# mount_nullfs -o rw,noatime ./storage ./test1
# mount_nullfs -o rw,noatime ./storage ./test2
# df -h | grep "pr-test"
zroot/data/pr-test 327G 112K 327G 0% /data/pr-test
/data/pr-test/storage 327G 112K 327G 0%
/data/pr-test/test1
/data/pr-test/storage 327G 112K 327G 0%
/data/pr-test/test2

# dd if=/dev/random of=./test1/random.dd bs=1M count=1024
# df -h | grep "pr-test"
zroot/data/pr-test 327G 1.0G 326G 0% /data/pr-test
/data/pr-test/storage 327G 1.0G 326G 0%
/data/pr-test/test1
/data/pr-test/storage 327G 1.0G 326G 0%
/data/pr-test/test2

# rm ./test2/random.dd
# df -h | grep "pr-test"
zroot/data/pr-test 327G 1.0G 326G 0% /data/pr-test
/data/pr-test/storage 327G 1.0G 326G 0%
/data/pr-test/test1
/data/pr-test/storage 327G 1.0G 326G 0%
/data/pr-test/test2

Unmount of test1 releases the blocks.

NB: mount_nullfs option nocache does not help for me.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

Konstantin Belousov <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

--- Comment #12 from Konstantin Belousov <***@FreeBSD.org> ---
Try https://reviews.freebsd.org/D48825
(I did not checked it)

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

--- Comment #13 from commit-***@FreeBSD.org ---
A commit in branch main references this bug:

URL:
https://cgit.FreeBSD.org/src/commit/?id=1a0cb938f7b461edc40316cb44a80438df674174

commit 1a0cb938f7b461edc40316cb44a80438df674174
Author: Konstantin Belousov <***@FreeBSD.org>
AuthorDate: 2025-02-04 00:23:05 +0000
Commit: Konstantin Belousov <***@FreeBSD.org>
CommitDate: 2025-02-04 22:19:50 +0000

unlink, rmdir: call notify_upper from VOP pre method instead of syscall

Suppose that there are two or more nullfs mounts over some fs, and
suppose the we unlink a file on one of the nullfs mount.
This way notify_upper get called from the lower vnode as well, allowing
the other nullfs mounts to note that and drop their caches for the
unlinked vnode.

PR: 254210
Reviewed by: olce
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D48825

sys/kern/vfs_subr.c | 2 ++
sys/kern/vfs_syscalls.c | 2 --
2 files changed, 2 insertions(+), 2 deletions(-)

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

Konstantin Belousov <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|Open |In Progress

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

--- Comment #14 from Ronald Klop <***@FreeBSD.org> ---
With the commit of comment #13 installed I can't reproduce the test case of
comment #11.

# uname -a
FreeBSD rpi4 15.0-CURRENT FreeBSD 15.0-CURRENT #20 main-123911af5e63-dirty: Sat
Feb 8 13:51:07 CET 2025
***@rpi4:/data/ronald/freebsd/obj/data/ronald/freebsd/src/main/arm64.aarch64/sys/GENERIC-NODEBUG
arm64

👍

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

--- Comment #15 from commit-***@FreeBSD.org ---
A commit in branch stable/14 references this bug:

URL:
https://cgit.FreeBSD.org/src/commit/?id=cba91bf59c79eadc3b60b83fe2185116496f08fb

commit cba91bf59c79eadc3b60b83fe2185116496f08fb
Author: Konstantin Belousov <***@FreeBSD.org>
AuthorDate: 2025-02-04 00:23:05 +0000
Commit: Konstantin Belousov <***@FreeBSD.org>
CommitDate: 2025-02-11 01:11:49 +0000

unlink, rmdir: call notify_upper from VOP pre method instead of syscall

PR: 254210

(cherry picked from commit 1a0cb938f7b461edc40316cb44a80438df674174)

sys/kern/vfs_subr.c | 2 ++
sys/kern/vfs_syscalls.c | 2 --
2 files changed, 2 insertions(+), 2 deletions(-)

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

Konstantin Belousov <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|In Progress |Closed
Resolution|--- |FIXED

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

--- Comment #16 from ***@rlwinm.de ---
Any chance to get this into the next errata patches?

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254210

--- Comment #17 from Danilo G. Baio <***@freebsd.org> ---
Hi,

Unfortunately, we are still facing this issue in the FreeBSD infra.
Two jails are involved, gitmir and cgit, plus ZFS.
We are using 14-STABLE with this specific update.
The gitmir dataset usually consumes only about 5GB.

$ uname -a
FreeBSD xxx.yyy.freebsd.org 14.2-STABLE FreeBSD 14.2-STABLE #0
stable/14-n270391-cba91bf59c79: Tue Feb 11 01:59:07 UTC 2025
***@build-14.freebsd.org:/usr/obj/usr/src/sys/CLUSTER14 amd64

$ cat /etc/fstab.cgit
/j/jails/gitmir/home/git/ /j/jails/cgit/home/git/ nullfs ro 0 0

zroot/j/jails/gitmir 440G 296G 440G /j/jails/gitmir

# /etc/rc.d/jail restart cgit

zroot/j/jails/gitmir 5.51G 731G 5.51G /j/jails/gitmir