New partition sizes in default recipes for d-i

[Holger Wansing]

Hi Pascal,

since your work from
https://salsa.debian.org/installer-team/partman-auto/-/merge_requests/15
and
https://salsa.debian.org/installer-team/partman-auto-lvm/-/merge_requests/5
has landed in the daily images, I did some testing with different disk sizes;
the results can be found in
https://people.debian.org/~holgerw/partman-auto__with-MR15-merged/
separated by BIOS and UEFI.

While it works fine so far, on glitch came up during testing:

Using the "separate home+var+tmp" recipe under UEFI variant on a 10G disk
results in an error

"Unable to satisfy all constraints on the partition"
"Failed to partition the selected disk. This probably happened because there
are too many (primary) partitions in the partition table."

See
Loading Image...
Loading Image...
Loading Image...

In the partitioning overview (screenshot 3) the /home partition is
missing, so this is the one which raises the error apparently.

However, since this is UEFI, and therefore gpt partition table, the amount
of partitions (here: 6) should be no issue here.
Since I was unable to reproduce this issue with d-i images from before these
partman-auto changings, it's somehow related to this recipe changes, and
the message regarding "too many partitions" is wrong.

As this only happens with a very small disk (10G):
Do we need to adjust the minimum disk size for this recipe maybe, or similar?



Holger

[Pascal Hambourg]

Hello Holger,

Post by Holger Wansing
Using the "separate home+var+tmp" recipe under UEFI variant on a 10G disk
results in an error
"Unable to satisfy all constraints on the partition"
"Failed to partition the selected disk. This probably happened because there
are too many (primary) partitions in the partition table."

(...)

Post by Holger Wansing
In the partitioning overview (screenshot 3) the /home partition is
missing, so this is the one which raises the error apparently.

Also, the /tmp partition size (468MB) is lower that the minimum size
defined in the EFI "multi" recipe (500MB). The minimum disk size for
this recipe is 11768MB, so the recipe should not be available on a
10.7GB (10GiB) disk. I suspect you hit the same bug on an already
partitioned disk as described in
<https://lists.debian.org/debian-boot/2024/08/msg00151.html>.

Summary: existing reusable partitions (efi and swap) minimum sizes may
wrongly be ignored when calculating the required minimum disk size even
when using an entire disk.

Last time I guess it did not trigger an error because /home could be
created, athough with the wrong size. This time, there was no free space
left to create /home at all.

The bug already existed previously and is totally unrelated with the MR,
it could be triggered with specific disk sizes. It just happened that
10GiB is such a specific size for the updated recipes.
It cannot happen on a disk with no efi nor swap partitions.

Post by Holger Wansing
Do we need to adjust the minimum disk size for this recipe maybe, or similar?

The minimum disk size for a recipe is calculated automatically by
summing up the minimum partition sizes in the recipe. Changing minimum
partition sizes would just change the disk size which triggers the bug.

You may want to try the fix I prepared but considered not essential
(useful only on small disks with sizes close to the required minimum
size). Who would be fool enough to use the multi recipe on a 10GB disk ?

<https://salsa.debian.org/pham/partman-auto/-/commits/choose_recipe-fixes?ref_type=heads>

[Pascal Hambourg]

Post by Pascal Hambourg
Summary: existing reusable partitions (efi and swap) minimum sizes may
wrongly be ignored when calculating the required minimum disk size even
when using an entire disk.

(...)

Post by Pascal Hambourg
You may want to try the fix I prepared (...)
<https://salsa.debian.org/pham/partman-auto/-/commits/choose_recipe-fixes?ref_type=heads>

Draft merge request:
<https://salsa.debian.org/installer-team/partman-auto/-/merge_requests/17>

[Holger Wansing]

Hi,

Post by Pascal Hambourg

Post by Pascal Hambourg
Summary: existing reusable partitions (efi and swap) minimum sizes may
wrongly be ignored when calculating the required minimum disk size even
when using an entire disk.

(...)

Post by Pascal Hambourg
You may want to try the fix I prepared (...)
<https://salsa.debian.org/pham/partman-auto/-/commits/choose_recipe-fixes?ref_type=heads>

<https://salsa.debian.org/installer-team/partman-auto/-/merge_requests/17>

I think, we can consider this being a corner case, and therefore leave that
as is.


Holger

[Holger Wansing]

Hi,

Post by Pascal Hambourg

Post by Holger Wansing
Using the "separate home+var+tmp" recipe under UEFI variant on a 10G disk
results in an error
"Unable to satisfy all constraints on the partition"
"Failed to partition the selected disk. This probably happened because there
are too many (primary) partitions in the partition table."

(...)

Post by Holger Wansing
In the partitioning overview (screenshot 3) the /home partition is
missing, so this is the one which raises the error apparently.

Also, the /tmp partition size (468MB) is lower that the minimum size
defined in the EFI "multi" recipe (500MB). The minimum disk size for
this recipe is 11768MB, so the recipe should not be available on a
10.7GB (10GiB) disk. I suspect you hit the same bug on an already
partitioned disk as described in
<https://lists.debian.org/debian-boot/2024/08/msg00151.html>.
Summary: existing reusable partitions (efi and swap) minimum sizes may
wrongly be ignored when calculating the required minimum disk size even
when using an entire disk.

You are right. A test on a freshly generated qemu disk does not trigger
this bug. (In fact, this recipe is not provided at all.)
So this is indeed an 'reuse' issue.

Post by Pascal Hambourg
Last time I guess it did not trigger an error because /home could be
created, athough with the wrong size. This time, there was no free space
left to create /home at all.
The bug already existed previously and is totally unrelated with the MR,
it could be triggered with specific disk sizes. It just happened that
10GiB is such a specific size for the updated recipes.
It cannot happen on a disk with no efi nor swap partitions.

Post by Holger Wansing
Do we need to adjust the minimum disk size for this recipe maybe, or similar?

The minimum disk size for a recipe is calculated automatically by
summing up the minimum partition sizes in the recipe. Changing minimum
partition sizes would just change the disk size which triggers the bug.
You may want to try the fix I prepared but considered not essential
(useful only on small disks with sizes close to the required minimum
size). Who would be fool enough to use the multi recipe on a 10GB disk ?

You are of course right here. We cannot make it fool-proof for all and
every situation.

Post by Pascal Hambourg
<https://salsa.debian.org/pham/partman-auto/-/commits/choose_recipe-fixes?ref_type=heads>

Holger

[Pascal Hambourg]

We cannot make it fool-proof for all and every situation.

Maybe not, but identified bugs with available fixes should not be left
unfixed.

[Holger Wansing]

We cannot make it fool-proof for all and every situation.

Maybe not, but identified bugs with available fixes should not be left unfixed.

Sounds senseful.

I thought I go for some testing then, but sadly the mechanism
to build mini.iso's from d-i commits seems broken for some
time.

Phil: are you aware of this?

I looks like an issue with the naming/versioning of linux-image,
but there have been some kernel uploads since then, and
the problem persists.


See
<https://salsa.debian.org/pham/partman-auto/-/pipelines/729206>:



The following packages have unmet dependencies:

builddeps:. : Depends: linux-image--amd64 but it is not installable

E: Unable to correct problems, you have held broken packages.




Holger

[Diederik de Haas]

Post by Holger Wansing
I thought I go for some testing then, but sadly the mechanism
to build mini.iso's from d-i commits seems broken for some
time.
Phil: are you aware of this?
I looks like an issue with the naming/versioning of linux-image,
but there have been some kernel uploads since then, and
the problem persists.
See
builddeps:. : Depends: linux-image--amd64 but it is not installable
E: Unable to correct problems, you have held broken packages.

That looks like the kernel 'ABI' detection is going wrong.

https://salsa.debian.org/installer-team/debian-installer/-/blob/master/debian/salsa-ci.yml#L43
would be my guess where 'something' needs to be updated?

HTH,
Diederik

[Pascal Hambourg]

Hi,

Post by Diederik de Haas

Post by Holger Wansing
I thought I go for some testing then, but sadly the mechanism
to build mini.iso's from d-i commits seems broken for some
time.
Phil: are you aware of this?

(...)

Post by Diederik de Haas
That looks like the kernel 'ABI' detection is going wrong.
https://salsa.debian.org/installer-team/debian-installer/-/blob/master/debian/salsa-ci.yml#L43
would be my guess where 'something' needs to be updated?

Good catch. Phil just fixed the issue, thanks to him.

<https://salsa.debian.org/installer-team/debian-installer/-/commit/7575895>

[Cyril Brulebois]

Hi,

I've been watching from afar, thanks to everyone involved in trying to
get that updated.

Post by Holger Wansing
Do we need to adjust the minimum disk size for this recipe maybe, or similar?

Testing a local build against unstable, with a regular BIOS-based setup,
I'm seeing a failure to automatically partition a 5G disk, which I'd
call a major regression.

Sure, people can use whatever templates or official Debian images their
virtualization environments might provide, but I've seen people upload
a netinst ISO and install their VMs via d-i, possibly with small disks
because all they need is installing a specific (set of) package(s) to
provide a service.

I'm not sure *where* I'd put the limit, but failing to partition a 5G
or 10G disk really doesn't seem acceptable to me.


Cheers,

[Pascal Hambourg]

Hi,

Post by Cyril Brulebois
Testing a local build against unstable, with a regular BIOS-based setup,
I'm seeing a failure to automatically partition a 5G disk, which I'd
call a major regression.

Which method and recipe did you use ?

With a 5GB disk, the only available recipe should be "small_disk"
without LVM. On BIOS setup it requires at least 2.5GB disk space so it
should not fail on a 5GB disk (I just tested).
Other recipes require 7.5-12.5GB, so they should not be offered.

Post by Cyril Brulebois
I'm not sure *where* I'd put the limit, but failing to partition a 5G
or 10G disk really doesn't seem acceptable to me.

Our assumption was that guided partitioning primarily targeted casual
desktop/laptop users who want to install a standard system with a
desktop environement, so we raised the limits of the "standard" recipes
(atomic, home, multi) to enforce this use case. The "small_disk" recipe
was added to support small disks in other use cases.

Maybe we went to far ?

[Holger Wansing]

Hi,

Post by Pascal Hambourg

Post by Cyril Brulebois
Testing a local build against unstable, with a regular BIOS-based setup,
I'm seeing a failure to automatically partition a 5G disk, which I'd
call a major regression.

Which method and recipe did you use ?
With a 5GB disk, the only available recipe should be "small_disk"
without LVM. On BIOS setup it requires at least 2.5GB disk space so it
should not fail on a 5GB disk (I just tested).
Other recipes require 7.5-12.5GB, so they should not be offered.

I guess that's just another case of the "reuse" feature leading to
unexpected results, as in
https://lists.debian.org/debian-boot/2024/09/msg00094.html

MR!17 has been filed to fix this bug. See
https://salsa.debian.org/installer-team/partman-auto/-/merge_requests/17

A test with a freshly generated QEMU disk with 5G size shows no errors here,
the only provided recipe is the "small disk (<10G) partitioning scheme",
leading to a 4.7G root partition and 701 MB swap.
The rest of the installation completed without error.

Post by Pascal Hambourg

Post by Cyril Brulebois
I'm not sure *where* I'd put the limit, but failing to partition a 5G
or 10G disk really doesn't seem acceptable to me.

Our assumption was that guided partitioning primarily targeted casual
desktop/laptop users who want to install a standard system with a
desktop environement, so we raised the limits of the "standard" recipes
(atomic, home, multi) to enforce this use case. The "small_disk" recipe
was added to support small disks in other use cases.
Maybe we went to far ?

Looks fine to me as it is...


Holger

[Pascal Hambourg]

Hi,

Post by Holger Wansing

Post by Pascal Hambourg

Post by Cyril Brulebois
Testing a local build against unstable, with a regular BIOS-based setup,
I'm seeing a failure to automatically partition a 5G disk, which I'd
call a major regression.

Which method and recipe did you use ?
With a 5GB disk, the only available recipe should be "small_disk"
without LVM. On BIOS setup it requires at least 2.5GB disk space so it
should not fail on a 5GB disk (I just tested).
Other recipes require 7.5-12.5GB, so they should not be offered.

I guess that's just another case of the "reuse" feature leading to
unexpected results, as in
https://lists.debian.org/debian-boot/2024/09/msg00094.html

Even when subtracting the reusable swap partition, all the other recipes
require more than 5GB so they should not be offered.

[Cyril Brulebois]

Post by Holger Wansing

Post by Pascal Hambourg
Which method and recipe did you use ?

Encrypted LVM.

The default option works though.

Post by Holger Wansing
I guess that's just another case of the "reuse" feature leading to
unexpected results, as in
https://lists.debian.org/debian-boot/2024/09/msg00094.html

No, this is a brand new disk image.


Cheers,

[Holger Wansing]

Post by Cyril Brulebois

Post by Pascal Hambourg
Which method and recipe did you use ?

Encrypted LVM.
The default option works though.

Ah, the small_disk recipe does not support LVM. It is intended, because I considered that using LVM does not make much sense on a < 10GB disk and takes more space for the separate /boot. See the part of the discussion starting with <https://lists.debian.org/debian-boot/2024/08/msg00149.html>. But I overlooked that it also disables encryption which requires LVM...
Enable LVM with small_disk (again) ?

+1


Holger

[Cyril Brulebois]

Post by Cyril Brulebois

Post by Pascal Hambourg
Which method and recipe did you use ?

Encrypted LVM.
The default option works though.

Ah, the small_disk recipe does not support LVM. It is intended, because I
considered that using LVM does not make much sense on a < 10GB disk and
takes more space for the separate /boot. See the part of the discussion
starting with <https://lists.debian.org/debian-boot/2024/08/msg00149.html>.
But I overlooked that it also disables encryption which requires LVM...
Enable LVM with small_disk (again) ?

Yes please.


Cheers,

[Pascal Hambourg]

Post by Cyril Brulebois

Enable LVM with small_disk (again) ?

Yes please.

<https://salsa.debian.org/installer-team/partman-auto/-/merge_requests/19>

[Cyril Brulebois]

Hi again,

Post by Cyril Brulebois

Post by Holger Wansing
Do we need to adjust the minimum disk size for this recipe maybe, or similar?

Testing a local build against unstable, with a regular BIOS-based
setup, I'm seeing a failure to automatically partition a 5G disk,
which I'd call a major regression.

This time, still testing with a 5G disk, but using the default choice
(“use entire disk”, without encrypted LVM), I'm getting both Debian
Desktop environment and GNOME selected, but that doesn't fit: the
installation aborted mid-way with a weird error about dictionaries
(French not being found/installed as expected, while the root issue is
ENOSPC).

Should that matter, the partitioning scheme is the small disk one (as
expected given the <10G in parens).


Cheers,

[Cyril Brulebois]

There's no correlation with partitioning/partition sizes here, I think.

That's what I thought as well, but that's the biggest, related change
I could think of.

I guess there has never been any check in tasksel, if the selected
tasks fit on the disk? (or the other way around: tasksel does not
filter out tasks, if don't fit on the disk)

Well, those two tasks were not enabled automatically until the last few
weeks
 (and I've been testing with that particular disk size for many
release cycles), so *something* changed.


Cheers,

[Holger Wansing]

Hi,

Post by Cyril Brulebois

There's no correlation with partitioning/partition sizes here, I think.

That's what I thought as well, but that's the biggest, related change
I could think of.

I guess there has never been any check in tasksel, if the selected
tasks fit on the disk? (or the other way around: tasksel does not
filter out tasks, if don't fit on the disk)

Well, those two tasks were not enabled automatically until the last few
weeks… (and I've been testing with that particular disk size for many
release cycles), so *something* changed.

There is infact some sort of (incomplete) check for enough disk space
in tasksel.

In 2007 there was:

* Increase the minimum free disk for automatic selection of the desktop task to 3 gb. Needed since the desktop task has grown somewhat, to around 2.5 gb, and since several hundred megabytes are needed for debs in /var (which might be on the same partition). 2.5 gb free is tested to currently work in a standard single-partition layout when installing standard+desktop+laptop, so this leaves some room for future bloat, and for localisation and other tasks.

<https://salsa.debian.org/installer-team/tasksel/-/commit/7648f17bc2b9486cd033663d6611095a7c858b13>

And that logic in tests/desktop controls indeed, if desktop task
is pre-selected.
Maybe it's time to raise that limit once again?


Of course, one might ask, what has changed in the last few weeks:
that might be the size of partition (however since you used the
small_disk recipe, that should only be marginal), or maybe some
dependencies have changed at some point, so that more packages
are now included in the default desktop.
But either way, at the end we will need to raise the limit above by
1 or 2G anyway, to get this solved IMO.


Holger

[Cyril Brulebois]

Post by Holger Wansing
And that logic in tests/desktop controls indeed, if desktop task
is pre-selected.
Maybe it's time to raise that limit once again?

Yeah, that looks plausible.

Post by Holger Wansing
that might be the size of partition (however since you used the
small_disk recipe, that should only be marginal), or maybe some
dependencies have changed at some point, so that more packages
are now included in the default desktop.
But either way, at the end we will need to raise the limit above by
1 or 2G anyway, to get this solved IMO.

That would work for me, yes: I think it's better to provide safe
defaults, and let people who want to have a desktop environment select
it; maybe they'll get lucky and it'll fit, maybe that won't be the case.

Seems better than selecting something by default without being sure it
does fit?


Cheers,

[Holger Wansing]

Hi,

Post by Cyril Brulebois

Post by Holger Wansing
And that logic in tests/desktop controls indeed, if desktop task
is pre-selected.
Maybe it's time to raise that limit once again?

Yeah, that looks plausible.

Post by Holger Wansing
that might be the size of partition (however since you used the
small_disk recipe, that should only be marginal), or maybe some
dependencies have changed at some point, so that more packages
are now included in the default desktop.
But either way, at the end we will need to raise the limit above by
1 or 2G anyway, to get this solved IMO.

That would work for me, yes: I think it's better to provide safe
defaults, and let people who want to have a desktop environment select
it; maybe they'll get lucky and it'll fit, maybe that won't be the case.
Seems better than selecting something by default without being sure it
does fit?

Sure.
Tests show, that raising the limit by 1G to 4G brings back the old behaviour
(no desktop task pre-selected), but to be a bit prepared for future growth,
I would propose to go to 5G here?


Holger

[Cyril Brulebois]

Post by Holger Wansing

Post by Cyril Brulebois
Seems better than selecting something by default without being sure it
does fit?

Sure.
Tests show, that raising the limit by 1G to 4G brings back the old behaviour
(no desktop task pre-selected), but to be a bit prepared for future growth,
I would propose to go to 5G here?

That would look good to me, thanks for the tests!


Cheers,

[Holger Wansing]

Hi,

Post by Cyril Brulebois

Post by Holger Wansing

Post by Cyril Brulebois
Seems better than selecting something by default without being sure it
does fit?

Sure.
Tests show, that raising the limit by 1G to 4G brings back the old behaviour
(no desktop task pre-selected), but to be a bit prepared for future growth,
I would propose to go to 5G here?

That would look good to me, thanks for the tests!

Just pushed to git.


Holger

[Holger Wansing]

Hi kibi,

Post by Holger Wansing
Hi,

Post by Cyril Brulebois

Post by Holger Wansing

Post by Cyril Brulebois
Seems better than selecting something by default without being sure it
does fit?

Sure.
Tests show, that raising the limit by 1G to 4G brings back the old behaviour
(no desktop task pre-selected), but to be a bit prepared for future growth,
I would propose to go to 5G here?

That would look good to me, thanks for the tests!

Just pushed to git.

This has now landed in trixie, and I double-checked that this issue is
now fixed.

Thanks for pointing us on this!


Holger

[Holger Wansing]

Hi,

Post by Cyril Brulebois

Post by Holger Wansing

Post by Cyril Brulebois
Seems better than selecting something by default without being sure it
does fit?

Sure.
Tests show, that raising the limit by 1G to 4G brings back the old behaviour
(no desktop task pre-selected), but to be a bit prepared for future growth,
I would propose to go to 5G here?

That would look good to me, thanks for the tests!

Also there is a test in tests/desktop, if enough RAM is installed, to make
sense for a desktop environment to be installed.
The actual value is 64MB, from the beginning of this logic, thus it's 19 years
old.
So, if less than 64MB of RAM is installed, the desktop task will not be
pre-selected.
Should we update this to real world as well?
Currently this test does nothing in fact...

In https://d-i.debian.org/manual/en.amd64/ch03s04.html we recommend at least
2GB of RAM for desktop systems.
I could think of using this value for the above check as well, or maybe
the double of it.

Comments?
Is it worth the effort at all?


Holger

[Pascal Hambourg]

Hello,

Post by Holger Wansing
Also there is a test in tests/desktop, if enough RAM is installed, to make
sense for a desktop environment to be installed.
The actual value is 64MB, from the beginning of this logic, thus it's 19 years
old.
So, if less than 64MB of RAM is installed, the desktop task will not be
pre-selected.
Should we update this to real world as well?

IMO it should be updated or removed. Keeping obsolete/useless code makes
no sense.

Post by Holger Wansing
In https://d-i.debian.org/manual/en.amd64/ch03s04.html we recommend at least
2GB of RAM for desktop systems.
I could think of using this value for the above check as well, or maybe
the double of it.

Matching the recommended RAM size sounds fair, assuming it is realistic.

[Pascal Hambourg]

Post by Holger Wansing
Tests show, that raising the limit by 1G to 4G brings back the old behaviour
(no desktop task pre-selected), but to be a bit prepared for future growth,
I would propose to go to 5G here?

FWIW, 5GB is fine with the "multi" recipe minimum root size (7GB).
Increasing it further may require to update the recipe.

[Holger Wansing]

Hi Pascal,

Post by Holger Wansing
Tests show, that raising the limit by 1G to 4G brings back the old behaviour
(no desktop task pre-selected), but to be a bit prepared for future growth,
I would propose to go to 5G here?

FWIW, 5GB is fine with the "multi" recipe minimum root size (7GB). Increasing it further may require to update the recipe.

I went with the 5G value, so we should be fine here.


Holger

[Pascal Hambourg]

Hello,

Post by Holger Wansing
There is infact some sort of (incomplete) check for enough disk space
in tasksel.
* Increase the minimum free disk for automatic selection of the desktop task to 3 gb.
And that logic in tests/desktop controls indeed, if desktop task
is pre-selected.
Maybe it's time to raise that limit once again?
that might be the size of partition

The previous "atomic" recipe allocated 4GB to the root partition on a
5GB disk.
The new "atomic" recipe does not fit in 5GB.
The new "small_disk" recipe allocates 4.3GB to the root partition on a
5GB disk.
When tasksel is executed, 800MB-1GB are used in the target root
filesystem. I guess the extra size make the difference.

[Holger Wansing]

Hi,

Post by Cyril Brulebois
Hi again,

Post by Cyril Brulebois

Post by Holger Wansing
Do we need to adjust the minimum disk size for this recipe maybe, or similar?

Testing a local build against unstable, with a regular BIOS-based
setup, I'm seeing a failure to automatically partition a 5G disk,
which I'd call a major regression.

This time, still testing with a 5G disk, but using the default choice
(“use entire disk”, without encrypted LVM), I'm getting both Debian
Desktop environment and GNOME selected, but that doesn't fit: the
installation aborted mid-way with a weird error about dictionaries
(French not being found/installed as expected, while the root issue is
ENOSPC).
Should that matter, the partitioning scheme is the small disk one (as
expected given the <10G in parens).

There's no correlation with partitioning/partition sizes here, I think.
I guess there has never been any check in tasksel, if the selected
tasks fit on the disk? (or the other way around: tasksel does not
filter out tasks, if don't fit on the disk)


Holger