I'm adding this answer because I'm not satisfied with how other answers handle the why part of the question to understand what's going on and choose the appropriate course of action.
Hopefully this will help someone avoid blindly running apt dist-upgrade
in despair!
Why is a package kept back?
To my knowledge, there are 3 categories of reasons for packages being kept back during apt upgrade
.
1) It is marked as held back
apt-mark
can do this:
sudo apt-mark hold <package>
hold is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed.
To list all packages marked on hold or find out if a package is on hold use:
apt-mark showhold
apt-mark showhold <package>
To remove a hold on a package and allow it to be upgraded:
sudo apt-mark unhold <package>
2) apt
detects a dependency change
The best authoritative source of information I could find regarding this is marked as obsolete, but it says:
[Kept back] means that there are new versions of these packages which will not be installed for some reason. Possible reasons are broken dependencies (a package on which it depends doesn't have a version available for download) or new dependencies (the package has come to depend on new packages since the last version)
This will tell you the current and candidate upgrade versions of the package:
$ apt list <package>
example output:
vim/bionic-updates,bionic-security 2:8.0.1453-1ubuntu1.4 amd64 [upgradable from: 2:8.0.1453-1ubuntu1.3]
N: There are 2 additional versions. Please use the '-a' switch to see them.
With the current version (e.g. 2:8.0.1453-1ubuntu1.3
) and new version (e.g. 2:8.0.1453-1ubuntu1.4
), we can figure out the changed dependencies with apt show
:
apt show <package>=<old version> <package>=<new version>
example:
apt show vim=2:8.0.1453-1ubuntu1.3 vim=2:8.0.1453-1ubuntu1.4
(or just use apt show -a
to view all versions directly, but it makes the version comparison harder in my opinion)
The important parts are the Depends
and Recommends
package lists. If there are new packages in those lists in the new version of the kept back package, apt won't automatically upgrade it.
At this point there are 2 options to upgrade the kept back package. Note that both solutions below have the proper arguments to avoid erroneously changing a package from "automatically installed" to "manually installed".
To upgrade the package and install any new "Recommended" packages (i.e. as if newly installed with apt install <package>
, use --with-new-pkgs
:
sudo apt upgrade --with-new-pkgs <package>
(Tip: add --dry-run
to see what will happen before doing it)
To upgrade the package without installing any newly added "Recommended" packages, use --only-upgrade
.
sudo apt install --only-upgrade <package>
3) Phased updates
PhasedUpdates
Once an update is released to -updates, the update is then phased so that the update is gradually made available to expanding subsets of Ubuntu users. This process allows us to automatically monitor for regressions and halt the update process if any are found.
You can check for phased updates with:
sudo apt -o APT::Get::Always-Include-Phased-Updates=true upgrade --dry-run
Lots more info here: What are phased updates, and why does Ubuntu use them?
Case study: upgrading the docker-ce
package
Upgrading the docker-ce
package on Ubuntu 18.04 is what brought me here in the first place so I thought it would be interesting to have a full concrete example.
$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
docker-ce
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
$ apt list docker-ce
Listing... Done
docker-ce/bionic 5:20.10.3~3-0~ubuntu-bionic amd64 [upgradable from: 5:19.03.12~3-0~ubuntu-bionic]
N: There are 34 additional versions. Please use the '-a' switch to see them.
Ok let's see what's holding back docker-ce
:
$ apt show docker-ce=5:19.03.12~3-0~ubuntu-bionic docker-ce=5:20.10.3~3-0~ubuntu-bionic
Package: docker-ce
Version: 5:19.03.12~3-0~ubuntu-bionic
Priority: optional
Section: admin
Maintainer: Docker <support@docker.com>
Installed-Size: 107 MB
Depends: docker-ce-cli, containerd.io (>= 1.2.2-3), iptables, libseccomp2 (>= 2.3.0), libc6 (>= 2.8), libdevmapper1.02.1 (>= 2:1.02.97), libsystemd0
Recommends: aufs-tools, ca-certificates, cgroupfs-mount | cgroup-lite, git, pigz, xz-utils, libltdl7, apparmor
Conflicts: docker (<< 1.5~), docker-engine, docker-engine-cs, docker.io, lxc-docker, lxc-docker-virtual-package
Replaces: docker-engine
Homepage: https://www.docker.com
Download-Size: 22.5 MB
APT-Manual-Installed: yes
APT-Sources: https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
Description: Docker: the open-source application container engine
Docker is a product for you to build, ship and run any application as a
lightweight container
.
Docker containers are both hardware-agnostic and platform-agnostic. This means
they can run anywhere, from your laptop to the largest cloud compute instance and
everything in between - and they don't require you to use a particular
language, framework or packaging system. That makes them great building blocks
for deploying and scaling web apps, databases, and backend services without
depending on a particular stack or provider.
Package: docker-ce
Version: 5:20.10.3~3-0~ubuntu-bionic
Priority: optional
Section: admin
Maintainer: Docker <support@docker.com>
Installed-Size: 121 MB
Depends: containerd.io (>= 1.4.1), docker-ce-cli, iptables, libseccomp2 (>= 2.3.0), libc6 (>= 2.8), libdevmapper1.02.1 (>= 2:1.02.97), libsystemd0
Recommends: apparmor, ca-certificates, docker-ce-rootless-extras, git, libltdl7, pigz, xz-utils
Suggests: aufs-tools, cgroupfs-mount | cgroup-lite
Conflicts: docker (<< 1.5~), docker-engine, docker-engine-cs, docker.io, lxc-docker, lxc-docker-virtual-package
Replaces: docker-engine
Homepage: https://www.docker.com
Download-Size: 24.8 MB
APT-Sources: https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
Description: Docker: the open-source application container engine
Docker is a product for you to build, ship and run any application as a
lightweight container
.
Docker containers are both hardware-agnostic and platform-agnostic. This means
they can run anywhere, from your laptop to the largest cloud compute instance and
everything in between - and they don't require you to use a particular
language, framework or packaging system. That makes them great building blocks
for deploying and scaling web apps, databases, and backend services without
depending on a particular stack or provider.
Version 5:20.10.3~3-0~ubuntu-bionic
has added docker-ce-rootless-extras
as a new recommended dependency. I wish apt would be more helpful and simply suggest installing it or something instead of leaving me with an old version... Anyhow, here are the 2 possible fixes (with --dry-run
for illustration purposes):
$ sudo apt upgrade --with-new-pkgs --dry-run docker-ce
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
docker-ce
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Inst docker-ce [5:19.03.12~3-0~ubuntu-bionic] (5:20.10.3~3-0~ubuntu-bionic Docker CE:bionic [amd64])
Conf docker-ce (5:20.10.3~3-0~ubuntu-bionic Docker CE:bionic [amd64])
$ sudo apt install --only-upgrade --dry-run docker-ce
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
docker-ce-rootless-extras
Recommended packages:
slirp4netns
The following NEW packages will be installed:
docker-ce-rootless-extras
The following packages will be upgraded:
docker-ce
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Inst docker-ce [5:19.03.12~3-0~ubuntu-bionic] (5:20.10.3~3-0~ubuntu-bionic Docker CE:bionic [amd64])
Inst docker-ce-rootless-extras (5:20.10.3~3-0~ubuntu-bionic Docker CE:bionic [amd64])
Conf docker-ce (5:20.10.3~3-0~ubuntu-bionic Docker CE:bionic [amd64])
Conf docker-ce-rootless-extras (5:20.10.3~3-0~ubuntu-bionic Docker CE:bionic [amd64])
apt show <package>=<new-version>
(where<package>
is one of the packages listed inapt list --upgradable
and<new-version>
is the version string listed as the 2nd (space-separated) field in the output of same) should show a line sayingPhased-Update-Percentage: <n>
for some<n>
that's the current percentage. Presumably once that gets to 100%, one will get the upgrade. – lindes Sep 17 '22 at 17:15apt show -a $(apt list --upgradable 2>&1 | grep / | cut -d/ -f1) 2>&1 | grep Phased | sort -n | uniq -c
-- shows how many packages are at what phased-update percentages. (e.g. I currently have 3 at 0%, and 14 at 70%). – lindes Sep 17 '22 at 17:17apt update
in the long run, and some new others will appear as held back. But if it does not affect the system negatively, would there be a reason to "fix" this excluding getting rid of the warning? – M.K Feb 08 '23 at 14:38