The big names mostly can’t yet, but some lesser-known Linux distributions offer the ability to undo updates and recover from damage, even automatically.
With a new version of Linux Mint available, a lot of people – especially less-technical types – are about to upgrade their PCs. As with almost all major upgrades, inevitably some some stuff won’t work with the new version. Going back is difficult, although Mint does have a solution, and so do some smaller distros.
Linux Mint includes the Timeshift utility, originally developed by Teejeetech, but that doesn’t make it invulnerable. No operating system is, as amply demonstrated recently by CrowdStrike. If installed on a file system that supports copy-on-write (COW) snapshots, Timeshift can use that, but if you chose some other disk format, it can also back up your system files using rsync to another drive.
For now, most Linux file systems can’t handle snapshotting, including the default ext4 or its predecessors, or XFS, or the now-vanishing ReiserFS. Several existing file systems that work on Linux do include snapshot support, including bcachefs, ZFS, and Btrfs. However, there are solid reasons why most distros don’t include the functionality.
For instance, while OpenZFS does support snapshots, and several distributions include ZFS, including TrueNAS Scale, Ubuntu, Proxmox, Void Linux, and NixOS, the ZFS code can’t be compiled into the Linux kernel due to an incompatible software license. Thus, from appropriate levels of caution, OpenZFS isn’t part of either SUSE’s or Red Hat’s enterprise offerings. Their compliance with multiple legal requirements and certifications is a big part of their corporate appeal. (For the curious, both companies maintain lists of their certifications. Red Hat has a 16-section list and SUSE a respectable dozen sections. Some companies will pay good money for this stuff.)
Red Hat removed Btrfs from RHEL six years ago, and is working on its own, all-GPL combined volume management system and file system called Stratis. Since Btrfs was invented there, Oracle had a simpler route available for its RHEL-compatible distro: Oracle Linux offers a kernel with Btrfs.
Bcachefs, meanwhile, is relatively new and immature. It only appeared at the beginning of this year in kernel 6.7, meaning that it’s only been in three releases of the kernel so far. It’s a little bit too new for anyone else to support it just yet.
So, for now, Btrfs is the most mature file system in the Linux kernel that can do snapshots. This is why SUSE’s Snapper uses Btrfs by default, although it also supports LVM, and can be configured to use that instead.
Btrfs + Timeshift saved my broken Linux Mint
I use NixOS btw.
Lol I came here to basically make that exact comment. I don’t even bother with timeshift on NixOS since the state that actually matters to me is described by my git repo and it’s got native rollbacks.
That said, I’d love for btrfs + timeshift to become the out of the box standard everywhere else, it’s just fantastic user experience.
I love btrfs+snapper. I have automatic snapshots taken before and after every
apt install, so if anything ever goes belly-up, no problem, I just roll back.It’s a little weird sometimes when I’m running out of disk space, so I delete some big downloads and…I get no disk space back, because those files still exist in old snapshots! I suspect there’s some way to finetune it to ignore certain directories (like ~/Downloads) in snapshots but I haven’t taken the time to dig into it. Anyway, it’s not a huge problem because the automatic snapshots are limited to a certain number, so they eventually get bumped out (or I can delete them manually if needed).
I haven’t tried bcachefs yet. Perhaps on my next build.
From what I understand, you would need to move your Downloads-folder to a separate subvolume, as btrfs always does a snapshot of a whole subvolume.
Here on openSUSE, the default setup has the whole /home-folder on a separate subvolume, though. Basically the thought is that, if you need to roll back your OS, you don’t want to also roll back the changes you made to your documents and whatnot.
Also, you should make proper backups of your home-directory anyways, although I guess, you could separately create snapshots of your home-directory, if you want them anyways.Ah, that makes sense! I probably should have split my /home off to its own subvolume. I’ll add that my list of things to think about next time I hop distros or rebuild (which I’m considering once again, because I have Plasma envy).
And yes, snapshots should NOT be treated as backups. A real backup saves your butt if your drive dies, while a snapshot goes down with that ship. I should really set up a better backup system, but for now I just periodically use Borg to back up to an external HD, and then copy that into an encrypted cloud drive.
… and then copy that into an encrypted cloud drive.
If you care about privacy you should encrypt the files first and then upload it to the cloud
Yep, totally agree. Borg supports encryption, so I have the backup encrypted locally, independently of my end-to-end encrypted cloud service. I guess I could store it on a non-e2ee cloud drive and save a little money, but I prefer to keep it simple with an “E2EE or bust” mentality, even for data that’s already encrypted.
For anyone who finds Borg intimidating, take a look at Vorta, an easy-to-use GUI frontend: https://github.com/borgbase/vorta
Yeah my biggest problem with snapper was the disk space.
Also it just… Isn’t super well documented? At least it felt like trying to use it on Fedora was not working as well as it had on opensuse.
I eventually found it to be more trouble than it was worth…
Snapper was too unintuitive for me to figure out and I didn’t want to dig through tutorials and potentially fuck something up. Timeshift has the benefit of being pretty streamlined. It is intuitive and easy to understand even on a first use. The only thing I once had issues was that the background service didn’t start, so it didn’t do auto backups. But that was easy to look up and basically just one command necessary.
Snapper has been wonderful, and saved me several times. I only keep root+home on btrfs and have a separate, larger data partition on ext4 where larger data and nonessential programs end up living.
Why use separate partitions over subvolumes within btrfs?
Mostly because it’s on a separate drive and I hadn’t used btrfs before. Ext4 is ubiquitous and a lot simpler. Besides, I don’t really need the extra features of btrfs for my data volume.
OpenSUSE is very well known and had btrfs snapshots for a very long time. I’ve used this feature several times to undo updates and it is extremely easy and best of all works perfectly without user having to configure anything.
And it still does.
I don’t know why it isn’t a so-called “big name” when it’s among the oldest distributions around. Those journalists are as well informed as a random forum poster :-/
Yup. Been using it for about 20 years now. It is more popular in Europe, but I wouldn’t call it a small player in US either.
Too bad OpenSUSE could not even change my resolution down to 1080p without bugging out.
This anecdote without any context certainly makes OpenSuse a low key player in OS game.
Desktop config has very little to do with OS since most are running kde or gnome and those handle all that.
Yeah, but no other distro caused my screen to go into stretched pancake mode when I selected 1080p.
Running Fedora Atomic (Silverblue) has definitely saved ms a few times already, being able to roll back to the previous state, or to a state I pinned. The first time was due to the ublue signing key change, the second had no apparent cause. Both issues would have given me more of a headache without the built in ability to roll back.
I would jump to NixOS in a heartbeat but I do not think I can manage to learn Nix. I think that if they manage to to make it noob friendly it would be the most popular distro out there(probably).
It’s called /var/cache/pacman/pkg.
You mean https://archive.archlinux.org/. I ain’t keeping no stinking obsolete packages around.
I ain’t keeping no stinking obsolete packages around.
That argument could be said about any backups.
Hell yeah! Who needs yesterday’s data when today’s data is so much better. Preach!
You need network drivers for that tho. And that’s not necessarily given when something goes wrong with drivers. Even though that’s like 1% of the time (one time for me, I didn’t remove the explicit driver after it was moved to the kernel/another driver, I don’t remember exactly). The main culprit is NVidia.
Timeshift has got me out of an uncomfortable situation at least twice at this point, and since I’m running a mostly LMDE frankenDebian right now (drivers tho, amirite?), it’s nice to know that this potentially delicate working state is backed up in some way should it decide to shatter into a thousand tiny pieces.
I was using Nix but ran into some software compatibility issues so I’m back on Arch. It’s obviously not immutable and doesn’t have generations, but using btrfs + snapper + snap-pac + grub-btrfs has given me an automated system of snapshots that let me roll back to my system state before or after each package change (As well as weekly snapshots). It has been rock solid and reliable for me. On my Debian server I just use ZFS and snapshot it weekly, which isn’t as easy to roll back but it does everything I need
honestly, Timeshift + btrfs was a big part of the reason I was willing to try switching to Mint from Windows in February when I read about it (the other big part being Proton and my experience with my Steam Deck), and I’ve been rather happy with it since. I still dual boot for the odd thing, but 90% of the time I’m in Linux and it just works; and on the single occasion I’ve had an update bork something, I just used Timeshift to restore a snapshot, tried updating again, and it worked fine the second time. Took me 10 minutes. I remember the heady days of the early 2010s when I first tried Linux, and that would’ve normally kicked off an entire evening’s worth of troubleshooting. (And, indeed, it was shit like that that pushed me back to Windows whenever I’d try Linux)
I’m really hoping more semi-computer-literate people start taking the plunge, it isn’t nearly so awful an adjustment to make as it was just ten years ago and Microsoft clearly needs the competition to encourage them to make not-shit products. I still wouldn’t give a Linux machine to your grandmother, but your average technically competent nerd who can use Google can actually use Mint nowadays without having to fuck with it too much; as opposed to having to almost become an expert on Linux and get up into its guts to make it work right for you (which, Windows is increasingly requiring itself, if you don’t want Microsoft knowing everything you do and serving you ads in your OS), and Timeshift is a big part of why, IMO. It’s not quite there to “perfectly suited for general use by your average idiot”, plenty of programs that don’t yet play nice with it; but it’s so much closer to that ideal now than it’s ever been before, and it’s still getting better.
I know, wrong community, y’all like to get up into its guts, poke around and tinker; but I and many others would rather work on our computers, as opposed to work on our computers, if you get my meaning.
I vaguely remember seeing something about this in Nobara news, does anyone knows if Nobara is also rollbackable like this, or am I out of luck?
It does not, unfortunately. Their update broke Plasma yesterday. Tried to use Timeshift, which did not work and instead broke my system further. Could not find anything to solve either one of the issues. Backed up my home folder (again), tried Bazzite, could not figure out how to import my old Firefox profile since FF is apparently sitting within the immutable environment. Then tried Fedora, after all people kept saying all my issues come from Nobara being a hobby project. Could not figure out how to make firefox play videos on various websites. Tried rpm fusion and installing codecs. System stopped responding. Hard reboot. Now my desktop environment is not loading anymore… I have no idea what distro to use now to be honest. Everything breaks at some point, or straight up is not usable out of the box. After several years I would rather not go back to Windows, especially since that pile of shit got even worse with 11.
I was just about to update my Nobara after some time, thanks for the heads up. It’s a shame, I really loke Nobara and I switched to it because I couldn’t get lot of game related stuff working properly on Fedora and I’d like to stick to Fedora-based distros since I’m used the most to that. I guess PopOS would be my next choice if I couldn’t get something working, Mint after that.
PopOS uses Gnome, which I cannot stand and have had only issues with as well. I was contemplating of going back to Mint, which I used a long long time ago, but Cinnamon is very limited compared to KDE and I hate apt and the whole ppa hell, which I would not look forward to getting back to.
Can’t you just use snapshots from the btrfs filesystem?
