Pacman 6.0.0alpha1

We have been slowly working towards the next pacman release. The major change for this release is the implementation of parallel downloads. Check out this video:

It is time for this feature to receive some wider testing. Grab the package using:
pacman -U http://allanmcrae.com/packages/pacman-6.0.0alpha1-1-x86_64.pkg.tar.zst

This should be “safe” – there are quite a number of us using this code for our system management. There may be issues with AUR helpers due to ABI changes… If you are concerned, keep a copy of the repo package around to reinstall if necessary.

There are still a few issues with the downloader we will fix before release. If one of your mirrors is broken, you will be spammed with error messages at the end of the downloads. And TotalDownload does not work. But otherwise, it is good to go!

Report any bugs to the bug tracker.

Distri – Comparing Apples and Oranges?

Last weekend we had what I consider to be the very successful gate.io trustpilot. This included a talk by Michael Stapelberg about distri, his Linux distribution to research fast package management.

Michael showed an example of installing QEMU in Arch vs distri, very similar to the results given on his post about Linux package managers being slow. There are a couple of examples of installing packages in that blog post – one pacman comes out looking quite good, one it comes out looking OK (at least it beats apt on Debian and dnf on Fedora…).

Before I get into the teardown of these comparisons, I will point out that I am not disagreeing that distri is a very fast package manager. It is designed for pure speed, so it should definitely be the fastest! Also, I am not disagreeing that pacman could be faster in many areas. What I am concerned about is the conclusion that “The difference between the slowest and fastest package managers is 30x!“. This conclusion assumes that the package manager is the difference and not how the distribution chooses to implement the use of the package manager.

Lets look at the first comparison – ack. That is not me agreeing with myself, but rather a small perl program that depends on a single perl library (perl-file-next). Assuming perl is installed in the base image for the distros being compared, this is a reasonable comparison, if not slightly small. However, looking at the output from the installs (which is not provided in full for all package manager + distro combinations), we can see that is not the case. Note, Fedora and dnf require 114MB of downloads compared to Arch and pacman with 6.5MB. It appears that Arch is the only distro to have perl installed by the base image, so this makes pacman look better. The NixOS image does not even have coreutils installed (and why is that needed for ack?)!

This gets worse for the qemu comparison. Given the video comparison in Michael’s talk, I will just focus on Arch + pacman vs distri. Lets look at the dependency lists for the two packages. For distri we get:

$ wget https://repo.distr1.org/distri/supersilverhaze/pkg/qemu-amd64.meta.textproto
$ grep runtime_dep qemu-amd64-4.2.0-9.meta.textproto
runtime_dep: "gcc-libs-amd64-9.3.0-4"
runtime_dep: "glibc-amd64-2.31-4"
runtime_dep: "gmp-amd64-6.2.0-4"
runtime_dep: "mpc-amd64-1.1.0-4"
runtime_dep: "mpfr-amd64-4.0.2-4"
runtime_dep: "glib-amd64-2.64.2-5"
runtime_dep: "libffi-amd64-3.3-4"
runtime_dep: "util-linux-amd64-2.32.1-7"
runtime_dep: "ncurses-amd64-6.2-9"
runtime_dep: "pam-amd64-1.3.1-11"
runtime_dep: "zlib-amd64-1.2.11-4"
runtime_dep: "libcap-amd64-2.33-4"
runtime_dep: "pixman-amd64-0.38.4-7"

On a fresh Arch install, the only package in that list that would not be installed is pixman, which has no futher dependencies (beyond glibc), so that would require two packages to be downloaded. However compare that to the Arch package:

$ pacman -Si qemu
...
Depends On : virglrenderer sdl2 vte3 libpulse brltty seabios gnutls
libpng libaio numactl libnfs lzo snappy curl vde2
libcap-ng spice libcacard usbredir libslirp libssh zstd
liburing

That is a lot more dependencies? Lets quantify this with a number:

$ pactree -s -u base | sort > base.deps
$ pactree -s -u qemu | sort > qemu.deps
$ comm -23 qemu.deps base.deps | wc -l
123

That is very different! Why? Because Arch’s qemu package provides the graphical frontend. However, Arch does have a qemu-headless package, and …

$ pactree -s -u qemu-headless | sort > qemu-headless.deps
$ comm -23 qemu-headless.deps base.deps | wc -l
20

That is getting to be a fairer comparison. But we still are not comparing only package manager differences. In fact, this comparison is heavily skewed towards how a distribution chooses to package the software. Pacman appears at a disadvantage, because Arch packages all files for the software in a single package, and does not split binaries, libraries, docs, include files, etc into separate packages. That is a distribution decision, and not a package manager decision – all package managers in the comparison list are capable of splitting packages into smaller units. So maybe not apples to oranges, but rather oranges to orange segments? I don’t think I am good at analogies!

Finally, I’ll note comparisons “include metadata updates such as transferring an up-to-date package list“. This seems reasonable at first glance. However, consider two distributions, where the one has a small package set, and the second includes all the packages from the first plus many more. The second distro would take longer to refresh the packages metadata, but that does not mean the package manager is slower! I need to look at distri closer, but my guess from looking at its repo layout is that it downloads the metadata for a package on demand and includes a full dependency list so does not rely on any transitivity. This appears to be a nice trick, but a distro using pacman could be set-up such that package database refreshes were only needed when performing a system update and not when installing packages. Again, Arch makes the choice not to support older versions of packages and removes them from download immediately on updates, but that is not a pacman limitation.

In summary, package managers are slow… But some distributions decide to make them slower! So we need to be careful when drawing conclusions of relative speed of package managers when more variables are at play.

Pacman 5.2 Release

Nothing like a new pacman release to make me locate the password to this site…

Tradition dictates I thank people who have contributed to the release (as well as genuinely meaning the thanks!). We had 29 people have a patch committed this release, with a few new names. Here is the top ten:

$ git shortlog -n -s v5.1.0..v5.2.0 | head -n10
   108  Eli Schwartz
    38  Allan McRae
    30  morganamilo
    24  Andrew Gregory
    20  Dave Reisner
     9  Jan Steffens
     6  Michael Straube
     4  Jonas Witschel
     4  Luke Shumaker
     3  Que Quotion

We have a clear winner. Although I’m sure that at least half of those are in responses to bugs he created! He claims it is a much smaller proportion… And a new contributor in third.

What has changed in this release? Nothing super exciting as far as I’m concerned, but check out the detailed list here.

We have completely removed support for delta packages. This was a massively underused feature, usually made updates slower for a slight saving on bandwidth, and had a massive security hole. Essentially, a malicious package database in combination with delta packages could run arbitrary commands on your system. This would be less of an issue if a certain Linux distro signed their package databases… Anyway, on balance I judged it better to remove this feature altogether. We may come back to this in the future with a different implementation, but I would not expect that any time soon. Note a similar vulnerability was found with using XferCommand to download packages, but we plugged that hole instead of removing it!

Support for downloading PGP keys using the new Web Key Directory (WKD) was added to pacman. Both pacman-key and makepkg will also look there by default with the latest GnuPG release. This prevents DoS attacks through people adding very large numbers of signatures to PGP keys. The attack scope was limited for Arch Linux anyway, as most people obtain the pacman keyring through the archlinux-keyring package.

The much maligned --force made its way to /dev/null. The --overwrite option has been a replacement for over a year and is a precision surgical instrument compared to the blunt hammer of --force.

There is a small user interface change for searching files databases with -F. Specifying the -s option was redundant, so removed. More information such as package group and installed status is shown in the search results, bringing the output inline with -Ss.

The split of makepkg into smaller and extendable components continued. You can now provide new source download and signature verification routines (e.g. if you are living in the past and want to support cvs:// style URLs). We also added support for lzip, lz4 and zst compressed packages. Arch Linux will switch zst by default in the near future.

Under the hood, we are in the process of changing our build system from autotools to meson. This is relatively complete, but there still was a decent churn of patches to meson files as we approached release. You can build pacman from the release tarball using meson if you want to test. Next release is likely to be meson only. (Edit: you can’t test meson with the 5.2.0 tarball as it is missing a couple of the meson build files.)

Expect the release to land in Arch Linux “soon”. Expect to see another blog post in a year or so when I make the next release…

Pacman-5.1 – Don’t Use the Force, Luke!

Wow… look at all the cobwebs around here! No posts in two years. But the need for a pacman release post has dragged me back. I clearly still remembered the password, so that is a bonus!

As is tradition, before I get in to details, I need to thank everyone for their help in making this release. Here are the top 10 committers:

$ git shortlog -n -s v5.0.0..v5.1.0
    82  Allan McRae
    60  Andrew Gregory
    45  Eli Schwartz
    16  Ivy Foster
    10  Dave Reisner
     9  Christian Hesse
     9  Gordian Edenhofer
     8  Alastair Hughes
     7  Rikard Falkeborn
     6  Michael Straube

(I win!) Lots of new names there which is always really appreciated. And as usual a long tail of contributors submitting the occasional patch – there were 48 contributors in total.

Onto what has changed in this release. There is a lack of what I would call a killer feature in this release. Mostly a lot of small changes that improve usability, which is why there was so much time between releases. Here is a detailed list of changes. However, there are a few things worth highlighting.

There is a new option --overwrite, which is a replacement for to often misused --force (hence the release name). This allows fine grained control of what files pacman is safe to ignore conflicts with. Handling the latest upgrade requiring user intervention in Arch Linux would now look like:
pacman -Syu --overwrite usr/lib/libmozjs-52.so.0You can even use globs when specifying the files to overwrite. Not only is specifying exact files to overwrite a lot safer than the old --force, there are also some common sense restrictions there too (you can’t overwrite a directory with a file, or force package installs with conflicting files).

We have also added a --sysroot option that will replace --root. Basically, this now works the way people will expect – for example, the configuration file used is the one in the specified root, and not the local one. This does require a bit more setup while creating a new install root, but hopefully will be a lot more robust.

We have also added support for reproducible builds. This was mostly ensuring all files had the same timestamp and obeyed the SOURCE_DATE_EPOCH standard. We also added a .BUILDINFO file within each package, recording information about the environment a package was built in. This allows scripts to regenerate the build environment to demonstrate a package is reproducible (particularly important in rolling release distros).

There was also improved support for debugging packages. Split packages now produce a single debug package instead of one for each split package. This makes it easier to get all required debug symbols for a particular package (and hopefully easier for distros to carry these packages…). Also, we include relevant source files in the debug packages, allowing us to step through the code.

Finally, I killed off the “contrib” directory as it was taking excessive amounts of pacman developer time. That means no more checkupdates, paccache, … However, this has been picked up as a separate project, which is available by installing pacman-contrib in Arch Linux.

As always, this is a bug free release. But if you spot something you think is a bug, please file a bug report and we can assign blame – which is more important than fixing! (The pool for developer who created the first pacman bug of this release is still open at the time of posting.)

Pacman-5.0 Released

As is becoming tradition, I need to make a blog post to accompany a pacman release! This is a big release with a long awaited feature so it needed a major version bump (and, most importantly, we now are back ahead of the Linux kernel in version numbers). I have reclaimed the title as most prolific committer, but that just means Andrew had more patches to point out mistakes in… Here are the top 10 committers:

$ git shortlog -n -s v4.2.0..v5.0.0
   176  Allan McRae
    85  Andrew Gregory
    16  Florian Pritz
     9  Dave Reisner
     9  Johannes Löthberg
     9  Rikard Falkeborn
     7  Pierre Neidhardt
     5  David Macek
     5  Evangelos Foutras
     4  Mohammad Alsaleh

As always, more regular contributors would be helpful. Just have a talk to us first before running ahead implementing a new feature (it is not nice to have to reject a patchset that obviously took a lot of work because it is already being handled in a different way…).

On to the more important stuff… What is new this release?

Hooks: This has been one of the most requested features for a long time, and Andrew gets all the credit for the implementation. So, what exactly are hooks? Hooks are scripts that are run at the beginning and end of a transaction. They can be triggered by either a file or a package name. This will allow us to (e.g.) update the desktop MIME type cache at the end of a transaction (if needed) and only do it one rather than after every package. Andrew has a git repo with some examples. Lets look at the desktop MIME cache one:

[Trigger]
Type = File
Operation = Install
Operation = Upgrade
Operation = Remove
Target = usr/share/applications/*.desktop
 
[Action]
When = PostTransaction
Exec = /bin/update-desktop-database --quiet

It should be fairly obvious what that does… See the alpm-hooks(5) man page for more information on the hook format.

Files database operations: pacman can now search repository file lists like pkgfile, but slower and probably less flexible. Sort of like Falcon to Captain America… still a super-hero! File bug reports for improvement requests. A separate files database is used so everyday package operations do not require downloading a much larger database. After updating the database (with “pacman -Fy“, you can do things like:

$ pacman -Fo /usr/bin/pacman
usr/bin/pacman is owned by core/pacman 4.2.1-4
 
$ pacman -Fl pacman
pacman etc/
pacman etc/makepkg.conf
pacman etc/pacman.conf
...
 
$ pacman -Fs libalpm.so
core/pacman 4.2.1-4
    usr/lib/libalpm.so
 
$ pacman -Fsx kcm.*print.*\.so
extra/print-manager 15.12.1-1
    usr/lib/qt/plugins/kcm_printer_manager.so
community/kmymoney 4.7.2-3
    usr/lib/kde4/kcm_kmm_printcheck.so

libmakepkg: makepkg in the pacman-4.2 release was a 3838 line shell script. A bit daunting, hard to test and not reusable… I have started the process of splitting this into a library containing scripts that are a more reasonable size. makepkg is still 2395 lines, so a lot of work remains (help!). One outcome of this splitting is we can drop in extra checks into the PKGBUILD and package checking steps, and even extra passes to (e.g.) optimize svg files. I have started rewriting namcap using this feature (see my github repo). This also provides tools for extracting variables from PKGBUILDs without sourcing the PKGBUILD itself, which does require ensuring that variables that should be arrays are actually arrays and those that are not are not.

There were a bunch of other small changes throughout the code base. Check out the NEWS file for more details.

Where Else?

Tweet

Australia’s “Shadow Minister for Justice” is the representative for the seat of Batman.

Posted in Tweet on by Allan – Comments Off on Where Else?

Anime Guide 2014

After many repeated requests (which was kind of surprising), here is my overview of the 2014 anime I watched. I am told that I was too negative in previous lists and I want to avoid that this time… probably not going to happen!

This year had a LOT of sequels. I prefer to watch something new unless the original was outstanding, so I ignored some of these (how can Fate/Stay have so many spin-offs?). And for some reason, many ~22 episode series were split in half and had a break in the middle. I am counting those as one series. Speaking of both sequels and split anime…

Anime of the Year

Mushi-Shi: Next Passage
(TV Series, 11+10 episodes + special)

This is how you do a sequel! And it came out of nowhere about 10 years after the original. The stunning visuals remain and are improved on as we expect in the HD era. The atmospheric soundtrack sits very well with the stories, with the intro “Shiver” by Lucy Rose fitting in perfectly. As with the original series, the story is largely episodic but there is a subtle underlying flow brings it together. Not only do I consider this the anime of 2014, it would be a strong contender for anime of the year in any year. Look out for the movie in 2015.

Recommended

Kill la Kill
(TV Series, 24 episodes)

Half naked girls fighting while being manipulated by clothing… This is made by the same people as “Gurren Lagann”, and it is just as insane, just with less robots and more fan-service. And there is a lot of fan-service, to the point where it would have been awkward if I watched this when taking public transport to work! That also caused some controversy around the internet. I watched this for the craziness and enjoyed it for that – the fan-service seemed to fit in to that insanity. This is not an anime that needs watched with extensive thought. (The uproar about the nudity makes it awkward to justify enjoying this series. But I am sure that if I only wanted to see naked people, I could just find some somewhere in the dark recesses of the internet…)

Ping Pong the Animation
(TV Series, 11 episodes)

I only watched this after it being repeatedly recommended to me, because I do not enjoy sport anime. It turns out, Ping Pong is not really a sport anime… despite being about ping pong. This is a show about the growth of the main characters, their friendships and rivalries develop while revolving around the game. Saying that, I saw some ping pong on TV the other day and I was now an expert on it, so there was some sport there. I found the distinctive art style aided in skimming over the actual games without losing the main focus of the show.

Samurai Flamenco
(TV Series, 22 episodes)

This is a series that people I have talked to either love or hate. I think it is because the series starts out reasonably serious and realistic before going off on wild tangents. I am still not sure if the script was planned, or if the writers genuinely went crazy (although I may lean towards planned after completing the series). But you can not go past this if you want a fun superhero story that has the ability to go from low-tech and gritty to the other extreme. I watch anime for fun, and not to search for some higher meaning, so this was a close contender for anime of the year!

Silver Spoon 2
(TV Series, 11 episodes)

Much like my comments on the first series, there is nothing individually outstanding about the sequel. It is just consistently good. The second season does well in moving on from the issues dealt with during the first (far less “this is where my food comes from” jokes) and focusing on new challenges. You know, exactly what a good sequel should do…

Average

Akame ga Kill!
(TV Series, 24 episodes)

This was one hyped anime for some reason. But there was nothing outstanding in it (some would call it “average” even…). The animation and music during the action sequences was well done, and that makes this fun to watch. However, it goes from dramatic battle to light-hearted romp with the flick of a switch, which ruins both.

Ghost in the Shell: Arise
(OVA, 4 episodes)

Sort of a prequel to Stand Alone Complex, but the backstories have changed… so not a prequel. But there is enough familiarity with the characters that it feels like a prequel (is any of this making sense?). The result is something neither better or worse than the original series.

Psycho Pass 2
(TV Series, 11 episodes)

Not how to do a sequel! I really enjoyed the darkness and mystery of the first series, but that had been mostly resolved and that left not a lot to add. The sequel proceeded to add that not a lot (or even just blatantly rip-off ideas from the first series), resulting in something that was both mildly entertaining and unnecessary at the same time.

Space Brothers
(TV Series, 99 episodes)

I usually check how many episodes a series has before I start watching it – I forgot for this one! Saying that, I watched each episode of this immediately as it broadcast. I would say this is highly recommended apart from the episodes being a bit slow at times and a bit predictable. Also, there was a trend to a substantial recap at the start of episodes. If they fixed that by reducing the series by ~20 episodes, this would be a great(er) anime.

Space Dandy
(TV Series, 13+13 episodes)

After picking a cow for Silver Spoon, I knew what picture I needed here too! I had great hope for this anime – it is by the makers of some of my favourites. However… I never got caught up in this series. Everything about it is simple, right down to the “comedy” which has none of the unexpected surprise needed to be funny. For example the “Boobies” restaurant, which is Hooters in space – oh, it is also shaped like boobs. That is the joke… There is also nothing riding on the outcome of an episode (spoiler follows). The entire cast die at the end of the first episode and are back as if nothing happened in episode two. So who cares what happens?

Tiger and Bunny the Movie: The Rising
(Movie, 100 minutes)

The second part of a two movie series that may or may not be a recap of the TV series (I can not be bothered checking…). If they were not a recap, it added very little beyond what the TV series did in terms of actual development of the storyline. Still enjoyable.

Tokyo Ghoul
(TV Series, 12 episodes)

I usually like short series (good risk/reward investment ratio), but I think being too short was the failure here. There was character development, building a world where ghouls have their own society, and then someone asked “how many episodes do we have left?”. Queue the final three episodes being packed with action and not a lot else. I realize there is a second series, but why would you watch it after what they did to the first?

Sub-par

A Certain Magical Index The Movie: The Miracle of Endymion
(Movie, 90 minutes)

This was the final straw in me watching “A Certain …” series. I really liked the first Scientific Railgun series (I think everyday superpowers is still a great concept), but this took much of what I did not enjoy about that series and added it to Magical Index (which was never as good), and come up with something that I really did not enjoy. Maybe this will serve as a reminder to me and I will not watch the next one (maybe).

Hozuki no Reitetsu
(TV Series, 13 episodes)

A slice-of-life anime set in Japanese Hell. It gets some points for being fairly original… However, I found the jokes fell flat more often than not, and that can leave a big gap when there is nothing else going on. With two stories per show (that is 10 minutes a plot line), nothing really caught my interest and I took a long time to finish watching this.

Log Horizon
(TV Series, 25 episodes)

Another “stuck in a game” anime, with the twist that there seems to be the lack of motivation to leave. Too much filler (especially near the end), forgettable action scenes, and a storyline that resembled a poor version of Spice and Wolf. The most entertained I was in this series was when other people started mocking the main character for always touching his glasses…

Noragami
(TV Series, 12 episodes)

I think this series was not particular bad in itself, but I think it could have been good and failed. I am still not sure what it was missing, but I find the whole thing completely forgettable (which may be why this “review” is so vague).

Terror in Resonance (Zankyou no Terror)
(TV Series, 11 episodes)

The initial build-up is OK – good even – but everything becomes overly contrived in the second half. Overall, this is probably a reasonable watch if you are not hoping for greatness, but I rate those series that screw up their potential harshly. And I just noticed the director was Shinichiro Watanabe, who has been involved in many of my favourite series. He did not have a great year according to this blog post!