Ubuntu Server

DV S I D
IN
+

D E
in the Azure Cloud

ISSUE 284 – JULY 2024

Laptop
Security
Easy steps for
protecting your
portable

thirty bees: Free and

community-focused NEON: Optimize code for
e-commerce solution the Rasp Pi and other
Safer Surfing: We round ARM systems
up some top browser Easy Reading: Save a web
extensions for security page to ebook format
and privacy
Git Tricks: Check your
code before you commit 10 TERRIFIC
FOSS FINDS!
W W W. L I N U X - M A G A Z I N E . C O M
 EDITORIAL
Welcome

VOYAGE TO WHEREVER
Dear Reader,
AI has continued its meteoric rise in the news headlines and Even the mighty Google is getting pushback over the
stock market reports. Companies are betting their futures on excesses of its AI vision. Last month, I wrote about
AI, and the whole tech world seems tuned in, breathlessly Google’s new plan to answer queries with AI-generated
waiting for a clue about where it might be going. But despite responses, rather than displaying links to the sites that
all the media hoopla, AI is not having a great month. served as the actual source of the information [6]. Based
It seems like every time I look at the news, something or on feedback (read: “outcry”), they have now announced
someone is pushing back. Sony just issued a warning not that they will make it possible to see the good ol’ web
to use its content to train AI [1]. Here in the open source links without scrolling to the bottom of the screen – al-
space, NetBSD announced that it is banning all AI-gener- though you will have to click a couple of menu options
ated code [2]. Even TikTok, which is no stranger to its own to get Google to cough up what we used to call the
controversies, announced that it will start watermarking “search results” [7].
AI-generated images posted on the platform [3]. We don’t know yet whether people will actually click on
Some of the scandals tear a bit deeper into the fabric of the these search menu options – and if they do, will Google go
culture. The US Publisher Wiley just announced that it is clos- back to the old way and stop trying to morph itself into the
ing down 19 scholarly journals, in part due to their publishing world’s AI answer-bot? Will the outcry over AI-generated
AI-generated articles from so-called paper mills that generate articles and fake product reviews cause us to renew our
academic papers for hire. The company has apparently had respect for journalism? Or are we merely “training” the AI
to withdraw 11,300 papers in the past two years due to au- to get better at faking?
thenticity issues. The article in The Register [4] also notes I have no illusions that these pushback efforts will stall
that the number of computer science papers submitted to the the rising momentum of AI, but the fact is, with our
online archive arXiv over the past four years (a time frame government leaders embroiled in the vital business of
coinciding with the rise of ChatGPT and other AI tools) is up investigating each other and raising money for the next
by 200 percent. (Are computer scientists that much more election cycle, these kinds of consumer-based checks
productive than they were four years ago, or is something are about the only meaningful restraints we have right
else going on?) Even National Public Radio got into the act, now on the AI industry. If nothing else, they promote
with a report on the content company AdVon, which has discussion, and we need a lot
passed off AI-generated articles and product reviews to more discussion to chart a
mainstream publications like Sports Illustrated, the Los An- safe course through these
geles Times, and the Miami Herald [5]. unknown waters.

Info
[1] Sony AI training opt out: [Link]
sonymusic/declaration-of-ai-training-opt-out/
[2] NetBSD Commit Guidelines: [Link] Joe Casad,
developers/[Link] Editor in Chief
[3] TikTok on AI transparency: [Link]
partnering-with-our-industry-to-advance-ai-
transparency-and-literacy
[4] “Wiley Shuts Down 19 Scholarly Journals Amid AI Paper Mill
Problems” by Thomas Claburn, The Register, May 16, 2024 :
[Link]
[5] “AI-Generated Articles are Permeating Major News Publications”
by Kathryn Fink, Christopher Intagliata, and Ailsa Chang, NPR,
May 16, 2024: [Link]
ai-generated-articles-are-permeating-major-news-publications
[6] “Sure You Need This Toy” by Joe Casad, Linux Magazine,
issue 283, June 2024: [Link]
2024/283/Welcome
[7] “Revolutionary New Google Feature Hidden Under ‘More’ Tab
Shows Links to Web Pages” by Samantha Cole, 404 Media,
May 15, 2024: [Link]
filter-ai-overview/

[Link] ISSUE 284 JULY 2024 3

 JULY 2024

ON THE COVER
28 thirty bees 62 Optimizing Pi Code
Build an online store with this simple and Use ARM NEON instructions to speed up
practical e-commerce solution. mathematically intensive tasks.

34 Security and Privacy Extensions 78 Web to Ebook

Protect your Internet presence from prying Save your eyeballs and disk space by
eyes. converting HTML pages to ebook format.

40 Git Pre-Commit 90 Ubuntu in the Azure Cloud

Configure the Git repository service to check A cloud instance will save you floor space and allow
for errors automatically. you to offload some of those pesky admin tasks.

NEWS IN-DEPTH
8 News 28 E-Commerce Solution
• Fedora Asahi 40 Remix Available for Macs with Apple Silicon Thirty bees offers a feature rich, open source e-commerce
• Red Hat Adds New Deployment Option for Enterprise solution for setting up your online store.
Linux Platforms
• OSJH and LPI Release 2024 Open Source Pros Job Survey 34 Security and Privacy Extensions
Results Many hands are hard at work on problems of Internet
• Proton 9.0-1 Released to Improve Gaming with Steam security and privacy. If you’re looking to lock down your
• So Long Neofetch and Thanks for the Info surfing experience, try these privacy-focused browser
• Ubuntu 24.04 Comes with a “Flaw” extensions.
• Canonical Releases Ubuntu 24.04
• Linux Servers Targeted by Akira Ransomware 40 Git Hooks
The pre-commit framework lets you automatically manage
12 Kernel News and maintain your Git hook scripts to deliver better Git
Zack Brown reports on developer trust. commits.

46 Command Line – Environmental Variables

COVER STORY Environmental variables often operate quietly in the
background, but knowing how to set, modify, and delete
16 Laptop Security them can come in handy.
Linux is quite secure compared to the alternatives, but
you’ll need a few additional steps if you really want to lock 50 Programming Snapshot – Go Bandwidth
it down. We’ll introduce you to some practical tools for Display
antivirus protection, firewall configuration, and A Go program running on a Raspberry Pi grabs metrics
sandboxing. from a pfSense firewall and displays them on a miniature
display to help Mike Schilli keep an eye on his Internet
connection’s bandwidth usage.
REVIEW
24 Distro Walk – Ubuntu Budgie
Ubuntu Budgie combines the simplicity of the Budgie
desktop with the power of Ubuntu, resulting in a 95 Back Issues 97 Call for Papers
customizable desktop experience. 96 Events 98 Coming Next Month

4 JULY 2024 ISSUE 284 [Link]

16 Laptop Security
In the scary world of the Internet,
“more secure than Windows” still
isn’t secure enough. If you want to
keep your traveling systems safe 71 Welcome
from the clutches of the espionage This month in Linux Voice.

economy, you’ll need some extra 72 Doghouse – Entrepreneurs

help. We show you how to outfit Advances in technology have opened up possibilities
for potential entrepreneurs, but running a small
your laptop with the extra defenses business still means doing many jobs.
you’ll need for life on the road.
73 Color on the Terminal
You don’t necessarily need color on the terminal, but still,
it does look good – and does not involve too much effort.
MakerSpace 78 Web to Ebook
Saving web pages to ebooks conserves space and
58 Snek leads to easier reading.
Reuse your old Arduino hardware while learning Python.
84 FOSSPicks
62 SIMD Code Optimization Our new columnist Nate Drake looks at Audacity, Endless
Coding for the ARM NEON vector hardware can Sky, GCompris, Switcheroo, MS-DOS, Qemu, and more!
significantly improve performance and help you get the
most out of low-power systems like 90 Tutorial – Ubuntu VM in the Azure Cloud
the Raspberry Pi. Are you ready to get started with the cloud? Microsoft’s
Azure Cloud Services provides easy access to an Ubuntu
virtual machine.

@linux_pro
TWO TERRIFIC DISTROS
@linuxpromagazine
DOUBLE-SIDED DVD!
Linux Magazine SEE PAGE 6 FOR DETAILS

@linuxmagazine

[Link] ISSUE 284 JULY 2024 5

DVD
This Month’s DVD

Ubuntu Budgie 24.04 LTS and Rescuezilla 2.5

Two Terrific Distros on a Double-Sided DVD!

Ubuntu Budgie 24.04 LTS Rescuezilla 2.5

64-bit 64-bit
Ubuntu Budgie is an official flavor of Ubuntu, the This special-purpose Linux serves as a counterpart
popular Debian-derivative. Ubuntu Budgie 24.04 LTS to the well-known Clonezilla disk cloning system.
(Noble Numbat) is a long term support (LTS) release, The goal of Rescuezilla is to let you back up, restore,
which will be supported until May 2027. Built around and recover your system – even if it won’t boot nor-
Ubuntu Core, Ubuntu Budgie ships with its own mally. Rescuezilla can save and restore Linux, Mac,
applications, such as the Control Center, as well as or Windows systems. You can also use Rescuezilla
accessibility tools and applets. It differs from the general to recover lost data, extract files from backup im-
Budgie desktop in its default use of icons rather than text ages, and access images created by virtual machine
and a dock for open apps. tools such as VirtualBox and VMWare. The 2.5 re-
The 20.24 release offers improved hot corners, lease is based on Ubuntu 24.04 and includes a new,
additional tiling options, and Bluetooth tethering experimental command-line interface, as well as
(i.e., the ability to access the Internet from a phone bug fixes and automated integration with test suite
linked to the desktop). Also featured are numerous new scripts.
and redesigned applets and additional administrative
tools, including new settings for Wacom tablets, battery
indicators for Bluetooth devices, and a Restore button
for the Trash. By any standards, Noble Numbat is a
major release for users of all levels, enhanced by a
strong aesthetic sensibility.

Defective discs will be replaced. Please send an email to subs@[Link].

Although this Linux Magazine disc has been tested and is to the best of our knowledge free of malicious software and defects, Linux Magazine
cannot be held responsible and is not liable for any disruption, loss, or damage to data and computer systems related to the use of this disc.

6 JULY 2024 ISSUE 284 [Link]

 NEWS
Updates on technologies, trends, and tools
THIS MONTH’S NEWS
08 • Fedora Asahi 40 Remix
Available for Macs with
Apple Silicon
• Red Hat Adds New
Deployment Option for
Enterprise Linux Platforms Fedora Asahi 40 Remix Available
09 • OSJH and LPI Release
2024 Open Source Pros
for Macs with Apple Silicon
Job Survey Results Last week, the Fedora Project announced the release of Fedora Remix 40 for Apple
• Proton 9.0-1 Released Silicon-powered Macs ([Link]
to Improve Gaming with available/ ). This new release includes Plasma 6 and was developed in conjunction with
Steam Asahi Linux project.
• More Online On top of the vast amount of improvements in Plasma 6, this new release also

10 • So Long Neofetch and

Thanks for the Info
adds OpenGL 4.6 support, which is conformant with the latest OpenGL versions
and brings compatibility with modern OpenGL workloads (such as with Blender).
• Ubuntu 24.04 Comes Conformant drivers must pass more than 100,000 tests to be included. You can
with a “Flaw” view the official list of drivers here ([Link]
conformant-products/opengl#submission_347).

11 • Canonical Releases
Ubuntu 24.04
The new release also features a customized Calamares-based initial setup wizard.
Along with the KDE Plasma remix, Fedora also released a Gnome variant that includes
• Linux Servers Targeted all of the latest features found in Gnome 46. As well, you’ll find a new Fedora Server
by Akira Ransomware variant of the Fedora Asahi remix, so you can turn your Mac into a powerful Linux server.
You can find official installation instructions here ([Link]
en-US/fedora-asahi-remix/installation/). The installation is as simple as running the
following command on your Apple Silicon-based Mac:

curl [Link] | sh

You’ll be prompted for your user password and the installation will begin. If you’d
prefer a nightly build, the command is:

curl [Link] | sh

Red Hat Adds New Deployment Option

for Enterprise Linux Platforms
If you work in an enterprise environment, you are probably familiar with Red Hat. What
you might not know is that the company just introduced image mode, which will serve
as a new deployment method for Red Hat Enterprise Linux (RHEL) that delivers the OS
as a container image.
Image mode is a container-native approach for the building, deploying, and manag-
ing of the Red Hat operating system and provides a single workflow to manage the
entirety of your IT landscape.
The reason image mode has come into being is an AI-centric future. According to
Matt Micene, Solution Architect at Red Hat, “...we’ve been exploring AI workloads.

8 JULY 2024 ISSUE 284 [Link]

 NEWS
Linux News

AI brings challenges of complicated software stacks and particular hardware

support to the forefront of application development.” MORE ONLINE
Micene continues, “And AI workloads are being built in every possible combina-
tion of cloud, edge, and on premises. Image mode for RHEL gives us a way to pull
all of these worlds together for tight dependency management across the applica- Linux Magazine
tions and the underlying hardware when building, testing, and deploying AI applica- [Link]
tions, both through its flexible nature and tight integration with Podman Desktop
and Podman AI Lab.” ADMIN HPC
Red Hat believes image mode will offer enterprise businesses a complete inven- [Link]
tory of standard images and environments, tracking of OS images, simple updates Desktop Blades (of Glory)
and rollbacks, faster experimentation, and the ability to explore containerized CI/CD. • Jeff Layton
Read more about image mode in the official Red Hat announcement (https:// The LattePanda Mu low-power HPC compute
[Link]/en/blog/introducing-image-mode-red-hat-enterprise-linux). module puts an HPC system on your desktop.

ADMIN Online
[Link]
OSJH and LPI Release 2024 Open Source
Recovering from a cyberattack in a hybrid
Pros Job Survey Results environment
• Evgenij Smirnov
Recently, the Open Source JobHub (OSJH) ([Link] ) and Restoring identity is an important part of
LPI ([Link] ) teams surveyed open source professionals to learn what disaster recovery, since it lays the founda-
they value most when seeking a new job role. tion for restoring normality and regular
“When looking at today’s tech job market, it’s important to understand the per- operations. We look into contingency
spective of those who are building their careers with FOSS,” says Brian Osborn, measures for hybrid directory services with
Entra ID, the Graph API, and its PowerShell
Founder of OSJH, and CEO and Publisher at Linux New Media. “This survey offers
implementation.
much-needed insight into what those open source professionals prioritize in terms
of both new opportunities and satisfaction with their current role.” Automatically terminate OpenSSH sessions
The results of this survey are now available in the free 2024 Open Source Profes- • Thorsten Scherf
sionals Job Survey Report ([Link] Disconnect OpenSSH user sessions after a
According to the findings, those who work with free and open source software certain period of inactivity with the systemd-
(FOSS) consider a variety of factors when seeking a new job role, including overall logind service.
work-life balance, open source policy, company culture, and training and certification
Intrusion Detection with OSSEC
opportunities. For example, 89 percent of respondents said they considered an • Thomas Joos
employer’s open source policy when making job choices. The OSSEC free intrusion detection and
Read the complete report at OSJH: [Link] host-based intrusion prevention system
detects and fixes security problems in real
time at the operating system level with
functions such as log analysis, file integrity
Proton 9.0-1 Released to Improve checks, Windows registry monitoring, and
Gaming with Steam rootkit detection. It can be deployed virtually
anywhere and supports the Linux, Windows,
and macOS platforms.
When Proton 9 was initially released, there was a persistent issue for Linux users
involving the download of games.
When using Proton 9, if you attempted to download any game the app would say
that the download was in progress but would remain at zero percent. This not only
happened on some Linux distributions but on the Steam Deck as well.
That issue has been resolved with the 9.0-1 release, so anyone having the down-
load issue should upgrade immediately to resume normal functionality.
The download issue isn’t the only change found in the new release. Users will
find even more Windows games now work on Linux (without having to resort to
Proton Experimental), such as:
• Dinogen Online
• Photography Simulator demo
• George McGeehan Gamer Hero
• The Finals
• Command & Conquer: Red Alert 2, Yuri’s Revenge, and Tiberian Sun
• Aisling and the Tavern of Elves
• Snares of Ruin 2
• Bloody Walls

[Link] ISSUE 284 JULY 2024 9

NEWS
Linux News

Other games are now available to play on high core count CPUs, such as Far
Cry 2/4, The Witcher 2: Assassins of Kings Enhanced Edition, Lara Croft and the
Guardian of Light, and more.
Several games saw fixes for various types of issues and Wine Mono was updated
to version 9.1.0
You can read the full changelog for version 9.0-1 on the Valve Software GitHub
page ([Link]

So Long Neofetch and Thanks for the Info

The developer, Dylan Araps, has officially archived the Neofetch GitHub repository,
making it read-only. To make his point, he added a [Link] file in the root of his
repository that includes the single line: Have taken up farming.
This should come as no surprise. According to It’s FOSS ([Link]
neofetch-rip/), the lead developer has gone AWOL before, such as three years ago
when the development of k1ss Linux (Araps project at the time) declined dramatically.
It happens in the open source community. Many of these projects are done on a
volunteer basis; when time becomes a problem, they get set aside. But at the
same time his development on k1ss Linux dropped off, Dylan went radio silent
([Link] ) and
no one could find him.
The good news is that the Neofetch code can still be cloned from GitHub (https://
[Link]/dylanaraps/[Link]). Should another developer decide to fork the
project, it’s all there and hasn’t been touched for three years.
My guess is that some open source developer will take up the torch and bring this
fan-favorite project back to life. If not, maybe another app will bubble up to the sur-
face that allows Linux users to show their support for their favorite distribution and
show off their systems.
Until then, we can only guess as to what hardware others are using and what
distribution they’ve chosen as their default.
Farewell, Neofetch. It was fun while it lasted.

Ubuntu 24.04 Comes with a “Flaw”

This was first reported by OMG Ubuntu ([Link]
dont-upgrade-to-ubuntu-24-04-yet) and It’s FOSS ([Link]
24-04-disappointment/), but it’s something I experienced early on when I was testing
the daily releases of Ubuntu 24.04.
The issue is how Canonical has secretly forced Snap installation on users. Previ-
ously, if you were to download a DEB file from the Internet, the software installer
GUI would open and prompt you to install the app.
That no longer happens. Instead, the file is downloaded and that’s that. If you dou-
ble-click the file, instead of an installer opening, the Archive tool opens, which is of
no help to users.
To take this further, it looks as if the Software app defaults to Snap packages for
everything now. I combed through various apps and found this to be the case.
I was able to find an exception with the Clementine audio player, which is no lon-
ger in development. When searching for that app, two versions appear – the Snap
and DEB packages. But if you only search for Clementine and hit Enter, the Snap
Get the latest news package is the only one you see. Run the search and wait for the drop-down to pop-
ulate, and you’ll see two different versions – one listed as a Snap package and one
in your inbox every listed as a Debian package.
week That means all is not lost for DEBs, but you have to be a bit sneaky. As far as the
auto-installation of downloaded DEB files, you’ll have to install something like gdebi
Subscribe FREE to bring back this feature.
to Linux Update It also seems (according to It’s FOSS) that Canonical has no intention of fixing this
[Link]/Linux-Update “flaw” and will, most likely, continue to migrate Ubuntu until it is a Snap-only system.

10 JULY 2024 ISSUE 284 [Link]

 NEWS
Linux News

Canonical Releases Ubuntu 24.04

Canonical has released Ubuntu 24.04 (Noble Numbat), which was delayed because
of the XZ Utils vulnerability that threatened to take down Linux distributions every-
where (but thankfully didn’t).
As far as what’s new, you’ll find Gnome 46 at the forefront, which vastly improves
the file manager experience and offers the best-performing version of the open
source desktop to date. Gnome 46 also brings a revamped Settings tool that arranges
certain sections (such as the Privacy option) in a much more logical fashion. Gnome
46 also improves the notifications feature, adding grouped notifications (per app) so
there’s less clutter.
In addition, Ubuntu 24.04 adds a brand new installer, which makes the installation
of the OS not only easier but more modern looking. As well, you’ll find some of the
settings in the installer have been rearranged (for example: all the accessibility set-
tings are in one location and can be customized).
The Software Center has been renamed to App Center and has been tweaked for
a more user-friendly experience. A new Manage sections offers an overview of in-
stalled and updated software, where you can check for and apply pending updates.
Other features and changes include Thunderbird as a Snap package, plenty of per-
formance tweaks, UI refinements, advanced ZFS filesystem features, kernel 6.8,
the enforcement of Retpoline ABI checks, and more.
You can download Ubuntu 24.04 from the official download page (https://
[Link]/download/desktop) and read about it in the official release notes
([Link]

Linux Servers Targeted by Akira Ransomware

Since March of 2023, the Akira ransomware has hit businesses and critical infra-
structure organizations in North America, Europe, and Australia.
According to this alert from the Cybersecurity & Infrastructure Security Agency
([Link]
visory-akira-ransomware), “Evolving from an initial focus on Windows systems to a
Linux variant targeting VMware ESXi virtual machines, Akira threat actors began de-
ploying Megazord (a Rust-based code) and Akira (written in C++), including Akira_v2
(also Rust-based) in August 2023. Akira ransomware has impacted a wide range of
businesses and critical infrastructure entities in North America, Europe, and Austra-
lia and claimed approximately $42 million (USD) in ransomware proceeds.”
This new Linux variant takes advantage of specific vulnerabilities found in Linux
for the purpose of credential theft and phishing.
Shortly after their Windows campaign started, the collective deployed a Linux vari-
ant of Akira that targeted VMWare ESXi virtual machines. This mutation of Akira to
target Linux enterprise environments is similar to what has been happening with
other ransomware groups, such as LockBit, CLOp, Royal, Monti, and RTM Locker.
According to K7 Security Labs ([Link]
play-with-linux/), “It appears that the ransomware operator dynamically constructs
the ransomware with a fresh public RSA key for each target, along with a corre-
sponding Unique ID appended in the ransomware note. The purpose of this Unique
ID is to facilitate the attacker in determining the specific ransomware build that
infected the victim, thereby identifying the corresponding private key required
for decrypting the compromised files.”
The Akira group has previously disclosed stolen data on its website, should victims
refuse to comply with their demands.

QQQ

[Link] ISSUE 284 JULY 2024 11

NEWS
Kernel News

Zack’s Kernel News

Developer Trust posted from his @[Link] email
In recent days, the infamous “XZ back- address, he used his @[Link]
door” has the entire open source world account because “@[Link] emails
reconsidering its development practices. resent from the mailing list are classified
Essentially, a bad actor joined an open by Gmail as spam, we are working on
source project, submitted some good fixing it.”
patches to gain the trust of the develop- Linus Torvalds replied, “I’ve pulled
ers, and eventually submitted some clev- from you before, but I still don’t have a
erly hidden security holes that were ac- signature chain for your key (not that I
tually accepted into the project. It was can even find the key itself, much less a
only when a regular user noticed some signature chain). Last time I pulled, it
odd timing behaviors in the tool and was after having everybody else just
decided to track down the issue that the verify the actual commit.”
whole thing came to light. The idea of keys and signature
Chronicler Zack Brown reports The open source project was not the chains in Linux kernel development
Linux kernel, but the tool came very comes from a lawsuit years ago, which
on the latest news, views, close to being included in many Linux asserted copyrighted materials were
dilemmas, and developments distributions. From there of course, it being incorporated into the Linux ker-
within the Linux kernel would have been inside the foundational nel source tree. Up until that time, es-
infrastructure of the entire Internet and tablishing the provenance of a given
community. almost every corporate network within piece of kernel code was a difficult
By Zack Brown the known universe. task, and the lawsuit dragged on for
It’s absolutely not the first time this quite a while before everything could
has been attempted, and, of course, be sorted out. After that, Linus and
there could be any number of similar others came up with the idea of kernel
backdoors that have not yet been discov- developers reviewing and signing off
ered. The whole experience has been a on each other’s patches and having
wake-up call for the open source world those details be part of the patch log
to re-examine their code review prac- entries. As part of this, each developer
tices. For example, one reason the XZ needed to generate a cryptographic key
backdoor was able to make it into the that would uniquely identify them; this
project was because the maintainer was key could be verified by other develop-
overworked and burnt out. So the issue ers who would meet that person in real
is about more than simply expecting ev- life. In this way, any future question
eryone to work harder. We’ll be seeing about where a patch came from could
the true effects of this wake-up call for be resolved quickly, and the legality or
years to come. authenticity of the code could be
At around the same time the XZ back- identified.
door was discovered, Roberto Sassu sub- As we can see from Linus’s interac-
mitted a security patch to Linus Torvalds tions with Roberto and others, the keys
for inclusion in the Linux kernel. To be and signatures are not always an abso-
clear, I’m not trying to imply any accusa- lute requirement for a patch to be ac-
tion against Roberto, but only to point cepted into the kernel. In general,
Author out an interesting moment between Linus will give gentle reminders,
The Linux kernel mailing list comprises kernel developers. which become more pointed over time,
the core of Linux development activities. Roberto’s patch “fixes a kernel panic until eventually it does become an ab-
Traffic volumes are immense, often in the newly introduced function secu- solute requirement. However, until
reaching 10,000 messages in a week, and rity_path_post_mknod(), when trying to then, patches may be accepted from
keeping up to date with the entire scope check if an inode is private. The panic that person without keys or signatures.
of development is a virtually impossible occurs because not all dentries have Of course, keys and signatures are not
task for one person. One of the few brave an inode attached to them.” Although a cure for hostile patches; they simply
souls to take on this task is Zack Brown. Roberto, a Huawei employee, usually allow Linus (or any project leader) to

12 JULY 2024 ISSUE 284 [Link]

 NEWS
Kernel News

have a reasonable confidence that some- working with the associated subsystem
one they already trust has vouched for a devs to get approval. In the cases where
person or their patch. we don’t get an explicit ACK, there is an
In this case, Linus looked the patch on-list approval, or several ignored on-
over with his own eyes and offered his list attempts over weeks/months/years.
evaluation: We want to be good neighbors. Roberto’s
“I have to say that I also think the secu- original patch which converted from the
rity layer code in question is ENTIRELY IMA/EVM hook to the LSM hook was
WRONG. ACK’d by the VFS folks.”
“IOW [in other words], as far as I can Al said in response, “Unkind com-
tell, the mknod() system call may indeed ments about the LSM folks’ tendency to
leave the dentry unhashed, and rely on shove hooks in places where they make
anybody who then wants to use the new no sense had been brought by many
special file to just do a “lookup()” to things […]. I’m not blaming Roberto –
actually use it. that really seems to be the general atti-
“HOWEVER. tude around LSM; I’ve seen a _lot_ of ‘it
“That also means that the whole no- doesn’t matter if it makes any sense,
tion of post_path_mknod() is complete somebody might figure out some use for
and utter hogwash. There is not any- the data we have at that point in control
thing that the security layer can possibly flow, eventually if not now’ kind of re-
validly do. sponses over the years. IME [in my expe-
“End result: instead of checking the rience] asking what this or that hook is
‘inode’ for NULL, I think the right fix is for and what it expects from the objects
to remove that meaningless security passed to it gets treated as invalid ques-
hook. It cannot do anything sane, since tion. Which invites treating hooks as
one option is always ‘the inode hasn’t black boxes….”
been initialized yet’. Paul replied in a conciliatory tone:
“Put another way: any security hook “It’s rather common for subsystems to
that checks inode in security_path_post_ push back on the number LSM hooks,
mknod() seems simply buggy.” which ends up resulting in patterns
Linus concluded that, partly because where LSM hooks are placed in as wide a
of these technical issues, and partly be- scope as possible both to satisfy the re-
cause of the key and signature issues, he quirements of the individual subsystems
couldn’t accept the patch without more as well as the LSM’s requirements on cov-
developers reading the code and signing erage. Clearly documenting hooks, their
off on it. inputs, return values, constraints, etc. is
There was a small discussion sur- important and we need to have those dis-
rounding the patch, including airing cussions as part of the hook. This is a big
some tensions between the Linux Secu- part of why we CC the subsystems when
rity Module (LSM) developers – of adding new hooks and why I make sure
whom Roberto is one – and other kernel we get an ACK or some other approval for
developers. In the context of what I’m a subsystem maintainer before we merge
talking about, the point is that there are a new hook. Is the system perfect, no,
often many moving parts to any discus- clearly not, but I don’t believe it is for a
sion, with keys and signatures not neces- lack of trying or any ill intent on the part
sarily being the highest profile part of of the LSM devs. We recently restored the
the conversation. LSM hook comment blocks in security/
For example, during the conversation, security.c (long story), I would gladly
Al Viro remarked, “LSM folks have a dis- welcome any comments/edits/sugges-
turbing fondness of inserting hooks in tions you, or anyone else may have,
various places, but IMO this one has no about the docs there – I will be the first
business being where they’d placed it. to admit those docs have rotted quite a
Bikeshedding regarding the name/argu- bit (once again, long story). If you have
ments/etc. for that thing is, IMO, not corrections, notes, or constraints that
interesting….” should be added please let me know
To which Paul Moore replied, “I know and/or send patches. Similarly, if you’re
it’s everyone’s favorite hobby to bash the aware of any hooks which are ill advised
LSM and LSM devs, but it’s important to and/or poorly placed, let us know so we
note that we don’t add hooks without can work together to fix things.

[Link] ISSUE 284 JULY 2024 13

NEWS
Kernel News

“I’m serious Al. These aren’t just words pains to do everything right all the
in an email. I realize you don’t have a lot time. Things like insisting on following
of free cycles, but if you do have feedback patch verification and testing proce-
on any of those things above, I’m dures may take a back seat to suspicion
listening. and resentment. Or, insisting on follow-
“I *really* want to see better collabora- ing procedures may become a battering
tion between various subsystems and the ram one developer may use to slow or
LSMs; that’s part of why I get annoyed stop contributions from another devel-
with LSM bashing, leaving the LSM devs oper. And, if the social difficulties re-
out of security/LSM related threads, etc. solve themselves, relaxing important
it only helps keep the divide up between procedures may feel like one way of
the groups which is bad for all of us.” extending an olive branch, while inad-
Meanwhile, Eric W. Biederman vertently also resulting in less careful
laughed a great guffaw right in Paul’s review of incoming code.
face, exclaiming: None of this is easy or simple. Linus,
“You merged a LSM hook that is only Al, Roberto, Paul, Eric, and many others
good for breaking chrome’s sandbox, must navigate daily development inter-
over my expressed objections. After I actions – along with their day jobs, in-
tested and verified that is what it does. volvements in other open source proj-
“I asked for testing. None was done. It ects, and personal lives – and still some-
was claimed that no security sensitive how stay on the ball when it comes to
code would ever fail to check and deal recognizing and stopping social engi-
with all return codes, so no testing was neering attacks from bad actors trying to
necessary. Then later a whole bunch of gently insert malicious code into their
security sensitive code that didn’t was projects.
found. The most amazing thing about all of
“The only redeeming grace has been this, to me, is that the open source devel-
that no-one ever actually uses that misbe- opment model itself evolves in the full
gotten security hook. light of day to address all of these issues.
“P.S. Sorry for this off topic rant but Back in the 1990s, the threats were seen
sheesh. At least from my perspective you to come from competitors like Microsoft,
deserve plenty of bashing.” and the developers had to weather the
Paul said merely, “Just in case people various storms publicly, with nothing
are reading this email and don’t recall but the correctness of their ideas to
the security_create_user_ns() hook guard against opponents who knew
discussions from 2022, I would suggest every detail of those ideas. Thirty years
reading those old threads and drawing later, Linux rules the world. In this
your own conclusions. A lore link is world, there are governments, corpora-
below: [Link] tions, black hat groups, and individuals,
linux-security-module/? who likewise can clearly see everything
q=s%3Asecurity_create_user_ns.” these open source projects like Linux
And the discussion came to an end. and others are doing, while the projects
It’s often difficult for developers who themselves again have nothing more
might have various grievances and his- than the correctness of their ideas to
tories with one another to take the wield in their own defense. Q Q Q

QQQ

14 JULY 2024 ISSUE 284 [Link]

COVER STORY
Laptop Security

Practical tools for locking down your Linux portable

Tight Ship
Linux is quite secure compared to the alternatives, but you’ll need a few additional steps if you
really want to lock it down. We’ll introduce you to some practical tools for antivirus protection,
firewall configuration, and sandboxing. By Chris Binnie

I
t occurred to me recently that the laptop I devote to my lots of interesting information [3], and I would definitely rec-
personal use did not have the same add-on protections I ommend putting LMD through its paces.
routinely expect from systems I use at work. In one sense, For this purpose, I’ll turn to one of the most popular antivi-
this is understandable. (No one gets paid for integrating rus solutions on Linux, ClamAV [4], which describes itself as
my personal laptop into a comprehensive security infrastruc- an open source engine for “detecting trojans, viruses, malware,
ture, and no one will get fired if I get hacked.) However, the and other malicious threats.”
threats posed by Internet activity are very real, especially for I will leave you to look at ClamAV’s impressive feature set
a laptop computer that operates in public spaces behind low- and will focus on how to get it up and running. There are mul-
tech coffee house firewalls that someone else configured. tiple ways to install ClamAV. I will opt for the package manager
When I read about the Infostealer malware targeting Linux [1], route on Debian derivatives such as Ubuntu Linux. The com-
I decided it was a good time to explore the options for using se- mand is:
curity sandboxing techniques to isolate applications. And while
I was at it, I took a closer look at antivirus options and local $ apt install clamav

firewall tools that would make me less dependent on the secu-

rity of whatever subnet I happen to have landed in. The following packages are installed, using up only 1.3MB of
Of course, users expect convenience and simplicity for their disk space:
home systems. Tools that are too elaborate or complicated are
often ignored – or set up once and then forgotten. For my sys- clamav clamav-base clamav-freshclam libclamav9 libmspack0 libtfm1

tem, I set out to find convenient yet powerful tools that could
provide virus protection, firewalling, and sandboxing support. To see what ClamAV is doing, I decided to run the following
Eventually I settled on the following cocktail: command:
• ClamAV for virus protection
• UFW for firewall configuration $ ps -ef | grep clam

• Firejail for sandboxing clamav 7503 1 9 08:41 ? [Link] /usr/bin/freshclam -d U

Linux Magazine has covered all of these tools at various points --foreground=true

in the past, but this article is an effort to bring them all to-
gether in a single configuration study for the Linux road The command reveals that straight after installation (I haven’t
warrior. run any commands yet) the busy ClamAV is running a process.
Checking with the trusty top command, there’s little CPU load
Clamming Up related to the process. A quick command after guessing the
Antivirus protection is an important component of any compre- manual page name, as shown, reveals the diligent ClamAV is
hensive laptop-protection strategy – even if you are using updating its virus definitions automatically:
Linux. Linux malware does exist, although it does not receive
as much attention as Windows malware. Cybercrime is becom- $ man freshclam

ing evermore sophisticated, and it is always possible that the

criminals will find a vulnerability before the security patch The manual points you at the configuration file /etc/clamav/
makers, and, even if your system itself is not specifically vul- [Link]. Among many settings, such as logging and
nerable, it is good policy to detect such threats and keep them where to store the virus definitions, the config file has two
off your system. lines that reveal an hourly update of the virus definitions
You’ll find a chapter on the sophisticated Linux Malware (which makes it all the more important that the system load
Detect (LMD) in one of my books [2]. LMD’s README file has is minimal):

16 JULY 2024 ISSUE 284 [Link]

 # Check for new database 24 times a day Listing 1: ClamAV Scan
Checks 24
----------- SCAN SUMMARY -----------

Known viruses: 8671660

If you have a look, you can find a systemd [5] unit file for the
Engine version: 0.103.8
service called [Link]. The package manager
Scanned directories: 2430
version holds 8,671,660 definitions at the time of writing,
which is pretty impressive out of the box. To manually update Scanned files: 44055

the definitions, the process involves stopping the freshclam Infected files: 2
service, running the command of the same name freshclam, Data scanned: 1222.22 MB
and then starting the service up again. Data read: 1087.28 MB (ratio 1.12:1)
Without further ado, I decide to run a virus scan on my
Time: 264.059 sec (4 m 24 s)
home directory without manually updating:

$ clamscan -r --bell -i /home/chris The ClamAV developers provide several different packages
for various tasks and Linux versions [6]. There are several
Here the -i only shows infected files to keep the output noise ways to install and run ClamAV, but most importantly, there is
to a minimum, the -r stands for recursive directory scanning, a daemon you can use to run the scanning periodically. You
and the -bell means beep the system bell should a nasty virus can skip this part if you are only interested in running manual
be found. scans.
Figure 1 shows why it is a good idea to use the -i option if I prefer to leave things ticking over in the background au-
you are manually running the scan. That’s because all the tomatically, so I install the daemon with the following
hidden files are displayed as they are scanned. It might sur- command:
prise you exactly what files Linux applications save in their
hidden “dot” directories within your home directory. $ apt install clamav-daemon

To do a full system command-line scan, just replace /home/

chris with / in the previous example. The final output (from The following packages are installed with a tiny (sub 300KB)
my home directory scan) will look like the output in Listing 1 footprint:
(after quite some time, if you have run ClamAV across your
whole system). clamav-daemon clamdscan

Figure 1: A tiny sample of what ClamAV is scanning as it delves into many hidden directories.

[Link] ISSUE 284 JULY 2024 17

COVER STORY
Laptop Security

Listing 2: Successful Catch $ apt install clamtk

% Total % Received % Xferd Average Speed Time Time Time

Dload Upload Total Spent Left

Open the app drawer in Ubuntu to find the icon for
100 68 100 68 0 0 105 0
ClamTK (Figure 2). Figure 3 shows the main menu.
stdin: [Link].EICAR_HDB-1 FOUND
The Ins and Outs
----------- SCAN SUMMARY -----------
For the small matter of preventing network traffic
Known viruses: 8671660
from damaging your computer, I have used iptables
(the front end for the Linux kernel’s Netfilter net-
Engine version: 0.103.8
work packet-filtering functionality) and have built
Scanned directories: 0
fairly lengthy rulesets in the past. On my laptop
Scanned files: 1
though, I want to be able to precisely control cer-
Infected files: 1
tain types of traffic without getting bogged down
Data scanned: 0.00 MB
with too much detail.
Data read: 0.00 MB (ratio 0.00:1)
I really like the commonly bundled Uncompli-
Time: 20.534 sec (0 m 20 s)
catedFirewall (often known as UFW), which is
popular in the Ubuntu space. UFW is a clever but
Listing 3: Default, Empty iptables Configuration simple to use application that still allows very
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
granular rules when necessary. UFW configura-
pkts bytes target prot opt in out source destination
tion is painless for beginners compared to the
learning curve required for starting from scratch
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
with iptables.
pkts bytes target prot opt in out source destination
UFW is installed by default in many modern
Ubuntu Linux
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
versions, but if
pkts bytes target prot opt in out source destination
you need to in-
stall it, use the
following
If I rummage around for systemd unit files again, I see the post- command:
installation triggers have installed and started this service for
the daemon: $ apt install -y ufw

$ systemctl list-unit-files | grep clam Before I show you UFW in

[Link] enabled enabled action, I’ll check what my
[Link] enabled enabled Ubuntu 22.04 laptop is Figure 2: The app drawer
showing for its standard icon for ClamAV’s UI,
That’s good news. This Arch Linux wiki page [7] offers a nice iptables configuration. ClamTK.
tip for testing ClamAV with a harmless virus signature, which
you will see in a moment. Otherwise, that page gives some top
advice about running ClamAV with alternative definition files
plus other surprisingly useful information. Have a read for fur-
ther detail.
Before testing ClamAV, a quick word about the configuration
file for the daemon, which is found at: /etc/clamav/[Link].
The manual pages (see: man clamd and man clamdscan) offer
more information.
To test ClamAV, the Arch Linux wiki page suggests this in-
nocuous test file:

$ curl U
[Link] | clamscan -

Abbreviated output is shown in Listing 2, showing a successful

catch by ClamAV with the output

stdin: [Link].EICAR_HDB-1 FOUND

A user interface takes the burden of command-line intrica- Figure 3: ClamTK offers a wealth of options to
cies away from users. Install a ClamAV user interface with: configure and operate ClamAV.

18 JULY 2024 ISSUE 284 [Link]

 COVER STORY
Laptop Security

In Listing 3, you can see an already-installed UFW instance In Figure 5, you can see how UFW appears on the desktop
(which is disabled) and what my iptables configuration looks with the default configuration, blocking incoming traffic by
like. (Look online for more information on tables, targets, and default and allowing all outbound traffic.
chains [8].) I don’t want to dwell on the UI because the output for the
The command used to check the iptables configuration is Rules and Report columns are constructed so clearly that lit-
as follows: tle explanation is needed. For the uninitiated however, open
up the Report column, which lists network connections, and
$ iptables -nvL click the plus sign at the bottom of the display to create a
new, relevant rule with minimum effort. Instead, I’ll dive a
The UI provided by UFW is very simple to use and, from the little deeper into how UFW works under the bonnet. Before I
app drawer in Ubuntu, offers the following view in Figure 4. To do that, now that I have enabled UFW’s default settings
find it, search for the word firewall and enter the sudo or root (with no to inbound traffic but yes to outgoing traffic), look
password to open it. at what UFW has changed in the iptables configuration (see
Figure 6). The output is generated by running the same
command as before.
Figure 6 is heavily abbreviated, and actually UFW has cre-
ated a whopping 177 lines of configuration in my case.
From the command line, the default UFW network settings
would look like:

$ ufw default deny incoming

$ ufw default allow outgoing

Easy isn’t it? Far quicker to get to grips with than iptables. If
you need to switch on UFW before entering these commands,
you can do so with:

$ ufw enable

Figure 4: The UFW UI is very easy to navigate. Figure 5: UFW switched on with its defaults.

Figure 6: New, abbreviated iptables config from UFW.

[Link] ISSUE 284 JULY 2024 19

COVER STORY
Laptop Security

Getting a little more advanced, you can also use the /etc/services the danger of opening something up or blocking access
file in Linux to allow an application through the firewall by name, accidentally:
HTTP in this case:
$ ufw reset

$ ufw allow http

Rule added The manual, which you can access with man ufw, states that the
Rule added (v6) reset option: “Disables and resets firewall to installation de-
faults. You can also give the --force option to perform the reset
As the output from the above command shows, you can choose without confirmation.”
to configure or disable IPv6. The bottom of the output from the There are loads of online guides to assist you if needed [9].
status command is now showing these two lines, one for IPv4 To deny a specific IP address, you can run this command:
and one for IPv6:
$ ufw deny from [Link] to any

80/tcp ALLOW Anywhere

80/tcp (v6) ALLOW Anywhere (v6) Just change [Link] to [Link]/24 to block all 254 IP
addresses in the referenced CIDR network range.
It’s worth saying at this point that adding the switch --dry-run You can also allow or block specific IP addresses per applica-
for each command is a wise approach when dealing with dis- tion or service:
ruptive firewall rules.
Removing a rule requires a couple of steps. In this case, $ ufw allow from [Link] to any port 80 proto tcp

deny doesn’t quite delete the rule but does disable the HTTP
rule just created: Just adjust the option after proto if required. See the manual
for more details.
$ ufw deny http The mighty UFW also has support for rate-limiting. That
Rule updated means you can restrict floods of traffic to protect your network
Rule updated (v6) stack from being overwhelmed. According to the man page:
“When a limit rule is used, ufw will normally allow the con-
The status command now shows this though: nection but will deny connections if an IP address attempts to
initiate 6 or more connections within 30 seconds.”
80/tcp DENY Anywhere You can set up rate limiting with this command:
80/tcp (v6) DENY Anywhere (v6)

$ ufw limit ssh/tcp

For good housekeeping, list the index number of the rule and
then delete the number in order to ensure that you are very You’ll find lots of excellent examples about how to use UFW. I
specifically targeting a rule: would suggest a skim through the available information to
make sure you don’t make unnecessary mistakes and get a grip
$ ufw status numbered on the massive list of features UFW provides.

The output is shown in Listing 4. Behind Bars

Then, to delete the HTTP rules (TCP port 80), simply run the Another useful tool for protecting your Linux machine from
following commands: attack is the security sandboxing application Firejail [10].
Firejail’s raison d’etre is to isolate specific applications in a
$ ufw delete 5 way that any damage to one will have limited blast radius on
Deleting: your system. You might not be aware that web browsers like
deny 80/tcp Google Chrome and Chromium and Docker containers make
Proceed with operation (y|n)? extensive use of isolation technologies. For more on
Rule deleted

Listing 4: UFW Status

One thing that has caught me out in the past is that, as soon as Status: active
one rule is deleted, the numbering of the following rules will
change. Be careful! To Action From
UFW does let you delete rules by name too, but I would rec- -- ------ ----
ommend using numbering, unless you are scripting around the [ 1] 4444 ALLOW IN Anywhere
functionality. Deletion by name looks like: [ 2] 8888 ALLOW IN Anywhere

[ 3] 8889 ALLOW IN Anywhere

$ ufw delete allow http
[ 4] 4445 ALLOW IN Anywhere

[ 5] 80/tcp DENY IN Anywhere

If it all goes horribly wrong then, only as a last resort and
[ 6] 80/tcp (v6) DENY IN Anywhere (v6)
very carefully, run the reset option, which does come with

20 JULY 2024 ISSUE 284 [Link]

 COVER STORY
Laptop Security

A Different Type of Sandbox for Devs

Google Chrome provides a privacy
sandbox (see Figure 7), which you can
find within its settings. According to
Google, “The Privacy Sandbox is a se-
ries of proposals to satisfy cross-site
use cases without third-party cookies
or other tracking mechanisms.”

sandboxing in Chrome, see the box enti-

tled “A Different Type of Sandbox for
Devs.” See the Chromium documenta-
tion for more on sandboxing [11].
For more on Privacy Sandbox, enter
the following URL in the Chrome address
bar: chrome://settings/privacySandbox.
According to the Firejail website, the
Firejail sandboxes are “… lightweight; Figure 7: Where to find Google Chrome’s Privacy Sandbox under Settings.
the overhead is low. There are no com-
plicated configuration files to edit, no socket connections open, • Permissions are then dropped.
no daemons running in the background. All security features • Firefox gets restarted as a non-root regular user.
are implemented directly in [the] Linux kernel and available on Firejail warns you that multiuser systems don’t suit SUID-
any Linux computer.” based sandboxes because other non-root users can potentially
Firejail is available in your package manager on Debian inherit unintended privileges. See the documentation for a
Linux derivatives and can be installed as follows: number of ways to mitigate the consequences of using SUID,
such as creating specific user groups. I like the following quote
$ apt install firejail from the docs about the popular ways of creating sandboxes:
“Currently there are exactly two technologies available: SUID
The resulting packages are tiny, at under 3MB in size: and user namespaces. Both of them are insecure. User
namespace has the advantage when things go wrong you can
firejail firejail-profiles blame it on kernel developers. For Firejail we use SUID.”

A quick word about Set Listing 5: Setting Up a Firejail

Owner ID (SUID): SUID files Configuring symlinks in /usr/local/bin based man created
are run by the owner of the on [Link] nslookup created
file when executed, as op- Xephyr created patch created
posed to the user that runs dconf-editor created pdftotext created
the file. If you are interested, dig created seahorse created

you can find any files on a dnsmasq created secret-tool created

system that may pose a secu- enchant-2 created ssh created

rity risk (if owned by the root enchant-lsmod-2 created strings created

user), with this command: eog created wget created

evince created whois created

xcalc created
$ find / -perm -u=s U evince-previewer created
evince-thumbnailer created yelp created
-type f 2>/dev/null
file-roller created zoom created

firefox created
Firejail uses this functional-
Adding user chris to Firejail access database
gapplication created
ity to run its sandboxes. It in /etc/firejail/[Link]
gcalccmd created
weaves its magic behind the
gedit created
scenes and drops otherwise Loading AppArmor profile
gimp created
powerful permissions as
gimp-2.10 created
soon as the sandbox gets Fixing desktop files in /home/chris/.local/
gnome-calculator created share/applications
started up. The documenta-
gnome-characters created [Link] created
tion takes the Firefox
gnome-font-viewer created [Link] created
browser as an example and
gnome-logs created [Link] created
describes the process as
google-chrome created [Link] created
follows: [Link] created
google-chrome-stable created
• The sandbox gets created [Link] created
host created
and built as the root user.

[Link] ISSUE 284 JULY 2024 21

COVER STORY
Laptop Security

I’ll use Gimp, the inimitable graphics tool, as an example of $ firejail --list

how to use Firejail. To set up Firejail with your desktop, run 17017:root::firejail --top

the following command. In my case, this command will run as 18752:chris::firejail gimp

the chris (non-root) user: 18823:chris::firejail zoom

$ sudo firecfg Firejail can give you a view of currently running applications
using the format of the top process management application:
Listing 5 shows the output from this command as well as a
number of interesting system resources being created or config- $ firejail --top

ured, ready for the sandbox. Note the following line:

In Figure 8, you can see both the Zoom application and the
Adding user chris to Firejail access database in U Gimp application running under top with active sandboxes.
/etc/firejail/[Link] To close down a sandbox, call its PID:

The following command will now run Gimp inside a sandbox $ firejail --shutdown=18752

as the chris user: Sending SIGTERM to 18752

$ firejail gimp You can then see that the sandbox no longer exists:

In Listing 6, you can see the output. $ firejail --list

The eagle-eyed among you might spot this error (as shown in 17017:root::firejail --top

Listing 6) for some applications: “Warning: an existing sandbox 18823:chris::firejail zoom

was detected. /usr/bin/gimp will run without any additional

sandboxing features.” This seems to be a bug as the sandboxing You can isolate the files in your home directory from the appli-
appears to work as hoped. Search online if you are concerned cations that you are running in a sandbox. The command is as
and see GitHub [12] for some additional information. follows, with Gimp as the example:
I have also opened up another sandbox for the video confer-
encing application Zoom, as you can see here if I run the $ firejail --private gimp

--list command to see which sandboxes are active:

The manual states that this option
Listing 6: Gimp in Firejail means you can: “Mount new /root
Reading profile /etc/firejail/[Link]
and /home/user directories in tempo-
Reading profile /etc/firejail/[Link]
rary file systems. All modifications are
Reading profile /etc/firejail/[Link]
discarded when the sandbox is
closed.” That’s certainly an appealing
Reading profile /etc/firejail/[Link]
feature if you are testing new applica-
Reading profile /etc/firejail/[Link]
tions or those with security concerns
Reading profile /etc/firejail/[Link]
and want to tidy up automatically
Reading profile /etc/firejail/[Link]
after testing.
Reading profile /etc/firejail/[Link]
An extension of that option is to spe-
Reading profile /etc/firejail/[Link]
cifically configure the sandboxed appli-
Seccomp list in: !mbind, check list: @default-keep, prelist: unknown,
cation’s working directory as so:
Parent pid 17112, child pid 17113

Warning: not remounting /run/user/1000/gvfs

firejail --private=<directory-name>
Warning: not remounting /run/user/1000/doc

Seccomp list in: !mbind, check list: @default-keep, prelist: unknown,

For ease of use, Firejail comes with “…
Blacklist violations are logged to syslog
over 1,000 security profiles, covering
Child process initialized in 152.44 ms
most common Linux applications.
Warning: an existing sandbox was detected. /usr/bin/gimp will run without any
Profile files have a friendly syntax and
additional sandboxing features
are stored in /etc/firejail directory.”
Gtk-Message: [Link].028: Failed to load module "canberra-gtk-module"
With a bit of practice, you can custom-
Could not connect: Permission denied
ize Firejail security profiles to suit your
gui_dbus_name_lost: connection to the bus cannot be established.
needs.

Figure 8: We have more than one sandbox running in Firejail.

22 JULY 2024 ISSUE 284 [Link]

 COVER STORY
Laptop Security

One final feature that I feel

compelled to write about, after
my extensive work on Docker
security, is Firejail’s ability to
use overlayFS [13], which cre-
ates an abstracted filesystem
(FS) separate from the ma-
chine’s filesystem. The docu-
mentation states that Firejail
makes use of an overlay file-
system so that the machine’s
filesystem isn’t ever directly
accessed (similar to the func-
tionality in many Linux con-
tainer systems).
All files written during instal-
lation and while an application
is in a running state are stored
in an overlay layer for maxi- Figure 9: The UI for Firejail, called Firetools.
mum isolation from the under-
lying system. I haven’t experimented with this functionality It is notable that many Linux applications are already dis-
yet, but the docs offer the following process for getting that creetly running in their own sandboxes, without breaking a
working. sweat. That is one of many reasons why Linux has become so
First, set up a sandbox with overlayFS, using this command popular. These days, there are so few barriers to using Linux in
(as the root user): a secure manner that I can comfortably recommend a Linux
laptop to users of all experience levels. Q Q Q
$ firejail --noprofile --overlay-tmpfs

Info
Become a non-root user, like chris, as shown: [1] What Is an Infostealer? Is It Dangerous?:
[Link]
$ su - chris [2] Binnie, Chris. Linux Server Security. Wiley, 2016:
[Link]
Run an application, as the chris user, as so: dp/1119277655
[3] Linux Malware Detect: [Link]
$ gimp [Link]
[4] ClamAV: [Link]
I should also mention that a UI is available for Firejail called
[5] systemd: [Link]
Firetools [14]. I would encourage you to have a look at the doc-
[6] ClamAV Packages:
umentation [15], which offers the following introduction: “It
[Link]
provides a sandbox launcher integrated with the system tray,
[7] ClamAV on Arch Wiki: [Link]
sandbox editing, management, and statistics.” Figure 9 shows
an example of the wizard that Firetools offers users for starting [8] Understanding iptables Chains and Targets:
[Link]
up a sandbox. If you plan to use Firejail, the Firetools user in-
[Link]
terface is definitely worth a look.
[9] How to Block an IP Address with UFW on Ubuntu Server:
[Link]
Conclusion with-ufw-on-ubuntu-linux-server/
I have touched on three important areas of concern for secur-
[10] Firejail Security Sandbox: [Link]
ing your Linux laptop, but there is still more to do. Install all
updates, pay attention to security advisories, and definitely [11] Linux Sandboxing in Chromium:
don’t click on suspicious links. You might also be interested in [Link]
docs/linux/[Link]
the article on browser security and privacy extensions else-
where in this issue. [12] Gimp Not Working in Firejail:
[Link]

Author [13] overlayfs: [Link]

filesystems/[Link]
Chris Binnie is a Cloud Native Security consultant and coauthor
of the book Cloud Native Security: [Link] [14] Firetools: [Link]
Cloud-Native-Security-Chris-Binnie/dp/1119782236. [15] Firetools documentation: [Link]

QQQ

[Link] ISSUE 284 JULY 2024 23

REVIEW
Distro Walk – Ubuntu Budgie

A polished, user-friendly desktop

Ubuntu Budgie
Takes Flight
Ubuntu Budgie combines the simplicity of the Budgie desktop with the power of Ubuntu,
resulting in a customizable desktop experience. By Bruce Byfield

U
buntu has 10 official flavors. Linux Magazine (LM): How did Ubuntu release. Back then, the idea was to have
Most are based on the default Budgie start? Tell readers about Budgie’s an alternative to already established desk-
desktop, while Edubuntu is cen- historical ups and downs. top environments such as Gnome and
tered on education and Kylin on KDE, with the focus on the traditional
the needs of Chinese users. Among these David: Budgie Remix was the original desktop paradigm and a simpler option.
variants, Ubuntu Budgie stands out, not name for Ubuntu Budgie – it came from With the release of Solus and the Budgie
only because Budgie is a desktop that is a 2016 Google+ reply from Mark Shut- desktop itself, it was the perfect opportu-
less than a decade old, but because the tleworth about an up-and-coming desk- nity to bring something new to Ubuntu,
development team has gone to great top environment called Budgie that for those users that wanted a more tradi-
lengths to produce a polished, user- could be a good fit for Ubuntu. The orig- tional desktop experience. At that time,
friendly desktop and many outstanding inal remix was put together over a cou- Ubuntu was still using Unity before
applications. Our request for more infor- ple of weeks and received really positive Gnome was adopted as the default desk-
mation about Ubuntu Budgie was an- feedback that has enabled the momen- top for the main release.
swered by David Mohammed, Ubuntu tum to keep rolling forward for the last
Budgie’s leader and founder, and Nikola eight years. LM: How does Ubuntu Budgie interact
Stojić, the project’s web manager. There haven’t been any actual downs; with other flavors of Ubuntu and with
like all open source projects, we would mainstream Ubuntu?
Author like to move faster taking the good ideas
Bruce Byfield is a computer journalist and from the Budgie community and imple- David: Ubuntu Budgie 24.04 LTS has
a freelance writer and editor specializing menting them. That takes commitment shown the flavor interaction at its best.
in free and open source software. In from all of our volunteers. As such, we Through the flavors Matrix channel, all
Photo by Navi on Unsplash

addition to his writing projects, he also are looking for inspirational people from the official flavors got together to help
teaches live and e-learning courses. In his
all walks of life to help out. each other out. Key community mem-
spare time, Bruce writes about Northwest
Coast art ([Link]
bers who have the relevant Ubuntu
com). He is also co-founder of Prentice Nikola: Everything started with 16.04, rights sponsored each other’s changes.
Pieces, a blog about writing and fantasy at better known as Xenial Xerus. I joined the Some dug into each other's issues and
[Link] team somewhere around the Yakkety Yak helped to resolve matters.

24 JULY 2024 ISSUE 284 [Link]

 REVIEW
Distro Walk – Ubuntu Budgie

In the 23.10 release, Ubuntu Budgie LM: How does Buddies of Budgie (the 2. Ability to create your own desktop
took the lead with the new Ubuntu in- Budgie developers’ organization) layout with various applets and fea-
staller, working hand-in-hand with operate? tures via Budgie Settings.
Ubuntu developers to be the first flavor 3. Modularity and adaptability to your
to ship with this new software. We then David: Buddies of Budgie is building a workflow.
helped flavors during the 24.04 release platform for Budgie Desktop and its as-
cycle to integrate the ever evolving in- sociated projects. This ensures that the LM: Name three or more features devel-
staller to make it a reality for the majority reach of Budgie Desktop is as wide as oped by Budgie/Ubuntu Budgie.
of official flavors. possible. It has a core team consisting of
Ubuntu Budgie, Fedora, Arch, and Solus. David: There are a plethora of different
LM: How does Ubuntu Budgie differ in We have contributors from a wide range applets, developed by Jacob Vlijm and
design philosophy or features from other of distros, such as NixOS and non-Linux Sam Lane, who are part of our team,
implementations of Budgie? platforms such as BSD. The core team aside from the default ones, the ones
works in a collegial approach; this en- available in Budgie itself, and ones
Nikola: Simplicity is elegance. Our de- sures decisions are consensus based made by the community. With that
sign philosophy is that you need to pro- and are not distro-specific driven. said, here are the three most important
vide an elegant but overall simple desk- ones that add essential features to the
top experience, from the selection of the LM: Who is the target audience for desktop:
software to the desktop layout. Our ap- Ubuntu Budgie? 1. Window Shuffler: Enables quarter til-
plets serve as a kitchen sink that you can ing via both keyboard and using a
use to shape the desktop to your liking, Nikola: Everyone from beginners who mouse via a feature we call Drag
so no two installations of Ubuntu Budgie want to try out Linux as a replacement Snap. Quarter and half tiling are pretty
are the same. for Windows or macOS, to the average much self-explanatory. For those who
and more experienced Linux users. come from Windows, it is one of the
LM: How is Ubuntu Budgie governed? When switching from another OS, we most used features. Dragging a win-
How are decisions made? know that the hardest part aside from dow to the top of the screen will tile
getting your applications is having the the window to the top half, but keep-
Nikola: When something needs to be de- desktop experience you are used to. ing the mouse button pressed will
cided, we bring it to the table for every- While the default layout with the dock progress the action into full-screen til-
one to discuss and give their opinion at the bottom is reminiscent of macOS ing. Also available is asymmetric tiling
and decide by consensus – especially for some, the flexibility of Budgie by pressing Alt or Ctrl while dragging.
when it comes to major changes. Ideas Desktop itself allows you to customize Alt + dragging will tile the window
are generated not only by the team, but it to the desktop routine you are al- into three-fifths of the screen width,
via our Discourse community and ideas ready used to. We offer different while Ctrl + dragging will tile the
through our GitHub tracker, as well as themes from Budgie Welcome as well window into two-fifths of the screen
building upon upstream discussions. as applets, so even the beginners can width.
On the individual issues, such as a have the desktops that are reminiscent 2. Budgie Weathershow: If you ever
new applet development or website re- of macOS or Windows or even wanted to have weather at glance,
design, each of us leading the change ChromeOS in just one click. For those without using it from the calendar,
takes the issue and brings it to the team, of us who are gamers, Ubuntu Budgie but also integrated into the taskbar
gathering feedback, and making sure Welcome offers the option to get the itself, Budgie Weathershow allows
that we are on the same page. The feed- latest drivers, various game software, you to do that.
back process is a very important part of and even some games. We try to ac- 3. Hotcorners: Developed by Jacob Vlijm,
how we function as a team. commodate for different use cases and this feature allows you to define what
let the users pick the stuff they need. happens when you move the mouse
LM: How does Ubuntu Budgie interact As users explore Budgie itself, they can cursor to the corner of the desktop. You
with other implementations of Budgie then customize it to their liking. can set it to lock screen, show desktop,
such as Solus? We also have a custom image for open a specific app, and so on. The
Raspberry Pi developed by our own Sam settings interface has a dedicated
David: Ubuntu Budgie supports directly Lane, for those who like to stay with window. There is an option to set the
both Debian and Ubuntu, and we wel- their desktop. pressure, so the user can fine-tune his
come Debian end users and Ubuntu- or her own touch intensity to prevent
based users to our Discourse-based com- LM: What are some reasons to use unintended firing. Also, when text is
munity. Our primary direct interaction Ubuntu Budgie? selected and you unavoidably hit the
with non-Debian and non-Ubuntu end edge of the screen and the correspond-
users is through our budgie-extras proj- Nikola: ing command could be triggered, Hot-
ect, and we welcome contributions such 1. Stability and familiarity of Ubuntu corners therefore checks for mouse
as ideas, issues, and code submissions with access to a huge software library button 1 to be pressed and skips the
through our GitHub projects [1]. thanks to Debian. action if that is the case.

[Link] ISSUE 284 JULY 2024 25

REVIEW
Distro Walk – Ubuntu Budgie

LM: What options does Ubuntu Budgie applets, Ubuntu Budgie Welcome has a desktop updates, as well the new ap-
give users for init system, X Windows/ special section dedicated to them, with a plets, you can. They are available right
Wayland, tiling, and container setup? screenshot of the applet and description from Ubuntu Budgie Welcome to enable.
of what it does with the option to install it Tiling support? Available out of the box.
David: Budgie Desktop is init agnostic right away or to remove it if you decide Want more control? You can use the ap-
and X11 based. In version 10.9, we have you do not need a specific applet. plet to control the keyboard shortcuts to
leveraged the work of the Xfce develop- Whether you want to get some gaming arrange open apps on the desktop.
ers through the libxfce4windowing proj- software or check for driver updates, you Working with different languages and
ect to begin the transition of the code- can do it right from Budgie Welcome. need to have keyboard support for one
base to Wayland. In the near future, There are a plethora of options to explore. language in your document processor
Buddies of Budgie will have developed and another language for the whole
a wlroots-based compositor called Mag- LM: What accessibility features are desktop? You can, with the Keyboard
pie. This, with libxfce4windowing, will included? AutoSwitch app. Want a total makeover?
provide a direct Wayland-compatible Simply select theme layouts, and you
desktop. Ubuntu Budgie will be show- David: Accessibility options are inher- can get a different desktop within a few
casing this work in its next series of in- ited from Gnome Mutter and are fea- seconds.
terim releases leading up to our next tured through Budgie Control Center.
26.04 LTS release, which will be Way- Out of the box, we ship Magnus to pro- LM: Any future plans for Ubuntu
land only. vide zoom capabilities as well are Orca. Budgie?
As for tiling, Budgie Desktop inherits We do acknowledge accessibility is key
the window management support of for our community and is an area we David: Our key focus is for the 24.10
Gnome Mutter. Ubuntu Budgie has built need talented individuals to join us to standard release. This will hopefully be
upon this approach for its Window Shuf- ensure the Budgie Desktop reaches as our first Wayland-only release, depend-
fler capability introducing a mouse- and wide a range of the user base as ing on progress made by our upstream
keyboard-driven tiling window possible. Buddies of Budgie. This first Wayland re-
approach. lease will be really experimental for us –
We have experimented with Ubuntu’s LM: How has Ubuntu Budgie enhanced we will use it to find out how Wayland
Core desktop and look forward to its routine features, such as system settings works for the community and what
final release. We hope to bring Budgie or the desktop menu? changes we need to do for the future. All
Desktop to the Core desktop, allowing our interim releases lead toward the next
Budgie as an option for end users of this Nikola: Let’s say you want to have a LTS, Ubuntu Budgie 26.04, where we
Snap-based approach of system- and ap- taskbar like on Windows. In previous want our offering to be super stable.
plication-based containerization. versions, you could have it on the top,
on the bottom, on the right, and on the Parting Thoughts
LM: What help features does Ubuntu left. Well, you can do the same with Many Ubuntu flavors offer little more
Budgie offer, especially for new users? Ubuntu Budgie via Budgie Desktop Set- than Ubuntu on another desktop, so my
tings. Want to have weather updates first close look at Ubuntu Budgie took
Nikola: I would say the biggest help fea- right on the taskbar? Simply install me by surprise. I would rank it with
ture for new users is the Ubuntu Budgie Weather Applet, and you can glance at Deepin and Zorin for its user experience
Welcome app. After you install the desk- the weather updates for the next five and the extent to which it has come into
top, it is the first thing that will greet you days as well as the icon showing temper- its own. In the future, it will be one of
and help you navigate your new Ubuntu ature and weather status. One of the no- the first Debian derivatives I will recom-
Budgie installation. For example, if you table features that Ubuntu Budgie offers mend to new users. Q Q Q
want to add a new browser you can do is support for backports. Most of the dis-
that just from the app in case you are tributions require you to update or en- Info
not a fan of Firefox, which comes as the able backports manually. If you want to [1] budgie-extras project:
default, or use something else as your stay on long-term support (LTS) but re- [Link]
default. In case you want to get new ceive the latest Budgie Environment budgie-extras

QQQ

26 JULY 2024 ISSUE 284 [Link]

IN-DEPTH
E-Commerce Solution

Setting up an e-commerce system

Open Source

Thirty bees offers a feature rich, open source e-commerce Getting Started
solution for setting up your online store. By Rubén Llorente Because thirty bees is intended to run on
a Linux, Apache, MySQL, PHP (LAMP)

I
stack, the official installation guide as-
f you are planning to create an third parties. OpenCart in particular sumes that you are using a commercial
online store, you will find plenty needs a lot of modules and add-ons to web-hosting service that provides the
of free, open source (FOSS) plat- turn it into a useful web store. After in- LAMP stack. Because the documentation
forms you can use to host your stalling all these extra plugins, you soon does not offer a guide for installing thirty
e-commerce site. In a previous Linux realize that you have either spent a bees on a fresh server of your own, I will
Magazine article [1], I reviewed Open- bunch of money buying the extensions, provide up-to-date, detailed instructions
Cart, the shopping cart service that or a bunch of time developing them if you want to try thirty bees on your
currently powers my online store. yourself. own machine.
While OpenCart works well enough, I In addition to the issue of extra pl-
find it a bit lacking after running it for ugins, the person in charge of OpenCart Installing an Environment
a couple of years. has been involved in some controversies Because I run my production environ-
Often, FOSS e-commerce solutions are regarding security advisories [2, 3] and ments on OpenBSD, I will use it as the
distributed on a disguised freemium version upgrades [4]. I once had Open- base here. However, the following steps
model. While the core of these solutions Cart break during a minor upgrade, should be easily adaptable to any popu-
are free and open source, they have just which did not inspire confidence. lar Linux distribution, such as Rocky
enough features to get by. If you need With all of this in mind, I couldn’t Linux or Devuan.
additional features, you must purchase help but wonder if there might be a bet- In order to get started, you need to in-
downloadable modules and extensions, ter alternative. In my search, I discov- stall a number of components on your
which are often pricey and developed by ered thirty bees [5], an e-commerce system of choice. You need a web server,
web application released under the a database, and PHP. The database will
Author Open Software License v3.0 (OSL-3.0). store thirty bees’s data, PHP will execute
Designed for end users, thirty bees the web application, and the web server
Lead Image © Kirsty Pargeter, [Link]

Rubén Llorente is a
mechanical engineer doesn’t require you to be an expert to will accept requests from the visitor’s
whose job is to deploy it. A fork of PrestaShop, thirty web browsers.
ensure that the bees was developed out of concerns From a fresh OpenBSD 7.5 install, you
security measures of about the direction PrestaShop was can fetch all the required components
the IT infrastructure taking with version 1.7 and onward. using the following command as root:
of a small clinic are Among other things, thirty bees aims to
both law compliant and safe. In addition, be a stable version of PrestaShop with a # pkg_add php-8.2.16 php-gd-8.2.16 U
he is an OpenBSD enthusiast and a focus on fixing bugs rather than adding php-zip-8.2.16 php-pdo_mysql-8.2.16 U

weapons collector. new features. php-intl-8.2.16 php-imap-8.2.16 U

28 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
E-Commerce Solution

Listing 1: /etc/hosts Listing 3: Enabling PHP extensions

[Link] localhost for file in `find /etc/[Link]/ -type f`; do

::1 localhost ln -s $file /etc/php-8.2/`basename $file`;

[Link] thirtybees [Link] done

Listing 2: Modifications in /etc/[Link] module in Apache by uncommenting the # ln -sf /var/www/conf/[Link]/U

max_input_vars = 10000
corresponding LoadModule line in /etc/ [Link] U
post_max_size = 32M
apache2/[Link] as well. /var/www/conf/modules/[Link]

Please note that this configures

upload_max_filesize = 16M
Apache to not use Transport Layer Secu- The final step for getting the execution
allow_url_fopen = On
rity (TLS) encryption, which a web store environment ready is to enable it and
definitively needs. The reason I do this is launch it. OpenBSD does not use sys-
php-curl-8.2.16 php-soap-8.2.16 U because I place my web services behind temd. Instead, services are managed
mariadb-server-10.9.8p0v1 U a load balancer that takes care of encryp- with the rcctl command:
apache-httpd-2.4.58p1 php-apache-8.2.16 U tion (a topic for another article).
unzip It is important to enable PHP support for # rcctl enable mysqld apache2

Apache, which can be done easily with # rcctl start mysqld apache2

Next, you need to configure the compo-

nents that have been installed. I always Listing 4: Modifications in /etc/apache2/[Link]
recommend editing /etc/hosts first to DocumentRoot "/var/www/htdocs"
ensure the operating system knows its <Directory "/var/www/htdocs">
own name (see Listing 1 for an #
example).
# Possible values for the Options directive are "None", "All",
While thirty bees supports the MySQL
# or any combination of:
database, OpenBSD uses MariaDB,
# Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
which is a compatible replacement (at
#
least for this example). To deploy the
# Note that "MultiViews" must be named *explicitly* --- "Options All"
database engine on OpenBSD, issue the
# doesn't give it to you.
following command as the superuser:
#

# mariadb-install-db # The Options directive is both complicated and important. Please see

# [Link]

PHP has set tight limits by default, so # for more information.

it needs some minor tweaks to work #

with thirty bees. First of all, you must Options Indexes FollowSymLinks
edit /etc/[Link] and modify the
values in Listing 2. This will allow visi- #
tors’ browsers to issue larger HTTP # AllowOverride controls what directives may be placed in .htaccess files.
POST messages, let thirty bees access ex- # It can be "All", "None", or any combination of the keywords:
ternal resources, and permit the admin- # AllowOverride FileInfo AuthConfig Limit
istrator to upload files to the shop. Once
#
this is done, you can enable the PHP
AllowOverride All
extensions required by thirty bees with
the script in Listing 3.
#
To configure the Apache web server,
# Controls who can get stuff from this server.
edit the file /etc/apache2/[Link].
You need to find the AllowOverride line #

within the definition for the directory you Require all granted

intend to use as the document root for </Directory>

your service and edit it as shown in List-

ing 4. This allows thirty bees to set its #

own redirection rules, which are handy # DirectoryIndex: sets the file that Apache will serve if a directory
for creating URLs that are compatible # is requested.
with search engine optimization (SEO). #
You also need to set the DirectoryIndex <IfModule dir_module>
file to [Link] (as shown in Listing 4). DirectoryIndex [Link]
If you intend to use SEO-friendly URLs, </IfModule>
then you must enable the mod_rewrite

[Link] ISSUE 284 JULY 2024 29

IN-DEPTH
E-Commerce Solution

Listing 5: Creating a Database It is recommended to script will be already installed on the

CREATE DATABASE thirtybees;
harden the MariaDB system. You can just call it as root and
GRANT ALL PRIVILEGES ON thirtybees.* TO
install before going let it perform its magic:
'thirtybees'@'localhost' IDENTIFIED BY 'somepassword'; into production (Fig-
FLUSH PRIVILEGES; ure 1). If the previous # mariadb-secure-installation

QUIT;
instructions have been
followed, a hardening The hardening script will ask you some
questions. Feel free to respond to them
with answers that make sense to you.

Installation
With the LAMP stack set, you are ready
to install thirty bees. Download thirty bees
into the web server folder and decompress
it. I like to use /var/www/htdocs/, the de-
fault web folder for OpenBSD installs. The
folder will have some demo content inside,
which you will have to remove:

# rm -rf /var/www/htdocs/*

Downloading and decompressing thirty

bees is trivial. Make sure the down-
loaded code is owned by the www user,
or the web server won’t be able to work
properly with it:

# cd /tmp

# ftp [Link]

[Link]

# unzip [Link] U
-d /var/www/htdocs

# chown -R www:www /var/www/htdocs

At this point, thirty bees is nearly installed,

but you still need to create a database for it
within your MariaDB install. Just invoke
Figure 1: Before going into production, you should use mariadb-secure- mysql and issue the SQL statements shown
installation to harden your database’s security. in Listing 5. Once the database is set, open
a web browser and visit your web server.
The install script will trigger automatically.
The installer is intuitive and self ex-
planatory (Figure 2). The only complex
step is configuring the database connec-
tion (Figure 3). If you are following
along with this example, the database
login will be thirtybees and the password
will be the one you defined in Listing 5.
Once finished, the installer will instruct
you to delete the install directory from
your web server and give you a link to
the admin dashboard. Please, bookmark
the dashboard link for later, because it is
randomly generated and you won’t be
able to easily log in as administrator if
you lose it. The install directory can be
Figure 2: As long as the web server is properly configured, the thirty bees deleted with a simple command:
web installer is very easy to use. The web installer will report any impor-
tant misconfigurations that are detected before the install is attempted. # rm -r /var/www/htdocs/install

30 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
E-Commerce Solution

PrestaShop 1.6 have a good chance of

working with thirty bees.
In the Preferences section, you can
configure most of site’s default behav-
iors. Many of the handy features sup-
ported by thirty bees can be managed
here. One outstanding feature is thirty
bees’ customer support service, which
can be integrated with an email service.
The idea is that customer tickets will
show up in thirty bees customer support
system in the back office, but conversa-
tions will also be readable from an email
client if necessary (Figure 7). The IMAP
integration configuration page is a bit
confusing, so it may take you some fid-
dling to get the results you need.
Figure 3: The installer will ask some questions regarding the database You can customize your store with
connection. I recommend using [Link] instead of localhost as the themes supported by thirty bees, but
database host in this example. Panda is the only remaining third-party
theme that still supports thirty bees.
Customization can still buy modules from either the Panda is very flexible and configurable,
PrestaShop store or from a third-party
Installing the thirty bees framework is only so it will let you build any sort of web-
vendor. Modules compatible with
the beginning. Once the store is installed, site you need. The main drawback is
you need to configure thirty bees
to prepare it for production. As
your first step, go to the Localiza-
tion section of the Dashboard and
configure the languages, curren-
cies, and tax rules for your site
(Figure 4). Settings in this section
are self-explanatory.
In the Shipping section, you
can select the carriers custom-
ers might choose for the deliv-
ery of their goods (Figure 5).
Shipping configuration is very
flexible: You can define different
carriers for different delivery
zones and set limits by size and
weight. You can also enable free Figure 4: You can choose from multiple free localization packs.
shipping for big orders.
Next, you need to set up
payment processing. By de-
fault, thirty bees only has a
Bankwire module, which al-
lows the store to accept wire
transfers. You can install a free
PayPal module, which in the-
ory would allow the store to
bill via PayPal and credit card
(Figure 6). The Custom pay-
ment methods module, also
available as a free download,
is handy for creating your own
billing methods (such as cash-
on-delivery or check) as long
as they are not complex. If Figure 5: Shipping lets you add as many carriers as desired to your e-commerce
these options fall short, you site and offers plenty of flexibility.

[Link] ISSUE 284 JULY 2024 31

IN-DEPTH
E-Commerce Solution

Figure 6: Multiple payment modules are available for free, including one for PayPal.

that Panda is not localized. If you want deliveries under Advanced Parameters | You can populate your store catalog in
to operate an e-commerce site that uses Email by choosing the PHPMailer Email the Products section. Products can be
languages other than English with transport. grouped by categories, and pretty, SEO-
Panda, you will have to translate the You will need to supply the contact in- friendly URLs are supported – just re-
theme yourself. Thankfully, thirty bees formation for your e-commerce site member to enable these URLs in Prefer-
includes an internationalization frame- under Preferences | Store Contacts. It is ences | SEO & URLs.
work you can use to translate any un- important to define the email address of
translated text string, without the need your store as the same email address Conclusion
to touch the website’s code directly. used with PHPMailer, because emails An acceptable e-commerce package,
Thanks to thirty bees’ content man- sent by thirty bees will use this address thirty bees is very complete when com-
agement system (CMS), your store can in the From header. Most email servers pared to other freemium FOSS alterna-
have an integrated blog if desired. The will reject your email if there is a mis- tives. The storefront feels responsive
CMS is also responsible for delivering match (ie., if the address in From is dif- during use, and the back office experi-
the Terms and Conditions page, along- ferent from the email account which ence is much more polished than the
side any privacy policy, payment sends the email). one offered by OpenCart.
terms, and similar information. Inte-
gration with Google Maps is also possi-
ble, and most often used to show the
location of the physical store (if it ex-
ists) on the map.
Proper email delivery is necessary in
order to let the store send password reset
emails to customers and deliver order
confirmations, among other things. Be-
fore email is properly configured, it is
necessary to install an email transport
module, which can be done from the
Modules and Services section. Mail via
PHPMailer is a popular email transport
module. Upon installation, you will be
taken to the configuration page in which
you will be able to fill the credentials
necessary for your store to use an exist- Figure 7: The thirty bees customer support system allows you to track
ing email account to deliver messages. issues brought up by customers. Email integration is available for this
You will then have to enable email function.

32 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
E-Commerce Solution

The main drawback to thirty bees is However, keep in mind that many of management features to prevent using
that it small size implies a small catalog these borrowed modules will work per- a given module in more than one shop.
of third-party modules, which would not fectly fine, but many won’t (Figure 8). Module cost is also a variable to con-
be an issue if third-party modules Third-party developers for sider, because a small working e-com-
weren’t essential for creating a fully PrestaShop modules are usually more merce site will need about $400 worth
functional shop. You can borrow mod- aggressive regarding license enforce- of third-party modules. This is actually
ules developed for PrestaShop 1.6. ment than, say, developers for Open- quite affordable for a store, but it is
Cart modules. still more than you’d spend on an
During my re- OpenCart site.
search, I ran into That said, I wholeheartedly recom-
many PrestaShop mend giving thirty bees a try if you are
modules that had looking for an e-commerce package. It is
digital rights certainly worth a look. Q Q Q

Info
[1] “Setting Up an E-Commerce OpenCart System” by Rubén
Llorente, Linux Magazine, issue 236, July 2020,
[Link]
[2] OpenCart CVE-2023-47444 disclosure timeline:
[Link]
opencart-cve-2023-47444/#disclosure-timeline
[3] Static Code Injections in OpenCart (CVE-2023-47444):
[Link]
Figure 8: Installing an incompatible module will
result in a server error. Before installing modules [4] OpenCart upgrade issue: [Link]
that don’t support thirty bees explicitly, ensure that opencart/issues/9236#issuecomment-774668513
you have a backup and restore strategy. [5] thirty bees: [Link]
IN-DEPTH
Security and Privacy Extensions

Browser extensions for safer surfing

Surf Protection
Many hands are hard at work on problems of Internet security and privacy. If you’re looking to
lock down your surfing experience, try these privacy-focused browser extensions. By Daniel LaSalle

T
he Internet can be a scary place, choice of privacy-centric search engines always known and enjoyed. Unlike
and if you’re going to spend and should be your default search en- other blockers, Privacy Badger actually
time on it, you’d best come pre- gine if wish to stay on the low. Duck- learns to blacklist invisible trackers
pared. These free browser exten- DuckGo eliminates ads and trackers and based on the experience you give it by
sions will help you safeguard your does not store your personal data. browsing. Privacy Badger learns to dis-
browsing experience. I’ll start with DuckDuckGo’s Privacy Essentials ex- cern the nuances between the trackers,
some extensions that are available for tension makes sure all of your searches based on its three-strike system. If it en-
both Firefox and Chrome, and I’ll also funnel through its service (Figure 1) and counters the same tracker on three dif-
show you some that are only officially also adds a layer of protection against ferent websites, it will automatically
supported by the privacy-conscious third-party trackers by automatically en- adapt its knowledge base to perma-
Firefox browser. If you use a different forcing encryption, defaulting to HTTPS, nently flag the tracker. To further help
browser, you might find equivalent and adding email protection so you don't users who are trying to avoid trackers,
functionality, either as a built-in or need to commit to giving your real email it will automatically mark all objects
through an add-on. Part of the purpose address in the numerous online forms that are deemed as unwanted (such as
of this article is to describe, not just the you will encounter this year. The email auto-play videos from external
tools, but the problems the tools are de- protection feature
signed to address, which should give allows you to cre-
you a better idea of the dangers faced ate an @duck.
by the casual Internet user. com email alias
Keep in mind that the first thing you that you can use
Photo by guille pozzi on Unsplash

should do after installing any operating in registration

system, browser, or other software is to forms to add an
apply all pending software patches and extra layer of
upgrades. Stay aware, and keep your sys- anonymity.
tem up to date.
Privacy
Chromium and Mozilla Badger
The Privacy Bad-
DuckDuckGo Privacy Essentials ger extension is Figure 1: The impact of the Duck search engine is
For those who don’t already know about not the typical ad immediately felt at the search level as it blocks
DuckDuckGo: It is the number one blocker you’ve undesired content.

34 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
Security and Privacy Extensions

basis. For example, if you wish to only to choose which proxy to default on.
accept cookies from [Link] and Git- FPS also supports advanced logging,
[Link], it is possible to deny every- custom lists, such as preventing certain
thing else by creating a whitelist entry domains from being accessed by cer-
for gi*[Link]. To manage this list of ex- tain of your proxies, and auto-switch-
pression, CAD supports JSON. ing, which allows you to redirect auto-
matically in case a website does not
FoxyProxy Standard meet a predefined condition, such as
FoxyProxy Standard (FPS), which has support for a secure version of HTTP.
been in active development since 2006, If you just wish to stick to the basics
is paramount when it comes to manag- of proxy management, such as using
ing proxies. Many users are aware of multiple proxy profiles, proxy import,
how tedious it is to continually alter and proxy switching, FoxyProxy has got
proxy settings via browser settings. FPS you covered with its basic edition [3].
simplifies the task, saving its users sev- Lastly, it is possible to purchase more
eral clicks. robust proxies and VPNs directly from
The FPS extension (Figure 4) sup- those guys for a monthly or yearly fee.
ports all the standard protocols, such
as HTTP(S) and SOCKS4-5, but also uBlock Origin
Psiphon, Privoxy, PAC, and TOR. FPS Not many software systems can change
also lets you have more than one proxy lives, but Raymond Hill has made that
enabled at once.
Perhaps the great-
est feature of FPS
is the URL pat-
terns (or patterns
by domain),
Figure 2: Privacy Badger allows which allow you
you to focus on the content that to default certain
really matters by removing URLs to a specific
another layer of distraction. proxy. For exam-
ple, you could
websites) (Figure 2) by adding a pri- prevent your
vacy badge over them, allowing you to proxy located in
be the judge when it comes to allowing the USA from
the content or not. ever accessing
Privacy Badger, which adds Global Google. Also, if
Privacy Control [1] to your sessions you have numer-
by default, was developed and is ous proxies con-
maintained by the Electronic Freedom figured and open
Foundation (EFF). a new tab, the Figure 3: Cookie AutoDelete is a cookie cleaner on
user is prompted steroids.
Cookie AutoDelete
Regular users typically only interact with
cookies to clean or accept/deny them.
The Cookie AutoDelete (CAD) extension
allows you to manage cookies by
browser tab, domain origin, or expira-
tion date (Figure 3). Cookie AutoDelete
also enhances regular browser cleanup
activities by handling some more ob-
scure cache entries such as low-level
API client-side storage (referred to as
IndexedDB) [2], as well as plugin-gen-
erated data.
Power users will enjoy CAD’s list of
expressions which allow you to push the
control up a notch by allowlisting (or in- Figure 4: Both the basic and standard versions of FoxyProxy grant the
spect-listing) cookies on a per-domain possibility to configure many different profiles.

[Link] ISSUE 284 JULY 2024 35

IN-DEPTH
Security and Privacy Extensions

happen for several millions of users by which is hosted and maintained by the script execution for trustworthy sites,
unleashing uBlock Origin (Figure 5). As Ghostery community. you need to specify them to NoScript.
with Privacy Badger, this one is not your The [Link] site is a tracker da- The NoScript extension comes with a
normal ad-blocking extension but rather tabase that was launched in 2018 and very minimal list of whitelisted default
defines itself as a “wide-spectrum con- has grown stronger than ever in 2024. domains, thus requiring a lot of man-
tent blocker with CPU and memory effi- The Ghostery community encourages ual user input for accepting all those
ciency as its primary feature.” [4] users to contribute to the database. You other websites you visit regularly. To
Aside from blocking the regular online can contribute by registering for a free do so, you can either interact with the
nags such as pop-ups, uBlock Origin also account via their extension or by pro- extension’s icon on top of your
lets you make any web page element you viding a donation. Either way, all of the browser or access its properties and
don’t wish to see disappear by adding features will remain the same for pay- look for the Per-site Permissions tab.
them to a zap list. If you wish to add ing or non-paying registered users. If After tweaking that list, you can export
them to a permanent set of blocked ele- you decide not to register, only the His- it for future usage.
ments, you can also do so by clicking on torical Stats por-
the element picker feature [5]. tion will be inac-
You can customize the exhaustive, cessible for you.
built-in list of filtered frames and scripts, The feature I
and uBlock Origin also supports a per- cherish the most
sonalized filter list that will complement is the one that
the built-in list while allowing you to ex- takes care of
port your changes locally for easier cookie prompts
portability. for you. Long
At the time of writing, uBlock Origin gone are the days
has been downloaded close to 8 million of a single cookie
times based only on the Mozilla website to rule them all.
figures. The Chrome web store claims Nowadays com-
that a whopping 36 million users have plex legal and
already downloaded this extension. A technical realities
lite version of uBlock Origin is also have led to the
available, but the scaled-down lite ver- emergence of
sion is missing the element picker func- what could best
tionality, as well as the dynamic filter- be described as a
ing and some of the policy features. In cookie consent
many settings, however, the lite version center. When
is more than enough for most people. you’re facing such Figure 5: uBlock Origin: Clicking on “more” gives you
a site, Ghostery a complete view of how many websites are being
Ghostery – Privacy Ad will automatically crawled when accessing the single site you wanted
Blocker answer and dis- to visit in the first place.
The three main components of Ghostery card this pseudo-
are: ad-blocking, anti-tracking, and a wizard for you,
never-consent flag activated at all times. not only excluding
Unlike most of the previously mentioned tracking but also
extensions, Ghostery does not contain saving you a cou-
any expert features, because everything ple of clicks in the
it can possibility do is accessible via its process.
interface (Figure 6). The rest of
Ghostery’s magic happens in the back- NoScript
ground. The available options are very Security
basic and easy to master: either always Suite
trusting a website (thus allowing track- NoScript Security
ing and not protecting your personal Suite’s main pur-
data), restricting access, or pausing the pose is to explic-
Ghostery extension. itly prevent the
If you’re feeling like the simple view execution of Ja-
has left you hungry for technical details, vaScript, Flash,
you can access a detailed view, which al- and other execut-
lows you to control the history of your able content (Fig-
browsing. But the real power of this tool ure 7). If you Figure 6: Ghostery’s Detailed View lets you manage
relies on the [Link] website [6], want to allow the tracker database.

36 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
Security and Privacy Extensions

image loading by
clicking on image,
which will highlight
the whole row of im-
ages shown. The same
applies to a list of do-
mains. As with some
of the previous
choices, users are
granted the flexibility
Figure 7: Out of the box, the NoScript Security Suite extension aggressively stops every to create and import
site that used to work fine prior its installation. That is, until you start paying attention to their own set of rules
that little icon at the top right hand counting the elements it blocked for you. (Figure 8).
uMatrix also offers
NoScript is based on its own ClearClick allowlist (and denylist) both domain re- advanced settings such as color-blind
technology, which was built to protect quests and webpage content. uMatrix is a friendliness, deletion of cache based on
users from cross-site scripting, cross-site straightforward “allow (or deny) wholly blocked hostnames, and deletion of
request forgery [7], clickjacking [8], and or partly” system that gives users the abil- non-blocked session cookies. As with
cross-zone DNS rebinding attacks [9]. ity to better manage incoming traffic and its little cousin, uBlock Origin, there is
This extension has a “shoot first and ask therefore consume less data. These fea- much more to appreciate about uMatrix
questions later” mentality. tures make uMatrix a great choice when than what is mentioned here.
paired with a proxy or a VPN.
2FAS – Two-Factor What it’s UI lacks in elegance, it cer- Mozilla Compatible
Authentication tainly makes up for in effectiveness. In Users who are concerned about privacy
With GitHub now enforcing two-factor that respect, one can easily decide should rally behind Mozilla, partly be-
authentication (2FA) [10], it is fair to say which of the cookies, images, scripts, cause of its excellence in matters of end-
that many readers have been looking for frames, and other components will be user privacy, but also because many ex-
a trustworthy 2FA mechanism. Big Tech allowed or denied. As previously tensions were developed strictly for
has been offering their own 2FA applica- stated, due to its complex nature, there Mozilla Firefox and its forks. At the time
tions for a while now, but 2FAS declares will be an adaptation period required of writing this article, the following ex-
on its site that it is "the Internet’s favor- before you can take full advantage of tensions were only available at https://
ite open source two-factor authenticator” it, but thanks to a temporary locking [Link], although many of
[11]. 2FAS needs two actions from its feature, you can test the settings at the them have Chromium-compatible coun-
users to start its work. First, install the session level before applying them per- terparts branded as something else.
application on your browser or your manently at the extension level.
smart phone. Next, enable 2FA on your You can block and unblock by do- Facebook Container
favorite websites, then scan the QR code main, by element, or even by selecting Here is further proof that the Mozilla
with your smart phone. While this ex- rows and columns, which will save you Firefox team is focusing development
tension is quite possibly the simplest precious time instead of having to cherry around user security: Arguably the fa-
from this whole article, it is also one of pick from the many possible entries. For vorite and most effective protection we
the most vital security extensions to in- example, it is possible to only enable have against Facebook intrusions is an
tegrate with your
browsing habits.

uMatrix
Raymond Hill is back
at it again with another
hit: the uMatrix fire-
wall. But beware, this
one is meant for ad-
vanced users and has a
learning curve to it.
By default uMatrix
blocks everything that
is not coming from
that first domain you
visit. It uses a matrix-
based interface that al- Figure 8: uMatrix offers so many options that even veteran users will have to learn
lows you to easily more before they can actually claim to take advantage of this powerful extension.

[Link] ISSUE 284 JULY 2024 37

IN-DEPTH
Security and Privacy Extensions

extension known as Facebook Container. intention. This extension comes with their web experience safe and fun. With
This extension does not leave a lot of four profiles: Personal, Work, Banking, the bounty of available extensions, it is
room for interpretation as it allows you and Shopping, each coded with a differ- now easier than ever to keep privacy at
to isolate communications coming in ent color (Figure 9). bay while browsing the Web. Modern tools
and out of the Zucked industry. The iso- The extension docks itself to your al- such as LibreWolf can even provide that
lation is done at the tab level as interac- ready-existing container tab menu (single privacy focus without the need for add-on
tions with your browser’s activities will right-click on the top bar then select Open extensions (see the “LibreWolf” box).
be prevented by it. This, of course, in new container tab). This is another sure Techniques such as filtering web page con-
makes it that much harder for third-party shot developed by the Mozilla Firefox tent, restricting sub-domains access to pri-
website cookies to track you. Visually team, cementing their stance on privacy vate data, and identifying trackers give end
speaking, the extension adds the logo of and awarding that group with that much users a multitude of choices. But, as his-
a fence to everything that is Facebook re- more legitimacy and love from end-users. tory shows, regardless of all the methods
lated. Even on those third-party websites and tools currently available right now, Big
that display any content relayed from FB [Link] Tech and authoritarian entities will always
(Such as an icon or a comment section), If you are registered to the [Link] look for ways to compromise your privacy,
the fence from this extension aims at service, your online experience will not so the race will continue. As of now, with
keeping everything contained in the tab, be complete unless you have also in- so many powerful security and privacy ex-
hence limiting the amount of data some stalled this extension. The [Link] ex- tensions available, it makes sense to arm
web pages can scrape off of you. tension gives an extra layer of control by up with the best available tools before
blocking adds, trackers, and WebRTC jumping into this big and beautiful jungle
Firefox Multi-Account [12]. This extension is only useful when that we now refer to as the Web 3.0. Q Q Q
Containers your OVPN tunnel is set to activate.
In a nutshell, the Firefox Multi-Account Info
Containers add-on helps compartmental- Conclusion [1] Global Privacy Control: https://
ize your online activities by allowing you Long gone are the days when netizens [Link]/
to mix and match them in the same open had to rely on external software to keep [2] IndexedDB:
browser, but restricting them to their [Link]
own activities at the tab level. Say LibreWolf [3] FoxyProxy: [Link]
you would wish to use both of your
The main focus of this article is on [4] uBlock Origin:
personal and work email addresses (and manually installed extensions for your [Link]
they are both hosted on the same do- browser. But what if you could make all
main) this extension allows you to do of this a little simpler by removing a [5] Element Picker: [Link]
that using the same browser and the few steps? At the turn of this last de- gorhill/uBlock/wiki/Element-picker
same session. It is then possible to open cade, a newcomer arrived in the [6] WhoTracksMe: [Link]
a different set of tabs based on your browser world that puts user security [7] Cross-Site Request Forgery:
at the center of its attention by includ-
[Link]
ing some of these very popular security
attacks/csrf
features out of the box.
[8] Clickjacking: [Link]
Built on the Mozilla Firefox code, while
www-community/attacks/Clickjacking
not being developed by Mozilla itself,
LibreWolf [13] is a community-based [9] DNS Rebinding Attacks:
solution that contains, by default, some [Link]
of the extensions described in this arti- dns-rebinding-explained/
cle. LibreWolf comes with uBlock Origin [10] GitHub 2FA:
and defaults to the DuckDuckGo search [Link]
engine. The developers stand by a strict raising-the-bar-for-software-security-
no-telemetry philosophy and have
github-2fa-begins-march-13/
made an effort to harden the LibreWolf
browser for enhanced privacy. Because [11] 2FAS: [Link]
Figure 9: Firefox Multi-Account it is built on the Firefox engine and fol- [12] WebRTC API:
Containers comes with four pro- lows the Firefox release cycle, Libre- [Link]
files: Personal, Work, Banking Wolf is of course compatible with all of docs/Web/API/WebRTC_API
the add-ons mentioned in this article.
and Shopping. [13] LibreWolf: [Link]

QQQ

38 JULY 2024 ISSUE 284 [Link]

IN-DEPTH
Git Hooks

Using Git hooks to check your commit code

Secure
Commitment
The pre-commit framework lets you automatically manage and maintain your Git hook
scripts to deliver better Git commits. By Koen Vervloesem

W
hen developing software in commit process is completed. It’s also it before committing changes. Put this
a public Git [1] repository, possible to run scripts before rebasing script in the .git/hooks directory of your
it’s recommended to check anything, after a successful git checkout Git repository and make it executable. Git
for common issues in your or git merge command, before pushing will automatically find and run it. It
code prior to committing your changes. your commits, and more. doesn’t matter what type of script this is,
Neglecting to do so could lead to your So, if you want to automatically do as long as it’s executable. Git hooks can
Git repository being cluttered with com- something before or after one of these Git be shell scripts, written in Python, Java-
mits that just fix some minor syntax or operations, just create an appropriately Script, Go, or anything you want.
style issue. To err is human. Conse- named script (without any extension), Most prevalent among users, Git’s
quently, relying solely on manual checks such as pre-commit if you want Git to run pre-commit hook allows you to run code
isn’t enough to deliver quality code.
To address this issue, the Git version
control system offers a way to start cus-
tom scripts when specific actions occur,
such as committing changes or merging
branches: Git hooks [2]. These hooks are
executable (often shell) scripts, stored in
the .git/hooks directory of a Git reposi-
tory. When you create a new repository
with the git init command, this direc-
tory is populated with several example
scripts (Figure 1). Removing the .sample
extension from a file name is all that’s
Photo by Rawpixel on Unsplash

necessary to enable this hook.

You can use these Git hooks to check
for code style on a snapshot that’s about
to be committed, to edit the default com-
mit message before the commit author
sees it, to validate a commit message be-
fore allowing a commit to go through, or
even to send a notification after the Figure 1: Each Git repository comes with a set of sample Git hooks.

40 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
Git Hooks

linters such as Stylelint [3], Ruff [4], pre-commit sample-config U repository’s root directory to install pre-
Vale [5], and more, and correct any er- > .[Link] commit’s Git hook scripts:
rors they discover prior to committing
your code. But what if you have a com- The generated configuration file, which $ pre-commit install

plex project where you need Stylelint looks like Listing 1, is a YAML file with pre-commit installed at U

(written in JavaScript) to check CSS only one mandatory top-level key, repos. .git/hooks/pre-commit

files, Ruff (written in Rust) to inspect The repos key’s value is a list of reposito-
Python code, and Vale (written in Go) ries where pre-commit can get the code As indicated by the command’s output,
to validate your documentation? Then, for the Git hooks. it sets up a Git hook script in .git/
you need to be sure that you can easily Listing 1 refers to a single repository, hooks/pre-commit. This is a shell script
install those linters and their language pre-commit’s own pre-commit-hooks [8]. that runs the pre-commit command
environments. The repo key refers to the repository’s with certain arguments (you can take a
URL; thus, pre-commit knows which re- peek at the file if you’re interested).
A Package Manager for pository to git clone. The rev key holds Thus when you now add files to the
Git Hooks the version (or Git tag) to install, and index with git add and then run git
This challenge of managing and main- the hooks key constitutes a list of map- commit, pre-commit will automatically
taining pre-commit hooks for Git reposi- pings describing which hooks to use run the hooks specified in the configu-
tories has spurred the creation of a from the repository. ration file (Figure 2).
dedicated framework, conveniently The trailing-whitespace hook trims In the command’s output, you see that
named pre-commit [6]. The pre-com- all white space from the ends of lines. pre-commit installs the hooks from the
mit framework identifies itself as a The end-of-file-fixer hook makes sure repository in its own environment and
“multi-language package manager for files end in a newline and only a new- runs the hook scripts on the added files.
pre-commit hooks.” All you need to do line. The check-yaml hook attempts to While this runs slow the first time due to
is list the hooks you wish to use in a load all YAML files to verify their syntax. the installation, pre-commit runs the
YAML [7] file located in your reposi- And the check-added-large-files hook hooks directly on subsequent commits,
tory. Then pre-commit manages the in- prevents large files (by default files which is much faster.
stallation of any hook written in any larger than 500KB)
supported programming language. It from being Listing 1: Default pre-commit Config File
automatically installs the necessary committed. 01 # See [Link] for more information
programming language environment in
02 # See [Link] for more hooks
an isolated environment (for example, Running
03 repos:
a Python virtual environment) without pre-commit
04 - repo: [Link]
the need for root access. Before modifying
You will find pre-commit in the pack- Listing 1 to meet 05 rev: v3.2.0

age manager of most distributions, but specific needs, I’ll 06 hooks:

it is often an outdated version. You can test how pre-com- 07 - id: trailing-whitespace
install the most recent release using mit works on a re- 08 - id: end-of-file-fixer
Python’s package manager, pip: pository. First run
09 - id: check-yaml
the following com-
10 - id: check-added-large-files
$ pip install pre-commit mand in your Git

Then check whether it’s installed

correctly:

$ pre-commit --version

pre-commit 3.7.0

A Basic Configuration
To manage your Git hooks with the pre-
commit package manager in your Git re-
pository, you need to create a configura-
tion file in the repository’s root directory,
named .[Link]. If you
don’t know where to begin, pre-commit
can generate a configuration file with
hooks for some basic checks. Just run
this command in your Git repository’s Figure 2: On running a git commit, pre-commit automatically checks
root directory: and fixes files you’ve added.

[Link] ISSUE 284 JULY 2024 41

IN-DEPTH
Git Hooks

For each of the four hooks that pre- - id: check-added-large-files - id: end-of-file-fixer

commit runs in this configuration, you args: ['--maxkb=1000'] types_or: [python, javascript]

get an indication of whether the files

pass or fail their checks. The commit is Note that you pass a list of arguments. Note that with types_or the hook runs if
only allowed to go through if all hooks This example only consists of a single the file is identified as either a Python or
pass their checks. argument, but you can pass multiple JavaScript file.
Some hooks are able to automatically arguments to a hook script using a
fix a file when it fails a check. For in- comma-separated list. Useful pre-commit Hooks
stance, if the trailing-whitespace hook You can also adjust a hook to either ex- Take a moment to explore pre-commit’s
finds white space at the end of a line, it clude or include specific files. For instance, pre-commit-hooks repository [8] for
removes it from the file, which is indi- to prevent the trailing-whitespace hook other useful hooks and their potential
cated in the output with the message from operating on files in a data directory, arguments. Note that many of these
‘Fixing [Link]’. Other hook scripts add an exclusion pattern as follows: hooks are checks for Python files, for
don’t automatically fix failures, requiring which there exist other, more compre-
you to manually make the necessary cor- - id: trailing-whitespace hensive pre-commit hooks, such as those
rections. This is often the case with code exclude: ^data/ from Ruff.
flagged by a code linter. Various code-linting tools publish
Regardless if a failure is fixed automat- On the other hand, if you want a hook pre-commit hooks in their repository. One
ically or manually, you need to re-add script to solely operate on specific files, of these is Ruff, a fast code linter and
the file with git add and commit again. add a file pattern like this: formatter for Python, written in Rust.
If all hooks now display success, the Running the code linter and formatter on
commit goes through. This approach en- - id: end-of-file-fixer all Python code in your repository is as
sures that all code entering the (local) files: *.py simple as adding a few lines to the repos
repository abides by the specific guide- list in your .[Link] file
lines verified by your pre-commit hooks. It’s also possible to exclude or include (Listing 2).
From time to time, it’s wise to update files on a global level for all hooks of all By default, the ruff hook merely
all your Git hooks to the latest available repositories. You only need to add ex- shows violations to its rules (Figure 3).
version. You can do this with the follow- clude or files keys on the top level of If you want Ruff to apply fixes for au-
ing command: the YAML file. Note that you can still tomatically resolvable violations,
override these patterns at the level of a simply add
$ pre-commit autoupdate specific hook as illustrated earlier.
[[Link] You don’t need to include or exclude args: [ --fix ]

pre-commit-hooks] updating v3.2.0 -> U files based on file patterns; you can use
v4.6.0 file types instead. To determine the file to the hook. The ruff-format hook al-
types of a specific file in your repository, ways fixes formatting violations.
This command checks for the latest tag run the identify-cli command with the Another useful tool, Vale, checks your
on the default branch of each repository file name as an argument: project’s documentation for consistency.
defined in pre-commit’s configuration However, if your Vale configuration re-
file and updates the rev key to this tag. $ identify-cli [Link] quires external packages (see my article
The next time you run pre-commit, it ["file", "markdown", "non-executable", U about Vale in Linux Magazine [9]), you
checks out the new version from the re- "plain-text", "text"] first need to run vale sync to download
pository and installs it before running and install these packages prior to run-
the hooks. If you now want to restrict a hook to all ning vale on your documentation files.
Markdown files, simply specify the Unfortunately, Vale doesn’t offer a
Modifying pre-commit hook as pre-commit hook for its sync command in
Hooks its repository. However, this is easy to
Listing 1 runs a few hooks with their de- - id: trailing-whitespace remedy by running the vale hook twice:
fault behavior, but you can modify this types: [markdown] once with the extra sync argument, and
behavior in some cases. While id is the once without (Listing 3).
only mandatory key in a hook mapping, You can also specify multiple file types Note that this configuration overrides
you can change a hook’s behavior with a for which a hook will run: the name of the first invocation of the
range of optional keys. For instance, you
can pass additional arguments to a hook. Listing 2: Using Ruff
After all, a hook is just an executable 01 - repo: [Link]
script that can take command-line 02 rev: v0.4.1
arguments.
03 hooks:
For example, you can use the follow-
04 - id: ruff
ing command to raise the maximum size
05 - id: ruff-format
of committed files to 1MB:

42 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
Git Hooks

where you specify the executable to be

run (Listing 5).

Running pre-commit Hooks

Manually
At any time, you can manually run all
pre-commit hooks in a repository. For
example, following some code modifica-
tions but prior to committing your
changes, you can run the hooks to reveal
any identified issues beforehand. Just
Figure 3: For each commit, Ruff checks your Python files to see if they run the following command:
violate its rules.
$ pre-commit run

hook, to show the user that this hook is If you find pre-commit beneficial,
performing a sync instead of the default you could add a variety of checks to Bear in mind that this checks only for
vale action. It also sets pass_filenames to your pre-commit hooks. However, make files added with git add.
false, so pre-commit doesn’t pass file sure that the hooks don’t take too long You can also run an individual hook
names to the hook. That is, by default to run, because this can lead to frustra- by referring to its ID:
pre-commit passes all file names of the tion, resulting in you or other collabo-
files changed in the commit to the hook, rators disabling pre-commit hooks, $ pre-commit run generate

so it knows which files it needs to check. which defeats their purpose. For exam-
However, the vale sync command ple, on a large codebase, mypy can be If you want to check all files in the re-
doesn’t need to check any files, because slow and may be better to run manu- pository, regardless of their state in the
it merely updates Vale’s packages. ally, rather than on every commit. Git database, add the --all-files
Mypy [10], a static type checker for Tests (for example, with pytest [11]) argument:
Python code, can also be run in a are something else that you should
pre-commit hook. The pre-commit project probably not run in pre-commit hooks. $ pre-commit run --all-files

has its own mirror holding a hook for

mypy. Because pre-commit runs the mypy Local Scripts This is always a good idea after adding a
command from an isolated Python vir- Running pre-commit hooks isn’t limited new hook. You can also combine this
tual environment, it’s advised to install to scripts from public Git repositories. with the restriction to an individual hook:
additional dependencies for more com- You can also run a local script as a Git
plete type checking. You can achieve this hook. For instance, to run a script in $ pre-commit run generate --all-files

using the additional_dependencies key scripts/[Link] in your repository,

(Listing 4). add a local repository in your pre- If a hook is too time-consuming, you
commit configu- can specify it as manual, so it won’t be
Listing 3: Running Vale with sync ration file, con- automatically run on each Git commit
01 - repo: [Link]
taining a hook (Listing 6).
02 rev: v3.4.1
Listing 5: Using a Local Script
03 hooks:
01 - repo: local
04 - id: vale
02 hooks:
05 name: vale-sync
03 - id: generate
06 pass_filenames: false
04 name: Generate Python modules
07 args: [sync]
05 entry: python scripts/[Link]
08 - id: vale
06 language: python

07 pass_filenames: false
Listing 4: Using mypy with Additional Dependencies
08 additional_dependencies: [jinja2]
01 - repo: [Link]

02 rev: v1.9.0
Listing 6: Specifying a pre-commit Hook as Manual
03 hooks:
01 - repo: [Link]
04 - id: mypy
02 rev: v1.9.0
05 additional_dependencies:
03 hooks:
06 - bleak>=0.19.0 04 - id: mypy

07 - bluetooth-adapters>=0.15.3 05 stages: [manual]

[Link] ISSUE 284 JULY 2024 43

IN-DEPTH
Git Hooks

Listing 7: Using pre-commit as a GitHub Action pre-commit/ac- Conclusion

01 name: pre-commit
tion [13] action to The pre-commit framework provides an
02
run the same Git easy way to automatically run checks on
03 on:
hooks that you any code committed to a Git repository.
04 pull_request:
run locally on It’s a powerful tool to help to improve
05 push:
GitHub’s end for code quality and consistency and get
06 branches: [main]
every pull request every contributor on the same page.
or push. This ap- Moreover, the pre-commit website offers
07
proach is benefi- comprehensive documentation about its
08 jobs:
cial as an addi- advanced features. Q Q Q
09 pre-commit:
tional safeguard if
10 runs-on: ubuntu-latest
not all collabora- Info
11 steps:
tors have pre- [1] Git: [Link]
12 - uses: actions/checkout@v3
commit installed
13 - uses: actions/setup-python@v3 [2] Git hooks: [Link]
locally.
14 - uses: pre-commit/action@v3.0.1 en/v2/Customizing-Git-Git-Hooks
A typical use of
[3] Stylelint: [Link]
pre-commit/action
You can still run the hook on demand checks out your repository, sets up [4] Ruff: [Link]
any time you want with Python, and then runs the pre-commit [5] Vale: [Link]
action (Listing 7). [6] pre-commit:
$ pre-commit run U Essentially, this runs pre-commit run [Link]
--hook-stage manual mypy on the changed files in the pull request
[7] YAML: [Link]
or push. With the extra_args argument,
Sometimes a pre-commit hook incorrectly you can pass options to pre-commit run, [8] pre-commit-hooks:
prevents you from committing your for example, to check all files or to spec- [Link]
changes. For example, I encountered a ify a single hook: pre-commit-hooks
temporary issue when one of the hooks [9] “Enforcing Text Style with Vale” by
failed to work due to a TLS certificate - uses: pre-commit/action@v3.0.1 Koen Vervloesem, Linux Magazine,
error. To skip a failed hook, specify it in with: issue 283, June 2024, pp. 78-82
the SKIP environment variable: extra_args: mypy --all-files
[10] mypy: [Link]
[11] pytest: [Link]
$ SKIP=vale-sync git commit -m "Add foo"

Using Other Git Hooks [12] GitHub Actions:

If you need to skip multiple hooks, use a In addition to supporting pre-commit [Link]
comma-separated list of hook IDs. Alter- hooks, pre-commit also supports com- [13] pre-commit/action:
natively, you can skip all pre-commit mit-msg, post-checkout, post-commit, [Link]
hooks with: post-merge, post-rewrite, pre-merge-com-
mit, pre-push, pre-rebase, and pre- Author
$ git commit -m "Add foobar" --no-verify pare-commit-msg. By default, pre-commit Koen Vervloesem has been writing about
only installs pre-commit hooks, but you Linux and open source, computer security,
Running pre-commit Hooks can specify a default set of Git hook privacy, programming, artificial intelli-
in GitHub Actions types to be installed by setting the top- gence, and the Internet of Things for more
If you’re using GitHub Actions [12] in level default_install_hook_types key to a than 20 years. You can find more on his
your GitHub repository, you can use the list of all desired hook types. website at [Link].

QQQ

44 JULY 2024 ISSUE 284 [Link]

IN-DEPTH
Command Line – Environmental Variables

Working with environmental variables

In the Know
Environmental variables often operate quietly in the background, but knowing how to
set, modify, and delete them can come in handy. By Bruce Byfield

R
oughly speaking, environmen- permanent Bash variables are stored in While other variables apply to a particu-
tal variables may be thought of .bash_profile in an account’s home di- lar application, environmental variables
as the configuration files for a rectory, while Bash variables such as can be used by any application run
user account. Operating be- aliases are stored in .bashrc. within an account.
hind day-to-day operations, environ- In contrast, environmental variables Environmental variables include a
mental variables define the resources are general settings for a particular ac- large variety of entries that varies with
available to an account. While it is per- count, rather than values for an entire each distribution and desktop environ-
fectly possible to ignore environmental system or a particular application. ment (Figure 1). Over the years, this list
variables when running a Linux ac- Confusion arises because all these has grown in some cases to two to three
count, you may need to edit them types of variables are similar in struc- screens full of values, thanks in large
sometimes to correct a gap in function- ture. When referred to in the abstract, part to the increasing complexity of
ality, especially after new packages are all these variables use the same struc- desktops environments and frameworks
installed. For this reason, it makes ture such as $HOME. The following three (e.g., Qt, GTK2, and GTK3) that need to
sense to know how to set, modify, and formats are used in all these define values, colors, cursor themes,
delete environmental variables. On circumstances: and window managers. Other variables
networks, you’ll also want to safeguard • Single variable: KDE_SESSION_VERSION=5 are as basic as the account’s $USER,
them against security breaches. • Variable with spaces: USER="marie $UID, $HOME, $LANGUAGE, $EDIT, $TERM (vir-
Structurally, environmental variables huxley" tual terminal), $PWD (present working
resemble the fields found in most appli- • Variable with multiple values: PATH=/ directory), and $MAIL location. In mod-
Lead Image © sinenkiy, [Link]

cations’ configuration files. For in- usr/local/bin:/usr/bin:/bin:/usr/ ern systems, there may be a path to
stance, Python’s [Link] contains local/games:/usr/games $SYSTEMD_EXEC_PID, as well as settings
such variables as $ENGINE, $HOST, and Note the use of uppercase characters for starting with XDG (e.g., $XDG_SESSION_ID
$POST. Some of these applications are the variable’s name and the lack of and $XDG_RUNTIME_DIR) for the X Win-
global, such as the systemd variables spaces unless quotation marks are used. dow System or, increasingly, for Way-
contained in /etc/experiment.d, which In addition, the variable’s value is case- land. Some 18 lines are devoted to LS_
include the resources that GTK and Qt sensitive. The differences lie mainly in COLORS, the color options for directories,
use to interact with system. Similarly, what part of the system they define. files, and extensions in the shell.

46 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
Command Line – Environmental Variables

Probably the most important variables variables are loaded from other files. queried at the same time in a space-
are the path to the $SHELL, usually /bin/ The first source is /etc/environment. separated list (Figure 2). By contrast,
bash, and the complete $PATH, which at From there, /etc/profile or /.bash_pro- env can create a temporary environ-
minimum is usually /usr/local/bin:/ file, ~/.bash_login, and ~/.profile in ment suitable for testing or a special-
usr/bin:/bin:/usr/local/games:/usr/ an account’s home may be used. With a ized one-time use by specifying a
games. Both $SHELL and $PATH define how graphical display manager, a successful space-separated list of variables. If
the account interacts with the rest of login follows init and starts an X ses- --ignore-environment (-i) is added to
the system. In particular, the complete sion, reading variables from all the pos- env, the temporary list consists only of
path allows binaries to run without sible file sources. For security, the ac- those listed in the command. If
being located in the present working count’s password will not be displayed. --unset=VARIABLE (-u) is added, the
directory. Either at the command line or in a variable listed is temporarily removed
graphical display manager, configuration from the environment.
Editing by Command in a text editor would involve so many To make permanent changes to the en-
Another distinguishing feature is that different files that the task would be vironment, you can use either export or
environmental variables are not edited both painstaking and slow. For this rea- set, followed by one or more variables
as text in a configuration file. In fact, son, it is easier to edit using commands. presented in the format VARIABLE=VALUE.
how environmental variables are stored Many of these commands duplicate the You’ll find export especially useful for
is frequently glossed over because the functions of others. adding to an existing variable. For exam-
explanation is not widely available. Ac- Because environmental sources ple, to add a directory to the $PATH, the
cording to the Debian Wiki [1], when come from multiple sources, there is command structure is
you boot from the command line, the no single file to edit. Instead, a com-
boot process ends with init running the posite list of variables can be created. export PATH=NEW-DIRECTORY:$PATH

scripts in /lib/systemd/system under If you enter the bare command print-

systemd or in the current run-level file, env or env, you will see the same com- However, set can be the safer of the two
rc [1-6].d under sysinit, in either case plete list of current system variables commands if you use -C to avoid acci-
loading variables from these sources. shown in Figure 1. In addition, print- dentally overwriting files, and the edits
When you type your username and env can display the value for a particu- can be tested before making changes by
password, $USER and $TERM are loaded. lar variable simply by adding its name adding -n. In addition, any deletions are
If both appear in etc/passwd and /etc/ after the command, without a $ in made deliberate by the requirement that
shadow, the remainder of the account’s front of it. Multiple variables can be they must be done with the companion

Figure 1: The start of the three screens of variables for KDE Plasma in Debian 12. Other distributions and
desktop environments will have different sets of variables.

[Link] ISSUE 284 JULY 2024 47

IN-DEPTH
Command Line – Environmental Variables

Your Account Summary

Most of the time, environmental vari-
ables operate in the background, with-
out needing any attention. However, it
is worth knowing how to read and edit
Figure 2: Using a space-separated list, printenv can display the values them when unexpected situations arise.
for multiple variables. Surprisingly often, an application may
refuse to run for no stronger reason
command unset. None of these com- Basically a general manager for en- than it is not in your path and you are
mands output any feedback, so the re- vironmental variables, envio runs on not in its directory. Or perhaps you
sults can only be seen when printenv Linux, macOS, and Windows. It can want to always use a shell other than
or env is entered. create, assign, and remove multiple Bash or you want to change your lan-
profiles for environmental variables, guage and locale. In such cases, know-
Managing Multiple Sets thereby controlling each user’s access ing your environmental variables can
of Variables to system resources. For added secu- provide a quick update. Q Q Q
On a standalone machine, environmental rity, envio can also protect profiles
variables are probably adequately pro- with passphrases or GPG encryption. Info
tected by passphrases and encryption. Although still in rapid development, [1] Debian Wiki: [Link]
However, on networks, the situation can envio is a much needed enhance- EnvironmentVariables
be more complicated. Often, not all users ment that applies existing security [2] envio: [Link]
should have access to all resources, and measures to environmental
intruders could edit resources such as variables. Author
the path to Bruce Byfield is a computer journalist and
gain full ac- a freelance writer and editor specializing
cess to a sys- in free and open source software. In
tem. For such addition to his writing projects, he also
teaches live and e-learning courses. In his
reasons, sys
spare time, Bruce writes about Northwest
admins may Coast art ([Link]
want to install Figure 3: Ideal for networks or testing, envio creates com). He is also cofounder of Prentice
envio [2] (Fig- encrypted profiles with different sets of environmental Pieces, a blog about writing and fantasy at
ure 3). variables for different uses or purposes. [Link]

QQQ

48 JULY 2024 ISSUE 284 [Link]

IN-DEPTH
Programming Snapshot – Go Bandwidth Display

Displaying bandwidth usage with Go

Measuring
the Flow
A Go program running on a Raspberry Pi grabs metrics
from a pfSense firewall and displays them on a
miniature display to help Mike Schilli keep an eye on
his Internet connection’s bandwidth usage. By Mike Schilli

M
easuring the active throughput but on a separate Raspberry Pi, which I “the Internet is down again” from the
of an Internet connection is equipped with a $50 color display for other room.
not entirely trivial, because no- continuous viewing pleasure (Figure 2).
body wants the measuring When I’m sitting at my desk, I can see Raspberry as a Helper
probe to slow down the data traffic. How- out of the corner of my eye the number The Go program from the source code
ever, the router at the Internet access point of bits zooming in or out every second. for this issue runs on a Raspberry Pi 4
has to view and forward all of the packets As a side effect, I can also see at a glance with an Ethernet connection. It retrieves
anyway, so why not let it also count them what’s going on when someone shouts the current packet throughput from the
and provide the results via an API?
At home, I use a pfSense firewall on a
fanless mini PC as my main router,
which also runs some apps with access
to the packet throughput (Figure 1). One
of these apps is ntopng, which shows
you in a browser which LAN client is
currently communicating with which
server on the Internet – among other
things. Ntopng also offers an API with
token authentication, which returns
counters for the bits transferred in both
directions.
I didn’t want to rely on the tool just
running on demand in a web browser,

Author
Lead Image © xyzproject, [Link]

Mike Schilli works as a

software engineer in the
San Francisco Bay Area,
California. Each month
in his column, which has
been running since 1997,
he researches practical applications of
various programming languages. If you
email him at mschilli@[Link]
he will gladly answer any questions. Figure 1: pfSense dashboard.

50 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
Programming Snapshot – Go Bandwidth Display

pfSense firewall’s API every five seconds go-chart library to draw a chart from the to display this chart on the Rasp Pi desk-
and saves the upload and download val- data of the last two and a half minutes, top, refreshing the display smoothly
ues with the current timestamp in a ring nicely illustrating the packet throughput every five seconds.
buffer. The program then uses the over time. Its GUI uses the Fyne library For example, if someone in the house-
hold is using Netflix, the graph (Fig-
ure 3) shows the streaming client’s inter-
mittent server requests at up to 10Mbps.
But if I run my ISP’s load test, which
first measures the maximum download
speed and shortly afterward the upload
speed, the display looks like what you
can see in Figure 4.

Flat Line Signals Outage

If an error occurs (e.g., if someone takes
out the Internet connection for test pur-
poses), the throughput visibly drops to
practically zero, as shown in Figure 5. I
can then see at a glance that something
is wrong and start troubleshooting.
How does the Rasp Pi retrieve the cur-
rent load values from the pfSense fire-
wall? It requires an API token for access;
ntopng generates one in the User Auth
Figure 2: The new display at work. Token tab of the Settings | Users menu
item (Figure 6). Listing 1 stores the re-
sulting hex string in a constant in line 11;
it can be used when calling fetchJSON()
starting in line 13 to retrieve the current
JSON data from the pfSense appliance
under the specified IP address along
with the ntopng app’s Lua path. The API
documentation [1] for ntopng contains
rudimentary documentation for the re-
quest paths, and the JSON content in the
response is kind of self-explanatory.
Line 20 specifies the router’s Internet
interface as 0 in the ifid parameter; this
is the first and only interface on my sim-
ple firewall. The client does not send the
API token as part of the URL, but adds it
Figure 3: A typical throughput pattern when watching Netflix. in line 26 as an HTTP header before the
actual URL request. The server then re-
turns detailed data on the firewall status
(Figure 7), from which the fetchUpDown()
function starting in line 44 of Listing 1
extracts the bits-per-second values for
upload and download.

JSON Navigation
The code makes quick work of extract-
ing the relevant bits by importing the
gjson library from GitHub and plumb-
ing the depths of the JSON structure in
XPath style using the [Link].
[Link] hierarchy. There are float-
ing-point numbers for upload and down-
load there, which gjson imports into Go
Figure 4: The ISP’s load test saturates the Internet connection. as Floats. The final return instruction

[Link] ISSUE 284 JULY 2024 51

IN-DEPTH
Programming Snapshot – Go Bandwidth Display

divides the value by 1,000 to produce a older values without further ado as Data Storage in Go
more manageable kilobits per second soon as the pointer has come round In Listing 2, line 12 defines the Dpoint
value. full circle (Figure 8). At any given structure as a container for individual
The ring buffer data structure de- time, the buffer only knows what the measurements; it stores the timestamp
fined in Listing 2 collects the individ- current element is, how many elements for each measured value, along with the
ual readings that then occur every five exist in the ring, and how to move floating-point values for the upload and
seconds until 30 measured values are from the current to the Next() element download in kilobits per second. Line 8
available; the chart library uses these or back from the current element to the molds the ring buffer from Go’s standard
values later on to generate the chart. previous one (Prev()). That’s it; it’s library container/ring into the Dpoints
Conveniently, the ring buffer drops simple, but powerful. (note the plural) structure.
This means that the NewRing() con-
structor can create a new ring object in
line 18. Starting in line 22, Add() uses
Go’s receiver mechanism to feed new
values into the ring, while All() later re-
turns all the values that exist in the ring
buffer in three array slices starting in
line 31. The first slice contains all the
timestamps for the measured values, the
second the floating-point values for the
upload measurements, and the third the
values for the download measurements.
Sounds awkward? The reason for this is
that the chart library later needs the val-
ues in this format to draw the chart in
the X/Y coordinate system.
Figure 5: A drop in throughput indicates connection problems. While moving within locations on the
ring, the code makes use of the fact that
uninitialized elements in the ring have a
zero value (nil) and that Len() returns
the total number of available elements.
The All() function moves backward until
it encounters an uninitialized element or
has turned a complete circle. It then
starts moving forward again and picks
up all the measured values it finds until
it reaches the starting point stored in n.

Fancy Colors
I used the go-chart project from GitHub
for drawing the charts in Figure 3, Fig-
ure 4, and Figure 5, utilizing two line
graphs for uploads and downloads.
The drawChart() function (Listing 3,
Figure 6: The pfSense ntopng app issues API tokens.

Figure 8: The simple ring buffer

Figure 7: The JSON data of the ntopng API contains the bps values for navigates forward and back-
upload and download. ward in a circle.

52 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
Programming Snapshot – Go Bandwidth Display

Listing 1: [Link]
01 package main 28 Transport: &[Link]{

02 29 TLSClientConfig: &[Link]{InsecureSkipVerify: true},

03 import ( 30 },
04 "crypto/tls" 31 }
05 "[Link]/tidwall/gjson" 32 resp, err := [Link](req)
06 "io/ioutil"
33 if err != nil {
07 "net/http"
34 return "", err
08 "net/url"
35 }
09 )
36 defer [Link]()
10
37 body, err := [Link]([Link])
11 const APIKEY = "35a3907943cdf9fdb85228627a06034c"
38 if err != nil {
12
39 return "", err
13 func fetchJSON() (string, error) {
40 }
14 u := [Link]{
41 return string(body), nil
15 Scheme: "https",
42 }
16 Host: "[Link]:3000",
43
17 Path: "/lua/rest/v2/get/interface/[Link]",

18 } 44 func fetchUpDown() (float64, float64, error) {

19 p := [Link]() 45 json, err := fetchJSON()

20 [Link]("ifid", "0") 46 if err != nil {

21 [Link] = [Link]() 47 return 0, 0, err

22 req, err := [Link]("GET", [Link](), nil) 48 }

23 if err != nil { 49 down := [Link](json, "[Link]").

24 return "", err Float()

25 } 50 up := [Link](json, "[Link]").Float()

26 [Link]("Authorization", "Token "+APIKEY) 51 return down / 1000, up / 1000, nil

27 client := &[Link]{ 52 }

Listing 2: [Link]
01 package main 28 [Link] = [Link]()

02 29 }
03 import ( 30
04 "container/ring" 31 func (d Dpoints) All() ([][Link], []float64, []
05 "time" float64) {

06 ) 32 ups, downs := []float64{}, []float64{}

07 33 times := [][Link]{}
08 type Dpoints struct { 34 r := [Link]
09 rp *[Link] 35 n := 0
10 } 36 for i := 0; i < [Link](); i++ {
11
37 r = [Link]()
12 type Dpoint struct {
38 if [Link] == nil {
13 dt [Link]
39 r = [Link]()
14 up float64
40 break
15 down float64
41 }
16 }
42 n++
17
43 }
18 func NewRing(n int) *Dpoints {
44 for i := 0; i < n; i++ {
19 return &Dpoints{rp: [Link](n)}
45 dp := [Link].(Dpoint)
20 }
46 times = append(times, [Link])
21
47 ups = append(ups, [Link])
22 func (d *Dpoints) Add(up, down float64) {

23 [Link] = Dpoint{ 48 downs = append(downs, [Link])

24 dt: [Link](), 49 r = [Link]()

25 up: up, 50 }

26 down: down, 51 return times, ups, downs

27 } 52 }

[Link] ISSUE 284 JULY 2024 53

IN-DEPTH
Programming Snapshot – Go Bandwidth Display

starting in line 14) expects a ring buf- the web browser and downloads a few This is why line 21 in Listing 3 does
fer and stores a finished chart file in images from the Internet, the measured not define anything special for the X-axis
[Link]. value quickly jumps up to the megabit with the time values; after all, the time-
Lines 34 and 43 define two chart. range. A Netflix connection for stream- stamps grow in a linear fashion as time
TimeSeries type time series. Each of ing a movie pushes the accelerator pedal progresses. In contrast to this, Logarith-
them is assigned an array slice of the right down to the floor in regular bursts, micRange transforms the readings on the
timestamps in Unix seconds in XValues, using up the entire available bandwidth Y-axis to an exponential display format
while the measured values are assigned of 50Mbps. in the code in line 27. The maximum
as floating-point values in YValues. The If you were to use a linear scale that value of 100,000 is equivalent to
color combinations cyan/green (upload) goes up to 50Mbps, though, a variation 100Mbps, going down to 10Mbps,
and red/baby blue (download) for the in the range of 1kbps would be imper- 1Mbps, 100kbps, and so on at equal in-
graphs and their fill area may seem arbi- ceptible – totally flat. Instead, I wanted tervals. This means both that even minor
trary, but not so fast! I spent decades the display to be able to distinguish be- variations remain visible and the graph
scouring the world’s museums for paint- tween absolute zero and a connection does not shoot over the top of the coor-
ings by Gerhard Richter just to create with low usage. I used a logarithmic dinate system in the case of brief peaks
this exquisite combination. scale for this to show the range from of high bandwidth.
100kbps to 1Mbps at the same level as The [Link] type object starting in
Logarithmic Scale the range between 1Mbps and 10Mbps. line 52 packages the two axes and the
Now the bandwidth used by an ISP con- This is perfect for observing variations in time series, while the Render() function
nection often varies by orders of magni- every order of magnitude – as long as draws a neat looking chart in the speci-
tude. If almost nothing is happening, there are no negative values, because as fied PNG file. To make sure that the li-
only a few kilobits whiz back and forth. you might remember from school, by brary cleanly labels the X-axis with the
But if someone presses the Enter key in definition, logarithms can’t handle these. times of the measurement points,

Listing 3: [Link]
01 package main 33 }

02 34 upseries := [Link]{

03 import ( 35 XValues: times,

04 "fmt" 36 YValues: ups,

05 "[Link]/wcharczuk/go-chart/v2" 37 Style: [Link]{

06 "os" 38 StrokeColor: [Link],

07 "time" 39 StrokeWidth: 10,

08 ) 40 FillColor: [Link](64),

09 41 },

10 const GRAPH_FILE = "[Link]" 42 }

11 const GRAPH_WIDTH = 1920 43 downseries := [Link]{

12 const GRAPH_HEIGHT = 1000 44 XValues: times,

13 45 YValues: downs,

14 func drawChart(ring *Dpoints) { 46 Style: [Link]{

15 up, down, err := fetchUpDown() 47 StrokeColor: [Link],

16 if err != nil { 48 StrokeWidth: 10,

17 panic(err) 49 FillColor: [Link](64),

18 } 50 },

19 [Link](up, down) 51 }

20 times, ups, downs := [Link]() 52 graph := [Link]{

21 xAxisCfg := [Link]{ 53 XAxis: xAxisCfg,

22 ValueFormatter: func(v interface{}) string { 54 YAxis: yAxisCfg,

23 return [Link](0, int64(v.(float64))). 55 Height: GRAPH_HEIGHT,

Format("[Link]") 56 Width: GRAPH_WIDTH,
24 }, 57 Series: [][Link]{
25 } 58 upseries,
26 yAxisCfg := [Link]{ 59 downseries,
27 Range: &[Link]{ 60 },

28 Max: 100000, 61 }

29 }, 62 f, _ := [Link](GRAPH_FILE)

30 ValueFormatter: func(v interface{}) string { 63 defer [Link]()

31 return [Link]("%.2f MBps", v.(float64)/1000.0) 64 [Link]([Link], f)

32 }, 65 }

54 JULY 2024 ISSUE 284 [Link]

 IN-DEPTH
Programming Snapshot – Go Bandwidth Display

ValueFormatter defines a function in line calling Refresh() for the container ob- Remember that the Rasp Pi uses an ARM
22 that first converts the X-values ject. If the user has enough and presses processor, whereas most Linux boxes
(which are available in Unix seconds as Q, the GUI jumps to the callback starting use an Intel-compatible CPU, which
you will recall) into [Link] objects in line 25, collapses the window, and makes things a bit challenging.
before then displaying them as hours, terminates the program. Normally, Go makes it easy to compile
minutes, and seconds using For- binaries for other operating systems or
mat("[Link]"). In contrast to this, the The Trouble with Go architectures from the same source code.
value formatter for the Y-axis in line 30 To compile the binary, go mod init/tidy However, the fun stops as soon a graph-
simply divides the incoming kilobit val- fetches all the dependent libraries off the ics library such as Fyne starts integrating
ues by 1,000 (i.e., it’s showing them in web, while calling go build with the native C code, such as the X11 library on
megabits per second units). source code files builds everything lo- Linux. The C compiler needs to be able
The main program in Listing 4 now cally. But how do you actually install the to cross-compile in this case. Thankfully,
only has to call the utility functions de- Go program on a Raspberry Pi? the Fyne team offers the fyne-cross [2]
fined previously to generate the chart
file, display the chart in an application
window, and refresh it at regular inter-
vals. To do this, the Fyne universal GUI
framework dumps the image object cre-
ated by updateChart() starting in line 43
into a container, which is waiting in the
application window.
The Go routine starting in line 31 runs
in an infinite loop with a timer waiting
five seconds on each round. The code
then calls updateChart() to create a new
image file, reads the file, and triggers the
GUI to refresh the displayed image by Figure 9: Cross-compile for a Rasp Pi binary.
IN-DEPTH
Programming Snapshot – Go Bandwidth Display

Listing 4: [Link]
01 package main 26 switch key {

02 27 case "Q":

03 import ( 28 [Link](0)

04 "[Link]/fyne/v2" 29 }

05 "[Link]/fyne/v2/app" 30 })

06 "[Link]/fyne/v2/canvas" 31 go func() {

07 "[Link]/fyne/v2/container" 32 for {

08 "os" 33 select {

09 "time" 34 case <-[Link](5 * [Link]):

10 ) 35 img = updateChart(ring, width, height)

11 36 [Link]()

12 func main() { 37 }
13 a := [Link]() 38 }
14 w := [Link]("Netgraph") 39 }()
15 width := float32(GRAPH_WIDTH) 40 [Link]()
16 height := float32(GRAPH_HEIGHT) 41 }
17 [Link]([Link](width, height)) 42
18 [Link](true) 43 func updateChart(ring *Dpoints, width, height float32)
19 ring := NewRing(30) *[Link] {

20 img := updateChart(ring, width, height) 44 drawChart(ring)

21 con := [Link](img) 45 img := [Link](GRAPH_FILE)

22 [Link](con) 46 [Link] = [Link]

23 [Link]().SetOnTypedKey( 47 [Link]([Link](width, height))

24 func(ev *[Link]) { 48 return img

25 key := string([Link]) 49 }

toolchain for this: fyne-cross creates a private server for downloading, and full-screen mode, you can do this with
Docker container and then executes the the application is ready to run. You the wmctrl tool using
desired cross-build in it. As a result, de- still need to adapt the API key and the
velopers do not need to rack their brains IP address for the firewall to your local wmctrl -r "Netgraph" U
working out numerous settings and conditions. -b toggle,fullscreen

dependencies.
As Figure 9 shows, the fyne-cross Automatic Start-Up In addition, you will want to create a new
cross-compiler makes itself at home in To tell the Raspberry Pi running on Pi [Link] file with the configura-
the user’s Go directory. If you call it OS to automatically log into the desktop tion from Listing 5 in your home directory
there with linux as the target and and launch the application immediately below ~/.config/autostart to start the ap-
--arch=arm64 for the 64-bit ARM archi- after booting, the Rasp Pi configuration plication right after a complete boot.
tecture (for a 32-bit Rasp Pi use arm), must be set to use Auto-Login. Make The shell script launched there can call
you can look forward to seeing a bi- sure you set Screen Blanking to Off in netgraph directly, or, if you fancy, first
nary for the target platform after a few the Raspberry Pi configuration as well download the latest version from the server
minutes; this time is mainly needed by to avoid the small-board computer acti- and then launch it. The Rasp Pi will then
the program to download several layers vating the screen saver. start to display the chart – first with just a
of a Docker image. Next, copy the bi- To avoid wearing out the perma- few values, and then more as time pro-
nary to a path on the Rasp Pi with an nently active display, it makes sense to gresses and readings accumulate. It’s fun to
Internet connection, preferably from a set a black background for the chart. watch out of the corner of your eye! Q Q Q
You can do that
Listing 5: [Link] using the Background Info
[Desktop Entry]
and FillColor op- [1] API documentation for ntopng: https://
Type Application
tions on the Chart [Link]/guides/ntopng/api/
object. If you want [2] Tool for cross-compiling Fyne applica-
Name=Netgraph
the application win- tions: [Link]
Exec=/bin/sh /home/pi/[Link]
dow to run in cross-compiling

QQQ

56 JULY 2024 ISSUE 284 [Link]

MAKERSPACE Snek

MakerSpace
Run Python on old Arduino modules

Old Dogs,
New Tricks
Reuse your old Arduino hardware while learning Python.
By Pete Metcalfe

S
nek [1] is a tiny embeddable lan- Python language. The Snek implementa-
guage that can run on processors tion on Arduino hardware allows you to
too small for MicroPython. It build some enjoyable Python projects
supports about 20 of the older with just a few lines of code. The pro-
Arduino modules (e.g., Duemilanove, gramming experience you gain in Snek
LilyPad, Mega, Nano, and Uno), as well can be used in larger Python 3 or Micro-
as Snekboard controllers and LEGO Python projects.
robotics projects that use the LEGO EV. In this article, I introduce Snek by
Snek is based on Python syntax, but it showing you how to install and set up
only supports a small subset of the full some simple Arduino projects. None of

Lead Image © margaritatkahcenko, [Link]

Figure 1: Snek firmware uploader.

58 JULY 2024 ISSUE 284 [Link]

 Snek MAKERSPACE

these examples use more than 10 lines $ ./snek-uno-install U code editor [3] is a simple and popular
of code. -hex [Link] tool that can be used with Python 3,
MicroPython, and Snek coding.
Installation It’s important to note that different hard- To use Mu with Snek, connect an up-
Arduino Uno, Nano, or Pro Mini mod- ware might support added features. For loaded Arduino module to your laptop
ules can load their firmware directly example, the Adafruit ItsyBitsy M0 module and select the Mode icon (Figure 2).
from a web page [2]. The web uploader supports NeoPixel LED arrays, and the After Mu is connected to the Arduino
page (Figure 1) installs Snek v1.5. To in- LEGO EV3 controllers can use servomotors module, the Serial option lets you man-
stall the latest firmware (version 1.9 at and color, touch, and ultrasonic sensors. ually enter and run Snek commands
press time) or to upload to modules not Because of limited amounts of RAM (Figure 3).
supported on the web page, enter the and memory on certain Arduino modules, The Put icon uploads and runs a Snek
following commands in a terminal basic and big firmware versions are avail- file. Figure 4 shows the code to blink
window: able. The big versions support added the on-board LED (digital pin 13) four
features, but they might not be install- times. Snek print() statements show
$ wget [Link] able on certain modules without some the program starting, iterating, and
dist/[Link] extra steps (see the Snek homepage [1] completing. Support for the general
$ chmod +x [Link] for more details). The Arduino Mega purpose input/output (GPIO) pins is
$ # Create a local dir with all files supports the big version as its default. built into Snek, so unlike Python and
$ ./[Link] For all the examples in this article, I MicroPython, no libraries need to be
have used the basic firmware on an imported. The talkto(pin) command
Once the files are stored locally, an Ar- Arduino Uno. (Figure 4, line 7) connects to a specific
duino Uno module can be uploaded with pin, and the on() (line 8) and off()
the latest version: Connecting and Testing (line 10) commands set a GPIO pin to
A number of different integrated devel- a 1 or 0 output state.
$ # Move to the Snek install directory opment environ-
$ cd Snek ments (IDEs) can
$ # Upload Uno module with the U be used with
basic v1.9 Snek. The Mu

Figure 4: Upload and run an LED blink program.

Figure 2: Connect a Snek module with Mu.

Figure 3: Manually run Snek commands on an

Arduino module. Figure 5: Touch sensor and fan setup.

[Link] ISSUE 284 JULY 2024 59

MAKERSPACE Snek

Listing 1: Toggle a Fan (ground), and SIG has a pair of buttons that can be used to
01 # [Link] - Toggle a fan with a touch sensor
(data signal). The turn on LEDs (Figure 6).
02 # Note: the touch sensor toggles and hold the last state
SIG pin is wired to In this example (Listing 2), I created a
03 fan_pin = D12
digital pin 3 (D3) dictionary with pairs of switch pin num-
04 touch_pin = D3
on the Arduino. bers and the LED pin numbers (line 5).
05
The Snek code The for loop (line 10) iterates through
(Listing 1) uses the pairs dictionary, and an if statement
06 talkto(fan_pin)
the talkto(<pin>) checks for a button push (line 11). It’s
07 while True:
function to set up important to note that the Snek default is
08 # toggle the fan state with each touch
output commands for GPIO pins to be pulled up, so an
09 if read(touch_pin) == 1:
to a GPIO pin open connection is 1 and a closed, ener-
10 on()
(line 6), and then gized, or (in this case) pressed button
11 else:
on() and off() reads as 0. If the button is pressed, the
12 off()
commands can be onfor(3) statement turns on the LED for
sent (lines 10 three seconds (line 14).
Reading and Writing to and 12). The read(<pin>) function reads For just two input/output pairs, a dic-
GPIO pins both analog and digital pins (line 9). tionary might seem like overkill, but this
In the next GPIO project, user input approach works well for projects with
generates output. The project uses: Arduino Add-On Components multiple inputs, such as controlling
• A Keyes L9110 fan motor (~$4) Snek only supports the basic GPIO func- motor pins for forward, stop, left, right,
• A touch sensor (~$5) tions of digital and analog reads and and reverse actions.
• An Arduino Uno writes, so unfortunately, equipment that This project could be enhanced to cre-
• Meccano construction pieces for has I2C, serial, or specialized communi- ate car or boat projects that use a remote
support cation solutions won’t be usable. Despite four-input radio frequency (RF) module
The touch sensor toggles and holds a this limitation, you still have a good selec- (XD-YK0) with a keypad (~$12) and
state with each touch. The first touch tion of Arduino add-on modules and motor or relay shields ($10-$15).
holds and reads as a 1, and then a sec- components you
ond touch holds and reads as a 0. This can use on your
1-0-1-0 holding of values removes any Snek projects. For
problems of quick touches being missed. the next example, I
Figure 5 shows my setup with some used an HY-M302
Meccano pieces securing the fan motor. multifunction
A simpler project to illustrate read/write shield (~$12), a
to GPIO pins could be done with a basic general-purpose
push button and an LED. The fan motor board with but-
has four pins: VCC (power), GND tons, buzzers,
(ground), INA (direction), and INB (on/ LEDs, and analog
off). The INB pin is wired to digital pin inputs that can be
12 (D12) on the Arduino, and the INA accessed through
(direction) pin isn’t used. The touch sen- the standard GPIO
sor has three pins: VCC (power), GND pins. The shield

Listing 2: Button Press LEDs Figure 6: Use the switches on the general-purpose
01 # [Link]
HY-M302 multifunction shield to turn on LEDs.
- Turn on an LEDs for 3 seconds with buttons

02 #

03 # create a dictionary of switches and LEDs

04 # pairs = {switch_pin : led_pin, ... }

05 pairs = { 2:13, 3:12 }

06

07 # Cycle and watch for a button press

08 # Note: LEDs are normally off

09 while True:

10 for switch in pairs:

11 if read(switch) == 0:

12 led_pin = pairs[switch]

13 talkto(led_pin)

14 onfor(3)
Figure 7: Block coding interface for Snek.

60 JULY 2024 ISSUE 284 [Link]

 Snek MAKERSPACE

The BIPES inte- because BIPES does not support Python

grated develop- dictionaries or the onfor(<seconds>)
ment environment statement.
(IDE) [5] offers a
palette of Snek Summary
blocks that can be Snek programming offers the new Python
dragged, dropped, enthusiast a simple approach for creating
and connected to- Arduino hardware projects. For older Ar-
gether into logic. duino modules, such as the Nano and
Figure 7 shows the Uno, Snek is the only Python interfacing
blocks for the pre- option available. I occasionally had Snek
vious project with versioning issues – for example, I wasn’t
onboard blinking able to use Mu with an Arduino Mega
LEDs. Block logic module running version 1.5 – but it
is automatically worked fine with BIPES. Q Q Q
generated into the
Figure 8: BIPES generates Snek code from block logic. equivalent Snek Info
code (Figure 8). [1] Snek: [Link]
Block Programming The BIPES Console tab is used to down- [2] Firmware uploader: [Link]
Interface load the generated Snek code and to br/snek-web-uploader/
The BIPES (block-based integrated plat- enter commands manually. [3] Mu code editor: [Link]
form for embedded systems) Project [4] Note that BIPES blocks are great for [4] BIPES Project: [Link]
offers a block programming environment simple tasks, but the interface doesn’t [5] BIPES IDE: [Link]
for MicroPython, micro:bits, and Snek support all the Snek and Python state-
projects. The Snek interface is tested ments. For example, the earlier project Author
with version 1.5, but other versions with the HY-M302 shield would need an For more of Pete’s projects, see:
might also work. alternative approach for the logic, [Link]
MAKERSPACE SIMD Code Optimization

MakerSpace
Code optimization with single
instruction, multiple data
Strong Arm Performance
Coding for the ARM NEON vector hardware can significantly
improve performance and help you get the most out of low-
power systems such as the Raspberry Pi. By D.R. Jordan

Y
Listing 1: findMax() Scalar Code ou have just coded that new Intrinsic functions that map to the un-
typedef struct{
algorithm, the one that han- derlying NEON instructions are available
int ind; dles all the complexities of in the GCC compiler [2], enabling as-
float val; your data to return exactly the sembly-style programming of NEON op-
} maxret_t; answers you need. As you launch the erations with the overall conveniences
program and wait, you realize that you of the C (or C++) language. The intrin-
maxret_t findMax(int N, float *xval) have a problem: You are still waiting. sic functions and NEON data types are
{ The code is slow, far too slow to be us- available with the
int n; able. You need faster processing, prefera-
float x, x2, x3; bly without upgrading your hardware. #include <arm_neon.h>

float val; Before you reach for the other cores on

maxret_t mret = {-1, -1.0e38}; your processor, it makes sense to see if directive in the C code. It can take
you can optimize the code for a single practice and a fair bit of imagination to
const float A = 0.052; core first. Any single-core optimizations master the data manipulation needed
const float B = 0.24; will reduce the number of cores ulti- for effective SIMD coding, but the per-
const float C = 3.3; mately needed. Taking advantage of any formance gains justify the effort.
const float D = 10.1; available single instruction, multiple data
(SIMD) hardware is an effective means of The Algorithm
for(n=0;n<N;n++) accelerating mathematically intensive The best way to learn SIMD vector cod-
{ problems. SIMD (vector) hardware uses ing is on real problems with simple but
x = xval[n]; parallel arithmetic units executing the non-trivial algorithms that demonstrate
x2 = x*x; same operation on multiple elements of a variety of SIMD techniques. The example
x3 = x2*x; data within the same clock cycle. algorithm
val = A*x3 + B*x2 + C*x + D; ARM’s implementation of SIMD,
called NEON, is relatively intuitive and for n in [0,N)
if(val > [Link])
effective. The NEON instructions operate y[n] = Ax^3[n] + Bx^2[n] + Cx[n] + D
{
on 128 bits (16 bytes) of data per clock,
Lead Image © vectortatu, [Link]

return(max(y),argmax(y))
[Link] = val;
either as sixteen 8-bit characters, eight
[Link] = n;
16-bit short integers, or even four 32-bit evaluates a third-order polynomial, but
} /* end if */
floating-point numbers. Modern compil- rather than return the resulting values,
} /* end for n */
ers, such as GCC, have the ability to vec- the maximum of the result and its index
torize code automatically during optimi- (argmax) are returned instead. The values
return(mret);
zation [1], but you often have room for of the input array x are randomly gener-
improvement if you are willing to use ated, so you cannot exploit any specific
} /* end findMax */
the NEON instructions directly. ordering.

62 JULY 2024 ISSUE 284 [Link]

 SIMD Code Optimization MAKERSPACE

Scalar Code whether it is larger than the current compiler to treat the memory as an array
The first step to writing the vector code maximum stored in the return structure. of vectors when using vxa, with each vec-
is to create a scalar (i.e., standard) C If so, the value and its index are stored tor consisting of four 32-bit floating-point
code implementation of the algorithm, in the return structure. values. The Nv variable is the number of
which provides a reference implemen- The initial maximum is set to a signifi- complete vectors in the N length array. If N
tation against which to validate the cantly large negative value so that it is is not a multiple of four, Nv will not encom-
eventual vector code. The x and y values overwritten in the loop. Should the loop pass the final few data points.
are represented as 32-bit floating point, still fail to find a new maximum, the caller The first loop in the function iterates
which is realistic and enables good can check to see whether the returned over the number of vectors (Nv), evaluat-
NEON improvement. index is set to -1, indicating this error. ing the polynomial with vector multiply
The findMax() function in Listing 1 (vmulq_f32) and vector multiply-accumu-
takes in the number of elements (N) and Vector Code late (vmlaq_f32) instructions (Listing 3).
a pointer to the input array (xval). The The NEON version of the code, findMax- The “f32” in the name indicates operation
return value is a structure that holds the Vec() in Listing 2, takes the same argu- on 32-bit floating-point data. The “q” in
maximum value and its index. A single ments as the scalar version. Inside the the name indicates operation on the full
loop evaluates the polynomial formula routine, the xval array pointer is recast to 128 bits of the NEON vector. NEON in-
for each entry of the xval array, storing it a float32x4_t pointer named vxa. This structions can operate on half vectors
to a temporary value. To implement the change does nothing to the underlying (non-q versions) but are not needed in
max and argmax functions in this same data; both xval and vxa point to the same this case. Here, all four of the 32-bit floats
loop, each output value is checked to see address, but the float32x4_t type tells the can be used for better acceleration.

Listing 2: findMaxVec() Vector Code

maxret_t findMaxVec(int N, float *xval) /* Use vector compare greater than

{ * and bit select to implement max and

int n; * argmax functions

int Nv = N/4; /*

vmask = vcgtq_f32(vtmp,vmax);

float32x4_t vx, vx2, vx3; vmax = vbslq_f32(vmask,vtmp,vmax);

float32x4_t vtmp; vmxind = vbslq_s32(vmask,vind,vmxind);

float32x4_t *vxa = (float32x4_t *) xval;

float vfload[4] __attribute__((aligned(16))); /* Update vector version of index */

int32_t viload[4] __attribute__((aligned(16))); vind = vaddq_s32(vind,vinc);

} /* end for n */

uint32x4_t vmask;

float32x4_t vmax; vst1q_f32((float32_t *) vfload,vmax);

int32x4_t vmxind; vst1q_s32(viload,vmxind);

int32x4_t vind = {0, 1, 2, 3};

int32x4_t vinc = {4, 4, 4, 4}; Nv *= 4; /* Convert Nv count to scalars */

if(Nv != N)

maxret_t mret = {-1, -1.0e-38}; {

mret = findMax(N-Nv,xval+Nv);

const float32x4_t vA = {0.052, 0.052, 0.052, 0.052}; [Link] += Nv;

const float32x4_t vB = {0.24, 0.24, 0.24, 0.24}; } /* end else */

const float32x4_t vC = {3.3, 3.3, 3.3, 3.3};

const float32x4_t vD = {10.1, 10.1, 10.1, 10.1}; for(n=0;n<4;n++)

vmax = vdupq_n_f32([Link]); if( (vfload[n] > [Link]) ||

vmxind = vdupq_n_s32([Link]); ((vfload[n] == [Link]) &&

for(n=0;n<Nv;n++) (viload[n] < [Link])) )

{ {

/* val = A*x3 + B*x2 + C*x + D; */ [Link] = vfload[n];

vx = vxa[n]; [Link] = viload[n];

vx2 = vmulq_f32(vx,vx); /* x2=x*x */ } /* end if */

vx3 = vmulq_f32(vx2,vx); /* x3=x2*x */ } /* end for n */

vtmp = vmlaq_f32(vD,vC,vx); /* tmp = C*x+D */

vtmp = vmlaq_f32(vtmp,vB,vx2); /* tmp += B*x2 */ return(mret);

vtmp = vmlaq_f32(vtmp,vA,vx3); /* tmp += A*x3 */

} /* end findMaxVec */

[Link] ISSUE 284 JULY 2024 63

MAKERSPACE SIMD Code Optimization

Listing 3: Polynomial Evaluation argument order is mask, maximum and index values. The global
/* val = A*x3 + B*x2 + C*x + D; */
true answer, false answer. maximum must be found among these
vx = vxa[n];
partial results to match the original
vx2 = vmulq_f32(vx,vx); /* x2=x*x */
Vector scalar function.
vx3 = vmulq_f32(vx2,vx); /* x3=x2*x */
Initialization To handle any leftover data, you can
To find the indices of the run the scalar function findMax() where
vtmp = vmlaq_f32(vD,vC,vx); /* tmp = C*x+D */
maximum, the vector loop the vector loop left off, namely at an
vtmp = vmlaq_f32(vtmp,vB,vx2); /* tmp += B*x2 */
must keep track of the offset of 4*Nv:
vtmp = vmlaq_f32(vtmp,vA,vx3); /* tmp += A*x3 */
scalar indices for the vec-
tor elements being oper- /* Convert Nv count to scalars */

The NEON unit has two multiply-accu- ated on. This is accomplished by creating Nv *= 4;

mulate instructions, the second one being a signed integer vector (int32x4_t vind) if(Nv != N)

the fused multiply-add (vfmaq_f32). The initially set to the sequence 0,1,2,3. Each {

vfmaq_f32 version rounds the floating- time through the vector loop, vind is in- mret = findMax(N-Nv,xval+Nv);

point result after the accumulate, cremented by a vector set to all fours [Link] += Nv;

whereas the vmlaq_f32 instruction used (4,4,4,4) because each vector element, or } /* end else */

here rounds after the multiply and again lane, is striding through the array by four
after the accumulate. As a result, the elements during the loop. The overhead of a function call is not
vector and scalar versions of the code The vind and vinc vectors are initial- ideal for a few samples, but it is worth
might have small rounding differences ized during the declaration statement the code simplification and is only called
when using either of these instructions. with bracket notation: once. The scalar function is called with
the proper offsets into xval (4*Nv) and
Conditionals int32x4_t vind = {0, 1, 2, 3}; size (N-4*Nv). The index value returned
To implement the max/argmax functional- int32x4_t vinc = {4, 4, 4, 4}; is relative to the offset data fed into it, so
ity, the polynomial values in vtmp must be ... 4*Nv is added to the returned index. No-
compared with the running maximums in /* Bottom of the for loop */ tice how Nv is multiplied by four first to
vmax. The vtmp and vmax vectors each con- /* Increment index values */ convert it from vector to scalar counts.
tain four values. A conditional could have vind = vaddq_s32(vind,vinc); The individual elements of the vectors
a different result for each element in the vmax and vmxind must be compared with
vector, which is not conducive to branch- This bracket notation for setting vectors the maximum in the return structure mret.
ing the code. The SIMD convention is not is only available in the declaration state- Listing 4 shows the relevant instructions
to branch, but to evaluate both branches ment, similar to the C language limita- from the findMaxVec() function. The vector
of a conditional and combine the results tions for initializing arrays and data is stored into regular C arrays to pro-
with a conditional mask. The mask is set structures. cess them in scalar code. Two arrays,
to all ones if the condition is true, and all When the vmax and vmxind vectors are vfload and viload, are declared as four-ele-
zeros if false. The true result is bitwise initialized to the values of the mret ment C arrays. The alignment of the arrays
ANDed with the mask, and the false re- structure, to 16-byte boundaries is not absolutely
sult is bitwise ANDed with the mask’s necessary for NEON. The vector store in-
complement. The two masked results are vmax = vdupq_n_f32([Link]); structions (vst1q_f32 and vst1q_s32) move
then added together for the desired out- vmxind = vdupq_n_U

put. This process is often called a select s32([Link]); Listing 4: Vector Cleanup
operation, and NEON provides the vector float vfload[4] __attribute__((aligned(16)));
bit select instruction (vbsl) to implement all the lanes are set to the int32_t viload[4] __attribute__((aligned(16)));
it. The bit select pseudocode is same value. The vector
...
duplicate instructions
vst1q_f32((float32_t *) vfload,vmax);
result = if(a==true)&result1 U (vdupq_n_f32 and vdupq_n_
vst1q_s32(viload,vmxind);
+ if(a==false)&result2 s32) generate vectors with
...
all of the lanes set to the
Listing 2 shows the conditional logic in input scalar value. This for(n=0;n<4;n++)

findMaxVec(). A vector greater-than in- initialization is placed {

struction (vcgtq_f32) generates the bit before the for loop. if( (vfload[n] > [Link]) ||

masks for vtmp > vmax. This bit mask vec- ((vfload[n] == [Link]) &&
tor (uint32x4_t vmask) is used to drive two Cleanup (viload[n] < [Link])) )
separate bit select instructions: one to up- If the input array length {
date vmax and one to update the maximum is not a multiple of four, [Link] = vfload[n];
index vmxind. The mask itself is of type the vector loop leaves up [Link] = viload[n];
uint32x4_t and operates with the bit select to three entries unpro-
} /* end if */
function for floating-point (vbslq_f32) and cessed. The vector loop
} /* end for n */
signed integer (vbslq_s32). The bit select itself results in four

64 JULY 2024 ISSUE 284 [Link]

 SIMD Code Optimization MAKERSPACE

data from the vector variables to the scalar allocated with Listing 5: Time Conversion Function
arrays. Note that the vector load instruction malloc() and then static inline double getTimeInSec(void)
(vld1q_f32) was not explicitly used in the filled with ran- {
first loop because the data was accessed by domly generated double dtime = -1.0;
a vector pointer to the allocated memory. floating-point data struct timeval tv;
The store into the vfload array could be (lines 109-113).
accomplished with pointer casting, but These random if(gettimeofday(&tv,NULL) == 0)
would not be as legible. numbers are {
The following lines are equivalent: scaled from 0 to dtime =
((double) tv.tv_sec) + ((double) tv.tv_usec)*1e-6;
10, with four frac-
} /* end if */
*((float32x4_t *) vfload) = vmax; tional digits.
vst1q_f32((float32_t *) vfload,vmax); The inline func-
return(dtime);
tion getTimeIn-
The final four maximums are run in a Sec() in Listing 5
} /* end getTimeInSec */
small loop at the end of the routine. is used to measure
During testing, one complication came runtimes. The
to light: The randomly generated xval function encapsulates the gettimeofday() but these can be considered overhead.
arrays are not diverse enough to avoid call to return time in seconds. Note that To optimize the code, this overhead
duplicate values, so multiple copies of gettimeofday measures real time, or wall should be reduced as much as feasible to
the maximum can be found from a sin- clock time, as opposed to the clock() concentrate the computer’s efforts on
gle input array. The scalar findMax() rou- function found in the C library, which the actual work of the algorithm.
tine returns the index of the first in- counts CPU cycles. If the operating sys- The operations count for the algo-
stance of the maximum in these cases. tem suspends the application, the time rithm does not have to be absolutely ac-
In the vector code, the first maximum suspended is not counted by the clock() curate, although it should be a reason-
is not guaranteed to occur in the first function and can result in erroneously able estimate. However, a consistent es-
vector element because of the strided na- good profile times. timate will be more useful than one that
ture of the computation (each lane of the The precision of gettimeofday() varies adapts to the peculiarities of a particu-
vector sees every fourth array value). with hardware but is generally good. lar processor. In the algorithm here, the
The final four comparisons must check The conversion to double precision in polynomial evaluation involves five
to see whether the values are equal and, seconds may even reduce the timer’s multiplies and three additions for each
if so, select the minimum of the indices precision, but not enough to affect the input value. The max and argmax func-
(Listing 4). With this in place, the vector tests performed here. tions are implemented by a single con-
code matches the scalar function results, The main routine runs both findMax() ditional if statement per input in the
except for some small rounding differ- and findMaxVec() in an iteration loop processing loop. The code estimates the
ences in the polynomial. (Listing 6), with calls to getTimeInSec() multiplications and additions as one
to time all iterations. The average run- operation each, and the conditional as
The Test Routine time is computed from this overall dura- four operations (compare, conditional
The main routine can be found in full at tion, smoothing out any fast or slow jump, operation, bad branch predic-
the end of the article (Listing 9). This pro- runs of the functions. Variability in the tion), which gives the basic equation
gram is set up to run and profile both the runtime of individual calls is difficult to for the operations count,
scalar findMax() and the NEON findMax- avoid in any complex operating system.
Vec() functions. The caller can specify the The number of iterations can be tuned to
number of elements to test on the com- get multiple seconds of runtime, at mini- Listing 6: Timing Loops
mand line, but if they do not, the code mum, to ensure a good average. time1 = getTimeInSec();
will use 1024*1024+1 elements in each call
Metrics
for(n=0;n<TIME_ITER;n++)
to the functions (line 46), which equates
{
to 4MB of memory in the input array Although you could use the total mea-
xval. This size should be larger than the sured times to report the average run- mret = findMax(N,xval);

L2 and L3 caches of the test machines so time per iteration, that information will } /* end for n */

they are forced to cycle data from DRAM only allow you to evaluate equivalent duration = getTimeInSec() - time1;
on each iteration, rather than using only versions of the same algorithm. To get a
cache memory. The plus one in the count more general sense of performance, you
time1 = getTimeInSec();
ensures the NEON code is executing the can also compute an estimate of the op-
cleanup call to the scalar code. erations performed per second by the for(n=0;n<TIME_ITER;n++)

Both functions use a 32-bit integer to code. This estimate should only include {

track the array index, so the main func- the operations in the algorithm itself. mret = findMaxVec(N,xval);
tion ensures the array is never large The computer is performing additional } /* end for n */
enough to roll over those indices (with logistical operations, such as moving
duration = getTimeInSec() - time1;
plenty of margin). The xval array is memory and controlling iteration loops,

[Link] ISSUE 284 JULY 2024 65

MAKERSPACE SIMD Code Optimization

where N is the number of elements in the limited or compute limited. For each iter- code will take some time to run the tests
array. Dividing this by the runtime gives ation, the scalar and vector C code ac- before displaying the scalar and vector
operations per second (Ops/s), which is cess the input array only once to com- results (Listing 7).
scaled to giga-operations per second pute the pair of output values (max,
(GOps/s, or 1x10 9 Ops/s): index). All other operations are carried Python Comparison Code
out on data stored in registers or lower For another point of reference, you can
level cache memory, which means you build the same algorithm in a high-level
The use of the processor’s clock rate to can estimate the memory access from language. In Python, the findMax() func-
compute the operations per clock (Ops/ the array size in bytes divided by the tion can be accomplished in just a few
clk) gives a sense of the code’s process- time per iteration: lines with two numpy functions (Listing 8).
ing efficiency: The test routine functionality also has
been duplicated, with iterations timed by
time.perf_counter(). Note that it ex-
Because I did not include any overhead pressly declares the input array as 32-bit
in the operations estimate, you would ex- Compile the Code floating point for the best comparison to
pect scalar code to execute not much more The neontut.c code in Listing 9 can be the C code.
than one operation per clock and the compiled with gcc on ARM computers One consequence of the Python imple-
NEON code no more than four operations with NEON capability. Check your CPU’s mentation is additional memory access.
per clock. As you will see, this rule of documentation or look at the feature The return from [Link]() must write
thumb is oversimplified but does provide flags in the /proc/cpuinfo file to see if a second array, which is then read by
an intuitive limit for initial assessment. your ARM has NEON support (check for [Link](). This results in at least three
In addition to the computational rate, neon or asimd). accesses of a memory array equal to the
the memory access rate of the code This tutorial is simple enough that you input size. The estimate for memory ac-
should be estimated, as well. Comparing can call gcc directly. The -O3 flag selects cess in the Python implementation is:
the code’s access rate to the computer’s relatively good, but safe, compiler opti-
memory bandwidth will help identify mizations. The -Wall flag enables extra
whether an implementation is memory compiler warnings. If you have a 32-bit
ARM processor,
Listing 7: Compile and Run you might need an Results
$ gcc -O3 -Wall -o neontut neontut.c
additional flag to Table 1 summarizes the statistics of the
$ ./neontut
enable NEON in- hardware used in the test, with the code
structions run on three different ARM platforms
Scalar: index = 2557, max = 119.098816,
duration = 4.190763 msec
(-mfpu=neon). After (Raspberry Pi 4B [3] and 5 [4] and
compilation, you NVIDIA Jetson Nano [5]). All tests were
rate = 3.002538 GOps/s, memory = 1000.845954 MB/s
can run the code run on the systems with no other active
Neon: index = 2557, max = 119.098824,
by calling neontut user processes.
duration = 2.392712 msec
from the com- Some manufacturers will list the mem-
rate = 5.258854 GOps/s, memory = 1752.951308 MB/s
mand line. The ory bandwidth for their processors, but

Listing 8: [Link]
01 import numpy as np 18 tmp = [Link]([Link](N)*20e4)*5.0e-5

02 import time 19 xval = [Link](np.float32)

03 20

04 def findMax(xval): 21 time1 = time.perf_counter()

05 P = [0.052, 0.24, 3.3, 10.1] 22 for n in range(TIME_ITER):

06 y = [Link](P,xval) 23 (mxval,mxind) = findMax(xval)
07 mxind = [Link](y) 24 #end for n
08 25 duration = time.perf_counter() - time1
09 return (y[mxind],mxind) 26 rate = TIME_ITER*N*12e-9/duration
10
27 membw = TIME_ITER*N*12e-6/duration
11 #end findMax
28
12
29 print("Python: index = %d, max = %f, duration = %f
13 if __name__ == "__main__": msec" % (mxind,mxval,1.0e3*duration/TIME_ITER))
14 TIME_ITER = 1000 30 print(" rate = %f GOps/s, memory = %f MB/s"
15 N = 1024*1024+1 %(rate,membw))

16 31

17 # Generate matching random sequence 32 #end if main

66 JULY 2024 ISSUE 284 [Link]

 SIMD Code Optimization MAKERSPACE

Table 1: Test Hardware These discrepancies pays dividends in significantly reduced

Computer Processor Clock (GHz) Cache (MB) in the model would runtimes for the lifetime of the code.
RPi4B Cortex A72 1.8 1 not normally be so ob- Your hardware also gets a new lease
vious except that this on life. As you can see from the tables,
RPi5 Cortex A76 2.4 2
code has very little the RPi4 NEON code is only 16 percent
Nano Cortex A57 1.43 2
overhead. The proces- slower than the scalar code on the RPi5,
sor needs to shuffle which would allow the NEON code to
you can also estimate this value by timing around few variables for the algorithm to compete on hardware a full generation
memory access operations. I do not ad- work, and most of the operations are the older than modern alternatives that use
dress this topic in detail here because the core math. Code, in general, exhibits scalar code.
reported memory rates are well below the much more overhead. As such, you The tests for the Jetson Nano provide
memory bandwidth of each system, which should not use the results from this algo- some additional insight. The scalar C
indicates that all tests are compute-bound rithm as a general benchmark, but rather code is two to three times less efficient
rather than memory-bound, and I can in- to measure the improvement from the than on the RPi4 or 5, which could be a
stead focus on the computational trends. NEON instructions. result of a different compiler and opti-
Tables 2, 3, and 4 summarize the test re- The results clearly show that the extra mizer version, or it could be indicative
sults for all three computers on the basis of work involved in coding for NEON was of improvements in the instruction per
time per iteration. Looking at the scalar C not in vain. Even on the Raspberry Pi clock (IPC) of the newer ARM architec-
code first, you can see it has unusually computers, where the scalar code opti- tures (the Nano is an older processor).
high operations per clock ratios, even ex- mized extremely well, NEON acceleration The NEON code on the Jetson pro-
ceeding the notional one operation per is 1.8 to 2.3 times, which is far from the vides more than three times acceleration
clock limit on the RPi4 and RPi5. So high ideal four times speedup one could ex- over the scalar code. Unlike the scalar
were these estimates that it warranted pect, but as you saw, the NEON code has code, the operations per clock figure of
double-checking that the code was pro- more overhead than the scalar code. The the NEON code on the Jetson is compa-
cessing the entire array. Remember, how- NEON code has a mix of scalar and vec- rable to that of the RPi4. When coded
ever, that the computational model was tor variables, with conversions in be- with NEON intrinsics, the compiler gets
not entirely accurate, so you are likely see- tween. It must also keep track of the vec- more explicit directions on which in-
ing some consequences of that. Unlike tor and scalar indices separately. Lastly, it structions to use, reducing the amount of
some processors, the ARM can use multi- contained the additional cleanup section freedom the optimizer has to function.
ply-accumulate instructions even in scalar to match the scalar functionality exactly. Although you do introduce an element
code, so it is getting a boost compared These additional costs reduce the of risk, you see it clearly benefits the
with the model. The conditional in the max gains made by the vector processing, but code when done correctly.
operator is also overcounted by the model. you still end up with substantial im- In contrast with the Nano and RPi4,
The vast majority of the conditional opera- provements, with roughly two times the the operations per clock dramatically im-
tions will be comparing small values with performance from the NEON code on the prove with the NEON code on the RPi5,
a larger one because it isolates the maxi- same hardware. Although it costs extra which might be an indication of im-
mum, meaning operations inside the if time and attention in the development of provements made to the NEON units in
statement are not executed very often. the code, it is a one-time expense that the RPi5’s processing cores.
In all cases, the Python code was
Table 2: Raspberry Pi 4B Test Results many times slower than even the scalar
Code Time (ms) Rate (GOps/s) Ops/clock Mem(MB/s) code. Only on the Jetson, where the
Scalar-C 4.184 3.007 1.671 1002.5
scalar C code was much less efficient,
was the Python code even close, being
NEON-C 2.372 5.305 2.947 1768.4
2.5 times slower. The low overhead
Python 40.656 0.309 0.172 309.5
achieved in the C code is less possible
with a higher level language that uses
Table 3: Raspberry Pi 5 Test Results more generic and complex data struc-
Code Time (ms) Rate (GOps/s) Ops/clk Mem (MB/s) tures. If you assume the two numpy rou-
Scalar-C 1.991 6.319 2.633 2106.4 tines are compiled functions under the
hood, they still have additional over-
NEON-C 0.861 14.613 6.089 4870.9
head traversing in and out of the Py-
Python 14.092 0.893 0.372 892.9
thon environment. The Python code
must also manage an intermediate array
Table 4: Jetson Nano Test Results and cannot conveniently combine the
Code Time (ms) Rate (GOps/s) Ops/clk Mem (MB/s) separate polynomial and argmax loops.
Considering all these factors, you can
Scalar-C 10.631 1.184 0.828 394.6
start to see why the Python code falls be-
NEON-C 3.162 3.980 2.783 1326.6
hind the scalar C code in this instance. The
Python 27.360 0.460 0.322 459.9
NEON code looks truly impressive by

[Link] ISSUE 284 JULY 2024 67

MAKERSPACE SIMD Code Optimization

Table 5: NEON Commands in neontut.c

Function Type Prototype Operation
Vector add integer int32x4_t vaddq_s32(int32x4_t vA,int32x4_t vB) Zi = Ai + Bi for i=0-3
Vector multiply float float32x4_t vmulq_f32 Zi = Ai x Bi for i =0-3
(float32x4_t vA,float32x4_t vB)
Vector multiply float float32x4_t vmlaq_f32 Zi = Ai x Bi + Ci for i =0-3
accumulate (float32x4_t vC,float32x4_t vA,float32x4_t vB)
Vector compare float uint32x4_t vcgtq_f32 if(Ai > Bi) Zi = mask('1') else Zi
greater than (float32x4_t vA,float32x4_t vB) = mask('0') for i =0-3
Vector bit select integer int32x4_t vbslq_s32 Zi = Ai&Mi + Bi&(~Mi) for i =0-3
(uint32x4_t vM,int32x4_t vA,int32x4_t vB)
Vector bit select float float32x4_t vbslq_f32 Zi = Ai&Mi + Bi&(~Mi) for i =0-3
(uint32x4_t vM,float32x4_t vA,float32x4_t vB)
Vector store integer void vst1q_s32(int32_t *A,int32x4_t vB) A[i] = Bi for i =0-3
Vector store float void vst1q_f32(float32_t *A,float32x4_t vB) A[i] = Bi for i =0-3
vector duplicate int int32x4_t vdupq_n_s32(int32_t A) Zi = A for i=0-3
vector duplicate float float32x4_t vdupq_n_f32(float32_t A) Zi = A for i=0-3

comparison, but you must be careful not to processors. Table 5 summarizes all of some general descriptions but is not ex-
draw too many conclusions from this part the NEON instructions used in the find- tensive. Additionally, you can mimic Intel
of the tests. This algorithm is too special- MaxVec() function, but so much more SSE and AltiVec code examples by con-
ized. Comparisons of C and Python perfor- capability is available. verting to NEOwN-equivalent instruc-
mance with the use of more complex algo- A number of resources are available for tions. These SIMD implementations (for
rithms will have dramatically different NEON instructions. ARM’s list of NEON x86 and PowerPC, respectively) have a lot
results, dependent on the algorithms. intrinsics [2] is complete but can lack de- in common with ARM’s implementation,
tail on the instructions’ operations. ARM’s so coding techniques are highly transfer-
Conclusion NEON introduction to developers [6] has able (with exceptions). Q Q Q
In this article, I touched on the usage of
ARM’s SIMD capabilities, demonstrated Info
how to integrate NEON instructions into [1] Automatic vectorization, ARM Developer, version 2.1,
C code, and measured the potential [Link]
improvements it can bring to computa- Developing-for-NEON/Automatic-vectorization
tional performance. In all cases, the [2] Intrinsics, ARM Developer:
NEON code produced truly impressive [Link]
processing rates on very low power ARM [3] Raspberry Pi 4 specs:
[Link]
[4] Raspberry Pi 5 specs:
Author
[Link]
D.R. Jordan works in the field of scientific
[5] NVIDIA Jetson Nano specs:
and embedded computing and enjoys [Link]
studying the implications of hardware [6] ARM. Introducing NEON: Development Article. ARM Limited publication DHT 0002A
design on software structure and ID060909, 2009:
performance. [Link]

Listing 9: neontut.c
001 /* neontut.c - A tutorial demonstrating the use 016
002 * of NEON intrinsic functions from the C language. 017 typedef struct{

003 */ 018 int ind;

004 019 float val;

005 #include <stdio.h> 020 } maxret_t;

006 #include <stdlib.h> 021

022 /* Function to return clock time in seconds */
007 #include <stdint.h>
023 static inline double getTimeInSec(void)
008 #include <memory.h>
024 {
009 #include <time.h>
025 double dtime = -1.0;
010 #include <sys/time.h>
026 struct timeval tv;
011
027
012 #include <arm_neon.h>
028 if(gettimeofday(&tv,NULL) == 0)
013
029 {
014 /* Define timing iteration count */
030 dtime = ((double) tv.tv_sec) + ((double) tv.tv_
015 #define TIME_ITER 5000 usec)*1e-6;

68 JULY 2024 ISSUE 284 [Link]

 SIMD Code Optimization MAKERSPACE

Listing 9: neontut.c (continued)

031 } /* end if */ 092 *
032 093 * Rederiving N from msize is somewhat overcautious
033 return(dtime); 094 * to protect against unexpected size_t conversion.
034 095 */
035 } /* end getTimeInSec */ 096 msize = ((size_t) N)*sizeof(float);
036 097 N = (int) msize/sizeof(float);
037 /* Prototypes to the test functions, 098 xval = (float *) malloc(msize);
038 * code located after main routine 099 if(xval == NULL)
039 */ 100 {

040 maxret_t findMax(int N, float *xval); 101 fprintf(stderr,"Memory allocation error: %s:%d\n",__
FILE__,__LINE__);
041 maxret_t findMaxVec(int N, float *xval);
102 return(-1);
042
103 } /* end if */
043 int main(int argc, char *argv[])
104
044 {
105 /* Load xval with random floating-point
045 int n;
106 * data between 0.00005 and 10
046 int N = 1024*1024 + 1;
107 */
047 size_t msize;
108 srand((unsigned int) time((time_t *) NULL));
048 maxret_t mret = {-1,0.0};
109 for(n=0;n<N;n++)
049 float *xval = NULL;
110 {
050 double time1, duration;
111 xval[n] = ((float) (rand()%200000))*5.0e-5;
051 double rate, membw;
112 } /* end for n */
052
113
053 /* Nlimit*sizeof(float) = 1GB */
114 /* Run the scalar and vector functions
054 const int Nlimit = 256*1024*1024;
115 * multiple times to get good timings
055
116 */
056 /* Parse command line argruments for
117
057 * simple experimentation
118 time1 = getTimeInSec();
058 */
119 for(n=0;n<TIME_ITER;n++)
059 for(n=1;n<argc;n++)
120 {
060 {
121 mret = findMax(N,xval);
061 if(strcmp(argv[n],"-h") == 0)
122 } /* end for n */
062 {
123 duration = getTimeInSec() - time1;
063 printf("neontut [-h] [-n #samples]\n");
124 rate = ((double) TIME_ITER)*((double) N)*12.0e-9;
064 return(0);
125 rate /= duration;
065 } /* end if */
126 membw = ((double) TIME_ITER)*((double) N)*4.0e-6;
066 else if(strcmp(argv[n],"-n") == 0)
127 membw /= duration;
067 {
128
068 if(++n >= argc)
129 printf("Scalar: index = %d, max = %f, duration = %f
069 { msec\n", [Link],[Link], 1e3*duration/
070 printf("-n option requires integer ((double) TIME_ITER));
argument\n"); 130 printf(" rate = %f GOps/s, memory = %f MB/s\n",
071 return(-1); rate, membw);

072 } /* end if */ 131

073 else 132 time1 = getTimeInSec();

074 { 133 for(n=0;n<TIME_ITER;n++)

134 {
075 N = atoi(argv[n]);
135 mret = findMaxVec(N,xval);
076 } /* end else n argument */
136 } /* end for n */
077 } /* end else if -n */
137 duration = getTimeInSec() - time1;
078 else
138 rate = ((double) TIME_ITER)*((double) N)*12.0e-9;
079 {
139 rate /= duration;
080 printf("Unknown argument [%s] ignoring\
n",argv[n]); 140 membw = ((double) TIME_ITER)*((double) N)*4.0e-6;

081 } /* end else */ 141 membw /= duration;

082 } /* end for n */ 142

083 143 printf("Neon: index = %d, max = %f, duration = %f

msec\n", [Link],[Link], 1e3*duration/
084 /* Check value of N */
((double) TIME_ITER));
085 if(N < 0) N = 0;
144 printf(" rate = %f GOps/s, memory = %f MB/s\n",
086 if(N > Nlimit) N = Nlimit; rate, membw);
087 145
088 /* Allocate the X value array. 146 if(xval != NULL) free((void *) xval);
089 * Unlike some SIMD, NEON appears to be 147 return(0);
090 * compatible with 4-byte alignment so we 148
091 * can use the standard malloc call. 149 } /* end main */

[Link] ISSUE 284 JULY 2024 69

MAKERSPACE SIMD Code Optimization

Listing 9: neontut.c (continued)

150 213 vx2 = vmulq_f32(vx,vx); /* x2=x*x */
151 /* Scalar version of the test function */ 214 vx3 = vmulq_f32(vx2,vx); /* x3=x2*x */
152 maxret_t findMax(int N, float *xval)
215 vtmp = vmlaq_f32(vD,vC,vx); /* tmp = C*x+D */
153 {
216 vtmp = vmlaq_f32(vtmp,vB,vx2); /* tmp += B*x2 */
154 int n;
155 float x, x2, x3; 217 vtmp = vmlaq_f32(vtmp,vA,vx3); /* tmp += A*x3 */

156 float val; 218

157 maxret_t mret = {-1, -1.0e38}; 219 /* Use vector compare greater than
158
220 * and bit select to implement max and
159 const float A = 0.052;
221 * argmax functions
160 const float B = 0.24;
161 const float C = 3.3; 222 */

162 const float D = 10.1; 223 vmask = vcgtq_f32(vtmp,vmax);

163 224 vmax = vbslq_f32(vmask,vtmp,vmax);
164 for(n=0;n<N;n++)
225 vmxind = vbslq_s32(vmask,vind,vmxind);
165 {
226
166 x = xval[n];
227 /* Update vector version of index */
167 x2 = x*x;
168 x3 = x2*x; 228 vind = vaddq_s32(vind,vinc);

169 val = A*x3 + B*x2 + C*x + D; 229 } /* end for n */

170 230
171 if(val > [Link])
231 /* Store the vector results into standard
172 {
232 * arrays to finish up using scalar logic
173 [Link] = val;
174 [Link] = n; 233 */

175 } /* end if */ 234 vst1q_f32((float32_t *) vfload,vmax);

176 } /* end for n */ 235 vst1q_s32(viload,vmxind);
177
236
178 return(mret);
237 Nv *= 4; /* Convert Nv count to scalar */
179
180 } /* end findMax */ 238 if(Nv != N)

181 239 {
182 /* Neon version of the test function */ 240 /* Use the scalar function to process
183 maxret_t findMaxVec(int N, float *xval)
241 * any data at the end of the array
184 {
242 * not covered by the vector loop
185 int n;
186 int Nv = N/4; 243 */

187 244 mret = findMax(N-Nv,xval+Nv);

188 float32x4_t vx, vx2, vx3; 245 [Link] += Nv;
189 float32x4_t vtmp;
246 } /* end else */
190 float32x4_t *vxa = (float32x4_t *) xval;
247
191 float vfload[4] __attribute__((aligned(16)));
192 int32_t viload[4] __attribute__((aligned(16))); 248 /* Finish the last 4 max operations

193 249 * from the vector loop. Take the

194 uint32x4_t vmask; 250 * first of any duplicate results.
195 float32x4_t vmax;
251 */
196 int32x4_t vmxind;
252 for(n=0;n<4;n++)
197 int32x4_t vind = {0, 1, 2, 3};
253 {
198 int32x4_t vinc = {4, 4, 4, 4};
199 254 if( (vfload[n] > [Link]) ||

200 maxret_t mret = {-1, -1.0e-38}; 255 ((vfload[n] == [Link]) &&

201 256 (viload[n] < [Link])) )
202 const float32x4_t vA = {0.052, 0.052, 0.052, 0.052};
257 {
203 const float32x4_t vB = {0.24, 0.24, 0.24, 0.24};
258 [Link] = vfload[n];
204 const float32x4_t vC = {3.3, 3.3, 3.3, 3.3};
205 const float32x4_t vD = {10.1, 10.1, 10.1, 10.1}; 259 [Link] = viload[n];

206 260 } /* end if */

207 vmax = vdupq_n_f32([Link]); 261 } /* end for n */
208 vmxind = vdupq_n_s32([Link]);
262
209 for(n=0;n<Nv;n++)
263 return(mret);
210 {
211 /* val = A*x3 + B*x2 + C*x + D; */ 264

212 vx = vxa[n]; 265 } /* end findMaxVec */

70 JULY 2024 ISSUE 284 [Link]

 INTRODUCTION LINUX VOICE

Some of us old-schoolers still have nightmares over the

ridiculous excesses of Microsoft’s Fear Uncertainty and
Doubt (FUD) era, in which they sought to portray Linux as
a scary virus that infects everything it touches. Through
court proceedings, corporate disinformation, and PR
shenanigans, the rascals of Redmond really tried to make
Linux disappear. But the truth is, they lost and Linux won.
Readers of this magazine don’t have to be told that, of Doghouse – Entrepreneurs 72
course, although sweet victories are always worth remem- Jon “maddog” Hall
bering. It is all so different now, with Microsoft contributing Advances in technology have opened up
to the Linux kernel and even serving possibilities for potential entrepreneurs,
but running a small business still means
as a Platinum member of the
doing many jobs.
Linux Foundation. You can even
set up your own Ubuntu Linux Color on the Terminal 73
Frank Hofmann
instance in Microsoft’s Azure You don’t necessarily need color on the
cloud, and in this month’s tu- terminal, but still, it does look good – and
torial, we show you how to does not involve too much effort.
do it. Also in Linux Voice, Web to Ebook 78
we pretty up the terminal Marco Fioretti
Image © Olexandr Moroz, [Link]

window and introduce Saving web pages to ebooks conserves

space and leads to easier reading.
you to some tools for
transforming web pages FOSSPicks 84
Nate Drake
into ebooks.
Our new columnist Nate Drake looks at
Audacity, Endless Sky, GCompris, Switcheroo,
MS-DOS, Qemu, and more!
Tutorial – Ubuntu VM in the Azure Cloud 90
Marcin Gastol
Are you ready to get started with the cloud?
Microsoft’s Azure Cloud Services provides
easy access to an Ubuntu virtual machine.

[Link] ISSUE 284 JULY 2024 71

LINUX VOICE DOGHOUSE – ENTREPRENEURS

MADDOG’S
Jon “maddog” Hall is an author,
educator, computer scientist,
and free software pioneer
who has been a passionate
DOGHOUSE
advocate for Linux since 1994
when he first met Linus Torvalds Advances in technology have opened up possibilities for potential
and facilitated the port of
Linux to a 64-bit system. He entrepreneurs, but running a small business still means doing
serves as president of Linux
International®. many jobs. BY JON “MADDOG” HALL
When I was in university, people bought stereo systems in
Starting a company today pieces. You bought a turntable, a tuner, an equalizer, a pream-
plifier, a power amplifier, and some speakers. They were all
was listening to a podcast today about the evils of working for made to (more or less) work with each other to produce the

I a company where you receive a portion of the profits versus

working for yourself or with a few people with whom you share
the profits of your labor. In the end, the conversation came down
music you wanted.
Later, some of these functions were combined into a stereo re-
ceiver, which combined the tuner, preamplifier, and power amplifier
to a comparison of capitalism vs. Marxism. along with inputs for the turntable and outputs for the speakers.
No matter how you feel about the two philosophies of eco- Often these components came from the same manufacturer,
nomics, there are good things and bad things about being your described as matched components. The manufacturer said
own boss. they were all designed to work together, so “naturally” you
The computer industry has probably supplied more opportu- bought them all at the same time from the same vendor.
nity for “lone wolf” types of companies where industry consul- Of course this tended to devolve into the all-in-one system for
tants can sell their knowledge than any other industry I know. I a lot of people.
have known many people who left the comfort and security Some people do buy high-end audio equipment, where am-
(such as it is) of a large company to pursue the greater freedom plifiers alone cost tens of thousands of dollars. But most
of the small business. people do not have that amount of money and are satisfied
Most of the time these businesses have just done software, with buying a system that provides a solution for listening to
either writing a software product and selling it or maintaining an music, news, etc.
existing set of applications for the end users. Sometimes the However, using free software and open hardware you can build a
small business was an “added value reseller” who resold a com- solution that uses a single board computer (SBC) to be the “tuner”
pany’s products. which uses digital mixing to bring in streaming music over the In-
When you work for a large company, there are usually products ternet (replacing the turntable, tuner, and other parts) and that
and services that other people market for you, sell, write the plugs into a digital amplifier or drives Bluetooth-powered speakers.
contracts for, and so forth. You perform the actual delivery of A person can find this hardware on the Internet, put Kodi (or
the services and do not have to worry about the rest. some other free software) on it, and make a multimedia system as
When you are a small company, you often have to wear all those good or better than what you can buy in “commercial” systems.
hats and more. You may not have a lot of resources (also known The SBC can be mounted on the back of an LCD panel which
as “money”) to start your company and keep it going until your can be used for video displays, and a digital phone (either iOS or
revenues outpace your expenses and you become “profitable.” Android) can be used to control the system.
Many times the small companies go out of business, but some- Once you have built the system, you can create a demonstra-
times they are very satisfying and profitable. And open software tion video, use a free software video editor to make a low, flashy
and open hardware can be used along with social networking video to demonstrate it, then use these in your social media to
and free media (to provide images and sounds) to be used for let people know what your “product” does.
your product and advertising. On your same social media site, you can have your contact
Previous columns have discussed how “in the old days” capable information and the price for both sales and installation.
computers were measured in kilobytes of main memory and mil- You should be aware, however, that even with good social media
lions of US dollars – compared to gigabytes of memory and a pow- the product and services do not sell themselves. Reach out to
erful system for less than the cost of a couple of hamburgers today. local user groups and try word-of-mouth to friends and family (and
Today a “development environment” could be a recycled laptop friends and family of your friends and family). Look for larger sales
with free software on it. opportunities like apartment houses and condominiums.
Finally, and most important, most people do not purchase hard- Or build a product and service useful to small business. You can
ware or software today; they buy solutions. be your own boss. Q Q Q

72 JULY 2024 ISSUE 284 [Link]

 COLOR ON THE TERMINAL LINUX VOICE

Use color for terminal output

Colorful
You don’t necessarily need color on the terminal, but still, it does
look good – and does not involve too much effort.
BY FRANK HOFMANN

f you look at the output of the common Linux associated with properties that can be found in the

I commands, there is always one impressive

feature: simplicity. Nothing flashes, wobbles,
or makes a noise – plain-vanilla information with-
/lib/terminfo directory.
You can use the echo command to determine
the value of the variable TERM, as shown in line 1
out any distracting trappings. In today’s world, of Listing 1. You can see from the output that
where every device or open tab in the web the call in the example was in a terminal emula-
browser begs for attention with a beep, this ap- tion that emulates the properties of an xterm
proach, which may seem anachronistic to new- with 256 colors. The infocmp command from the
comers, supports a focused work approach and ncurses-bin [1] package provides the current
lets you fully concentrate on solving the problem settings in the terminal. You need to pass in the Table 1: Color Values
at hand. Simply fantastic. content of TERM (line 2) as a parameter to see 0 Black
output revealing all of the terminal’s settings in
1 Red
Why Color? the form of cryptic-looking abbreviations and
If everything is so perfect, why bother with color? values. 2 Green
Color helps to highlight things and draw your at- Examples of the individual values for colors 3 Yellow
tention to a particular point or fact. Let’s take a and text effects can be found in a post by If- 4 Blue
look at how we can spice up the output on the enna on [Link] [2]. If you want to read about the
5 Magenta
command line with some color. Note that how the subject in detail, take a look at the Bash Prompt
output is actually displayed depends on the termi- HOWTO [3]. 6 Cyan
nal, its size, the fonts used, and other settings. As nice as the color values are, there is massive 7 White
After all, you still want output to remain legible. potential for errors due to transposed letters 8 Unused
Without color, neither the developer nor the pro- and numbers in the color values. The tput [4]
9 Reset to
gram have to worry about the properties of the command shows that this can also be done
default color
terminal currently in use. Nobody needs to know more simply. Instead of the cryptic color values,
how the user’s terminal is set up and what output
the program uses (e.g., whether it is a simple dis- Listing 1: Terminal Info (Excerpt)
play on the terminal, a redirect to a file, or further 01 $ echo $TERM
processing via a pipe). Remember: Color in- 02 xterm-256color
creases complexity. 03 $ infocmp xterm-256color

04 # Reconstructed via infocmp from file: /lib/terminfo/x/

On the Terminal xterm-256color
In the shell, ANSI escape sequences control the 05 xterm-256color|xterm with 256 colors,
cursor in the terminal. They can be used not only 06 am, bce, ccc, km, mc5i, mir, msgr, npc, xenl,
to set colors, but also to make text corrections 07 colors#0x100, cols#80, it#8, lines#24, pairs#0x10000,
and control the cursor. In this article, the focus is 08 acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
on color output, with changes to the foreground 09 bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l,
and background colors, plus additional highlight- 10 clear=\E[H\E[2J, cnorm=\E[?12l\E[?25h, cr=\r,
ing such as bold and underlining.
11 csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
First of all, it is important to clarify what your cur-
12 cud=\E[%p1%dB, cud1=\n, cuf=\E[%p1%dC, cuf1=\E[C,
rent terminal can do. A good, but not perfect indica-
13 cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
tor of this is the environment variable TERM. It con-
14 [...]
tains not the name of the shell or the terminal emu-
15 $
lation, but the terminal category. This category is

[Link] ISSUE 284 JULY 2024 73

LINUX VOICE COLOR ON THE TERMINAL

tput accepts numbers outputs the text. Last but not least, tput sgr0
Table 2: Text Effects (Table 1) or text effects resets all the attributes to their original states.
Abbreviation Meaning (Table 2), which it then What you need for orientation purposes is a
translates into the cor- GSPSVWTIGXVYQ0MWXMRKɄ?AGVIEXIWXLMWYWMRKE
bold Bold (start) responding ANSI con- shell script with two nested for loops. Figure 2
smul Underline (start) trol codes. shows what this looks like on the terminal.
rmul Figure 1 shows text If you do not want to generate this output your-
Underline (end)
output via tput in ma- self, but prefer to use something that already exists,
rev Inver (start) genta and bold. First, you can use the colortest [5] tool. It paints color
blink Flashing text (start) the call to tput bold acti- spectra of 8, 16, or 256 colors in an impressive way
invis Invisible text (start) vates bold, and then the (Figure 3).
command
smso Standout mode (start)
With Python
rmso Standout mode (end) tput setaf 5 Things that work in Bash also work in a similar
sgr0 Disable all attributes way with programming languages such as Py-
setaf VALUE switches to a colored thon. In the first step, you can again use ANSI
Set foreground color
text display. The echo escape sequences (see the blog post by Li
setab VALUE Set background color
command that follows Haoyi [6]). The program code required for this is
quite simple [7]; Listing 3 shows an adapted
Listing 2: Testing Colors version.
01 #!/bin/bash
Lines 2 to 11 first define a dictionary named
basicColorSet, which uses the names of the col-
02 for fg_col in {0..7}; do # values 0 to 7 ...
ors as keys. Each key has a matching color
03 set_foregrnd=$(tput setaf $fg_col) # ... as foreground color
value in the form of a character string. The code
04 for bg_col in {0..7}; do # values 0 to 7 ...
for resetting the color is defined in line 14. A for
05 set_bkgrnd=$(tput setab $bg_col) # ... as background color loop in lines 17 and 18 runs through the diction-
06 echo -n $set_bkgrnd$set_foregrnd # enable color combo ary and outputs the name of the color in the cor-
07 printf ' F:%s B:%s ' $fg_col $bg_col # output text responding color. Line 21 finally resets all color
08 done
settings using the previously defined reset
code. Figure 4 shows the output after calling
09 echo $(tput sgr0) # reset colors
the Python script.
10 done
The same applies to Listing 3 as to Listing 2: If
the ANSI escape sequences are not correct, in the
best case the output will be the wrong color, but if

Listing 3: ANSI Escape Sequences

01 # Define colors

02 basicColorSet = {

03 "black": "\u001b[30m",

04 "red": "\u001b[31m",

05 "green": "\u001b[32m",

06 "yellow": "\u001b[33m",

07 "blue": "\u001b[34m",

08 "magenta": "\u001b[35m",

Figure 1: tput in action. 09 "cyan": "\u001b[36m",

10 "white": "\u001b[37m"

11 }

12

13 # Code for resetting color properties

14 resetCode = "\u001b[0m"

15

16 # Output text in respective color

17 for item in basicColorSet:

18 print ("%s %s" % (basicColorSet[item],item))

19

20 # Reset color settings

21 print (resetCode)
Figure 2: Generating a color spectrum.

74 JULY 2024 ISSUE 284 [Link]

 COLOR ON THE TERMINAL LINUX VOICE

Figure 3: You can generate a spectrum with 256 colors with colortest.

you’re unlucky it will be total nonsense. To help

you avoid this as an error source, Python comes
with two useful libraries: Colorama [8] and
termcolor [9]. Both are available for supplemen-
tary installation via the Python Package Index
(PyPI). I found Colorama to be more intuitive, so
that’s why I’m focusing on it here.
Listing 4 implements the same behavior as List-
ings 2 and 3, but using the Colorama library. Very
little has changed: Line 1 imports two classes,
Fore (foreground) and Style (text properties), from
the Colorama library, while lines 4 to 13 define the
Figure 4: The result of calling the code from Listing 3. color codes in a reader-friendly way using the
constants defined in Fore. The output of the pro-
Listing 4: Colorama gram is identical to that of Listing 3 (Figure 4).
01 from colorama import Fore, Style
For background colors, Colorama supports the
02
Back class with similar predefined color values. It
can be used in exactly the same way as the Fore
03 # Define colors
class. I will cover highlighting and decorating the
04 basicColorSet = {
text later. Figure 5: Like colortest,
05 "black": [Link],
Before that, however, I’ll look at a practical tool: colortest-pythonprovides
06 "red": [Link],
colortest-python from the Debian package of the a handy color spectrum.
07 "green": [Link],

08 "yellow": [Link],

09 "blue": [Link],

10 "magenta": [Link],

11 "cyan": [Link],

12 "white": [Link]

13 }

14

15 # Define reset code

16 resetCode = Style.RESET_ALL

17

18 # Output text in respective color

19 for item in basicColorSet:

20 print ("%s %s" % (basicColorSet[item],item))

21

22 # Reset color settings

23 print (resetCode)

[Link] ISSUE 284 JULY 2024 75

LINUX VOICE COLOR ON THE TERMINAL

Table 3: Decorating Text (Excerpt)

Description ANSI Escape Sequence For Python Constant in Colorama
Bold \E[1m \u001b[1m [Link]

Invert \E[7m \u001b[7m Does not exist

Dim \E[2m \u001b[2m [Link]

same name [10], which fills the terminal with a visual impairments such as color blindness make
color spectrum. Figure 5 shows the output on a it difficult or even impossible to interpret the color.
Gnome terminal. When choosing a color combination, it is always
important to pay attention to the contrast. The
Color Combinations combination [14] of foreground and background
Annoyingly, it is not easy to find out which back- color [15] must match and provide the highest
ground color is used by the terminal on which a possible contrast. The higher the contrast, the
Bash command or Python script is currently run- greater the legibility of the output, even in poor
ning. Libraries such as term-background [11] read lighting conditions.
the environment variables and rely on escape The darkest possible text colors, such as black
sequences, which not every terminal emulation or dark blue, go well with a light background and
understands [12]. vice versa. A dark background tends to “swallow
But if you’re going to tinker with the color combi- up” light-colored text, making such text easier to
nation, you at least have to make sure that the out- read with a text style such as bold.
put remains legible. The use of colors and their To decorate the text appropriately, you can
acceptance by users is very subjective and always use the examples from the Table 3. It compares
influenced by culture. In Central European culture, the ANSI escape sequence [16] for Bash and the
red is associated with an error, yellow with a warn- predefined value in the Colorama library for Py-
ing, and green with success (see “Visual & Design thon [17]. Listing 5 shows text output in white
Principles” on GitHub [13]). Other cultures have dif-
ferent relationships to the same colors. It therefore
depends on how the user interprets the selected
color and what action they derive from it. In addition,

Listing 5: Decorated Text

01 from colorama import Fore, Back, Style

02

03 # Define foreground and background

04 foreground = [Link]

05 background = [Link]

06

07 # Define bold Figure 6: Decorated text from Listing 5.

08 bold = [Link]

09 Listing 6: Call-Up Test

10 # Define reset flag 01 # Load os and sys libraries

11 resetCode = Style.RESET_ALL 02 import os, sys

12 03 # Default false

13 # Define text 04 colouredOutput = False

14 textNormal = " white on blue " 05 # On a terminal?

15 textBold = " white on blue (bold " 06 if [Link]():

16 07 # An Xterm with 256 colors?

17 # Output text 08 if [Link]["TERM"] == 'xterm-256color':

18 print ("%s%s%s" % (background, foreground, textNormal)) 09 # yes -> Use color output

19 print ("%s%s%s%s" % (background, bold, foreground, textBold)) 10 colouredOutput = True

20 11 else:

21 # Reset color values 12 # No terminal

22 print (resetCode) 13 [...]

76 JULY 2024 ISSUE 284 [Link]

 COLOR ON THE TERMINAL LINUX VOICE

font on a blue background, both in normal and redirected. However, both factors can be resolved
in bold output. The bold output is far easier to with just a few lines of Python code.
read (Figure 6). Let’s start by using the Python module os to de-
termine the terminal type through the TERM [18] en-
Why Bother? vironment variable [19]. I will then use the sys
Finally, I’ll return to the basic question of whether module’s isatty() [20] method to determine
color is useful in output at all – after all, some whether the script was called on a terminal. List-
terminals only support a limited set of colors. An ing 6 summarizes the two tests.
xterm only supports eight basic colors; its suc-
cessors support more variants. Conclusions
On top of this, colored output makes no sense if Even in Bash, you can add splashes of color to the
a Python script is part of a pipe or its output is terminal output of your own programs with very
little effort. The Python library Colorama makes
The Author things even easier. But that is by no means the
end of the story: There are small tools that can be
Frank Hofmann works on the road, integrated into existing software to spice up its
preferably in Berlin, Geneva, and Cape output with color. Q Q Q
Town, as a developer, LPI-certified trainer,
and author. He is one of the authors of
the Debian Package Management book The author would like to thank Benjamin Schieder
([Link] and Axel Beckert for their criticism and support in
writing this article.

Info
[1] Debian ncurses-bin package: https:// [12] xterm color queries:
[Link]/bookworm/ncurses-bin [Link]
[2] “Adding colors to Bash scripts”: documents/xterm-color-queries/
[Link] [13] “Visual & Design Principles,” by Frank Hof-
bash-scripts-48g4 mann: [Link]
[3] Colors in the Bash Prompt HOWTO: visual-design-principles
[Link] [14] “26 Best Color Combinations for Your Next
HOWTO/[Link] Design,” by Naja Wade: [Link]
blog/best-color-combinations
[4] tput: [Link]
[15] “19 Color Combinations to use in Your Cam-
[5] Debian colortest package:
paigns,” by Ashly Winchester:
[Link]
[Link]
colortest
combinations-cheat-sheet
[6] “Build Your Own Command Line with ANSI Es-
[17] ANSI escape sequences:
cape Codes,” Haoyi’s Programming Blog:
[Link]
[Link]
458719343aabd01cfb17a3a4f7296797
[Link]
[18] Python predefined colors: [Link]
[7] training-python:
[Link]/print-colors-python-
[Link]
terminal/
training-python/tree/master/ansicolor
[19] TERM:[Link]
[8] Colorama: [Link] environment-variable
[9] termcolor: [Link] [20] Python environment variables:
[10] Debian colortest-python package: [Link]
[Link] env-vars-how-to-get-an-environment-variable-
colortest-python in-python/
[11] term-background: [21] isatty(): [Link]
[Link] html#[Link]

QQQ

[Link] ISSUE 284 JULY 2024 77

LINUX VOICE WEB TO EBOOK

Make a web archive using ePub

Book Binder
Saving web pages to ebooks conserves space and leads to easier reading.
he World Wide Web is rich with interesting menus and sidebars, or any interactive scripts

T
BY MARCO FIORETTI
articles, essays, and tutorials that are or multimedia features, you are better off retain-
worth having close at hand. After 30 years ing the HTML.
of web history, it is now clear that sooner or later However, if you only care about the actual con-
most web pages disappear. tent of a page – its text, images, and links – the
This is why, in a previous Linux Magazine, I looked ebook format becomes a much better choice, for
at how to create a private archive – viewable with at least two reasons. To begin with, modern web
any web browser – of full copies of all one's book- pages are often incredibly bloated and ebooks use
marks using Shaarli and archiveBox [1]. Another less space.
helpful way to preserve the content you read online As just one example, the CNN article discussed
is to save it in ebook format, which is then viewable in this article takes 2.7MB (or 1.7 if compressed) if
through an ebook reader. In this article, I’ll introduce saved as a complete HTML page. Saving it in
you to three different open source tools that convert ebook format, instead, only takes 240KB, seven
web pages to ebooks, and I’ll show you how to auto- times smaller than the compressed HTML version.
matically save all the web pages you want as eb- The other reason for saving the page as an
ooks, by passing their URLs to a simple shell script. ebook is efficiency. An ebook contains all (and
Note that, in some jurisdictions, you might run only) the parts of an article that matter without
into legal issues with downloading some content any of the distractions: It is also readable on
from the web – especially if you try to distribute or ebook readers that are easier on the eyes than a
reuse it. This is not a legal article – it is about the computer or smartphone and can hold a charge
technology. If you have any doubts, check the for weeks. Last but not least, once you convert
laws for you region and read the copyright or li- your bookmarks to ebook format, ebook manag-
censing notice for the web page. ers such as Calibre [2] can catalog them better
than most bookmark managers.
Why Ebooks?
Why bother with an ebook when you could just Three Ways
save the web page in HTML format? Actually, When I decided that I wanted to save my book-
saving a full web page in its native format is marks also as ebooks and started to search for
only necessary if you really want to preserve solutions, I had four requirements: First, the ar-
the full appearance and functionality of the con- chive should be private and local, on computers I
tent. If you want to preserve the drop-down fully control. Second, the software should not only
run on Linux (of course!), but also be easy to in-
stall and use. Third, it should save web pages in
The ePub Standard the ePub format (see the “The ePub Standard”
EPUB or ePub (a shorthand for “electronic publication” [3]) is an open box), which is the most portable, most widely sup-
ebook file format published by the International Digital Publishing ported open standard in this field. Finally, I wanted
Forum (IDPF), recognizable by the .epub file extension. something I could run from a shell script, to save
Version 3 of ePub is the most widely supported, vendor-independent many pages automatically.
ebook format, which almost all available hardware and software ebook Eventually I restricted my choice to the three
readers can handle. programs: ePub Creator [4], rePocketable [5], and
percollate [6]. ePub Creator is a Firefox extension,
Under the hood, ePub files are just ZIP archives that store one XHTML file
whose declared goal is to save in ePub format ev-
with the actual text, plus all the images and other files that contain the
erything you can see in Firefox’s “reader mode.”
table of contents and other metadata used by ebook software managers.
Being a browser extension, ePub Creator is not
This means that it is also easy to index and reformat them automatically,
scriptable (not easily, at least), but it’s the only op-
with shell scripts and other open source text-processing tools.
tion on websites that require subscriptions, and it

78 JULY 2024 ISSUE 284 [Link]

 WEB TO EBOOK LINUX VOICE

is so simple to use that it would have been wrong After running the Docker command, you will find
not to mention it. RePocketable, written in the Go an ePub version of the desired web page saved in-
language, was created because “reading anything side your computer’s LOCALDIR directory. In practice,
on the Internet has become a full-on nightmare” [7], there are a couple of things to deal with, which I will
a pain I too really feel. Percollate is a [Link] com- cover later in the Scripting Everything section.
mand-line tool that “turns web pages into beauti-
fully formatted PDF, EPUB, HTML, or Markdown Comparing Results
files” [6], making them very easy to reuse for gen- Using any of these three programs on any Linux
erating archives in those other formats, should I system is no big deal. Even with rePocketable and
decide to do so in the future. percollate, the only prerequisite is basic familiarity
Installing these three programs on any Linux with the command line. The obvious question then
distribution is simple. For ePub Creator, just visit becomes “which of these program is better?” The
its home page with Firefox, click on Install, and equally obvious answer is that there isn’t a single
then launch it from the browser every time you answer, and there never can be one, because the
want to make an ebook of a web page. best program heavily depends on which websites
RePocketable is actually a bundle of three pro- you read and need to save more frequently, and on
grams available as statically linked binaries for Linux, how frequently those websites change the layout
Darwin, and Windows platforms. Two of these pro- of their pages. Just keep reading to see why.
grams are only needed to interact with the Pocket In order to help you to decide, and to see how to
social bookmarking service [8]. The third program, make the whole process automatic (in the space
called To ePub, is the only one you need to convert available for this article), I have followed a pro-
web pages to ebooks. To install it, unpack the com- cess which may be unscientific, but is still ade-
pressed archive from the website, make the To ePub quate to at least give you an idea of how to per-
file executable, and move it to a folder in your $PATH: form your own tests: I chose seven random book-
marks from my collection, saved each of them
#> chmod 755 toEpub with all three programs, and took screenshots of
the resulting ebooks, side by side.
#> mv toEpub /usr/local/bin The first web page I saved is a ZDNET report of
a conversation with Linus Torvalds. The three re-
(the second command should be run as root). sulting ebooks are visible in Figure 1, in Ubuntu’s
Next, to save a web page you must type toEpub default ebook viewer. In that and all the other
at the command prompt, followed by its URL. comparison screenshots, the leftmost ebook is
Percollate can be installed with npm, the [Link] the one generated by rePocketable, the middle
package manager (again, as root): one by ePub Creator, and the rightmost one by Figure 1: A Linus conversa-
percollate. tion, as converted to ePub by
#> npm install -g percollate Some differences are evident at first sight: ReP- (left to right) rePocketable,
ocketable is the only tool that generates an ebook ePub Creator, and percollate.
but personally I found it more convenient to just
use the Docker container available on the website
with this command, which is much simpler than it
may look at first sight:

#> docker run -v "LOCALDIR:/tmp" xiangronglin/fl

percollate-alpine percollate epub URL -o /tmp/fl

[Link]

In plain English, this command:

Q downloads and runs the Linux container image
called percollate-alpine, which is a full working
copy of the percollate program, wrapped inside
a ready-to-run virtual Linux system;
Q binds, with the -v option, the /tmp directory in-
side that virtual system to a LOCALDIR directory
on your physical computer;
Q tells the percollate program inside the container
to download the web page at the address URL
and save it in ePub format (but it can also be
Markdown or PDF), with the name EPUBNAME.
epub in the container’s /tmp directory.

[Link] ISSUE 284 JULY 2024 79

LINUX VOICE WEB TO EBOOK

and styles all get the same general look when they
are automatically converted to ePub. I consider
this uniformity as one less cause of distraction, a
feature. The article shown in Figure 4 was also the
first with which I noticed a feature unique to ePub
Creator, which may be irrelevant for some users
and really important for others: the ability to check
and modify the author’s name before saving the
page (Figure 5).
Of course, I also wanted to check what the con-
verters make of Linux Magazine pages, so I pointed
them to my Obsidian tutorial [9], which is visible in
Figure 6 both in Firefox (right side) and in the Cali-
bre viewer (left side). The main thing to notice in
Figure 6 is that image captions are harder to recog-
nize than in the original document, because they
are saved with the same style as generic text. This,
however, doesn’t depend on the website; it is an-
other aspect of the uniformity I just mentioned.
Converting that tutorial allowed me to notice an-
other difference in the behavior of the three pro-
Figure 2: Firefox showing the web page used for the ebooks grams, which I highlighted with the arrows in Figure
in Figure 3. 7: They do not render code and other specially for-
matted text in the same way, and on this specific
cover and, at least for the website in Figure 1, per- point I would say ePub Creator does the best job.
collate recognizes the author’s name, but ePub Moving on, Figure 8 is one more proof of why
Creator doesn’t. there cannot be any single answer to the “which
Figure 2 shows a CNN article viewed in Firefox, web-to-ebook converter is best” question. Figure
and Figure 3 shows how the three programs con- 8 shows a post from my own blog, which is made
verted that article. This highlights another issue with the Hugo static site generator. It seems that,
that, in general, may be solved only by manually unlike what happened with the websites in the
editing the ebook file: Depending on the layout of previous figures, Hugo, or at least the specific
the web page, a converter (ePub Creator in this Hugo theme I chose for that blog, does not mark
case) may be unable to filter out side content up images in a way that makes them recogniz-
such as all the CNN menus. able as cover material by rePocketable.
Figure 4, from the Rest of World online maga- As far as this tutorial is concerned, that’s OK, be-
zine, is interesting because it is further proof of cause it makes evident another feature that’s unique
something that may have already been evident by to rePocketable: the metadata page that this pro-
comparing the previous figures: No matter which gram always puts at the beginning of each ebook.
tool you use, websites with very different layouts Figures 9 and 10 contain only two ebooks (by
ePub Creator and percollate, in both cases),
Figure 3: Ebook conversion doesn’t always remove all the overhead of mod-
ern web pages. Here, ePub Creator kept all of the original article’s menus. Figure 4: These ebooks have the same look and feel of those in Figures 1
and 3, even if they come from very different websites.

80 JULY 2024 ISSUE 284 [Link]

 WEB TO EBOOK LINUX VOICE

Figure 5: ePub Creator is the only tool covered here that

lets users change the authors of a page before conversion.

because they illustrate a bigger problem than the

inability to generate covers for certain websites. I Figure 6: A Linux Magazine tutorial converted to ebook: Looking good, even if figure captions
tried to create ebooks from two different Substack don’t stand out.
blogs (to be sure it was a site-level issue), and in
both cases rePocketable just quit without gener-
ating any file, with an error message like this:

Cannot fill document: bad set[Link]

[Link]/image/fetch/ ...

Right after this discovery, I also realized that the

only program mentioned here that can turn Sub- Figure 7: While their default
stack articles into complete ebooks with both text Scripting Everything styles are very similar, dif-
and images is percollate; as you can see in Figure Listing 2 is the script I used to generate all the ferent converters treat code
11, ePub Creator failed to insert images! rePocketable and percollate ebooks automatically. markup in different ways.
As a final element to consider for your choice of You may modify the script as desired to do the
ebook generator, Listing 1 lists the sizes in bytes same thing for all your bookmarks or any other col-
of all the ebooks generated for this tutorial. Seven lection of links. It takes one argument, which is a
pages are surely too few to produce really reliable plain-text file containing one URL per line.
results. With that disclaimer, I found that ePub After saving the name of that file in the $LIST vari-
Creator always generated the smallest files and able (line 3), the script removes and then recreates
percollate always the biggest, even if rePocketable (lines 4 and 5) the directory $EPUBDIR, where all the
is the only tool that adds a cover to each ebook. ebooks will be saved. Line 7 makes $EPUBDIR world-
writeable because otherwise the
Figure 8: Even when it fails to create a cover, rePocketable always puts an Docker container that runs percollate
information page at the beginning of an ebook. could not write into it.

Figure 9: Not all converters work on all websites. Only ePub Creator and
percollate, for example, could render this Substack post.

[Link] ISSUE 284 JULY 2024 81

LINUX VOICE WEB TO EBOOK

any trailing slash, and then everything before the

Listing 1: Sizes of Ebooks Generated with actual file name part of the URL.
Different Tools After that, the for loop creates two ebooks for
15054 [Link]
each URL: When the loop variable $CONV (for “con-
74905 [Link]
verter”) is equal to repo (line 19), the script tempo-
305696 [Link]
rarily moves into the folder $EPUBDIR/repotemp be-
76179 [Link]
83207 [Link]
fore running the toEpub program (line 21 and 22).
74029 [Link] Then, the ePub file created by To ePub is moved to
204368 [Link] $EPUBDIR, with the new name $EPUBNAME-$[Link].
416233 [Link] The reason to run To ePub into an empty folder
488768 [Link] and then move its ebook back into $EPUBDIR is that
115340 [Link] it is the only way to give its ebook a custom name,
230797 [Link] instead of the non-configurable one that To ePub
197324 [Link] generates by itself. If To ePub ran inside $EPUBDIR,
1841220 [Link] which contains all the ebooks generated in previ-
3205030 [Link]
ous runs, the mv (move) command in line 23 would
156895 [Link]
fail, because it would have more than one file to
238847 [Link]
move to the new name.
185811 [Link]
When the $CONV variable is equal to percollate, in-
231596 [Link]
624306 [Link]
stead, the script just launches the Docker container
as I already explained (lines 26 to 29), with the appro-
priate values for the output directory and file name.
Because it must run Docker, on many Linux dis- As is, the script generates two ebooks for every
tributions this script will have to be run as the root URL written in the $LIST file, because that is ex-
user, unless you set up your own Linux account to actly what I needed to generate complete screen-
use Docker directly. For the same reason, it is nec- shots for this tutorial, but such a redundancy
essary to specify the full, absolute path of the would obviously be unnecessary when creating an
$EPUBDIR directory, as in line 4. actual archive. However, it is a good starting point
After clearing the screen and moving to the for archiving many URLs as ebooks automatically,
$EPUBDIR directory (lines 9 and 10) the script with just a couple of relatively easy improvements
runs the main while loop of lines 12 to 31. This that I leave as exercises for the reader.
loop reads the $LIST file (line 31) one line at a The first improvement would consist of making
time, loading each URL to convert in the $line the script only run one converter per URL, depend-
variable. ing on the website it belongs to. For example, you
The sed command in line 14 generates an may decide to always use rePocketable, because it
$EPUBNAME file name for each ebook by stripping also adds covers and metadata sheets, except on
from the URL loaded into $line (which always URLs that contain the “[Link]” string, which
begin with [Link] all the characters before the rePocketable cannot handle but percollate can.
two con- The other improvement that would be essential
Figure 10: A confirmation of Figure 9: Conversion of another Substack blog secutive in real-world archiving is related to the fact that
only works with percollate and ePub Creator. slashes, rePocketable, and maybe even percollate, don’t

Figure 11: Only percollate captures the pictures from the same post shown
in Figure 9.

82 JULY 2024 ISSUE 284 [Link]

 WEB TO EBOOK LINUX VOICE

get the web page title always right. For example,

if a URL ends with [Link], rePocketable will Listing 2: Script for Web to Ebook Automatic Conversion
generate an ebook called [Link], which 01 #! /bin/bash
02
not only is useless, but would overwrite any other
03 LIST=$1
ebook generated in previous runs from other
04 EPUBDIR="/home/marco/epub-temp"
URLs ending with [Link].
05 rm -rf $EPUBDIR
Luckily there is an easy solution for this: Re-
06 mkdir -p $EPUBDIR/repotemp
place line 14 in Listing 2 with one that loads into
07 chmod 777 $EPUBDIR
$EPUBNAME the actual title of the current URL, after
08
fetching it with the Xidel data-extraction program 09 clear
that I covered in another tutorial [10]. 10 cd $EPUBDIR
Even before making these improvements, how- 11
ever, there are two changes that you will surely 12 while read line
want to make to this script if you plan to retrieve 13 do
and convert lots of web pages. One is to make the 14 EPUBNAME=`echo $line | sed -e 's|.*//||' -e 's|/$||' -e 's|.*/||' `
script sleep a few seconds after every conversion 15
to avoid overloading any website from which you 16 for CONV in repo percollate
plan to download many pages back to back. The 17 do

other, even more important, especially with slow 18 echo "$CONV: Converting $line to $EPUBNAME-$[Link]"

connections, is to make Docker load a local image 19 if [[ "$CONV" == "repo" ]]

of the percollate-alpine system, rather than down- 20 then

loading it from the Internet at every call. 21 cd $EPUBDIR/repotemp

22 toEpub $line

Conclusions 23 mv ./*.epub $EPUBDIR/$EPUBNAME-$[Link]

24 cd $EPUBDIR
The web is huge and highly transitory. Many in-
25 else
teresting web pages are also bloated and full of
26 docker run -v "$EPUBDIR:/tmp" \
distractions. Under such conditions, many readers
27 xiangronglin/percollate-alpine percollate \
like the ability to keep a clutter-free, personal,
28 epub $line -o /tmp/$EPUBNAME-$[Link]
permanent, fully private copy of something they
29 fi
find online. Ebooks in the ePub format are the
30 done
most future-proof way to achieve that goal, even 31 done <$LIST
if you don’t have and don’t plan to have an actual
ebook reader.
The conversion tools described in this article will Info
occasionally fail or give suboptimal results, as hap-
[1] “Preserve Your Favorite Pages” by Marco Fioretti, Linux Magazine, issue
pened to rePocketable and ePub Creator in my little
test, and it’s impossible to know in advance which 232, March 2020, [Link]
websites will create problems. But that’s just an Create-a-Personal-Web-Archive
unavoidable consequence of the general messi-
[2] Calibre: [Link]
ness and continual evolution of the Web, which is
probably more a feature than a bug overall. [3] ePub file format: [Link]/publishing/epub3/
Even in these conditions, however, percollate,
[4] ePub Creator: [Link]
and rePocketable can create usable ebooks while
you take a nap, and ePub Creator can take care of [5] rePocketable: [Link]
password-protected pages. Q Q Q
[6] percollate: [Link]

The Author [7] “Reading from the web offline and distraction-free” by Olivier Wulveryck,
owulveryck‘s blog, October 7, 2021, [Link]
Marco Fioretti (https://
[Link]) is a free- info/2021/10/07/[Link]
lance author, trainer, and [8] Pocket: [Link]
researcher based in
Rome, Italy, who has been [9] “Tutorial – Obsidian” by Marco Fioretti, Linux Magazine, issue 247,
working with free/open June 2021, [Link]
source software since
sonal-Knowledge-Managers/(language)/eng-US
1995, and on open digital
standards since 2005. Marco also is a board [10] “An XML, HTML, and JSON Data Extraction Tool” by Marco Fioretti, Linux
member of the Free Knowledge Institute Magazine, issue 276, November 2023,
([Link]
[Link]

[Link] ISSUE 284 JULY 2024 83

LINUX VOICE FOSSPICKS

FOSSPicks Sparkling gems and new

releases from the world of
Free and Open Source Software

This month Nate looks at Audacity, Endless Sky, GCompris,

Switcheroo, MS-DOS, Qemu, and more! BY NATE DRAKE
This is Nate’s first time writing FOSSPicks after Graham passed on the baton. On behalf of all
at Linux Magazine, we’re eternally grateful for Graham’s years of spotting the top FOSS picks.

Audio editor customer data with the head of- available for download in most Linux repositories. However,

Audacity
fice in Russia and US legal coun- for the most recent version at the time of writing (3.5.1),
sel. The FOSS Post team also users need to visit the main site to download an AppImage.
published a damning indictment The release notes correctly state that more modern ver-
of the new terms and conditions sions of Linux – like our Ubuntu 24.04 test machine – will
hen Graham reviewed in November 2022, claiming that need to install libfuse2 in order to launch the editor.

W the Tenacity fork of the

popular audio editor
Audacity last year, he touched on
unhashed IP addresses were
being stored temporarily on Au-
dacity servers. They also cited a
After the controversy of Muse Group’s data collection
practices, Linux users may prefer to use Audacity entirely
offline. Still, doing so will cause them to miss out on the lat-
Audacity’s acquisition by Muse provision which stated that Au- est “cloud save” feature, which allows uploading of projects
Group in 2021. Controversy was dacity wasn’t permitted for users to [Link] via a linked account. The Audacity Support
sparked when a draft proposal under 13, which technically would pages note this should make collaborating, sharing and
was introduced to the code for be a violation of the GPL. restoring previous versions of projects much simpler.
opt-in telemetry to record app Nevertheless, Audacity remains In keeping with Audacity’s tradition of providing an intu-
usage, leading to accusations that nominally open source software itive UI, the editor now also has a simplified method for
Audacity had become spyware. under GPLv2, as well as one of the changing pitch on a clip-by-clip basis. Users now only
Muse quickly backpedaled but most popular audio editors for need to hold Alt and use the up/down arrows or use the
managed to provoke yet another Linux. It’s drawn praise in particu- overflow menu to make changes. Once the pitch has
backlash over changes to the Au- lar for its simple interface, as well been altered, a small indicator appears at the top of the
dacity policy that would have as its extensibility via various pl- clip. The latest version of the editor also makes importing
allowed the company to share ugins. The editor remains loops much simpler by detecting their tempo via audio
analysis and meta checking. Audacity can then automati-
cally adjust the imported loops to keep them in tempo.
This feature can be toggled via the Preferences menu.
Some features have also now been removed. These in-
clude printing, taking automatic screenshots, and even Ka-
raoke, which would display a bouncing ball in time with lyr-
ics. On the plus side, Audacity has added subtitle formats
for labels. It can also now import and export SubRip files.
The plugin manager has also been overhauled, with an em-
phasis on making multiple extensions easier to manage.
The interface is more spartan but can also now be filtered.
Users can also find specific plugins via the search box.
For all the accusations of the owners being tone deaf to
1. Cloud save: Audacity now supports saving projects to the cloud through a linked users, Audacity does automatically check for and apply
[Link] account. 2. Tempo detection: Each time a loop is imported, Audacity updates upon launch. Newer versions also seem to be
now automatically adjusts it to be in tempo. 3. Pitch shifting: Users can non- available more regularly than for competing offerings
destructively change pitch via the overflow menu or by holding Alt and pressing the such as Tenacity. As always we encourage readers to do
up and down arrow keys. 4. Centered zero line: When zooming in vertically, the zero their own research and configure firewall settings to
line now remains centered. 5. No Karaoke: Sadly the “bouncing ball” Karaoke option
specify network permissions for apps.
has been removed. 6. Timeline options: Use the new settings gear to switch
between timelines and configure looping. 7. Simplified plugins: The updated plugin
manager now has a much simpler, filterable interface, as well as a search box. Project Website
8. Easier effects: Audacity now places OK/Cancel at the bottom of effectswindows. [Link]

84 JULY 2024 ISSUE 284 [Link]

 FOSSPICKS LINUX VOICE

Educational software

GCompris
he aim to make learning divided into a number of broad

T fun may trigger an eye

roll from teenagers, but
GCompris’s target market is 2- to
categories including Computer
Discovery, Numeracy, Science,
Geography, Games, and Reading.
10-year-olds. It consists of a Some of the activities clearly
suite of games (currently around are designed to teach the same
190) known in GCompris par- concepts, such as “Mixing Paint The canal lock activity requires players to assist a very familiar-
lance as activities, with an em- Colors” and “Mixing Light Colors,” looking penguin’s boat through gates.
phasis on education. The suite is a variety which can help keep
available for Linux but there are things interesting for young fully translated into a number of languages including
also versions for BSD, Android, learners. There’s also a built-in Dutch, Polish, Brazilian Portuguese, and French. After ap-
macOS, and Windows. The proj- “difficulty filter” that can hide parently nine years of work, Gcompris graphics have also
ect has been in active develop- more advanced activities from been reworked to match the project guidelines. Version
ment since 2000 and was origi- younger children. This can be ac- 4.0 also includes eight new activities. These include Cal-
nally released under the GNU cessed via the Settings menu, cudoku, in which players have to fill a grid with numbers
GPL by French software engineer which also allows users to con- according to specific rules. Other new titles make refer-
Bruno Coudoin. These days figure speech and background ences to “graduated lines.” This use of advanced words,
GCompris is developed and music. such as “enumerate the fruit,” seems to persist through-
maintained by the KDE commu- The latest version of GCompris out GCompris making the suite more appropriate for kids
nity. The name for the suite is a (4.0) addresses some of the criti- with a wide vocabulary.
play on the French expression cisms previously leveled by par-
J’ai compris (“I have under- ents that the suite is too Anglo- Project Website
stood”). Activities for kids are centric. The suite has now been [Link]

Image Converter

Switcheroo
his extremely versatile and GIF. By default, more ob-

T utility first began its life in

2022 as Converter – a
GTK4 plus libadwaita application
scure export formats such as
AVIF aren’t listed in the relevant
menu, but users can click into
that served as a graphical front the settings menu at the top right
end for ImageMagick. Converter to choose Show Less Popular
has since been rebranded as Datatypes. From here, users can
Switcheroo. Written almost en- also paste images directly from Switcheroo’s preferences menu supports exporting image formats
tirely in Rust, the app is now a the clipboard, as well as clear the such as AVIF. Use the + at the top left to add more images.
proud member of Gnome Circle. current window and launch
On launch, Switcheroo’s inter- another. multiple images, these can also be exported as a sin-
face allows users to click Open Once the export format is se- gle ZIP file. Upon clicking Convert, the program
Images to browse for files, lected, Switcheroo also allows prompts users to select the destination for the newly
though you can also simply drag users to configure options such created images.
and drop pictures for conversion. as image quality and back- Switcheroo is only available as a Flatpak package,
The program supports both indi- ground color (which replaces meaning Ubuntu users will have to install Flatpak using
vidual and batch conversion. the original transparency layer). the command line to download and set up Switcheroo
One of the most impressive The program also allows for re- via Flathub. The project’s GitLab page also lists steps
features of Switcheroo is the sizing of images either via a to clone the Git repository using Gnome Builder from
number of supported image for- percentage (default) or pixels. Flathub.
mats. At the time of writing these Users can also switch between
include PNG, JPG, WebP, HEIF, the Default and Pixel Art resizing Project Website
HEIC, BMP, AVIF, JXL, TIFF, PDF, algorithms. If a user selects [Link]

[Link] ISSUE 284 JULY 2024 85

LINUX VOICE FOSSPICKS

Machine Emulator/Virtualizer

Qemu
emu’s popularity stems B-L475E-IOT01A IoT node, mp3-

Q from its ability to emulate

various processor archi-
tectures, allowing it to run operat-
an536 (MPS3 dev board plus
AN536 firmware), and crucially,
the Raspberry Pi 4 Model B. The
ing systems for virtually any ma- virtio-blk device has also been
chine, on any supported architec- overhauled and now has true
ture. It can also run KVM and Xen multiqueue support. This means
virtual machines with almost-na- different queues on a single disk Boxes uses QEMU, KVM, and libvirt virtualization, offering a graphi-
tive performance. Despite being a can be processed by different I/O cal interface that supports downloading a guest OS.
favorite of command-line lovers, threads. The Qemu changelog
there are also various graphical notes that this could improve during TLS handshakes has also been resolved. Version
front ends available. This can be scalability in cases where the 9.0 also fixes a flaw in Qemu 8.2, which accidentally al-
a preferable way to interface with guest submits enough I/O to sat- lowed users to create memory back ends with sizes that
Qemu, especially given that some urate the host CPU. Users can weren’t aligned to page size. Qemu 9.0 also has improved
front ends such as Gnome Boxes also now configure multiple I/O security: The luks block driver now supports creating and
support downloading the required threads using the new io- using detached LUKS header files. The block driver also
installation media. thread-vq-mapping property. now supports the SM4 cipher algorithm. Qemu’s cipher
The most recent release of The ESP SCSI (am53c974/ test suite will also now automatically skip testing algo-
Qemu (9.0) includes support for dc390) device has also had a rithms that have been disabled in the underlying OS
RISC-V, LoongArch, s390x, and substantial upgrade, fixing sev- crypto library at build time.
HPPA emulation. There’s also eral long-standing bugs – one
much better support for ARM which could potentially cause Project Website
boards including the crashes if the connection is lost [Link]

x86 Operating System

MS-DOS 4.0
n April 25, Microsoft, in One of the intended key fea-

O partnership with IBM,

released MS-DOS 4.00
under the open source MIT Li-
tures of MS-DOS 4.0 was its mul-
titasking abilities, including the
system component session man-
cense. Admittedly, this comes 36 ager ([Link]), which in theory al-
years after the OS’s original re- lowed easy switching between up The MS-DOS GitHub page also hosts versions 1.25 and 2.0. Open the
lease, from a time when Micro- to six applications, which were v4.0-ozzie/bin folder to download both original disk images.
soft had a very troubled relation- listed in [Link] along with their
ship with open source. Still, the corresponding hotkeys. Unfortu- compile and run MS-DOS 4.0 despite all the build tools
fruits of this project came from nately, in this beta, [Link] only being available for download from the GitHub project
English researcher Connor “Star- checks the directory from which page. He has suggested that Microsoft should make
frost" Hyde, who contacted Mi- you ran it, meaning it cannot read images of the original floppy disks available instead.
crosoft chief technical officer [Link] and consequently is unable There’s a certain historical irony to these troubles, given
Ray Ozzie about his software col- to launch any apps. MS-DOS 4.0 also had running woes back when users first
lection, which included unre- Unfortunately MS-DOS 4.0’s ran it in 1988. In a precursor to the Windows 98 “blue
leased beta binaries of DOS 4.0. woes don’t end there, because it screen of death,” popular programs such as Lotus 1-2-3
These were on floppy disks he’d seems that the process of up- and Doom could cause the OS to freeze. This may be be-
received while working at Lotus. loading to GitHub removed time- cause MS-DOS 4.0 could consume as much as 92KB of
Working with eager volunteers, stamps from the source, as well RAM, a huge increase on the approximately 56KB used by
the floppies were imaged and the as encoding with UTF-8. Michal MS-DOS 3.31, to which users returned in droves.
original paper manuals were digi- Necasek of the OS/2 Museum
tized for the benefit of the rest of has noted how these bugs have Project Website
the Internet. made it extremely difficult to [Link]

86 JULY 2024 ISSUE 284 [Link]

 FOSSPICKS LINUX VOICE

Web browser

Mozilla Firefox
his November, it will have version 125 where Firefox security

T been 20 years since the

release of version 1.0 of
Firefox. Since 2004 it has lost
settings were being too proactive
about blocking potentially untrust-
worthy URLs. Mozilla hopes to re-
much of its market share but store this feature in Firefox
usage hovers around three per- 125.0.3. The latest version of the Firefox will now automatically suggest URLs that have been copied to
cent, making it still the fourth browser also supports highlighting the clipboard, saving users from manually pasting them in.
most popular graphical web PDFs, as well as storing addresses
browser after Chrome, Safari, for users in the US or Canada. mechanism (X25519+Kyber768). While this can quantum-
and Edge. Firefox remains a Firefox now supports resolving proof TLS connections, the Mozilla blog warns that this can
popular choice for Linux users, HTTPS DNS records via the Linux result in slow TLS handshakes or failed connections on net-
which is reflected in Mozilla’s de- OS’s DNS resolver. There’s even works with TLS intercepting middleboxes. Fortunately the
cision in January 2024 to create support for the AV1 codec for En- feature can be disabled in Firefox preferences. Naturally
an official Apt repository for crypted Media Extensions (EME), readers who are concerned with security may prefer to use
Debian-based Linux distributions enabling higher-quality playback popular Firefox forks such as LibreWolf, which has been
such as Ubuntu, where it re- from video-streaming providers. hardened to block cookies, prevent fingerprinting, and uses
mains the default browser. The Firefox View has been overhauled the privacy-centric DuckDuck Go search engine instead of
final build of Firefox 125 was re- and now displays pinned tabs in Google. Still, Firefox proper is updated much more fre-
leased on April 15. the open tabs section. quently than its forks, and it’s still possible to tweak the
At the time of writing, the latest As of version 125, Firefox also stock browser’s privacy settings to suit users’ needs.
point release is version 125.0.2. now attempts to establish TLS
This includes a URL suggestion connections using a hybrid post- Project Website
feature, as well as a fix for a bug in quantum key agreement [Link]

Process viewer

htop
ost Linux sys admins Because system monitoring

M will have encountered

this interactive pro-
cess viewer, originally released in
interfaces are not standardized
for Unix-like operating systems,
there’s no one definitive htop,
May 2004 as an alternative to but most major Linux distros
Unix’s own top. Its advantages in- have a bespoke version in their
clude a more interactive inter- repositories, making it simple to The colorful, interactive, and highly customizable layout makes htop
face, as well as its use of color to install via dnf or apt-get. Admit- an ideal process viewer. Most importantly, it runs Doom.
provide detailed descriptions of tedly, with time and patience it
the processor, swap, and mem- is possible to create a colorful designed for porting the game to other plat-
ory status. It can also be used in- custom configuration of layout forms. The dedicated GitHub page also in-
teractively as a system monitor, and fields in top itself. Still, the cludes build instructions as well as a WAD from
supporting both hotkeys and general consensus in the Red- the open Freedoom project to avoid any copy-
mouse clicks to allow users to dit-sphere is that htop is better right issues. The GitHub page also contains a
search for, sort, and kill relevant at doing this out of the box. link to the original Doom shareware WAD. If you
processes. It’s written in the C Indeed, this process viewer is so decide to play this colorful ASCII-only version of
programming language, using visually rich that in April, software Doom, first make sure to install and run htop it-
the ncurses library. Any informa- developer 0x0mer (aka Omer self, and then quit via F10 to create the relevant
tion displayed can be configured Goldzweig) decided to adapt it to config file. You can then run doom-htop from
through htop’s graphical setup answer the all-important question: the doomgeneric folder.
and can be sorted and filtered in- “Does it run Doom?” The doom-
teractively, such as displaying htop project is a fork of doomge- Project Website
processes as a tree. neric, which is specifically [Link]

[Link] ISSUE 284 JULY 2024 87

LINUX VOICE FOSSPICKS

RTS Game

Widelands
was the year that as interact with
1996 gave the world
the first flip phone and DVD
other clans.
From various
video. It’s also the year that the online reviews
German Blue Byte Software in Linux publi-
(now Ubisoft Blue Byte) gave cations, I noted
the world The Settlers II: Veni, the game has
Vidi, Vici – a DOS city-building received praise
game with real-time strategy for its enemy
(RTS) elements. Today the artificial intelli-
game lives on indirectly through gence (AI), Players must first complete the Barbarian campaign. Once the “raging flames” are
Widelands, an open source RTS though both re- quenched, other campaigns are unlocked.
game, inspired by many of the viewers and
concepts and mechanics found the website state the real fun be- Since the release of version 1.1, the Widelands devel-
in the Settlers series. In keeping gins with Internet/network play. opment team has pulled out all the stops to fix bugs
with the slow pace of city-build- Players can choose to engage in and make the game more customizable. One of the
ing games, Widelands initially diplomacy and trade with other major changes is the introduction of UI plugins, which
became available in 2002, but it clans or form armies to fight. can add more functionality to the game interface.
wasn’t until 2021 that the devel- Widelands also has several play- There’s also tentative support for naval warfare, allowing
opers finally released the first able tutorials for newcomers. players to invade foreign coasts using warships. (Be-
stable version. They have since Despite the stable release, some cause this feature is experimental, it must be enabled
picked up the pace – Widelands features are still experimental: Dur- specifically from the setup screen). In-game ports and
1.2 was released at the end of ing my tests I found, for instance, HQs also now have soldier garrisons. Most units have
March. that the game was unable to gen- also had an image refresh, and are now displayed in a
The Widelands GitHub page erate a random map. On the plus higher resolution.
states that the game “has signifi- side, there are many options for Version 1.2 has expanded the Frisian campaign,
cantly more variety and depth” playing the game on Linux. I found adding a fifth scenario. There are improved tweaks to
than Settlers II. The project web- it available for installation in how the AI handles diplomacy and the setup screen
site explains that initially players Gnome software. The project also now offers more options, including durations for
are the regent of a small clan, has a dedicated personal package timed win conditions. Players can even stick pinned
with only a small HQ where all re- archive (PPA) for Debian-based notes onto map fields to mark important points. In
sources are stored. As time goes systems and can also be down- addition to English, Widelands is also now available in
on, the player manages their clan loaded and run as a Flatpak/ Catalan, German, Hungarian, Low German, and Rus-
to gather more resources as well AppImage. sian. After reviewing the changelog, I also discovered
that the random number gener-
ator (RNG) now must be
seeded to start a new game,
which is why I suspect it en-
countered an error when run-
ning in my test virtual machine,
which had very little available
entropy.
The project website states
the game is and always will be
in active development and so
will always need 2D/3D artists,
sound effects creators, and
playtesters. Anyone interested
in contributing can visit the
Widelands forum.

As with most RTS games, Widelands starts with one base. Players can then gather resources and build fur- Project Website
ther structures. [Link]

88 JULY 2024 ISSUE 284 [Link]

 FOSSPICKS LINUX VOICE

SDK

Steam Audio
hile Valve is a for-profit features they work on are deter-

W company and the

Steam client is proprie-
tary software, the gaming giant re-
mined by internal projects. The
example given was the develop-
ment of Half Life: Alyx, where the
cently announced in a blog post team devoted a lot of time to hy-
that the entire code for their Steam brid and pathing features, which
Audio SDK is now available via the made it into Steam Audio 4.0.0. Steam Audio has many applications beyond porting video games
open source Apache 2.0 license. Naturally, making the SDK avail- more easily, including acoustics modeling for architectural design.
The stated goal is to “provide more able will allow partners to port it
control to developers, which will themselves, such as to a games specific GameObject. The Unreal Engine plugin has also
lead to better experiences for their console even if Valve isn’t work- been tweaked to prevent crashing when simulating pathing.
users.” In a blog post, Valve also ing on such a project. The Steam Community pages further explain how
announced that this decision was At the time of writing, the most developers benefit from Steam Audio by noting that
made after receiving feedback recent release is Steam Audio SDK the SDK adds physics-based sound propagation on top
from the community relating to pl- 4.5.3. This includes fixes for a of HRTF-based binaural audio. It can also simulate
ugins that are already open number of bugs that were causing how objects occlude sound sources. In plain English,
source, such as Unity and Unreal. crashes, as well as an update to this means that sounds can actually interact with and
The company has also promised Embree build settings to allow in- bounce off of in-game geometry, and sound from par-
to continue its work on Steam stanced meshes to work again. tially hidden sources can be reduced. This provides
Audio, so it will not be solely com- The Steam Audio Unity plugin has players with a more immersive experience.
munity maintained. also been upgraded with a new
On this note, in the same blog property to notify Steam Audio Project Website
post Valve admitted that the that the listener has changed to a [Link]

Space trading and combat

Endless Sky
t only takes a few minutes of Steam online store states that

I playing this space explora-

tion, trading, and combat
game to see that it draws its in-
the main storyline takes 8-16
hours to play through. Not only
are there hundreds of planets and
spiration from the likes of Elite dozens of ships to choose from
and Escape Velocity. Players but as an open source game re-
begin on their home planet of leased under GPL 3.0, Endless
New Boston with a substantial Sky also incorporates a plugin Endless Sky walks players through the basics of selecting a ship and
bank loan, which they can use to system allowing anyone with ac- “jumping” to other planets via step-by-step text prompts.
purchase one of three ships. cess to a basic text editor and
They’re then free to start explor- image editor to create new ships, is also an option for players who don’t want to waste time
ing the galaxy, trading goods, weapons and missions. transporting passengers and goods, though certain ship
outfitting their vessels and en- The most recent version modifying “outfits,” like the Gatling turret, must now be op-
gaging in space combat as they (0.10.6) has an extensive change- erated by a dedicated crew member.
go. At the time of writing, players log. The Free Worlds campaign To quote the project website, players begin with the
can undertake missions involv- has been updated to reflect cer- most “wimpy” ship in the game and have to work their
ing a civil war breaking out in the tain worlds shifting their alle- way up to advanced weaponry. The website also hosts a
Space Republic, though the giances. The “hail” panel also now comprehensive manual that covers exploring the galaxy,
game teases other missions may correctly renders animated ship how to fly the ship itself, and things to do when landing
be playable in the future. and planet sprites. Space mer- on planets – such as picking up passengers for profit.
Although, sadly, I haven’t had chants also can now provide tips
the chance to test this for myself, on dumping cargo in order to ward Project Website
the game description in the off pirates. Naturally, space piracy [Link]

[Link] ISSUE 284 JULY 2024 89

LINUX VOICE TUTORIAL – UBUNTU VM IN THE AZURE CLOUD

Setting up an Ubuntu instance in Azure

Sky Server
Are you ready to get started with the cloud? Microsoft’s Azure Cloud Services
provides easy access to an Ubuntu virtual machine.

f you’re ready to implement a basic server Setting Up an Ubuntu VM on Azure

I
BY MARCIN GASTOL
system, and you want to avoid the complica- Ubuntu, known for its stability, security, and ease
tions of dealing with the hardware, cloud of use, is a popular Linux distribution for cloud en-
computing is a convenient and surprisingly afford- vironments. By choosing Ubuntu, you can lever-
able option. The cloud offers a wide range of com- age the stability and power of Linux, along with
puting services – including servers, storage, data- the wide range of applications and open source
bases, networking, software, analytics, and intelli- tools that Linux supports.
gence – with resources and economies of scale. Setting up an Ubuntu virtual machine (VM) on
Most cloud vendors offer a pay-as-you-go model, Azure involves several key steps: selecting the ap-
thereby reducing the need for significant upfront propriate VM image from the Azure Marketplace,
capital expenditures. configuring the VM’s specifications (such as size,
Microsoft Azure is a leading cloud service storage, and network settings), and finally, deploy-
provider (along with Amazon Web Services and ing and connecting to the VM. Throughout this
Google Cloud Platform). Microsoft used to be guide, I aim to clarify this process, providing clear,
known for its opposition to Linux, but the com- step-by-step instructions that enable you to effi-
pany has come a long way in recent years. ciently launch and manage Ubuntu VMs on
Azure now offers several options for Linux cloud Azure’s cloud platform.
systems, including Red Hat, SUSE, Ubuntu, and
Debian. Azure’s comprehensive suite of services Setting Up an Azure Account
encompasses everything from simple web apps Setting up an Azure account is the initial step. This
Figure 1: The initial screen to Internet-scale solutions with big data and arti- process begins by visiting the Azure portal where
for creating a new resource ficial intelligence capabilities. users can manage and monitor their cloud deploy-
group. ments. A Microsoft account is required to sign in or
register for Azure, act-
ing as the primary
gateway to accessing
Azure’s cloud ser-
vices. During registra-
tion, users select an
Azure subscription
that suits their needs
and budget, from op-
tions such as pay-as-
you-go to Azure Free
Account. The setup in-
cludes a verification
process for security
and fraud prevention,
typically requiring a
phone number and a
credit card. Once the
account is active, it is
beneficial to explore
the Azure portal to

90 JULY 2024 ISSUE 284 [Link]

 TUTORIAL – UBUNTU VM IN THE AZURE CLOUD LINUX VOICE

select the appropriate subscription from the drop-

down menu labeled Subscription. Then, you input
a name for the resource group in the field labeled
Resource group name. This name should be
unique within the selected subscription and typi-
cally follows a naming convention that reflects the
user’s organizational standards.
You must then select the geographical region
for the resource group by choosing from the Re-
gion drop-down list. It’s important to select a re-
gion that is close to your location or customer
base to minimize latency and ensure compliance
with data residency requirements.
Before finalizing the creation of the resource
group, review all settings (Figure 2). Navigate to
the Review + create tab, where Azure will validate
Figure 2: A view of the Create a resource group form. the configuration. Once validation passes, the Cre-
ate button at the bottom of the panel becomes
become acquainted with its features and function- clickable, allowing you to complete the creation
alities, including where to find the Ubuntu VM process of the resource group.
image for deployment. Understanding the sub- After you click Create, Azure will deploy the re-
scription details and how the Azure portal operates source group according to the specified configu-
is crucial for efficiently managing cloud services. rations. This process may take a few moments,
after which the re-
Resource Group Creation source group will be
From the Azure dashboard, navigate to the Re- available for use, en-
source groups section (Figure 1). Here you can abling you to manage
start the process of organizing related resources and allocate resources
for your Azure solutions by clicking on the + Create as needed for your
button, indicated by a plus sign. Azure solution.

Configuring a New Resource Group Creating a New VM

After initiating the creation of a new resource Once the resource
group, the user is presented with a form to specify group is in place, click
the details of the resource group. You should on the + Create button

Figure 4: Configuration for creating a new VM in Azure,

detailing the subscription and resource group selection, and
Figure 3: The Azure Marketplace search results, highlighting the query for “virtual machine.” instance details such as VM name, region, and size.

[Link] ISSUE 284 JULY 2024 91

LINUX VOICE TUTORIAL – UBUNTU VM IN THE AZURE CLOUD

account and choose whether you want to access

the account using a password or SSH. If you have
an existing SSH key pair, you can use it; otherwise,
Azure can generate a new key pair for you. Config-
ure the inbound port rules to define how the VM
can be accessed. If you’re using SSH, be sure
TSVXɄMWEPPS[IH
Note that the dialog in Figure 4 is just one tab in
an extensive configuration interface. Click through
the rest of the tabs to complete the configuration.
The Disks tab takes you to a dialog that lets you
complete configuration for the VM (Figure 5).
Azure provides a default size for the OS disk
based on the image you have chosen. For most
scenarios, the default size is sufficient, but if you
anticipate needing more space for the OS disk,
you can select a larger size from the drop-down
menu. Select the type of disk based on your per-
formance and cost requirements. Options typi-
cally include Premium SSD, Standard SSD, or
Standard HDD. Premium SSDs are recommended
Figure 5: The Disks tab. for production workloads due to their high perfor-
mance, but they come at a higher cost. Standard
to begin provisioning new resources within the SSDs can be a cost-effective solution for develop-
group. This action initiates the process for creat- ment and test environments, and standard HDDs
ing a new VM. are usually used for infrequent access.
Locate the Virtual machine resource within the Azure offers encryption at rest by default with
Azure Marketplace. This can be done by using the platform-managed keys. You can leave this set-
search bar and typing “virtual machine,” then se- ting as-is for most use cases. If you have specific
PIGXMRKMXJVSQXLIWIEVGLVIWYPXW *MKYVIɄ %^YVI compliance requirements, you can manage your
Marketplace provides a variety of VM images to keys through Azure Key Vault by changing the
choose from. Key management option. If you require high
From the options available, select Virtual ma- throughput and low latency, consider enabling
chine to create a new VM. This selection will lead Ultra Disks compatibility. Note that this feature
you to the configuration page, where you can may not be available in all regions and can incur
WTIGMJ]XLIWIXXMRKWJSVXLIRI[:1 *MKYVIɄ  additional costs.
Confirm the subscription and resource group for The Networking tab lets you define the network
the VM and enter a name for the VM. This name connectivity for the VM, including the virtual
should comply with
Azure VM naming con-
ventions and be unique
within the resource
group.
You can also select a
region that matches
the resource group or
is closest to your loca-
tion, choose an avail-
ability zone if high
availability is required,
and select the desired
Ubuntu Server image
from the list of avail-
able images. Choose a
VM size that fits your
performance and cost
requirements.
You’ll also need to
set up an administrator Figure 6: Creating a new virtual network in Azure.

92 JULY 2024 ISSUE 284 [Link]

 TUTORIAL – UBUNTU VM IN THE AZURE CLOUD LINUX VOICE

network (VNet) and subnets. If you do not have an

existing VNet or need a new one, click on Create
new under the Virtual network section. In the Cre-
ate virtual network pane (Figure 6), assign a name
to your VNet, such as AdminMagazine-vnet1. This
VNet will provide a private network for your Azure
resources. Define the address space in CIDR nota-
tion; Azure may autofill this with a default range,
such as [Link]/16. You can adjust this range
based on your network planning requirements.
Azure populates a default subnet, but you can
add more if needed.
After creating the VNet and configuring the ad-
dress space and subnets, submit the form to pro-
vision the new VNet. Azure will validate and create
the network. Once the VNet is created, select it in
the Networking tab of the VM creation process.
The associated subnet should also be selected by
default.
You’ll need to decide whether your VM needs a
public IP address. For a VM that needs to be ac-
cessible from the Internet, such as a web server,
choose Create new to assign a public IP. If the VM
does not require direct Internet access, you can
select None.
For the NIC network security group (NSG),
choose between Basic for a simplified ruleset or
Advanced for full control over inbound and out-
bound rules. If you’re creating a new NSG, you’ll
need to specify rules for allowing or denying traf-
fic to and from the VM.
The Management tab lets you elect to use the
Microsoft Defender for Cloud service, which
provides enhanced security features and threat
protection. No action is needed if this is already
active. You can also decide whether the VM will
be part of an Active Directory (AD) system. If
you plan to use Microsoft Entra ID (formerly
Azure AD), ensure that the role-based access
control (RBAC) settings are properly configured.
Other management settings pertain to auto-
shutdown, backup, and patch orchestration.
The Monitoring tab lets you configure alert
rules if you want to receive notifications for spe- Figure 7: The Advanced tab and its options.
cific activities or metrics that require attention.
You can also enable boot diagnostics with a
managed storage account to capture logs and
screenshots that can help troubleshoot VM
startup issues, and you can select Enable OS
guest diagnostics to collect additional data from
the operating system.
The Advanced tab (Figure 7) lets you add exten-
sions or VM applications for additional functional-
ity or automation after the VM is deployed. The
Custom Data and Cloud Init section allows you to
pass configuration files, scripts, or data to the VM
upon creation.
The Tags tab (Figure 8) lets you apply tags to
the VM for organizational, billing, or management Figure 8: The Tags page.

[Link] ISSUE 284 JULY 2024 93

LINUX VOICE TUTORIAL – UBUNTU VM IN THE AZURE CLOUD

Connecting to the VM
Azure provides a service that offers secure and
seamless RDP/SSH connectivity to your VMs over
SSL without exposing them to the public Internet.
To set up the Bastion service for your VM, select
Bastion under the Connect section. Provide a name
for the Bastion host and ensure it is within the cor-
rect resource group and virtual network. You’ll also
need to allocate a public IP address to the Bastion
service. Review the cost and click Deploy Bastion
(Figure 10). Alternatively, you can select Configure
manually for more advanced settings.
Figure 9: The dialog box for generating a new SSH key pair. Once the Bastion service is deployed, you can
connect to your VM over the Internet or through a
purposes. Tags are key-value pairs that help VPN for secure access alongside other options
categorize resources and can make it easier to such as SSH and RDP.
allocate costs or manage resources across a
large organization. Conclusion
When you’re finished with configuring the new Microsoft Azure offers easy access to an
VM, click the Review + create tab to review the Ubuntu server VM, and Azure Bastion and SSH
configuration settings. You’ll need to agree to the provide secure access to the virtual system. As
terms of use and enter contact information, and your needs increase, you can add additional re-
then click the Review + create button. sources in the cloud without cluttering your
If you elected to use SSH, you’ll see a dialog box work space. Q Q Q
for generating a new SSH key pair within the Azure
portal (Figure 9). Click on the Download private key The Author
and create resource button. This action generates
a new SSH key pair – the public key will be at- Marcin Gastol is a Senior
tached to your Azure VM, and the private key will DevOps Engineer and
be downloaded to your computer. It is crucial to Microsoft Certified Trainer
with extensive experience in
understand that Azure does not store the private
Azure technologies and
key. Once you download it, Azure cannot retrieve it
teaching various IT subjects.
for you. Therefore, save the private key file in a se- Marcin hosts a blog covering
cure and memorable location on your local ma- multiple IT areas at
chine. You will need this key to establish SSH con- [Link]
nections to your VM.

Figure 10: Azure portal interface for setting up Azure Bastion.

QQQ

94 JULY 2024 ISSUE 284 [Link]

 SERVICE
Back Issues

LINUX
NEWSSTAND
Order online:
[Link]

Linux Magazine is your guide to the world of Linux. Monthly issues are packed with advanced technical
articles and tutorials you won't find anywhere else. Explore our full catalog of back issues for specific
topics or to complete your collection.

#283/June 2024
AI Tools
Everyone is fascinated with AI right now, but at the end of all the articles and interviews and
research, it is fair to ask, what can I do with it really? This month we highlight some AI-based
tools that will help you build your own chatbot, sharpen photo images, and more.
On the DVD: Nobara 39 and Manjaro 23.14 Gnome

#282/May 2024
D-Bus
The D-Bus architecture creates a powerful channel for applications to communicate. A
deeper understanding of D-Bus will help you with troubleshooting. Also, if you know how
D-Bus works, you can customize the interaction of audio tools, text editors, and other apps
to save time and simplify your life.
On the DVD: Kubuntu 23.10 and Clonezilla Live 3.1.2-9

#281/April 2024
Virtual Memory
The classic vision of random access memory is just the beginning of the story. Modern hardware –
and modern operating systems – manage memory in ways that old-school programmers could
only have imagined. This month we take a look at virtual memory in Linux.
On the DVD: elementary OS 7.1 and Mageia 9

#280/March 2024
Plasma 6
KDE’s classic Plasma desktop can be as simple as you need it to be or as complicated as you
want to make it. This month we explore the powerful Plasma 6 release that is making its way
to your Linux distribution.
On the DVD: Linux Mint 21.3 MATE and Zorin OS 17 Core

#279/February 2024
Intrusion Detection
You don’t need a fancy appliance to watch for intruders – just Suricata and a Raspberry Pi.
On the DVD: EndeavourOS Galileo 11 and Arch Linux 2023.12.01

#278/January 2024
Scientific Computing
A crypto mining rig is built for math. Can an old rig find a second life solving science problems?
That all depends on the problem. Also this month, we explore a few popular data analysis
techniques and stir up some analysis of our own with the R programming language.
On the DVD: Kubuntu 23.10 and Fedora 39

[Link] ISSUE 284 JULY 2024 95

SERVICE
Events

FEATURED EVENTS
Users, developers, and vendors meet at Linux events around the world.
We at Linux Magazine are proud to sponsor the Featured Events shown here.
For other events near you, check our extensive events calendar online at
[Link]
If you know of another Linux event you would like us to add to our calendar,
please send a message with all the details to info@[Link].

GUADEC 2024 Flock 2024 Akademy 2024

Date: July 19-24, 2024 Date: August 7-10, 2024 Date: September 7-12, 2024
Location: Denver, Colorado Location: Rochester, New York Location: Würzburg, Germany + Online
Website: [Link] Website: [Link] Website: [Link]
event/209/
Flock is the Fedora Project's annual Akademy is the annual world summit of
GUADEC is the GNOME Foundation’s contributor-focused conference. This KDE, one of the largest free software
main annual event. Held since 2000, year's event will communicate the Fedora communities in the world. It is a free,
GUADEC brings together free software strategy, help you make connections that non-commercial event organized by the
enthusiasts and professionals from all lead to action, and celebrate our successes KDE Community. Join us in Würzburg
over the world. Join us in Denver or and our community. While Flock is open to or online, and meet hundreds of KDE
online to hear about the latest technical anyone, the majority of talks at Flock are contributors and other actors in the
developments, attend talks, participate focused on existing contributors and those open source world.
in workshops, and celebrate GNOME! looking to increase their involvement.

Events
SUSECON 2024 June 17-19 Berlin, Germany [Link]
stackconf June 18-19 Berlin, Germany [Link]
OpenSouthCode June 21-22 Málaga, Spain [Link]
opensouthcode2024
Design Automation Conference June 23-27 San Francisco, California [Link]
openSUSE Conference 2024 June 27-29 Nuremberg, Germany [Link]
useR! July 8-11 Salzburg, Austria and Virtual [Link]
GUADEC 2024 July 19-24 Denver, Colorado [Link]
Flock 2024 Aug 7-10 Rochester, New York [Link]
Akademy 2024 Sep 7-12 Würzburg, Germany + Online [Link]
RustConf 2024 Sep 10-13 Montreal, Canada [Link]
Open Source Summit Europe Sep 16-18 Vienna, Austria [Link]
DrupalCon Barcelona 2024 Sep 24-27 Barcelona, Spain [Link]
CARLA 2024: Latin America Sep 30 - Oct 4 Santiago, Chile [Link]
HPC Conference
Linux App Summit Oct 4-5 Monterrey, Mexico [Link]
Images © Alex White, [Link]

MSP GLOBAL Oct 9-10 Barcelona, Spain [Link]

2024 WISH (Women in Oct 12 San Jose, California [Link]
Semiconductor Hardware) lt-event?id=a1URi000000JahJMAS
All Things Open 2024 Oct 27-29 Raleigh, North Carolina [Link]
SFSCON 2024 Nov 8-9 Bolzano, Italy [Link]
SeaGL 2024 Nov 8-9 Seattle, Washington [Link]

96 JULY 2024 ISSUE 284 [Link]

 SERVICE
Contact Info / Authors

Contact Info
Editor in Chief
Joe Casad, jcasad@[Link]
Copy Editors
WRITE FOR US
Amy Pettle, Aubrey Vaughn Linux Magazine is looking for authors to write articles on Linux and the
News Editors tools of the Linux environment. We like articles on useful solutions that
Jack Wallen, Amber Ankerholz
solve practical problems. The topic could be a desktop tool, a command-
Editor Emerita Nomadica
Rita L Sooby line utility, a network monitoring application, a homegrown script, or
Managing Editor anything else with the potential to save a Linux user trouble and time.
Lori White Our goal is to tell our readers stories they haven’t already heard, so we’re
Localization & Translation
especially interested in original fixes and hacks, new tools, and useful ap-
Ian Travis
Layout plications that our readers might not know about. We also love articles on
Dena Friesen, Lori White advanced uses for tools our readers do know about – stories that take a
Cover Design traditional application and put it to work in a novel or creative way.
Dena Friesen
Cover Image We are currently seeking articles on the following topics for upcoming
© kirillm, Nah Ting Feng, & Oleksiy Mark cover themes:
[Link]
Advertising • Cool Rasp Pi Projects
Brian Osborn, bosborn@[Link]
phone +49 8093 7679420
• Linux Shortcuts and Hacks
Marketing Communications • System Rescue
Gwen Clark, gclark@[Link]
Linux New Media USA, LLC Let us know if you have ideas for articles on these themes, but keep in
4840 Bob Billings Parkway, Ste 104 mind that our interests extend through the full range of Linux technical
Lawrence, KS 66049 USA
topics, including:
Publisher
Brian Osborn • Security
Customer Service / Subscription
For USA and Canada: • Advanced Linux tuning and configuration
Email: cs@[Link] • Internet of Things
Phone: 1-866-247-2802
(Toll Free from the US and Canada) • Networking
For all other countries: • Scripting
Email: subs@[Link] • Artificial intelligence
[Link] • Open protocols and open standards
While every care has been taken in the content of the
magazine, the publishers cannot be held responsible If you have a worthy topic that isn’t on this list, try us out – we might be
for the accuracy of the information contained within
it or any consequences arising from the use of it. The interested!
use of the disc provided with the magazine or any
material provided on it is at your own risk.
Please don’t send us articles about products made by a company you
Copyright and Trademarks © 2024 Linux New Media work for, unless it is an open source tool that is freely available to every-
USA, LLC. one. Don’t send us webzine-style “Top 10 Tips” articles or other superfi-
No material may be reproduced in any form
whatsoever in whole or in part without the written
cial treatments that leave all the work to the reader. We like complete so-
permission of the publishers. It is assumed that all lutions, with examples and lots of details. Go deep, not wide.
correspondence sent, for example, letters, email,
faxes, photographs, articles, drawings, are supplied Describe your idea in 1-2 paragraphs and send it to: edit@[Link].
for publication or license to third parties on a non-
exclusive worldwide basis by Linux New Media USA, Please indicate in the subject line that your message is an article proposal.
LLC, unless otherwise stated in writing.
Linux is a trademark of Linus Torvalds.
All brand or product names are trademarks of their
respective owners. Contact us if we haven’t cred-
ited your copyright; we will always correct any Authors
oversight.
Printed in Nuremberg, Germany by Kolibri Druck.
Chris Binnie 16 Frank Hofmann 73
Distributed by Seymour Distribution Ltd, United Zack Brown 12 Dean Jordan 62
Kingdom
Represented in Europe and other territories by: Bruce Byfield 6, 24, 46 Daniel LaSalle 34
Sparkhaus Media GmbH, Bialasstr. 1a, 85625
Glonn, Germany. Joe Casad 3 Rubén Llorente 28
Linux Magazine (Print ISSN: 1471-5678, Online Mark Crutch 71 Vincent Mealing 71
ISSN: 2833-3950, USPS No: 347-942) is published
monthly by Linux New Media USA, LLC, and dis- Nate Drake 84 Pete Metcalfe 58
tributed in the USA by Asendia USA, 701 Ashland
Ave, Folcroft PA. Application to Mail at Periodicals Marco Fioretti 78 Mike Schilli 50
Postage Prices is pending at Philadelphia, PA and
additional mailing offices. POSTMASTER: send ad- Marcin Gastol 90 Koen Vervloesem 40
dress changes to Linux Magazine, 4840 Bob Billings
Parkway, Ste 104, Lawrence, KS 66049, USA.
Jon “maddog” Hall 72 Jack Wallen 8

[Link] ISSUE 284 JULY 2024 97

NEXT MONTH
Issue 285
Available Starting
July 5
Issue 285 / August 2024

Kernel Exploits
The kernel is the heart and soul of any Linux system,
and if there is a way in, you’ll want to know about it.
Next month we look at kernel exploits and what you
can do to stay ahead of intruders.

Preview Newsletter
The Linux Magazine Preview is a monthly email newsletter that gives you
a sneak peek at the next issue, including links to articles posted online.
Sign up at: [Link]
Image © peshkov, [Link]

98 JULY 2024 ISSUE 284 [Link]