Visual Coding with Processing: COVID-19 Tracker

This project uses the public API for COVID-19 mortality data, and maps the regional totals. So long as you understand the basics of Processing, you should be prepared for the step-by-step instructions in the video.

Get the code on Github. The repository has a improvements on the version in the video, but is largely similar.

A review of Processing books

Processing is the free and open Java development environment that targets artists who are intrigued by generative code. In essence it is the Java language with a friendly development interface and built-in libraries to get you started.

There are plenty of ways to learn Processing, including the tutorials on the organisation's website, and the built-in examples that come with the distribution. But if you prefer a printed book, keep reading. This article will review nine available publications, so you can make an informed purchase decision.

For the sake of completeness I will also append information on two books I haven't had a chance to read.

A crash course in project management

Introduction

A project starts as soon as you have a team of people working towards a goal. The naive approach is to start emailing backwards and forwards. This results in a mess of transactions that hides important patterns. Adding in some shared documents, say using Google Docs, only creates greater confusion. This quagmire is avoided by structuring work through project management software.

But which app? There are several overlapping categories of software, and within each category are dozens, if not hundreds, of potential solutions. This article will help simplify, by explaining the core concepts and defining terms. If you are new to project management, this article should save you a few dozen confusing hours.

Arduino IDE: Best practices and gotchas

Programming for the Arduino is designed to be easy for beginners. The Integrated Development Environment (IDE) provides a safe place to write code, and handles the make and compiler steps that are required to create processor instructions from your C++ code.

This is fine for trivial applications and school exercises. But as soon as you try to use structured code (including classes and custom libraries) on a larger project, mysterious errors and roadblocks become the order of the day.

This article will consider best practices for working within the IDE. I will document a number of common errors and their workarounds. My perspective is of an experienced Python coder who finds C++ full of needless obfuscation. But we can make it work!

How to building a website in 2016

After many years I have updated my personal website. In the process I had to consider what might be the "right way" to build a site in the current era. This post will discuss different options, categorised under five headings.

I have been creating websites for almost as long as there has been a web. For many years I was CTO of a business that did things then thought impossible (namely, large data transfers and processing). For two years I wrote Perl scripts. I have many hours of server configuration in my distant past, now thankfully (mostly) forgotten.

I now require a simple, open, extensible, standards-based solution. I am technically capable and don't mind programming a solution. But any coding must be in Python, since other languages waste my time. (I trust that is partisan enough!)

I will now summarise five different website development methods, as I see them.

1. Static
This is the simplest way to create a website. Create the HTML, CSS, and Javascript manually. Combine with JPG graphics. Upload to any old web host. Done.

The downsides of this method soon become apparent: maintenance, repetition, and sheer drudgery.

2. Generated static
Desktop tools can automate some or all of the process. Templates can be used to provide a basic pattern for a set of web pages. Specific data then fills out each instance. Code modules can be added to provide extra functionality. But the end result is still a set of static HTML files that can be viewed from anywhere.

Back when there were few available solutions I wrote my own programme, Wasp. This app worked in two modes, one of which was generated static files. And I have been using it ever since!

3. Dynamic server, fixed structure
This includes blogging sites like Blogger and most "easy" web front-ends. Content is delivered from a web server using CGI, but the process is completely hidden. The user is responsible for the content, and can modify the visuals to a degree.

This blog is one example. It is written on Blogger, which makes the basic process of posting articles quite easy. But the customisations you see here are the result of many hours of hacking and deciphering secret tech. When I dive into the code now, I have no idea what is going on.

A bigger issue is that the solution is closed. When (not "if") the Blogger engine disappear, my site disappears... and migration may not be an option. The knowledge learnt in one proprietary system is not transferable to another. They are a poor investment.

4. Dynamic server, dynamic structure
This category includes Content Management Systems (CMS), which range from the simple and cheap (Wordpress) to the complicated and terribly expensive (many corporate solutions). These provide a great deal of control over the structure and look of your website, and also easy methods to enter content.

This flexibility requires an increased commitment of time and effort. The same comments as above regarding the proprietary nature apply.

It's a given that a CMS option places constraints on server requirements and is often associated with increased monthly costs. It also restricts the choice of hosts. But these days the most popular solutions are supported by a variety of vendors.

I have developed client sites in Textpattern, which is open and free. Years later, these sites are still functional, but eventually will fade. All sites require maintenance as the underlying software versions change.

A CMS is a complicated beast. If you are building sites for a living, the investment you make in learning one might well pay off. Your customisations will be re-used often enough to be worth the time it takes to develop them.

But this isn't the case if you're using a system only occasionally. You will never remember all the little techniques and "gotchas". (That's why there's a thriving industry of Wordpress consultants.)

5. Dynamic framework
These code skeletons provide hooks into the web server and some useful scaffolding. They incorporate a CGI interface, so that dynamic websites can be served from Apache and the like. They assume that you have considerable programming skills, since you will be coding your own website structure and functionality.

My own Wasp was a workable solution. But it was made in an era when standard interfaces like WSGI did not even exist. It is out-dated and inferior to contemporary tools.

Comparing CMS
I spent a good deal of time evaluating the Python-based CMS solutions, but I am only going to present a summary here. The complexity of these systems, each with their own terminologies and assumptions, makes any comparison time-consuming and potentially error-prone.

The biggest issue is the amount of commitment necessary to properly review a product. Something simple and flexible at first glance may turn out to have hidden limitations.

We can break a CMS down to three areas of functionality: the templating system, data layer, and coding framework.

I am picky about templating. Certain products, typified by CherryPy, use "Pythonic" templating. What this means is that you mix in your code with your HTML template. To me this is a bad idea. Anything in a template should look like HTML (or meta-HTML). The Python code should stay in the code engine. This is doubly true in team development, where one expects a division of labour between the designer and programmer.

However, pragmatics comes into play. Sometimes it is easier to put logic in the template. A good solution will not require this but will support it.

Luckily, the situation in the last few years has stabilised and it seems that Jinja is the winner. This clear and powerful templating language is used in all of the CMS that made my final cut. The same was not true even a year ago, so be wary of reviews on the internet that are stale.

When it comes to the data, some products (eg. Django) bundle in an Object Relational Mapper (ORM), which is a way of translating from object-oriented code to set-oriented information in a database. My experience with these has been tainted, since they are terribly inefficient for large data sets. But for a website I am sure they are just fine. Nonetheless, I prefer that simpler options are also available.

Some coding frameworks are massive and bundle in as much functionality as possible (Django). Others are minimal and expect the developer to fill in the rest (Flask, Pyramid). I prefer the latter approach, since I won't be trapped by pre-made decisions.

This is true even if there is a rich ecosystem of developed plugins. There are several problems with using a plugin to add functionality. First, the plugin architecture itself my be restrictive or require excessive cognitive overhead. Second, you have just added a point of weakness, especially when it comes to security (Wordpress, cough). Third, you now rely on the plugin developer for timely updates as the main system evolves.

I will spare you further details of the forty or so systems I investigated. For me, Flask would be the system I'd choose when developing a dynamic website. But I'd keep an eye on Pyramid as well.

However...

Reconsidering
For a personal site, do I need dynamic content? Well, what does it have to do? (Have a look.)

I need to display a snazzy home page incorporating a couple hundred informative entries on my various activities. I want to filter these articles by category and show related articles for each, based on tags. The articles need to be sorted in reverse chronology, like a blog. And I will be adding more all the time.

So, yes, it does at first seem like dynamic content is needed.

But, in fact, all of this can actually be done with static files. The different views of the data can all be determined ahead of time (so long as there are not too many). Certain changing content can be provided on the front-end using Javascript, without requiring server-side actions.

What is needed is a darned good static site generator.

So I wrote one.

(To be continued.)

Bootstrap 4 column templates

I am building a new website. Each time I do this, the underlying technologies evolve and I play catch-up.

The big problem has always been supporting the variety of software platforms in use. It's a significant challenge creating a unified experience for users of different web browsers on different operating systems. The recent proliferation of hand-held devices has increased the number of software platforms (Android, iOS), the number of rendering engines, and the variety of screen sizes in use.

It is imperative to use a platform built to unify the common technologies of HTML, CSS, and Javascript. All the messy details should be hidden in the implementation, so that a web designer can focus on their structure and content.

In addition, the principle of responsive design holds that one single code-base should suffice for all viewer implementations.

That is why I am using Bootstrap, and specifically the new version 4 release of this platform. (Though this is still in alpha, I don't want to dedicate time to version 3, knowing the structure is imminently to change.)

Though the internal docs are good, there isn't much third party documentation on Bootstrap 4. I've been experimenting with layout and this article is the result.

How to retrieve photo metadata in Python

You'd think it would be easy to retrieve and even edit photo metadata. After all, we are living in the 21st century. But, no, some things prove more difficult than they should. A search for applications turned up quite a few that would display the metadata but none that would easily edit it.

OK, so there's always the programmatic approach. And for that I turn to Python. Let's see what the state of the art holds for us. (Hint: It's a bumpy ride.)

SendEmail automates boiler-plate email messages

Last week I needed to email out marks to my students. How to automate a bulk email project? No doubt the Internet will turn up hundreds of spyware-infested programmes of dubious lineage and intent, all designed for just this purpose.

So I turned to Python. Because if I write my own code I can trust it to do exactly what I wish, no more, no less. And so memories of Paul's first Python programme came flooding back... ah, sweet nostalgia! If you have no idea what I am talking about here, read Sunday's article. In it, I presented the minimal four line Python snippet that sends email. In this article I will develop that core into a fully-functional command-line application.

Python... and a letter to an editor

Python, the programming language, seems to be in the news more than usual. Or perhaps it's just the "Python For Beginners" article in Communications of the ACM that got me thinking. The article seems unreasonably critical for one ostensibly promoting the language. And so, for the first time, I've written a letter to the editor of that esteemed journal. More on that in a moment.

If I think through all of the technologies that have made my life easier and better, the first one that comes to mind is, quite naturally, the Internet. The second is Python, created by Guido van Rossum. Allow me to reminisce for a moment.

Reaktor reaches end of life?

I have been a big fan of Reaktor for some time, and use it exclusively in my live work. Recently I have encountered some absolute stupidity on the part of its developer Native Instruments. And this is forcing me to rethink my commitment.

I started with Reaktor back in version 3 and have been hacking away steadily for 15 years. It's a fantastic system for developing your own instruments, effects, and audio utilities. One does this either by hacking together pieces from existing tools or simply building virtual circuits from scratch in a graphical interface. It's a lot like Max/MSP except easier to get started, and with thousands of ready-to-go ensembles, many of excellent quality. Back when I started all this, Max had a poor interface and provided no way to customise the look and feel for performance. Also I didn't like the sound. Both of these factors are no longer the case, but still Reaktor has the edge in some departments. (For one, it works as a VST so it is easy to integrate with any DAW.)

Reaktor Multitimbral Sampler Released

= strata = is a sample layering instrument can be used as a multitimbral sampler, a generator of ambient soundscapes, or a synthesis engine in its own right. Included is a sample map and three presets to get you started. It is built in Native Instruments Reaktor and requires the full version of NI Reaktor to run.

However the instrument itself is completely free and licensed under Creative Commons. Registered users can download it from the Reaktor Library. Read on for details.

Blogger "Related Posts" Code

In a previous post I noted that there is no easy way to write conditional code based on which categories a post belongs to. In Blogger a category is called a "label". In this site I use labels to denote which section an article belongs to. So for me "category", "label" and "section" are the same thing. (Just saying, in case my terminology gets confusing!)

This is an issue because before now, if you were reading a specific post on this site, you would have no indication of what section that article belonged to. I wish to not only make this obvious, but also present other page code based on the current section. I found the hints of a solution from an article on WebUpd8. As a step towards integration, I decided to implement their method of listing Related Posts. This article will present my optimisations to their code.

Simple Tips For Editing Blogger Code

Blogger sometimes has a mind of its own. If you visited here recently you may have noticed that there were two sets of social networking buttons below each post. That is because the default Blogger set were showing up alongside the custom set I had inserted -- even though I had told Blogger not to display their own. Nothing I could do in the Page Elements screen seemed to effect this, so instead I deleted the offending code directly in the Edit HTML section. I'll now offer you some simple tips for making this editing process safer and easier.

Pyglet: Audio Support

I promised I would cover audio support in Pyglet, even though, when all is said and done, there is little to say. Support is quite rudimentary, though what is there works well. I was able to write a simple application on Windows 7, test it on LINUX and then install it for OS X without any bugs, code changes or even configuration issues. That is cross-platform programming as it should be. (Though I would accept the need for config changes to address hardware-specific issues.)

Audio and video playback is handled by the same package, pyglet.media, which automatically detects the media type and acts appropriately. As I mentioned in Pyglet: Helpful Tips, you will want to install the "optional" AVbin to support anything other than plain uncompressed formats.

Though you might want to refer to the API docs for all the details, I'll explain here how to set up sound playback. In brief, you use a Source instance to decode an audio file and then queue this on a Player for playback. By default this decoding is done on-the-fly as a StreamingSource. If you want a sound to be decoded in advance, you can designate it as a StaticSource. This is useful for short sound effects and the like. There's not much you can do with a source except read its duration.

A Player can be used to play, pause and restart a sound. You can get the current playback point and seek to a position in the source. There is support for pitch shifting and attenuation, but I did not try these. There is no time stretching, however, and even basic filtering or EQ is outside the realm of what this simple library is designed to do. I would like to see these features, as well as the ability to cross-fade sources, as this would provide commonly needed functionality.

Here is the basic code I used:

import pyglet

wavfile = 'some/path/to/file.wav'
sound = pyglet.media.load(wavfile)

core = pyglet.media.Player()
core.queue(sound)
core.play()


There is only one event provided for responding to audio, on_eos(), which triggers when the player has reached the end of the current source. It would be great to be able to set up triggers at other times (so one could have advanced notice of the approaching end of a file) and on other events (for example, pausing).

This library for sufficient for my simple case, but I would have to look at something more complete for more than the rudiments... a search that still continues!

Running Python Scripts on Mac OS X

[Warning! This is complete newbie information!]

One of the things I found difficult when first using Python on a Mac was simply getting my scripts to run as programmes, with a simple double-click. The nub of the problem is that one must run a script using the python interpreter, like so:
python script.py

This is easy enough to do in a command window, terminal window, at the shell prompt etc. (The terminology is different depending on the operating system, but the reality is the same.)

On Windows I would write a one-line batch file containing the command above. This could be then be treated in all ways like a self-contained application. Simple.

I had difficulty making anything like this work on Mac OS X until I did the following three things:

1. Make the script executable:
chmod a+x

2. Ensure the first line in the script is a "shebang line", telling it where to find the interpreter:
#!/usr/bin/env python

Er, hold on, that looks correct for LINUX but not the Mac. Anyway, be sure to put in the proper path!

3. Associate the script file with PythonLauncher. One does this by right-clicking (option-click) the script, selecting "open with", and then finding this programme.

In my case I had difficulties because the first PythonLauncher to be available in the menu was actually for an older Python version. It is quite common on the Mac and LINUX for there to be multiple versions of Python. This is because the operating systems come with one installed, and it is often best to leave that in situ so that any system tools that need it can still use it properly. But one inevitably wants to upgrade to the latest greatest version. The solution is to install this in parallel.

I had my script running with Python 2.3 when it needed 2.6. I got no error or indeed any feedback that something was wrong. This took me a while to figure out!

I had a second big problem, but that will wait for my next article.

Pyglet: Helpful Tips

In my last article I introduced Pyglet, a small and elegant cross-platform library that implements OpenGL, 2D sprites and a complete windowing interface in minimal Python code. In this article I will outline a few issues I had setting up the library. I hope that in this way I will save others time and headache.

These issues came about due to the fact I am running a 64-bit version of Windows 7. It seems that many programmes are still not ready for 64-bit operating systems, and if they are it can still be quite confusing getting and installing the correct versions.

To start with, the Pyglet download page makes available a Windows installer in MSI format. This failed to find my Python installation. Apparently this is a common issue; the solutions I found online involved copying registry entries from one place to another. Unfortunately the specific entries mentioned were not to be found on my system, so I abandoned this approach.

A simpler solution was to grab the source distribution and run "python setup.py install" to do all the work.

This appeared to do the trick, but the example applications I tried failed with various errors. Then I resorted to the test suite (very nice that one is included) and this could not even create a window. Houston, we have a problem!

From the support group I determined that Pyglet supports 64-bit Windows but does not support 64-bit Python. You need to have 32-bit Python running on 64-bit Windows to use Pyglet. With the correct version of Python 2.6.4 installed, everything went fine. It sure would be nice if somewhere, in the install docs or FAQ or on the download page, this was mentioned.

My second issue had to do with the declaration on the homepage that Pyglet has "No external dependencies or installation requirements". We have just seen that the last part of this is not true, but neither is the first part, in any real sense. A couple sentences later we find out that "pyglet can optionally use AVbin to play back audio formats such as MP3, OGG/Vorbis and WMA, and video formats such as DivX, MPEG-2, H.264, WMV and Xvid".

What is going on here is a matter of expectation. Other Python graphics toolkits have one or more external dependencies that require installing or compiling third-party libraries. Sometimes this can be quite an involved process. Further, these dependent libraries are not always free or open source. So long as one is using only uncompressed media, Pyglet is free from these dependencies. So, relative to other windowing libraries, this is in fact a Pyglet advantage.

However, to do any real work one will soon need to support some of the common compressed file formats and codecs that AVbin gives access to. (It does so by wrapping the extensive FFmpeg library.)

The AVbin site has one download for Windows, a file named "avbin-win32-5.zip". Hmmm.... does this mean it will not work on 64-bit Windows? I thought I'd give it a try. The ZIP contains only a readme and a DLL file. The former says to place the latter in your "\windows\system32" folder. I did this but applications that require AVbin still failed to work.

Pyglet has a Google Group to provide support. It averages about 150 messages a month. It is easy to get spoiled by groups that provide six answers within 3 hours; this is not one of those, so patience is required. My first query was answered in 12 hours, so I dashed off another request for information. This time it took 24 hours before I was told I would need to put the DLL in my "WOW64" folder. I found the similarly named "\windows\SysWOW64" folder. I put the DLL there and it worked perfectly.

The lesson to be learned is that if you want a 32-bit DLL to work in Windows 7 64-bit then put the file in SysWOW64.

There are two things that should be improved here. First, the zip file should be named in a way that makes it clear that it will in fact work on win64. Second, the readme needs more complete installation instructions.

Getting back to Pyglet itself, it would be marvellous if the correct version of the DLL could be included with the installation, saving a step and making the programme truly self-contained. (Actually, this is in fact the case for the full installer version, but not the manual process I needed to use.)

Better yet would be if AVbin could be dynamically linked as a Python library (a PYD file). This would have the distinct advantage of being loadable from elsewhere in the file system. No mucking about with Windows folders would be necessary and it would then be easier to distribute finished applications to clients.

Does someone want to create a PYD from AVbin? I have no idea what that entails. But it would be a real boon!

(I think that one disadvantage is that PYD files are specific to Python versions. This creates a maintenance issue.)

Pyglet: A Python Gaming Library.

In a previous article I noted the difficulty in finding a complete, well-documented and active audio library for Python. That was two and a half years ago, but the situation does not seem to have improved. Many of the libraries are moribund or have no developer community. Hardcore coders can piece together what to do from doc strings embedded in the libraries themselves, but the rest of us are left waiting.

One of the anonymous comments on that article mentioned Pyglet. In this series of articles I will provide an overview of that library with a special look at its audio facilities. I will highlight any limitations I have come across in the short time I have been working with it. My viewpoint is that of the casual programmer who wants to pick up a tool and get work done. I do not code every day or for its own sake, but to get real tasks accomplished efficiently in the real world. That is one of the reasons I love Python; even after a year away all the syntax comes readily back to mind. My goal is to find tools that are equally efficient.

You can tell a lot about a tool from its website. The Pyglet site provides detailed API documentation and a handy walk-through guide. Sample applications come with the install. This makes it a lot easier to get started than with many other small tools. (As far as I can tell, Pyglet is the work of a single developer.) However, the FAQ is minimal and many "before sales" questions are not answered, especially those concerning scope and limitations. I find this common enough for developer tools, though it is frustrating. Why would I choose Pyglet over other tools? When would I not want to use it? These are the questions a website should answer simply.

I will say from the beginning that Pyglet has only minimal audio facilities. There is nothing in the way of signal processing, audio effects or algorithmic composition. But as a quick way to play back existing audio in a nice interface, it might be just the ticket.

But now, on to the library!

Pyglet is a game development library for Python that supports Windows, Mac OS X and LINUX. In this day and age I treat those as a minimum target. As a test I will be developing a simple app on Windows 7 64-bit for testing on LINUX and distribution on a Mac mini. This is not an arbitrary test; these are the real-world requirements of the programme, dictated by the availability of computers at my disposal.

Pyglet includes a windowing interface library along with keyboard and mouse detection. Text output can be formatted using a subset of HTML (which does not support CSS, however) or Pyglet-specific markup (about which I could find little). You can even work with fonts and glyphs at a primitive level if you need to design your own.

Images can be read from files or file-like objects, so you can easily store your programme resources in an archive, for instance. In fact the module "image.atlas" provides special support for image bins. A full image or some part thereof can be displayed and transformed within a particular view-port or projection. Images have full sprite control (position, scale and rotation). The demo applications include a convincing clone of the classic Asteroids game.

Pyglet is notable for supporting a complete OpenGL and GLU functionality set, including basic graphics primitives. I am certainly in no position to verify this claim but it seems that anyone familiar with OpenGL can get right to work.

A clock module provides not only scheduled events at specific time slices, but can calculate or limit the frame rate of your application.

Some time has been spent solving common application development problems. For example the "resource" module handles locating data files relative to the application root. Everyone coding in Python ends up writing something similar, but it's good to see the application framework support it.

Without further ado, here is the canonical Hello World programme, taken from the examples provided.

import pyglet

window = pyglet.window.Window()
label = pyglet.text.Label('Hello, world',
    font_name='Times New Roman',
    font_size=36,
    x=window.width//2, y=window.height//2,
    anchor_x='center', anchor_y='center')

@window.event
def on_draw():
  window.clear()
  label.draw()

pyglet.app.run()

We first define a window instance and a label instance. The "@window.event" decorator provides an easy way to modify the class methods, in this case the "on_draw" event. The "app" module hides the application event loop in a single line of code.

And that is all! I don't think I have ever seen a simpler windowing application framework.

This should whet your appetite for what seems to be an uncommonly useful little library. In the second article I will look at some of the configuration "gotchas" I referred to above.

just stopping in to dust the shelves

Don't forget I am still very active over on Theatre of Noise, even if this site has been moribund for a year. I'd like to tell you I've been spending my time researching Quantum Computing, but the truth is even more fantastic... I've been taking photographs and composing music.

I also took a year to get my Masters. This in fact did involve a significant amount of programming but I was too busy getting grades to write here.

Maybe later.

So You Want To Learn Python?

First, let me congratulate you on wanting to give Python a try! I have used many languages but always return to Python as it is the easiest to read, the fastest to programme, works on any platform and is bug-free. It's very much like an easier Java, in the same way that Java is an easier C++. There are many tutorials and books available to ease you into the language. Read on and I'll recommend my favourites.

Csound Revisited

From my silence here you might imagine I've been doing no programming at all. Nothing could be further from the truth. While it is true that my commercial and open source projects are pretty well at a standstill, I have been working with a variety of languages for music composition and processing, so you can soon expect some sort of follow-up to the articles I wrote on music control languages, Chuck, etc.