Mike Walmsley

Zoobot v2

Fri, 05 Apr 2024 09:00:00 GMT

Zoobot 2.0: Scaling Up

This blog summarises the paper Scaling Laws for Galaxy Images. Use our foundation models with the Zoobot 2.0 Python package. Colab, code, docs, models, embeddings.

All my best work has come from giving a talk, getting asked a good question, and replying with totally the wrong answer.

I spent the last couple of years making and then refining a set of AI models called Zoobot. Zoobot is designed to be adapted to new galaxy tasks with minimal new labels. I often call Zoobot a foundation model ¹. But after my talks, people often say: “Zoobot works, but it’s not really a foundation model. It’s too tiny! Foundation models should have billions of parameters. What happens if you make it bigger?”

Like literally every AI researcher in the world, I enjoy training big models. Of course I tried making Zoobot bigger. It didn’t help much. I might even go further and say

“There is no evidence that bigger AI models work better in astronomy”

It’s clear that big models do better when fueled by limitless labelled data (think billions of web-scraped images with alt-text). But astronomers work with thousands of labelled images, not billions. Even ImageNet, with 1.3M images, is about four times larger than the largest astronomy equivalent (GZ DESI). I think that’s why there are no astronomy papers showing a convincing² improvement from (say) large vision transformers.

But if data is the limiting factor - what if we scaled that as well?

Pretraining Experiments

I spent (insert silly amount of time here) pulling together a dataset of 800k+ galaxies with 100M+ Galaxy Zoo volunteer annotations. That’s more than half of all human galaxy annotations ever collected. Then I trained models to predict those annotations. I systematically worked through every popular architecture (ConvNexT, EffNetV2, MaxViT, etc.) and every model size, from 1M to 200M parameters. I then I did it twice more for error bars.

Here’s what I found.

Adding data predictably improves performance for every model

We expect that more labels help. What’s interesting is that the amount of improvement for each label added is totally predictable. For every architecture, we see an almost-identical power law relationship between labels and performance. This means we can predict how much our models will improve as we add new labels.

Each time we double the training data, we get the same improvement in performance. But we can’t keep doubling training data; this paper used more than half of all the annotations ever collected. There’s not much left to add!

Bigger models do better, up to a point

When we use a quarter of our dataset, training bigger models does almost no better: they overfit. But our full dataset provides enough labelled galaxies to meaningfully train models up to around 100M parameters. Our best models (ConvNeXT-Base and MaxViT-Base) outperform the small (5M) EfficientNetB0 model used for GZ DECaLS and GZ DESI.

Beyond around 100M parameters, we start overfitting again. We could add some tricks like learning rate scheduling, aggressive regularization, etc., but it’s clear that we can’t scale indefinitely and expect better performance “for free”.

More data helps every task, but more parameters help only some tasks

We can measure performance separately for each of our Galaxy Zoo questions (88 in total). Adding data improves performance on every question. Remember the power law above? That applies to individual questions! But adding parameters only helps for some questions (possibly the more subjectively challenging ones).

This is subtly different from the typical view that models are either data-limited or parameter-limited. Our models are data-limited for all questions and parameter-limited for some questions.

Finetuning Experiments

I then finetuned our models on five real scientific tasks: finding and characterising faint tidal features, classifying JWST galaxies, detecting ringed galaxies, and (okay, slightly less real) galaxy morphology in Galaxy10 DECaLS.

We can do well at new tasks with just hundreds to thousands of labelled examples

For every new task, the galaxy-pretrained models did better than models pretrained only on ImageNet (31% better, on average). They were good enough to do useful science on every task.

Galaxy pretraining was most helpful when finetuning labels were scarce. This is exactly what Zoobot is for!

For new tasks, pretraining on galaxy images is more important than using big models

Bigger models did measurably better at the finetuning tasks, but the improvement was much smaller than from galaxy pretraining. A large ImageNet-trained model is much worse than a small galaxy-pretrained model, and a large galaxy-pretrained model only adds a little extra benefit.

That suggests that if you want good performance on galaxy images, you should focus on using a similar pretraining dataset rather than downloading the biggest possible model and hoping it works.

Where Next?

Alexander wept when there were no more worlds to conquer, and I weep now I’m out of human labels. Bigger models will require other techniques.

Self-supervised learning is an obvious choice. I and my collaborators have been experimenting with this for a couple of years (e.g. here, here). Scientific foundation models are attracting ever more interest and are still in their infancy in astronomy. I’m excited to see what we build together.

The other side of the coin is finetuning. We can now use the new labels Galaxy Zoo volunteers are creating to quickly build specialized models, including for difficult tasks like segmentation. Every label is now more valuable, not less. What new questions might we answer?

If you’d like to join in, reach out - mike.walmsley@dunlap.utoronto.ca. I’m also part of the deepskies and universeTBD open AI communities.

I’m grateful to Prof. Anna Scaife for encouraging me in building Zoobot, even when it meant spending more time writing code than papers. It was a gamble and I’m glad it’s paying off. I’m also grateful to the Dunlap Institute for continuing to support my work building these tools for the community and applying them to Euclid. Finally, I am, as ever, grateful to the Galaxy Zoo volunteers who make this all possible.

In the first Zoobot paper, I studiously avoided using the phrase `foundation model’ because it was controversial at the time; some researchers felt it dangerously implied that AI models could be a solid foundation. Instead I just described what they are: “adaptable” models that “learn meaningful semantic representations of galaxies that are useful for new tasks on which the models were never trained”.
↩
With the possible exception of Daglit 2023, who recently trained a large CoAtNet-like model (~300M params) on Galaxy10 DECaLS (27k images) and hit about 94% top-1 accuracy. That’s much better than baselines with typical models, and about the same as another paper focused on using equivariance (and small models). I’m doubtful that the underlying Galaxy10 DECaLS labels are 94% accurate - but that’s a topic for another blog.
↩

Dunlap Fellowship Plans

Sat, 15 Jul 2023 23:46:37 GMT

I’ll be starting as a Dunlap Fellow at the University of Toronto from September 2023. I’m hyped to start officially running my own research program.

In this post, I’d like to share what I’m hoping to do. The truth is I can’t do it on my own. If you’re interested in building shared software tools across surveys, in crowdsourcing and human-AI collaboration, and of course in applying deep learning to answer science questions, then please reach out. I’d love to chat.

Detailed Morphology for Euclid and Rubin

I want to measure the detailed morphology of every resolved galaxy in Euclid and Rubin.

I’ve spent the last four years building the tools to make this possible. The keystone is Zoobot (paper). Zoobot trains models that accurately answer every question we’ve ever asked at Galaxy Zoo. These “foundation” models can then be adapted to new problems and new surveys using very little new labels. Other folks have adapted Zoobot models to find interacting galaxies in HSC and the Hubble archives, segment bars in SDSS, locate clumps in DESI galaxies, find interesting anomalies like strong lenses, and more.

Zoobot is effective because of the training data. It’s not a large model - the current best-performing architecture is MaX-ViT Tiny, with 30M parameters, and larger architectures don’t (currently) perform better on my benchmarks. But it’s trained on a unique pretext task - 92M human clicks from Galaxy Zoo volunteers selecting an answer to a Galaxy Zoo question. Predicting the answer to any question is a broad task (like predicting the next word for any sentence). This broad task helps Zoobot learn a general representation that’s easily adaptable to new questions.

My Fellowship plan is less about trying the very latest architectures (that’s easy thanks to Zoobot’s timm support) and more about thoughtfully including more data and more pretext tasks. Practitioners sometimes call this data-centric ML.

The outline below is simply Plan A for applying Zoobot to Euclid and Rubin. Perhaps you have better ideas - if so, tell me!

Label-efficient Learning: Contrastive Learning, Active Learning, and Domain Adaption

I want to have a morphology catalog ready on the day of Euclid’s DR1 release (Dec 2024, internally).

First, we can use contrastive learning to exploit unlabelled images. We’ve tested this on DESI and on Radio Galaxy Zoo and found that contrastive learning reduces the amount of labels needed to achieve a given performance. We did this with BYOL, and might next try some more recent frameworks like MAE/DINO.

Second, we can use active learning to select the most informative galaxies to label (from Euclid’s preview release, DP1). I first tested this back in 2019 for SDSS galaxies (in collab. with Lewis Smith and Yarin Gal, from Oxford CS). Since then we’ve simplified the system and put it serverside on the Zooniverse. We ran it in anger from GZ HSC and we’re now testing it again for GZ JWST.

Third, we can use domain adaption to make our models more robust to differences between surveys. Work by others, particularly A. Ciprianovic and Y. Mes, shows domain adaption can be effecive for galaxy images. I haven’t tried this yet!

Going Beyond Classification: Segmentation, Personalised Anomaly Detection

We know we can build accurate classification models, especially with many (50k+) labelled galaxies. Our DECaLS and DESI models are as accurate as 5 to 15 human volunteers. So let’s up our ambition. Let’s do segmentation.

Accurate classification models must be able to recognise morphological features within the image. So in some fundamental sense, they must “know” where those features are. Work by P. Bhambra showed this for barred galaxy classifiers using SmoothCAM. Separate early work by A. Spindler shows UNet works straightforwardly on galaxy images. Let’s extend our pretrained classifiers to do segmentation as well. I have a small grant from Meta to develop annotation tools with human-in-the-loop active learning (i.e. you start drawing on the image, and the model completes your brushstroke).

I’m especially keen to do segmentation for low surface brightness features. Unlike detailed morphology in general (bars, spiral arms, etc) accurately identifying LSB features is an unsolved problem. I think segmentation annotations will be much better for this than classification labels, because these would explicitly tell the model which small part of the image contains the LSB feature. I’m putting together a cross-survey collaboration to give this a try.

Get It Right: Benchmarking, Annotators as Individuals

Visual galaxy morphology measurements from volunteers currently lacks the statistical rigor of, say, weak lensing inference. I think this is partly because, unlike weak lensing, we have historically not been able to simulate datasets where the right answer is known. Comparing to expert labels has been used as a work-around - but there simply aren’t many large expert morphology catalogs for modern (post-SDSS) surveys. If we want visual morphology to be taken as a serious quantative galaxy property in the 21st century, we need to know how often we get the right answer. We could do this with modern galaxy-scale sims (such as FIRE) with realistic synthetic images and particle/kinematic or new expert labels.

If we can create a set of galaxies with known “ground truth” labels, we can then adapt our pipeline design to best recover these. I’m especially interested in understanding how individual volunteers learn to answer different questions. When collecting labels we could add personalised advice and examples to guide volunteers who struggle on a question. And we could build models that understand that annotators (both volunteer and expert) as individuals with individual preferences.

Closing Appeal

They say that man makes plans and God laughs. Perhaps I’ll look back on this post as a moment of hubris. But I think everything I’m pitching here is technically feasible. If we can pull it off, we’ll transform our understanding of why galaxies look the way they do. And along the way we’ll enjoy a unique playground for building human-AI collaborative systems.

It’s a lot to do. I’d love to put together a team of people interested in working on this together. If you’d like to join in, reach out - mike.walmsley@dunlap.utoronto.ca.

What exactly do HDBSCAN's parameters do?

Mon, 08 May 2023 23:46:37 GMT

The HDBSCAN docs have an excellent high-level overview of how the algorithm works. But they don’t explain exactly how the HDSCAN code parameters fit in. This is my best effort at working that out.

`min_points`

HDBSCAN constructs a graph linking all nearby points (single linkage tree). But we don’t want to accidentally link two clusters with a single random noise point halfway between. To avoid this, we should try to move noise points further away from other points. That’s done by defining a new distance measure - the mutual reachability distance.

core distance: distance to min_point’th nearest neighbour mutual reachability distance: max(core(a), core(b), distance(a, b))

If a or b is in a high density area, core(a) or core(b) is low and MRD=distance i.e. no effect.
If a and b are in low density areas, core(a) and core(b) are high and MRD > distance

So the general effect of using MRD as a distance metric is to push points in low density areas further away from points in high density areas. Use this distance to safely construct the graph without accidentally including noise points.

As you increase min_samples, this increases how many points count as a high density area. Consider two points in a moderate density area. If min_samples is low, distance(a, b) > core(a) and core(b) and MRD=distance(a, b) i.e. MRD does nothing. But if min_samples is high, core(a) and core(b) > distance(a, b) and so MRD > distance(a, b) i.e. MRD is effectively pushing the points apart before constructing the graph. Greater min_samples means you need more neighbours to avoid having your graph distance (MRD) penalised.

Further, min_samples affects whether you’re ultimately considered in a cluster at all. Points are considered noise once the distance (edge weight) being dropped from the graph reaches that points core distance (i.e. distance to that point’s min_point’th nearest neighbor).

`min_cluster_size`

This is a bit more intuitive, thankfully.

We have the graph and now we’re cutting edges to create clusters. Edges are cut in order of longest first. Clearly we can count the number of connected nodes as we go. But not every set of still-connected nodes should be considered a cluster - some will be just a few points falling away. When cutting the graph, how many connected points should count as a cluster?

If you make a cut and the now-split part of the graph includes more than min_cluster_size nodes, consider the cluster as split - we now have two clusters, each with more than min_cluster_size points.

If, instead, fewer than min_cluster_size points were disconnected, consider that not a true split creating a new cluster - we just lost a few points in the course of refining our one cluster.

I hope this helps. Good luck!

Hooks in PyTorch Lightning v2

Sun, 19 Mar 2023 23:46:37 GMT

PyTorch Lightning v2.0 has made two major changes to pl.LightningModule methods.

First, training_epoch_end(outputs) methods have been removed in favor of on_training_epoch_end().

Notice how the _on function is not quite the same: the outputs argument has been removed, requiring users to explicitly store outputs as a self. property. The release notes have a good example:

Before:

class LitModel(L.LightningModule):
    
    def training_step(self, batch, batch_idx):
        ...
        return {"loss": loss, "banana": banana}
    
    # `outputs` is a list of all bananas returned in the epoch
    def training_epoch_end(self, outputs):
        avg_banana = torch.cat(out["banana"] for out in outputs).mean()

Now:

class LitModel(L.LightningModule):
    def __init__(self):
        super().__init__()
        # 1. Create a list to hold the outputs of `*_step`
        self.bananas = []
    
    def training_step(self, batch, batch_idx):
        ...
        # 2. Add the outputs to the list
        # You should be aware of the implications on memory usage
        self.bananas.append(banana)
        return loss
    
    # 3. Rename the hook to `on_*_epoch_end`
    def on_training_epoch_end(self):
        # 4. Do something with all outputs
        avg_banana = torch.cat(self.bananas).mean()
        # Don't forget to clear the memory for the next epoch!
        self.bananas.clear()

The idea is to make it obvious that using epoch_end actions requires all the batch-level outputs to be stored until the end of the epoch rolls around.

If you were using training_epoch_end(outputs) to update your TorchMetric metrics, as the TorchMetrics docs suggest (my issue), this will now raise an error.

You might think you could replace the TorchMetrics updates in the similar training_step_end(outputs) methods. Unfortunately, and much less obviously from the release notes, these have also been removed - and will now silently do nothing.

Instead, you can use on_train_batch_end(outputs). For example, in Zoobot:

def on_train_batch_end(self, outputs, *args):
    self.train_loss_metric(outputs['loss'])
    self.log(
        "finetuning/train_loss", 
        self.train_loss_metric, 
        prog_bar=self.prog_bar, 
        on_step=False,
        on_epoch=True
    )

This appears to work well for multi-GPU training. It also avoids storing batch outputs in memory. Overall I’m grateful to the Lightning team for the new release - I hadn’t noticed the previous memory-consuming behaviour before.

The Zoobot changes are now on dev and will be live on main and PyPI in the next few days.

Typesetting Figures for MNRAS

Tue, 03 Jan 2023 23:46:37 GMT

Figures look best when they match the style of the document. How can you make matplotlib figures that match the style of journal paper?

I mostly publish in MNRAS, so I’ll use that as an example, but this approach should work for any latex paper format (ApJ, NeurIPS, etc.).

There’s two parts that need adjusting:

graphics elements: primarily figure size, which affects line widths, tick spacing, etc.
text: primarily the font and size.

The same figure, before (above) and after (below) changing matplotlib's settings to match the journal style.

Graphics Elements

The first and most important step is setting the right figure size. Everything else follows.

The default matplotlib figure size is 6.4 by 4.8 inches. To fit in a two column paper format, figures need to be about half that default size. I used to resize the final figure image with [width=\columnwidth]. But this makes everything proportionally smaller - the text, the line widths, etc. I think this is why a lot of student reports (including mine) have borderline-unreadable figures.

Instead, make the figure the correct size in the first place. You can do this by trial and error, but it’s easiest if you know the actual width required. You can check this in LaTeX with \the\columnwidth (StackOverflow).

For MNRAS, the column width is 240 points (pt). 1pt is 1/72 of an inch, so the column width is exactly 10/3 inches. Set every figure 10/3 inches wide and they will fit perfectly into the MNRAS columns.

fig, ax = plt.subplots(figsize=(10/3, 3))  # height can be anything
ax.plot(...)
fig.tight_layout()

Matplotlib and Seaborn both have sensible defaults for the size of graphics elements provided you set the figure the same size that it will appear on the page. If you want to tinker further, check out sns.set_context and sns.set.

Text

Figure text looks professional when it is the same font and of a similar size to the body text.

Sizing

If we resize the figure to make it fit in a column ([width=\columnwidth]) then the text will also be resized. So it’s crucial to get the figure size right first. Check the section above.

Got that done? Good. Now let’s set the font size.

You can check the font size of the body text using the following LaTeX command:

\documentclass{article}

\makeatletter
\newcommand\thefontsize[1]{{#1 The current font size is: \f@size pt\par}}
\makeatother

\begin{document}
    \thefontsize\small
    \thefontsize\normalsize
    \thefontsize\large
\end{document}

This is also measured in points (pt). For MNRAS, \normalsize text is 9pt.

Now set matplotlib to make text in that size. You can use the fontsize argument with most matplotlib commands e.g. ax.set_xlabel(..., fontsize=9).

I like to make my figures in a dedicated Jupyter notebook with Matplotlib’s global text parameters set in a cell at the top at the top.

# in points - start with the body text size and play around
SMALL_SIZE = 9
MEDIUM_SIZE = 9
BIGGER_SIZE = 9

plt.rc('font', size=SMALL_SIZE)          # controls default text sizes
plt.rc('axes', titlesize=SMALL_SIZE)     # fontsize of the axes title
plt.rc('axes', labelsize=MEDIUM_SIZE)    # fontsize of the x and y labels
plt.rc('xtick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
plt.rc('ytick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
plt.rc('legend', fontsize=SMALL_SIZE)    # legend fontsize
plt.rc('figure', titlesize=BIGGER_SIZE)

There’s no need for the size to match exactly. I find slightly larger axes labels look good.But now the text sizing will be deliberately chosen and consistent for all your figures.

Before tinkering with the sizing, remember to match the font (below).

Fonts

Seaborn and Matplotlib use minimalist fonts, while journals tend to use fonts that look more serious and typewriter-esque. Changing the figures to use the serious journal font will help keep a consistent visual style. It will also help you precisely match the sizing (above) as different fonts can look different sizes even if they are technically the same points.

LaTeX sets fonts via \usepackage{} in the main document (not in the style file, confusingly) so take a look at which packages your document uses to find the font. This may require some sleuthing.

MNRAS uses \usepackage{newtxtext} which is part of newtx. The LaTeX Font Catalog lists newtx as using “URW Nimbus Roman”. We can check a sample on fontsquirrel.

Matplotlib can only use fonts which are installed on your computer. Download URW Nimbus Roman (.otf file) from fontsquirrel. Installing fonts is different for different OS, so you might need to google what to do next. For me, on CentOS, I placed the .otf in the new folder /home/{user}/.fonts and ran fc-cache -v to check for new fonts.

Now check the font is installed correctly. This command will list the install location of every font. You should see the path to your new .otf file, if you installed it.

import matplotlib.font_manager
matplotlib.font_manager.findSystemFonts(fontpaths=None, fontext='ttf')

Finally, we’re ready to set matplotlib to use that font. We need to find what the font is called, using a similar command:

fpaths = matplotlib.font_manager.findSystemFonts()

for i in fpaths:
    f = matplotlib.font_manager.get_font(i)
    print(f.family_name)

Now we can set matplotlib to use that font with plt.rc in just the same way as setting the text size.

plt.rc('font', family='Nimbus Roman')

Your figures should now have the same font and text size as the journal body text.

Now all you have to do is write the paper…

Towards Galaxy Foundation Models

Sun, 26 Jun 2022 23:46:37 GMT

This blog is a summary of my ICML Machine Learning for Astrophysics paper “Towards Galaxy Foundation Models with Hybrid Contrastive Learning”.

Computer science is coming to terms with a result that’s both exciting and tragic. For nearly any task, training a model directly on that task works less well than finetuning some existing large model that was pretrained on a huge and diverse dataset. These models are (controversially) called foundation models.

This is exciting because we can build better models than before - models which are not only more accurate but also more robust. It’s tragic because it risks entrenching well-resourced labs as the highly-paid gatekeepers of those models. But first let’s consider what this means for you.

My impression of our field is that many extragalactic astronomers:

Have a specific galaxy morphology question not answered by GZ’s questions (e.g. finding a rare galaxy type)
Have 100-10,000 expert-labelled galaxies where the answer is known
Want to solve the problem with deep learning (for effectiveness and for hype/career impact) but are not DL experts

Foundation models are perfect for this. Thanks to pretraining, they can be adapted to new tasks using very little data. I want to create foundation models for galaxies using Galaxy Zoo.

Finetuning pretrained "foundation" models can answer new questions with minimal data. These ring galaxies were found by finetuning Zoobot, a CNN pretrained on those questions, with ~200 examples of rings.

I previously open-sourced Zoobot - a CNN pretrained to solve every GZ DECaLS DR5 question. My tests showed that adapting Zoobot worked better than adapting from ImageNet or training the same model directly.

This new paper takes the same idea and scales it up. I make two key changes:

More labels: I train on four separate Galaxy Zoo campaigns, with nearly 100M human clicks between them.
More images: I use contrastive learning to benefit from 1.3M images that haven’t been labelled by humans.

Training on different Galaxy Zoo campaigns is tricky because the questions and answers are different between campaigns and images are usually only labelled in a single campaign. So what should the model predict? Happily, the Dirichlet loss function I introduced in GZ DECaLS can handle this naturally. The loss essentially measures $p(k|w, N)$ i.e. the odds of $k$ out of $N$ volunteers giving some answer to some question and for some deep learning model $w$ . But when $N$ is 0, $p(k=0|N=0, w) = 1$ and so the w doesn’t affect the loss - hence unanswered questions have no effect on training. It’s therefore easy to learn from galaxies with only a few answered questions: if the question is not answered, the model simply ignores it.

To train on totally unlabelled images - where no questions in any GZ campaign have been answered - I used the contrastive learning framework Bootstrap Your Own Latent (BYOL). This essentially presents two randomly-augmented versions of the same image to two networks, and asks one network to predict the internal representation of the other network. Because they see differently-augmented images, the representations of both must be invariant to those augmentations, which provides a learning signal. But BYOL is unsupervised, and I want to benefit from both unlabelled and labelled images. I did this by adding a supervised prediction head to one of BYOL’s networks, forcing it to both predict the representation of another network and solve the supervised Dirichlet loss task at the same time.

Adding a new supervised head (top) to BYOL (Grill 2020) helps guide the otherwise-unsupervised method to solve downstream tasks astronomers care about.

Making these two changes creates a model which substantially outperforms both direct training and pretraining, for the specific task of finding ring galaxies given scarce labels.

Historically, Galaxy Zoo has shared catalogs of vote counts for questions we hoped would be generally useful to many people. Choosing these questions is a compromise between scientific precision, broad applicability, and limited volunteer time. Now we no longer need to compromise. I believe Galaxy Zoo can shift towards creating adaptable foundation models that you can use to answer exactly the questions you care about.

I’d love to test this on more than just ring galaxies. If you have a small expert-labelled dataset centered on a specific question, please reach out (email, twitter).

Practical Galaxy Morphology Tools

Thu, 21 Oct 2021 23:46:37 GMT

This blog is a summary of my paper “Practical Galaxy Morphology Tools from Deep Supervised Representation Learning”.

What does a CNN learn?

At heart, deep learning is all about learning to make useful representations. Models need to learn a function to map an image (say) onto a lower-dimensional vector that represents the content of the image. That representation can then be used to make predictions, often by feeding it to a few final dense layers.

Representations have recently become really important in the natural language (NLP) community. By training models to predict masked words in sentences, essentially all digital text becomes useful training data. This allows groups with enough cash and engineers to train ridiculously large models with billions to trillions of parameters. While there’s a degree of “mine is bigger than yours”, larger models are genuinely useful: thanks to all that training data, the representations they learn are excellent for fine-tuning to downstream tasks like summarising news articles or writing code.

Could such an approach work for astronomy? Self-supervised methods have made huge progress in the last year (see e.g. simCLR). That said, self-supervised methods (and unsupervised methods more broadly) are not informed by human labels and thus might learn representations that are not as scientifically meaningful as supervised methods.

Schematic of our approach. By training to answer all the diverse Galaxy Zoo questions, our models learn a general representation that is useful for tasks on which the model was never trained.

In my paper, I show that we can train excellent galaxy representations using a supervised approach. The key is Galaxy Zoo. Because the GZ questions are very broad, and my models have to learn to answer all those questions with the same representation, they end up learning a general representation that is useful for new tasks on which the model was never trained.

This is much like how ImageNet pretraining is helpful for other terrestrial tasks because ImageNet is so broad. Pretraining on a broad galaxy task, however, works much better than a broad terrestrial task. I test this by trying to find ring galaxies using networks pretrained on GZ, pretrained on ImageNet, or trained from scratch. The GZ-pretrained network performs best. Interestingly, finetuning layers below the head is much less important than for the ImageNet-pretrained network, exactly as expected if the GZ representation is more useful.

When fine-tuning to find ring galaxies, it is much more effective to pretrain on Galaxy Zoo (blue) than ImageNet (orange, purple) or train from scratch (black dashed). This is because the Galaxy Zoo representation is already useful to find rings, and likely other kinds of galaxies as well.

The obvious upshot is that you can finetune our GZ models to solve your own galaxy morphology tasks using just hundreds of labelled examples. Want to find irregular galaxies? You can do that. Close pairs? Probably! Tidal features? Go for it.

You can find code, documentation and pretrained models at https://github.com/mwalmsley/zoobot. I’ve spent a long time writing guides and minimal working examples for you to adapt. Everything is designed to be accessible to researchers with no previous deep learning experience.

Our representations are also already useful to solve certain new tasks for which the models were never trained. I show two such tasks in the paper: a similarity search, where you can find the most similar galaxies to a query galaxy, and a human-in-the-loop anomaly detection system, which uses the models’ understanding of similarity to learn which galaxies a user is personally most interested in. Both work very well thanks to the representations.

I think the future of ML for galaxy morphology will start to include the thoughtful sharing of representations. The jury is still out whether those representations are supervised, self-supervised, some combination of the two, or some method that hasn’t been found yet.

Getting Started with Deep Learning for Astronomers

Sat, 07 Aug 2021 12:46:37 GMT

Welcome - come on in. Sorry for the debris. Grab a chair - steal Nick’s, he’s always in the lab.

Your supervisor said you’ll be using deep learning? But nobody actually told you how? Yes, you’ll find that a lot around here. Let’s see - what do you need to know…

What are you trying to do?

It is tempting to take the latest, fanciest model design you can find on paperswithcode and throw it at your problem. If you are doing “CV-driven research”, with the aim of moving into industry or just riding the deep learning hype, that’s a perfectly fair approach - just be honest with your collaborators about what you’re trying to achieve. However, if you’re trying to do good science, this will cause two headaches. First, fancy models have drawbacks; more code and more parameters make them generally slow, data-hungry, and hard to debug. Second, your paper will quickly be superceded by the next person using the inevitable slightly-fancier model.

A better strategy is to think carefully about what makes your problem hard, and then search for an approach that targets the hard parts. Maybe you are classifying faint tidal features in galaxy images, and so clever preprocessing might help. Maybe you have very little labelled data, and so you might look for ways to learn from unlabelled data or to intelligently choose which data to go out and label. Maybe the physics of the problem involves some symmetries or rules you can bake into your model. Some challenges will be unique to your astronomical context and this might be an opportunity to do some novel computer science research. When your method follows from your problem, your models will work better and your paper will stand out as thoughtful and original.

Most importantly, stay focused on the science question. The most common audience response at machine-learning-in-astro conference tracks is “So what?“. You have to be able to say what your method will be able to tell us about the universe. Even if you don’t have a single specific science question - perhaps you’re aiming to make an automated catalog or optimize a telescope schedule - you should be clear on how those outcomes will support other people’s science. This is the difference between making an impact in your field and spending a year carefully solving the wrong problem.

What can I read?

The good news is that there are thousands of websites to learn from. The bad news is that there thousands of websites to learn from. Turns out, anyone can make one.

Books (remember them?) are surprisingly helpful. I suggest www.deeplearningbook.org. It’s written by the inventor of GANs and strikes a good balance between words and math.

What can I ignore at first?

Many blogs tell you all the things you need to learn. I found this overwhelming. Let me instead tell you what not to learn.

The big thing to ignore is math. You’ve made it through a Physics degree (probably) and that puts you ahead of most folks trying to pick up deep learning. You have all the rusty tools you need to get the basic math.

You should learn the idea of backpropogation (calculating first derivatives at each layer, then combining them with the chain rule ) but you don’t have to work through the equations - that’s what software is for. You should be able to explain to your supervisor how layers work (try at your own risk) but you don’t need to write out the operations with Einstein notation. You don’t need to brush up on linear algebra (phew).

Math is foundational and, like all foundations, is best left buried until an extension is needed.

The other big thing to ignore is deployment. Much of the online wisdom is written by engineers, for engineers. They want machine learning pipelines running on multiple GPUs, on the cloud, on Kubernetes, on microcontrollers, in the browser, inside your phone and behind your eyes. You need it to run on your laptop first. Most astronomers will probably never need it to run anywhere else.

This brings me to:

Don’t I need a GPU?

Yes and no.

You want to train models quickly so you can find out what works. The nightmare scenario is setting a model training, spending the rest of the day hiding from imposter syndrome on Netflix, waking up tomorrow to find it didn’t work, and repeating until running out of either funding or sanity.

GPUs are unquestionably useful in shortening the cycle. Switching from a standard CPU to a standard gaming GPU might speed up your training by roughly a factor of five. But here are some things which will speed you up much more:

Solving the right problem in the first place (see above)
Using a “dummy” model to check your data pipeline and metrics
Training for one epoch to make sure your model doesn’t break
Using live logging like TensorBoard to monitor your model and cancel training if it’s going nowhere
Using “early stopping”, where training is cancelled if a model fails to improve over a set number of epochs, similarly

In short, train smart not hard.

I’m ready for a GPU. How do I get access to one?

Buy one. Departments and supervisors apply for funding to carry out research, and this funding often includes money for research equipment. GPUs are useful (sometimes vital) equipment for your research; you may be able to pitch your supervisor or department to buy you one. You should be able to explain how it will help (tensors go brrr) and why the current computing resources aren’t appropriate. It might feel like a lot to ask for as a lowly PhD student, and it might be, but it also might be less of a big deal than you think. If you don’t ask, you’ll never know. But do talk to your supervisor first.

Share one. Most Physics departments and universities will have compute clusters available to staff and graduate students. They grow organically by accreting hardware bought with grant money for particular projects. IT support can vary from “I’ve had a look at your submission script and fixed it for you, and by the way here’s how to do better logging” to ”obviously* you do it like this” to radio silence.

Borrow one.

Google has an excellent free resource called Collab. It is essentially Jupyter Notebook, but running on Google Cloud hardware instead of your computer. You launch a notebook, install the packages you want, off you go. The awkward part is moving data into the cloud so you can use it. One option is using GDrive. GDrive costs £2/month for 100GB and, since it’s also hosted by Google, can be instantly attached to your notebook.

Rent one.

AWS and GCP will both rent you GPUs. They are quite cheap - around $0.40 an hour for “Spot” (i.e. may, rarely, be turned off at any time) reservations. However, you will also need to pay to store your data and probably your environment. This can add up over time and, if you aren’t using the GPUs very much, might well cost more than the GPUs themselves. It’s also the most complicated option to set up. If you want to go on and work in industry as a cloud engineer, it’s a great CV building option, but for a quick project I would avoid it if possible.

Which package should I use?

It doesn’t matter.

You’ve got (sane) two choices; TensorFlow (by Google) and PyTorch (by Facebook & friends). They’re very similar. Put briefly: training simple models is easier with TensorFlow, while loading unusual data and debugging are both easier with PyTorch.

Both will let you write your own GPU-efficient functions in a style so teasingly close to numpy it’ll constantly trip you up:

np.mean(a)
tf.reduce_mean(a)
torch.mean(a)

np.concat([a, b])
tf.concat([a, b])
torch.cat([a, b])

Both will let you define models and use them like scikit-learn estimators. TensorFlow is friendlier for this than PyTorch thanks to the Keras module*:

from tensorflow import keras

model = keras.Sequential([
    keras.layers.Conv2D(...)
    keras.layers.MaxPool(...)
    keras.layers.Dense(...)
])
model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(X, y)
preds = model.predict(X)

Told you the deep learning code would be the easy bit.

PyTorch, on the other hand, requires two tutorials to show you how to define and fit a model. Fitting is a minefield; you should absolutely not write the training loop for the model yourself. Let other people solve your problem by using the extension package PyTorch Lightning:

from torch.utils.data import DataLoader
import pytorch_lightning as pl

train_loader = DataLoader(dataset)

trainer = pl.Trainer()
trainer.fit(model, train_loader)
preds = model.predict(train_loader)

Ultimately, the most important difference is what your collaborators use. Sharing tips and debugging help is crucial and much easier if you all use the same framework.

If you are still not sure - use PyTorch. It’s slightly more popular among computer scientists and so has slightly more of the latest and greatest models. Now stop worrying about it and go do some science.

GCR; Permission Denied - Unable to Determine Upload Size

Sun, 22 Sep 2019 23:46:37 GMT

Google Cloud has excellent container support, including GCR - a private container registry that automatically stores your containers on GCS. But GCR has a weird bug.

The first time I try to push a new image:

docker push $IMAGE_URI

The upload runs for a few seconds and then fails with the curious error:

denied: Unable to determine the upload's size

What’s going on?

GCR works by creating a bucket in GCS for your container. For reasons best known to Google, project owners do not automatically get write access to GCR buckets. You need to explicitly give yourself permission.

To do this, follow this guide. Click the bucket, then permissions, add member. Enter your Google or Service account. Search for and select the role ‘Storage Admin’.

You should know be able to push to your own GCR.

Happy coding,

Mike

Scaling Galaxy Zoo with Bayesian Neural Networks

Tue, 21 May 2019 23:46:37 GMT

This is a technical overview of our recent paper (Walmsley 2019) aimed at astronomers. If you’d like an introduction to how machine learning improves Galaxy Zoo, check out this blog.

I’d love to be able to take every galaxy and say something about it’s morphology. The more galaxies we label, the more specific questions we can answer. When you want to know what fraction of low-mass barred spiral galaxies host AGN, suddenly it really matters that you have a lot of labelled galaxies to divide up

But there’s a problem: humans don’t scale. Surveys keep getting bigger, but we will always have the same number of volunteers (applying order-of-magnitude astronomer math).

We’re struggling to keep pace now. When EUCLID (2022), LSST (2023) and WFIRST (2025ish) come online, we’ll start to look silly.

Galaxies/day required to keep pace with upcoming surveys now, by 2019 year-end, and by 2022 year-end. Estimates from internal science plan.

To keep up, Galaxy Zoo needs an automatic classifier. Other researchers have used responses that we’ve already collected from volunteers to train classifiers. The best performing of these are convolutional neural networks (CNNs) - a type of deep learning model tailored for image recognition. But CNNs have a drawback. They don’t easily handle uncertainty.

When learning, they implicitly assume that all labels are equally confident - which is definitely not the case for Galaxy Zoo (more in the section below). And when making (regression) predictions, they only give a ‘best guess’ answer with no error bars.

In our paper, we use Bayesian CNNs for morphology classification. Our Bayesian CNNs provide two key improvements:

They account for varying uncertainty when learning from volunteer responses
They predict full posteriors over the morphology of each galaxy

Using our Bayesian CNN, we can learn from noisy labels and make reliable predictions (with error bars) for hundreds of millions of galaxies.

How Bayesian Convolutional Neural Networks Work

There’s two key steps to creating Bayesian CNNs.

Predict the parameters of a probability distribution, not the label itself

Training neural networks is much like any other fitting problem: you tweak the model to match the observations. If all the labels are equally uncertain, you can just minimise the difference between your predictions and the observed values. But for Galaxy Zoo, many labels are more confident than others. If I observe that, for some galaxy, 30% of volunteers say “barred”, my confidence in that 30% massively depends on how many people replied - was it 4 or 40?

Instead, we predict the probability that a typical volunteer will say “Bar”, and minimise how surprised we should be given the total number of volunteers who replied. This way, our model understands that errors on galaxies where many volunteers replied are worse than errors on galaxies where few volunteers replied - letting it learn from every galaxy.

Use Dropout to Pretend to Train Many Networks

Our model now makes probabilistic predictions. But what if we had trained a different model? It would make slightly different probabilistic predictions. We need to marginalise over the possible models we might have trained. To do this, we use dropout. Dropout turns off many random neurons in our model, permuting our network into a new one each time we make predictions.

Below, you can see our Bayesian CNN in action. Each row is a galaxy (shown to the left). In the central column, our CNN makes a single probabilistic prediction (the probability that a typical volunteer would say “Bar”). We can interpret that as a posterior for the probability that k of N volunteers would say “Bar” - shown in black. On the right, we marginalise over many CNN using dropout. Each CNN posterior (grey) is different, but we can marginalise over them to get the posterior over many CNN (green) - our Bayesian prediction.

Active Learning

Modern surveys will image hundreds of millions of galaxies - more than we can show to volunteers. Given that, which galaxies should we classify with volunteers, and which by our Bayesian CNN?

Ideally we would only show volunteers the images that the model would find most informative. The model should be able to ask – hey, these galaxies would be really helpful to learn from– can you label them for me please? Then the humans would label them and the model would retrain. This is active learning.

In our experiments, applying active learning reduces the number of galaxies needed to reach a given performance level by up to 35-60% (See the paper).

We can use our posteriors to work out which galaxies are most informative. Remember that we use dropout to approximate training many models (see above). We show in the paper that informative galaxies are galaxies where those models confidently disagree.

Informative galaxies are galaxies where the each model is confident (entropy H in the posterior from each model is low) but the average prediction over all the models is uncertain (entropy across all averaged posteriors is high). See the paper for more.

This is only possible because we think about labels probabilistically and approximate training many models.

What galaxies are informative? Exactly the galaxies you would intuitively expect.

The model strongly prefers diverse featured galaxies over ellipticals
For identifying bars, the model prefers galaxies which are better resolved (lower redshift)

This selection is completely automatic. I didn’t realise the lower redshift preference until I looked at the images!

I’m excited to see what science can be done as we move from morphology catalogs of hundreds of thousands of galaxies to hundreds of millions.

Cheers,

Mike

Note: This post is cross-posted on the official Galaxy Zoo blog here.