330-apache-airflow.vtt

WEBVTT

00:00:00.000 --> 00:00:05.560
Hey everyone out there in the live stream, thank you so much for being here.


00:00:05.560 --> 00:00:10.280
If you have comments, thoughts, please put them into the live chat and I'll try to make


00:00:10.280 --> 00:00:11.280
it part of the show.


00:00:11.280 --> 00:00:14.560
If you're watching later, thanks for being here.


00:00:14.560 --> 00:00:16.600
Oh, hold on.


00:00:16.600 --> 00:00:18.520
Hold on.


00:00:18.520 --> 00:00:20.520
There's a super echo.


00:00:20.520 --> 00:00:24.480
There we go.


00:00:24.480 --> 00:00:26.000
Yeah.


00:00:26.000 --> 00:00:36.000
Does anyone in the meeting have the YouTube stream open, or is there something wrong with


00:00:36.000 --> 00:00:37.000
the stream?


00:00:37.000 --> 00:00:38.000
>> One moment.


00:00:38.000 --> 00:00:39.000
Maybe it's me.


00:00:39.000 --> 00:00:42.640
>> Actually, that brings up a good question.


00:00:42.640 --> 00:00:45.480
Do you want us to be muted when we're not talking?


00:00:45.480 --> 00:00:47.480
>> No, it's okay.


00:00:47.480 --> 00:00:49.880
This has never happened before with this thing.


00:00:49.880 --> 00:00:54.000
It's like I hear my own -- >> Is it better now?


00:00:54.000 --> 00:00:58.120
It's weird because I don't hear.


00:00:58.120 --> 00:01:00.640
I don't hear it either.


00:01:00.640 --> 00:01:01.640
Wow.


00:01:01.640 --> 00:01:02.640
Okay.


00:01:02.640 --> 00:01:07.240
I did see in Zencaster, you could turn on or off an option to hear your own voice.


00:01:07.240 --> 00:01:10.680
I don't know if that's a thing that could do that.


00:01:10.680 --> 00:01:11.680
All right, hold on.


00:01:11.680 --> 00:01:14.880
Let me just hang tight.


00:01:14.880 --> 00:01:18.280
The highly scientific turn it off and turn it on again.


00:01:18.280 --> 00:01:19.280
Yes.


00:01:19.280 --> 00:01:20.280
Yes.


00:01:20.280 --> 00:01:23.280
Hopefully, it's going to work.


00:01:23.280 --> 00:01:28.280
- Yeah, I think, can you say something?


00:01:28.280 --> 00:01:35.400
- Me?


00:01:35.400 --> 00:01:36.240
Hi.


00:01:36.240 --> 00:01:37.060
- Yeah.


00:01:37.060 --> 00:01:38.640
Okay, good.


00:01:38.640 --> 00:01:41.840
I was afraid it's from my setup, but I think it's okay.


00:01:41.840 --> 00:01:44.060
- I think we're good.


00:01:44.060 --> 00:01:48.120
- So now we're alone and we can, I don't know, are we live?


00:01:48.120 --> 00:01:50.560
- I don't know.


00:01:52.640 --> 00:01:53.640
Here's what's happened, everyone.


00:01:53.640 --> 00:01:54.480
Sorry about that.


00:01:54.480 --> 00:01:55.600
We're about to get there.


00:01:55.600 --> 00:01:56.960
So on my other account,


00:01:56.960 --> 00:01:58.760
a different account on my computer I'm logged into,


00:01:58.760 --> 00:02:00.200
I had the YouTube video open


00:02:00.200 --> 00:02:01.480
'cause I was copying the URL


00:02:01.480 --> 00:02:02.480
and it started playing,


00:02:02.480 --> 00:02:03.600
but not on my account.


00:02:03.600 --> 00:02:05.520
I mean, there was no windows I had.


00:02:05.520 --> 00:02:06.360
Anyway, all right.


00:02:06.360 --> 00:02:07.920
(all laughing)


00:02:07.920 --> 00:02:10.400
- In conclusion, it's a ghost.


00:02:10.400 --> 00:02:11.240
- It was.


00:02:11.240 --> 00:02:12.720
- The mystery is high. - It's really bizarre.


00:02:12.720 --> 00:02:13.560
Okay. - Yeah.


00:02:13.560 --> 00:02:16.140
(all laughing)


00:02:16.140 --> 00:02:18.040
- Yeah.


00:02:18.040 --> 00:02:19.520
Hello, everyone out there.


00:02:19.520 --> 00:02:20.960
Thanks for being here in the live stream.


00:02:20.960 --> 00:02:22.040
So, all right.


00:02:22.040 --> 00:02:26.160
Let's now that I've dispelled the ghost,


00:02:26.160 --> 00:02:28.560
thank you, we'll get started.


00:02:28.560 --> 00:02:32.920
Leah, Jarek, Axel, welcome to Talk Python To Me.


00:02:32.920 --> 00:02:34.840
It's good to have you all here.


00:02:34.840 --> 00:02:36.520
- Thanks for having us.


00:02:36.520 --> 00:02:38.120
- Yeah, absolutely.


00:02:38.120 --> 00:02:41.360
It's really fun to be talking about Airflow.


00:02:41.360 --> 00:02:43.600
These are the types of tools that I think


00:02:43.600 --> 00:02:46.000
they don't get that much awareness,


00:02:46.000 --> 00:02:48.460
but they're the kind of thing that can be the real backbone


00:02:48.460 --> 00:02:53.460
of a lot of teams, a lot of organizations and so on.


00:02:53.460 --> 00:02:56.680
So I think that'll be super fun to dive into


00:02:56.680 --> 00:02:57.840
and we'll all learn a lot.


00:02:57.840 --> 00:02:59.880
And I suspect a lot of people listening will realize,


00:02:59.880 --> 00:03:02.760
oh, here's a whole class of tools I didn't even realize


00:03:02.760 --> 00:03:05.280
I should have considered to solve my problems.


00:03:05.280 --> 00:03:06.640
But before we get down to that,


00:03:06.640 --> 00:03:08.640
let's start with your stories.


00:03:08.640 --> 00:03:09.860
Leah, you go first.


00:03:09.860 --> 00:03:12.780
How'd you get into programming Python?


00:03:12.780 --> 00:03:14.760
- Oh man, okay.


00:03:14.760 --> 00:03:19.760
So let's see, Python was the first language that I learned.


00:03:19.760 --> 00:03:22.720
I do have a bachelor's in computer science


00:03:22.720 --> 00:03:25.360
and at the school I went to,


00:03:25.360 --> 00:03:29.000
that is the language that Intro to CS is taught in.


00:03:29.000 --> 00:03:30.680
- I am so jealous.


00:03:30.680 --> 00:03:35.160
My Intro to CS class was in a scheme,


00:03:35.160 --> 00:03:36.560
which is derivative of Lisp,


00:03:36.560 --> 00:03:37.900
which didn't seem that practical.


00:03:37.900 --> 00:03:39.680
And then I was told I had to learn Fortran.


00:03:39.680 --> 00:03:42.360
It would be the most useful language I'd ever learn.


00:03:42.360 --> 00:03:44.180
Neither of which turned out to be true.


00:03:44.180 --> 00:03:46.220
I wish I learned Python.


00:03:46.220 --> 00:03:47.940
- I mean, I got lucky.


00:03:47.940 --> 00:03:52.020
Thanks Carleton College, Northfield, Minnesota


00:03:52.020 --> 00:03:53.980
for giving me Python early.


00:03:53.980 --> 00:03:57.980
And yeah, so I loved Python from the beginning.


00:03:57.980 --> 00:03:59.900
You asked how I got into programming.


00:03:59.900 --> 00:04:02.300
So I actually do have a parent in tech.


00:04:02.300 --> 00:04:04.540
It is my dad and he tried to get me


00:04:04.540 --> 00:04:06.420
into programming a lot earlier.


00:04:06.420 --> 00:04:10.260
And like a true teen, I said, absolutely not


00:04:10.260 --> 00:04:12.500
because it was suggested by my dad.


00:04:12.500 --> 00:04:14.740
It really wasn't until I got to school


00:04:14.740 --> 00:04:19.740
and I heard people say that Intro to CS was a fun elective.


00:04:19.740 --> 00:04:22.620
I was only listening, that's totally in quotes


00:04:22.620 --> 00:04:25.420
as I'm saying it, that I decided to take it.


00:04:25.420 --> 00:04:27.220
And it turned out I really liked it.


00:04:27.220 --> 00:04:29.900
And I decided to pivot from being a math major,


00:04:29.900 --> 00:04:31.980
which wasn't going very well,


00:04:31.980 --> 00:04:34.020
to being a computer science major.


00:04:34.020 --> 00:04:35.060
- Wow, that's fantastic.


00:04:35.060 --> 00:04:40.060
I was also a math major and I find the programming side,


00:04:40.060 --> 00:04:42.400
a lot of the same skillset you have to use,


00:04:42.400 --> 00:04:44.280
like the thinking through problem solving,


00:04:44.280 --> 00:04:47.520
you have these constraints or axioms in math


00:04:47.520 --> 00:04:49.480
and you like work from them.


00:04:49.480 --> 00:04:51.720
But in math, you just come up with sort of like


00:04:51.720 --> 00:04:53.880
the next idea that is the next problem


00:04:53.880 --> 00:04:54.800
that is the next idea.


00:04:54.800 --> 00:04:57.800
And in computers, you build stuff that people use.


00:04:57.800 --> 00:05:00.360
And it's such a difference I find.


00:05:00.360 --> 00:05:02.640
- It's puzzles is the programming.


00:05:02.640 --> 00:05:04.720
And that was always the part of math that I liked.


00:05:04.720 --> 00:05:06.820
I never liked the writing proofs


00:05:06.820 --> 00:05:08.880
or the theoretical side of things.


00:05:08.880 --> 00:05:12.800
I just wanted to solve puzzles with logic and rules.


00:05:12.800 --> 00:05:13.640
- Yeah, fantastic.


00:05:13.640 --> 00:05:15.440
Well, it sounds like you landed in the right spot.


00:05:15.440 --> 00:05:16.280
That's awesome.


00:05:16.280 --> 00:05:17.120
- I'm doing okay.


00:05:17.120 --> 00:05:18.360
(both laughing)


00:05:18.360 --> 00:05:19.520
- Jarek, how about you?


00:05:19.520 --> 00:05:23.960
- So, yes, you talked about your first language.


00:05:23.960 --> 00:05:27.200
So my first language in computer science


00:05:27.200 --> 00:05:32.000
during the studies was, I think, Delphi or Pascal.


00:05:32.000 --> 00:05:33.920
I can't even remember that.


00:05:33.920 --> 00:05:36.880
But actually the first language I started programming


00:05:36.880 --> 00:05:41.080
in the real work was, listen to that, COBOL.


00:05:41.080 --> 00:05:45.000
So if you've, if, so I tend to joke


00:05:45.000 --> 00:05:48.840
that when I'm retiring, I will be very well paid


00:05:48.840 --> 00:05:52.760
for five hours a week programmer of COBOL


00:05:52.760 --> 00:05:54.720
because nobody else will know it.


00:05:54.720 --> 00:05:56.480
- You're gonna keep the trucks delivering


00:05:56.480 --> 00:05:58.960
and the warehouse is open and just.


00:05:58.960 --> 00:05:59.800
- Exactly.


00:05:59.800 --> 00:06:00.880
- Yeah, beautiful.


00:06:00.880 --> 00:06:01.720
You'll be on routine.


00:06:01.720 --> 00:06:03.400
- Just five hours a week.


00:06:03.400 --> 00:06:04.880
Yeah, that's super cool.


00:06:04.880 --> 00:06:12.360
I think. But then Python is actually quite new in my portfolio, let's say, of languages.


00:06:12.360 --> 00:06:21.600
I've learned it maybe six years ago. And with my experience and years of working in CS,


00:06:21.600 --> 00:06:30.560
it's relatively late. But I loved it from the first glance. I used to work in like C,


00:06:30.560 --> 00:06:39.040
C++, hundreds of other, a lot of other languages, but Python was just super easy from the start


00:06:39.040 --> 00:06:42.320
and super nice too, and super, super friendly.


00:06:42.320 --> 00:06:48.400
Like I was, after years of programming in Java, I was like so much, Oh, in one line


00:06:48.400 --> 00:06:51.480
you can do what I would do in five pages.


00:06:51.480 --> 00:06:52.480
Yeah.


00:06:52.480 --> 00:06:53.480
Yeah.


00:06:53.480 --> 00:06:55.040
And you can understand it.


00:06:55.040 --> 00:06:56.040
Yes.


00:06:56.040 --> 00:06:57.040
Yes.


00:06:57.040 --> 00:06:57.880
(laughs)


00:06:57.880 --> 00:06:59.640
- Oh yeah, I fell in love immediately.


00:06:59.640 --> 00:07:00.480
- Oh, fantastic.


00:07:00.480 --> 00:07:04.960
- And it's my absolutely favorite language right now.


00:07:04.960 --> 00:07:06.360
- Yeah, same here.


00:07:06.360 --> 00:07:08.540
Kaxil.


00:07:08.540 --> 00:07:10.920
- Yeah, for me, I did my bachelor's


00:07:10.920 --> 00:07:14.880
in electrical engineering, so didn't do anything over there.


00:07:14.880 --> 00:07:18.200
But when I came in the UK to do my master's,


00:07:18.200 --> 00:07:21.820
we were taught our language and Java.


00:07:21.820 --> 00:07:25.400
One fine day, we were ending the college


00:07:25.400 --> 00:07:26.640
in just a month or two.


00:07:26.640 --> 00:07:30.000
And there was a presentation from someone


00:07:30.000 --> 00:07:32.240
in the university who was telling us


00:07:32.240 --> 00:07:34.840
how to use data science in the industry and everything.


00:07:34.840 --> 00:07:36.320
They said, you should know Python.


00:07:36.320 --> 00:07:39.440
And we were like, oh, but we were not taught Python.


00:07:39.440 --> 00:07:41.600
And we are just one or two months away


00:07:41.600 --> 00:07:43.360
from doing our internships and everything,


00:07:43.360 --> 00:07:44.440
and we don't know Python.


00:07:44.440 --> 00:07:47.600
So that's when I started looking into Python.


00:07:47.600 --> 00:07:50.720
I got an internship, and then I actually


00:07:50.720 --> 00:07:52.400
started learning more of Python.


00:07:52.400 --> 00:07:55.880
So this was 2016 I'm talking about.


00:07:55.880 --> 00:08:00.360
And yeah, since then it has been a wild, wild ride.


00:08:00.360 --> 00:08:04.680
I have written a lot of Java, a lot of R,


00:08:04.680 --> 00:08:07.360
but Python seems to be very easy to write,


00:08:07.360 --> 00:08:10.640
easy to understand, plus the community behind it


00:08:10.640 --> 00:08:13.400
and the packages behind it are so vast


00:08:13.400 --> 00:08:15.240
that you can use it for anything.


00:08:15.240 --> 00:08:18.080
- Yeah, I saw a funny T-shirt once that said,


00:08:18.080 --> 00:08:20.920
"I learned Python, it was a great weekend."


00:08:20.920 --> 00:08:22.920
which I think is really funny, right?


00:08:22.920 --> 00:08:24.100
(laughing)


00:08:24.100 --> 00:08:27.080
Because on one hand, yeah, sure, you can go through,


00:08:27.080 --> 00:08:28.360
and actually the language is simple,


00:08:28.360 --> 00:08:29.620
especially if you know something else


00:08:29.620 --> 00:08:31.440
that's like Java or C++,


00:08:31.440 --> 00:08:33.920
this is a breath of fresh air, right?


00:08:33.920 --> 00:08:35.160
But on the other hand,


00:08:35.160 --> 00:08:36.600
I've been doing this for a long time,


00:08:36.600 --> 00:08:37.440
all day, every day,


00:08:37.440 --> 00:08:39.520
and I'm still learning Python every day, right?


00:08:39.520 --> 00:08:42.240
So it's this really interesting juxtaposition


00:08:42.240 --> 00:08:44.360
of you can learn the language really easily,


00:08:44.360 --> 00:08:46.120
but then there's the standard library,


00:08:46.120 --> 00:08:48.420
and then there's 300,000 PyPI packages,


00:08:48.420 --> 00:08:50.080
like Airflow is just one of them,


00:08:50.080 --> 00:08:51.540
and that's our whole topic today, right?


00:08:51.540 --> 00:08:54.480
So it's kind of both, right?


00:08:54.480 --> 00:08:56.640
- Yeah, and the language keeps growing.


00:08:56.640 --> 00:08:59.880
So you gotta keep track of the cool new things


00:08:59.880 --> 00:09:02.720
that are released and things that are true in Python 2.7


00:09:02.720 --> 00:09:07.240
are definitely not true today with Python 3.10.


00:09:07.240 --> 00:09:08.440
It's grown a lot.


00:09:08.440 --> 00:09:09.620
- It definitely has.


00:09:09.620 --> 00:09:12.800
And I saw that Airflow is not supporting


00:09:12.800 --> 00:09:14.360
the older versions of Python,


00:09:14.360 --> 00:09:15.760
basically as they get deprecated.


00:09:15.760 --> 00:09:17.920
So yay for that, right?


00:09:17.920 --> 00:09:21.320
- Yeah, we have actually very, very strong rules


00:09:21.320 --> 00:09:23.280
following the Python release rules.


00:09:23.280 --> 00:09:26.280
So we've learned from what Python learned


00:09:26.280 --> 00:09:28.960
on the release schedule and we just follow it


00:09:28.960 --> 00:09:30.320
very, very closely.


00:09:30.320 --> 00:09:32.680
We like how much we have when we support,


00:09:32.680 --> 00:09:33.920
when we stop with support.


00:09:33.920 --> 00:09:34.760
- Yeah, that makes sense.


00:09:34.760 --> 00:09:36.080
- Difficult to maintain compatibility


00:09:36.080 --> 00:09:37.600
between Python two and three.


00:09:37.600 --> 00:09:39.140
That's a lot of overhead.


00:09:39.140 --> 00:09:42.680
- Actually, yes.


00:09:42.680 --> 00:09:44.600
- I have nightmares about it,


00:09:44.600 --> 00:09:46.760
cherry picking all the stuff from the main branch


00:09:46.760 --> 00:09:50.720
to the old release branch and adding Python to support.


00:09:50.720 --> 00:09:52.560
- But Kaxil, you cannot complain.


00:09:52.560 --> 00:09:55.200
I mean, we both, but Kaxil did that a lot.


00:09:55.200 --> 00:09:58.880
And thanks to that, we've been several times


00:09:58.880 --> 00:10:02.400
top committers on Apache organization.


00:10:02.400 --> 00:10:06.560
Like there is a like this week, most commits made.


00:10:06.560 --> 00:10:08.600
And that was us doing cherry picks


00:10:08.600 --> 00:10:11.560
between the three version and two seven version.


00:10:11.560 --> 00:10:14.320
And once we made it both at the same time


00:10:14.320 --> 00:10:17.080
top commuters on Apache.


00:10:17.080 --> 00:10:18.960
- I'm mad at GitHub though.


00:10:18.960 --> 00:10:22.480
GitHub does not count commits on any other branch


00:10:22.480 --> 00:10:24.040
except main or master.


00:10:24.040 --> 00:10:26.820
- Oh, not fair, not fair.


00:10:26.820 --> 00:10:30.400
- We'll have to do our own visualizations.


00:10:30.400 --> 00:10:31.560
- Yeah, exactly.


00:10:31.560 --> 00:10:34.640
All right, before we move on out of the live stream,


00:10:34.640 --> 00:10:37.240
Hawaii Girl says Python is awesome, like all of us.


00:10:37.240 --> 00:10:39.480
Yes, definitely, thanks for being here.


00:10:39.480 --> 00:10:41.720
All right, well, let's start this


00:10:41.720 --> 00:10:46.560
at a slightly higher conversation than just Airflow.


00:10:46.560 --> 00:10:51.560
So Airflow is one of these workflow management frameworks.


00:10:51.560 --> 00:10:53.640
Like whoever wants to take this,


00:10:53.640 --> 00:10:55.720
what is that, why do I need that?


00:10:55.720 --> 00:10:56.900
When do I need that?


00:10:56.900 --> 00:10:59.540
What are these tools as I hinted at the beginning?


00:10:59.540 --> 00:11:02.560
- Can't save my deal.


00:11:02.560 --> 00:11:08.560
- Yeah, so I want to walk through some of the history


00:11:08.560 --> 00:11:12.400
though, like in 2014, 2015, where like data engineering was


00:11:12.400 --> 00:11:18.040
not mainstream, and everyone was just using cron for scheduling


00:11:18.040 --> 00:11:23.320
their task. And then there came Luigi, where people were using


00:11:23.320 --> 00:11:30.840
XML and those sort of languages to write their DAGs workflows to


00:11:30.840 --> 00:11:33.280
make sure that the task runs on schedule.


00:11:33.760 --> 00:11:37.800
- Yeah, so DAGs directed acyclic graphs.


00:11:37.800 --> 00:11:38.640
- Yes. - Yeah.


00:11:38.640 --> 00:11:40.240
- Okay. - The DAGs, yeah.


00:11:40.240 --> 00:11:41.440
Let's be very clear.


00:11:41.440 --> 00:11:44.880
You cannot have a circle for tasks.


00:11:44.880 --> 00:11:46.960
The dependencies cannot be that.


00:11:46.960 --> 00:11:49.360
People have tried that and--


00:11:49.360 --> 00:11:51.280
- It takes a long time to finish those.


00:11:51.280 --> 00:11:52.520
- Yes. (laughs)


00:11:52.520 --> 00:11:53.680
A long time.


00:11:53.680 --> 00:11:56.360
Some of those DAGs are still running, so. (laughs)


00:11:56.360 --> 00:11:58.520
- Takes a long time for them to start sometimes too.


00:11:58.520 --> 00:12:01.280
(all laughing)


00:12:01.280 --> 00:12:05.000
But yeah, I think to complete the history part,


00:12:05.000 --> 00:12:09.640
people just got bored writing the XML syntax.


00:12:09.640 --> 00:12:11.840
And it's difficult to understand.


00:12:11.840 --> 00:12:14.280
Similar to what we were talking about Java and Python,


00:12:14.280 --> 00:12:17.480
like Python is much easier to read, easier to understand.


00:12:17.480 --> 00:12:19.880
There came Airflow.


00:12:19.880 --> 00:12:25.080
Maxim, who wrote Airflow in his time at Airbnb


00:12:25.080 --> 00:12:27.880
and open sourced it to a purchase of the foundation,


00:12:27.880 --> 00:12:32.080
And that had a sigh of relief for people working on Luigi


00:12:32.080 --> 00:12:36.200
and others as well, because then you could write your workflows


00:12:36.200 --> 00:12:38.440
in an easy to understand language


00:12:38.440 --> 00:12:40.600
that you're already very familiar with.


00:12:40.600 --> 00:12:43.680
You don't need to write those XMLs.


00:12:43.680 --> 00:12:49.520
And who loves writing XMLs, first of all?


00:12:49.520 --> 00:12:53.520
And so it's easy to understand, just configuration is good.


00:12:53.520 --> 00:12:56.960
And there was also, I think, a slight move


00:12:56.960 --> 00:13:02.640
towards everything as code, infrastructure as code with Terraform and Ansible and whatnot.


00:13:02.640 --> 00:13:10.580
And Airflow was just a perfect tool for workflow as code or DAGs as code.


00:13:10.580 --> 00:13:17.140
And since I think 2016 to 2018, Airflow's popularity has skyrocketed with the advent


00:13:17.140 --> 00:13:20.780
of like separate specialized data engineering field.


00:13:20.780 --> 00:13:24.340
Previously, I think software engineers used to do everything.


00:13:24.340 --> 00:13:28.340
But then people or companies also realize that it's a separate field.


00:13:28.340 --> 00:13:29.300
It's a lot of work.


00:13:29.300 --> 00:13:31.220
It's not you can also just not include


00:13:31.220 --> 00:13:34.180
machine learning engineer and let him do everything.


00:13:34.180 --> 00:13:37.460
So separate data engineers job to write


00:13:37.460 --> 00:13:42.340
a pipeline, knows how to handle the data and everything from start to space.


00:13:42.340 --> 00:13:46.220
We try that thousands of things, which you're first of all, you're


00:13:46.220 --> 00:13:50.180
cron expressions or cron alone cannot handle those.


00:13:50.180 --> 00:13:53.780
The task dependencies, the SLAs and whatnot.


00:13:53.980 --> 00:13:57.820
So I think that's when, with the advent of data engineering,


00:13:57.820 --> 00:14:00.020
people realizing the importance of data,


00:14:00.020 --> 00:14:05.020
Airflow's popularity gained massively between 2018,


00:14:05.020 --> 00:14:08.420
which also, by the way, coincided with where Airflow


00:14:08.420 --> 00:14:10.500
became their top level project


00:14:10.500 --> 00:14:12.260
in Apache Software Foundation.


00:14:12.260 --> 00:14:16.460
Until then, Airflow was just an incubating project in ASF,


00:14:16.460 --> 00:14:18.420
and then it became a top level project,


00:14:18.420 --> 00:14:22.060
and that was a big milestone for Airflow and the community.


00:14:22.060 --> 00:14:24.020
- I think data engineering is really interesting


00:14:24.020 --> 00:14:26.340
because a lot of people when they think of,


00:14:26.340 --> 00:14:30.180
well, what are the divisions of what you do with programming,


00:14:30.180 --> 00:14:31.380
you know, especially in Python,


00:14:31.380 --> 00:14:33.780
well, we've got like web programming,


00:14:33.780 --> 00:14:36.180
some to some degree UI programming,


00:14:36.180 --> 00:14:38.380
and then we've got data science,


00:14:38.380 --> 00:14:40.060
sort of web and data science are the two,


00:14:40.060 --> 00:14:42.820
but there's this middle ground


00:14:42.820 --> 00:14:45.980
where I feel like people kind of don't want to go there,


00:14:45.980 --> 00:14:47.060
or that's the data, right?


00:14:47.060 --> 00:14:48.340
You want to make sure, you know,


00:14:48.340 --> 00:14:50.580
if you get a bunch of data and you feed it to your model,


00:14:50.580 --> 00:14:53.860
your model is only as good as the data you get, right?


00:14:53.860 --> 00:14:56.940
If you're trying to automate some ingest of data


00:14:56.940 --> 00:14:58.820
or warehousing reporting,


00:14:58.820 --> 00:15:02.140
it's only as good as the reliability of the data coming in,


00:15:02.140 --> 00:15:03.900
the accuracy, right?


00:15:03.900 --> 00:15:06.460
We've got things, what is it, great expectations


00:15:06.460 --> 00:15:08.240
and stuff like that for testing,


00:15:08.240 --> 00:15:10.700
actually testing against the data,


00:15:10.700 --> 00:15:13.780
not the code that works with the data.


00:15:13.780 --> 00:15:17.020
- Yes, and let me just add,


00:15:17.020 --> 00:15:19.980
also Airflow is really an orchestrator.


00:15:19.980 --> 00:15:23.140
So I used to sing in a choir for many years.


00:15:23.140 --> 00:15:25.740
And for me, this is really like this parallel


00:15:25.740 --> 00:15:28.620
between the conductor and the team playing.


00:15:28.620 --> 00:15:30.700
So we don't do stuff in Airflow.


00:15:30.700 --> 00:15:32.420
Airflow doesn't do stuff.


00:15:32.420 --> 00:15:34.500
It just tells others what to do.


00:15:34.500 --> 00:15:36.740
And there's data processing stuff.


00:15:36.740 --> 00:15:38.580
So we don't know how--


00:15:38.580 --> 00:15:40.860
basically, we as data engineers--


00:15:40.860 --> 00:15:44.980
because we are actually software engineers writing things


00:15:44.980 --> 00:15:46.540
for data engineers.


00:15:46.540 --> 00:15:49.300
So when you think about this cross of these both,


00:15:49.300 --> 00:15:50.940
like software engineer and data engineer.


00:15:50.940 --> 00:15:54.340
So we don't know how to actually make a model,


00:15:54.340 --> 00:15:56.940
machine learning model, or we don't know machine learning.


00:15:56.940 --> 00:15:58.640
We don't know how to,


00:15:58.640 --> 00:16:01.140
even we don't know how to do map reduce.


00:16:01.140 --> 00:16:03.300
I mean, if you want to process a lot of data,


00:16:03.300 --> 00:16:06.620
but we know what to do with the data when it comes,


00:16:06.620 --> 00:16:08.020
who should do next, what,


00:16:08.020 --> 00:16:10.860
and how to pass it somewhere else.


00:16:10.860 --> 00:16:14.980
And we can make it super complex to define


00:16:14.980 --> 00:16:18.020
or complex in terms of composed


00:16:18.020 --> 00:16:20.740
on many, many different steps and different relations,


00:16:20.740 --> 00:16:24.180
but Airflow makes it super easy to manage the whole thing


00:16:24.180 --> 00:16:26.900
so that it runs smoothly and you can operate it


00:16:26.900 --> 00:16:31.900
and you can deal with any problems that arise on the go.


00:16:31.900 --> 00:16:35.620
- So Jarek, this point, I want to expand on it real quick.


00:16:35.620 --> 00:16:39.020
There's a very human aspect to the workflow orchestration


00:16:39.020 --> 00:16:42.980
that I think both you, both Kaxel and Jarek have touched on,


00:16:42.980 --> 00:16:46.020
which is that having a workflow orchestrator


00:16:46.020 --> 00:16:51.860
really enables you to move from having like the data scientist in their silo working on this


00:16:51.860 --> 00:16:57.620
pipeline alone to having a whole team of data scientists and data engineers working together


00:16:57.620 --> 00:17:02.820
because you have really specialized folks who can work on building those models and that might not


00:17:02.820 --> 00:17:07.540
be the same group of people that's figuring out how to get the data from A to B and making sure


00:17:07.540 --> 00:17:14.260
that it's healthy and is what the model and the data scientists are expecting. So I think it just


00:17:14.260 --> 00:17:16.460
just enables a lot more collaboration


00:17:16.460 --> 00:17:20.500
and helps you have more specialists working together.


00:17:20.500 --> 00:17:25.220
- Yeah, it becomes that well-known, well-tested way


00:17:25.220 --> 00:17:29.740
to flow data down into the specialties


00:17:29.740 --> 00:17:31.340
that people need, right?


00:17:31.340 --> 00:17:32.180
- Exactly.


00:17:32.180 --> 00:17:35.360
- One of the things you do in these types of frameworks


00:17:35.360 --> 00:17:37.520
is you build these tasks, right?


00:17:37.520 --> 00:17:40.820
Give us an idea of what some of the tasks look like.


00:17:40.820 --> 00:17:42.620
You actually have a whole bunch of,


00:17:42.620 --> 00:17:46.180
Would that be the integrations in there or is that something different?


00:17:46.180 --> 00:17:50.220
Providers, that's the name that we are using in Ergo 2.


00:17:50.220 --> 00:17:53.100
Yes, so we have like more than 70 of those right now.


00:17:53.100 --> 00:18:01.220
We do for 70 services, we talk to external services or databases or whatnot.


00:18:01.220 --> 00:18:08.460
70 entities, but within that we have several hundreds of these so-called operators


00:18:08.460 --> 00:18:14.300
or sensors or transfer operators, which perform the task.


00:18:14.300 --> 00:18:19.100
And they're actually super easy. It's just one method, execute. That's it.


00:18:19.100 --> 00:18:21.100
- Right. - That's pretty much it.


00:18:21.100 --> 00:18:26.540
- So, yeah, there's the three things, the sensors, the operators, the transfers.


00:18:26.540 --> 00:18:34.540
Like an example of a sensor might be waiting to see if an object is in S3 or in Google Cloud Storage.


00:18:34.540 --> 00:18:38.700
And a transfer is moving something from A to B.


00:18:38.700 --> 00:18:42.700
And an operator, those are the ones we probably have the most of, right,


00:18:42.700 --> 00:18:47.980
Yarak and Kaxel? And that, it can be anything in a service.


00:18:47.980 --> 00:18:51.420
I don't know, like starting, running, so I work in Google Cloud,


00:18:51.420 --> 00:18:53.660
so the operators I'm most familiar with are the Google ones.


00:18:53.660 --> 00:18:58.780
So like spinning up a data proc cluster and then running a Spark job on it,


00:18:58.780 --> 00:19:03.020
or running something on a Kubernetes pod.


00:19:03.020 --> 00:19:08.700
there. Yeah, if you can dream it, either there is an operator for it, or you can write an operator


00:19:08.700 --> 00:19:15.660
for it. Yeah. Yeah. When I started with Airflow, back in 2017, we used Airflow for the same reason,


00:19:15.660 --> 00:19:21.660
like Airflow was designed for being a classic ETL tool, or being an enabler of sorts. So


00:19:21.660 --> 00:19:28.780
a lot of companies are migrating from on-premises to cloud. We were doing a project with, in


00:19:28.780 --> 00:19:32.500
in partnership with Google to move customers' data to cloud.


00:19:32.500 --> 00:19:37.260
And we were using NiFi for data to be on GCS.


00:19:37.260 --> 00:19:41.740
But from there, everything was orchestrated by Airflow.


00:19:41.740 --> 00:19:44.300
So once the data lands in Google Cloud Storage,


00:19:44.300 --> 00:19:48.380
then there's classic ETL, that extract, transform, load.


00:19:48.380 --> 00:19:50.580
From GCS, it goes to BigQuery.


00:19:50.580 --> 00:19:53.300
BigQuery does some manipulation.


00:19:53.300 --> 00:19:55.940
Then the data goes to--


00:19:55.940 --> 00:19:57.820
There's a dashboard, a data studio


00:19:57.820 --> 00:19:59.940
that shows a rich dashboard behind it.


00:19:59.940 --> 00:20:03.020
And this is all managed by Airflow.


00:20:03.020 --> 00:20:06.540
And it was so easy because we separated this using task


00:20:06.540 --> 00:20:08.820
and we were using all the hooks and operators


00:20:08.820 --> 00:20:11.620
that Leah and Jarek were talking about,


00:20:11.620 --> 00:20:14.100
which was like GCS to GCS operator,


00:20:14.100 --> 00:20:17.740
move the data from the landing area to staging.


00:20:17.740 --> 00:20:19.940
So your landing area remains untouched.


00:20:19.940 --> 00:20:21.420
So you can verify with your vendor


00:20:21.420 --> 00:20:25.620
that the data is as they sent even in future.


00:20:25.620 --> 00:20:29.660
And then there were BigQuery operator to run SQL query.


00:20:29.660 --> 00:20:31.660
And then there are other operators


00:20:31.660 --> 00:20:33.860
for different GCS services.


00:20:33.860 --> 00:20:35.580
So I think with Google,


00:20:35.580 --> 00:20:38.580
there was already good amount of integrations back


00:20:38.580 --> 00:20:43.580
34 years back, similarly for Spark and other operators.


00:20:43.580 --> 00:20:45.380
- Yeah.


00:20:45.380 --> 00:20:47.340
One of the things that stands out to me


00:20:47.340 --> 00:20:50.260
that might be really useful here is if something goes wrong,


00:20:50.260 --> 00:20:52.900
you talked about the contrast being Cron jobs


00:20:52.900 --> 00:20:53.900
or something like that.


00:20:53.900 --> 00:20:56.060
And if something goes wrong with that,


00:20:56.060 --> 00:20:58.480
or you need to scale out across different machines


00:20:58.480 --> 00:21:01.060
or whatever, and how do you get those timings right


00:21:01.060 --> 00:21:02.900
or other weird things?


00:21:02.900 --> 00:21:05.760
So what's the mechanism for dealing with,


00:21:05.760 --> 00:21:07.220
you know, I'm gonna get some data,


00:21:07.220 --> 00:21:09.280
it drops in the cloud, I'm gonna pull it over,


00:21:09.280 --> 00:21:11.940
but then maybe it's invalid data or something.


00:21:11.940 --> 00:21:14.020
What's that look like?


00:21:14.020 --> 00:21:15.740
- So at least for Airflow,


00:21:15.740 --> 00:21:18.540
all the operators that were written previously


00:21:18.540 --> 00:21:21.780
or the idea behind them were that task,


00:21:21.780 --> 00:21:23.620
a single operator or a single task,


00:21:23.620 --> 00:21:28.980
be idempotent. So even if you run them multiple times, it should produce the same result.


00:21:28.980 --> 00:21:35.340
So if a task for whatever reason fails, you could add more retries to it as a retry parameter


00:21:35.340 --> 00:21:40.060
that the base class takes and you could say retries is four, retries is five and Airflow


00:21:40.060 --> 00:21:43.980
will handle that for you. So if a task fails, it will rerun it for that amount of time.


00:21:43.980 --> 00:21:47.700
It could fail because the database server is down or it could fail because it's never


00:21:47.700 --> 00:21:49.780
going to work, right? It could be either one.


00:21:49.780 --> 00:21:52.340
- Exactly, and you want to be notified as well.


00:21:52.340 --> 00:21:55.380
So then we had all those on failure callback,


00:21:55.380 --> 00:21:58.820
on success callback, those emails get sent out saying


00:21:58.820 --> 00:22:00.280
the data didn't arrive at all


00:22:00.280 --> 00:22:02.540
or whatever the reason may be.


00:22:02.540 --> 00:22:04.700
- And there is even more to that


00:22:04.700 --> 00:22:08.500
because we also have the mechanism of backfilling the data.


00:22:08.500 --> 00:22:11.140
So even in this case, it's not like not a server failure


00:22:11.140 --> 00:22:15.220
but your data has improved because you got a new metadata


00:22:15.220 --> 00:22:16.860
and you want to reprocess the data


00:22:16.860 --> 00:22:19.720
you've already processed for like last week


00:22:19.720 --> 00:22:22.280
or only process part of the data


00:22:22.280 --> 00:22:24.720
because it takes a lot of time


00:22:24.720 --> 00:22:28.920
and you know that the data up to a certain point is good,


00:22:28.920 --> 00:22:32.400
but then you have to process just part of your workflow,


00:22:32.400 --> 00:22:35.000
a part of your DAG for the last week.


00:22:35.000 --> 00:22:36.120
You can do that with Airflow.


00:22:36.120 --> 00:22:39.120
So you can just tell, make a comment, run a comment,


00:22:39.120 --> 00:22:44.120
just reprocess that data for this period of time,


00:22:44.120 --> 00:22:45.720
starting from this task,


00:22:45.720 --> 00:22:48.760
because this is where we know we have to reprocess the data


00:22:48.760 --> 00:22:52.280
because the data has been cleaned up, for example.


00:22:52.280 --> 00:22:54.120
- Right, you don't have to detect it,


00:22:54.120 --> 00:22:55.280
you don't have to copy it down,


00:22:55.280 --> 00:22:57.320
you've changed it locally and you want it to get fixed.


00:22:57.320 --> 00:22:58.160
I see, okay.


00:22:58.160 --> 00:23:00.280
- And the super cool thing there is that


00:23:00.280 --> 00:23:01.720
this can be done by one person


00:23:01.720 --> 00:23:04.600
who doesn't know what those tasks are doing at all.


00:23:04.600 --> 00:23:07.360
Like they are just all the kind of language


00:23:07.360 --> 00:23:09.400
of how the tasks are written.


00:23:09.400 --> 00:23:11.460
The specification is written in the way


00:23:11.460 --> 00:23:12.440
that anyone can do that.


00:23:12.440 --> 00:23:15.440
And then this person operating can very safely


00:23:15.440 --> 00:23:17.240
just rerun parts of it and be sure


00:23:17.240 --> 00:23:19.040
but what comes out at the end


00:23:19.040 --> 00:23:21.100
is just what they are expecting.


00:23:21.100 --> 00:23:23.680
And if you have like hundreds and thousands of docs


00:23:23.680 --> 00:23:28.220
written by tens and twenties or hundreds of people,


00:23:28.220 --> 00:23:32.020
just one person can sit down and operate the whole of it


00:23:32.020 --> 00:23:35.640
without understanding a single thing, how it works inside.


00:23:35.640 --> 00:23:38.040
But with knowing, with seeing what happens,


00:23:38.040 --> 00:23:42.500
this is like so powerful part of our flow.


00:23:42.500 --> 00:23:44.380
- It lets you focus on just the steps


00:23:44.380 --> 00:23:47.120
and not how they fit together, right?


00:23:47.120 --> 00:23:51.160
So yeah, let's focus on a couple of things


00:23:51.160 --> 00:23:52.320
on the website here that I think


00:23:52.320 --> 00:23:54.560
are maybe worth calling out.


00:23:54.560 --> 00:23:57.880
One of the things here is that the project


00:23:57.880 --> 00:24:02.680
has four principles that are really nice.


00:24:02.680 --> 00:24:04.840
Maybe you wanna highlight those for people?


00:24:04.840 --> 00:24:09.320
- Yeah, I think, okay, so the four principles,


00:24:09.320 --> 00:24:12.320
it's that Airflow is dynamic, extensible,


00:24:12.320 --> 00:24:13.880
elegant, and scalable.


00:24:13.880 --> 00:24:17.880
And I am going to go ahead and pick my favorite one right here.


00:24:17.880 --> 00:24:21.400
And it's one that we've kind of touched upon without spelling it out clearly,


00:24:21.400 --> 00:24:23.960
which is that Airflow is extensible.


00:24:23.960 --> 00:24:28.760
Jarek talked about how we have these 70 plus providers,


00:24:28.760 --> 00:24:32.800
these various integrations with all kinds of services,


00:24:32.800 --> 00:24:37.760
from the big cloud providers to things like Slack, Snowflake,


00:24:37.760 --> 00:24:40.720
which I know are also kind of big, to much smaller ones.


00:24:40.720 --> 00:24:44.560
And if a provider doesn't exist,


00:24:44.560 --> 00:24:47.940
or if an operator doesn't exist for a task


00:24:47.940 --> 00:24:50.960
that you need to perform, you can write it,


00:24:50.960 --> 00:24:52.960
and you can either write it and be running it


00:24:52.960 --> 00:24:54.800
in your instance of Airflow,


00:24:54.800 --> 00:24:57.440
or if you're being a good steward of open source,


00:24:57.440 --> 00:24:59.880
you can write it and contribute it back to the community


00:24:59.880 --> 00:25:02.800
so other people who need to do that task


00:25:02.800 --> 00:25:06.280
can also benefit from what you've already figured out.


00:25:06.280 --> 00:25:07.440
- Yeah, that's really neat.


00:25:07.440 --> 00:25:11.340
So a lot of these would be things like down here,


00:25:11.340 --> 00:25:13.760
like if only one person has to write,


00:25:13.760 --> 00:25:16.080
how do I connect to Hadoop?


00:25:16.080 --> 00:25:18.880
Or this is if you go to airflow.apache.org,


00:25:18.880 --> 00:25:19.720
you can go to the bottom,


00:25:19.720 --> 00:25:22.080
there's all these different,


00:25:22.080 --> 00:25:24.480
are these the operators or what are these?


00:25:24.480 --> 00:25:25.680
Or the tasks?


00:25:25.680 --> 00:25:27.720
- Those are integrations.


00:25:27.720 --> 00:25:28.560
- Integrations, right.


00:25:28.560 --> 00:25:31.800
- Integrations with the different services you have in.


00:25:31.800 --> 00:25:34.640
So like Google, for example, is a big provider,


00:25:34.640 --> 00:25:36.600
but it consists of integration like Google Cloud,


00:25:36.600 --> 00:25:38.980
PMS, data store, machine learning.


00:25:38.980 --> 00:25:41.280
So you have a number of integrations


00:25:41.280 --> 00:25:43.500
per provider even sometimes.


00:25:43.500 --> 00:25:44.340
Yeah.


00:25:44.340 --> 00:25:45.160
- Okay, cool.


00:25:45.160 --> 00:25:46.460
And Leah, when these,


00:25:46.460 --> 00:25:49.500
if I was gonna create one of these,


00:25:49.500 --> 00:25:51.200
if I was gonna be a good citizen and like,


00:25:51.200 --> 00:25:54.920
oh, I want to create one with AWS Lambda,


00:25:54.920 --> 00:25:56.920
that exists, you know, something like that, right?


00:25:56.920 --> 00:25:57.760
- Yeah.


00:25:57.760 --> 00:25:59.500
- Does that get contributed back to Airflow?


00:25:59.500 --> 00:26:01.560
So when I pip install Airflow, does that come with it?


00:26:01.560 --> 00:26:04.560
Or is there some external way to bring in?


00:26:04.560 --> 00:26:05.800
- Yes.


00:26:05.800 --> 00:26:08.380
So we do actually, well, I'll have to double check


00:26:08.380 --> 00:26:10.820
with Yarek and Caxel because I know we've been messing


00:26:10.820 --> 00:26:13.140
around with how we do the installs lately.


00:26:13.140 --> 00:26:18.020
So it used to be that Airflow operators were packaged


00:26:18.020 --> 00:26:19.460
along with Airflow.


00:26:19.460 --> 00:26:21.440
And when you did pip install Airflow,


00:26:21.440 --> 00:26:23.420
you would get everything.


00:26:23.420 --> 00:26:26.800
And I think that you do still get a certain number


00:26:26.800 --> 00:26:31.660
of base operators that are kind of like provider agnostic


00:26:31.660 --> 00:26:32.900
that come with Airflow.


00:26:32.900 --> 00:26:35.700
But the way we have things now is that all


00:26:35.700 --> 00:26:43.500
these provider-based operator sensors, all these provider task things are packaged separately,


00:26:43.500 --> 00:26:46.580
and you add them just like you would any other kind of Python package.


00:26:46.580 --> 00:26:52.180
So for example, if you want to install the Google Cloud operators, you have that separately.


00:26:52.180 --> 00:26:58.220
And the advantage of that is that they're released on a separate release schedule and


00:26:58.220 --> 00:27:03.700
follow versioning that ensures they're compatible with versions of Airflow, and they're very


00:27:03.700 --> 00:27:10.900
clear about that and it's a lot easier for Airflow users to upgrade just the


00:27:10.900 --> 00:27:15.820
providers package than it is to upgrade the entirety of Airflow which for folks


00:27:15.820 --> 00:27:21.040
running in production that is not always feasible or practical. Yeah you can


00:27:21.040 --> 00:27:25.460
actually click on documentation link on this page Michael and then you'll see


00:27:25.460 --> 00:27:30.780
all of all of those providers so you see little provider packages and you


00:27:30.780 --> 00:27:34.380
can see the documentation of that versions, the different versions.


00:27:34.380 --> 00:27:37.060
We release them very frequently.


00:27:37.060 --> 00:27:41.060
Like every month we have a bunch of providers released which are adding new


00:27:41.060 --> 00:27:45.220
functionality and they are done completely separately, as Lia said,


00:27:45.220 --> 00:27:51.300
not the same release schedule as Airflow and you can start using them


00:27:51.300 --> 00:27:57.220
faster. And this is actually super cool that you can actually


00:27:57.220 --> 00:27:59.340
always find something there.


00:27:59.700 --> 00:28:04.420
But if you don't, we don't actually force you to go this community route.


00:28:04.420 --> 00:28:09.860
Like those are all providers which are developed by community and maintained by the community of Apache Airflow


00:28:09.860 --> 00:28:14.580
under the Apache Software Foundation rules, which is called like Apache Way.


00:28:14.580 --> 00:28:17.540
So the way how Apache releases software.


00:28:17.540 --> 00:28:20.980
But if you want, you can actually build your own custom provider.


00:28:20.980 --> 00:28:25.060
You can build your own custom operators and you can release them separately.


00:28:25.060 --> 00:28:26.340
And somebody can install that.


00:28:26.340 --> 00:28:28.900
And we even have integration points


00:28:28.900 --> 00:28:31.340
that if people are writing the custom providers,


00:28:31.340 --> 00:28:33.180
they can use exactly the same feature


00:28:33.180 --> 00:28:34.620
as the community driven ones.


00:28:34.620 --> 00:28:36.940
And you can install them as a package,


00:28:36.940 --> 00:28:38.620
as another Python package,


00:28:38.620 --> 00:28:40.820
completely independent from Airflow.


00:28:40.820 --> 00:28:44.140
And it just plugs in the UI of Airflow,


00:28:44.140 --> 00:28:47.580
plugs into the whole framework and you can start using it.


00:28:47.580 --> 00:28:50.600
So it's both community and custom.


00:28:50.600 --> 00:28:53.260
- Yeah, you can go either path, right?


00:28:53.260 --> 00:28:54.100
That's neat.


00:28:55.460 --> 00:28:58.780
I think Leah, what you're saying about the cadence,


00:28:58.780 --> 00:28:59.940
the release frequency,


00:28:59.940 --> 00:29:04.580
and maybe even the degree of seriousness


00:29:04.580 --> 00:29:07.500
with which you have to apply to these,


00:29:07.500 --> 00:29:10.740
you might want the main airflow to be treated differently


00:29:10.740 --> 00:29:15.740
than some edge package or integration, right?


00:29:15.740 --> 00:29:18.780
- Yes, definitely.


00:29:18.780 --> 00:29:23.060
- Yeah, there was a proposal for requests,


00:29:23.060 --> 00:29:28.060
the very popular HTTP library to be integrated into Python


00:29:28.060 --> 00:29:32.260
to replace Python's HTTP layer.


00:29:32.260 --> 00:29:36.140
And the decision of the core devs, I believe was,


00:29:36.140 --> 00:29:38.140
we don't wanna do that to requests.


00:29:38.140 --> 00:29:40.700
Like it will actually make requests go much slower


00:29:40.700 --> 00:29:43.080
and only get released once a year with changes


00:29:43.080 --> 00:29:46.540
rather than, you know, as quickly as it needs to go.


00:29:46.540 --> 00:29:47.900
Same thing for you all, right?


00:29:47.900 --> 00:29:50.140
- That was one of the biggest reasons


00:29:50.140 --> 00:29:51.820
for us to separate the providers


00:29:51.820 --> 00:29:56.340
because when we were releasing 1.10.2, 1.10.3, 1.10.4,


00:29:56.340 --> 00:29:59.100
it meant that all the development was happening


00:29:59.100 --> 00:30:01.220
in the main or master branch.


00:30:01.220 --> 00:30:03.840
And we were not releasing from master branch


00:30:03.840 --> 00:30:05.540
because we were just releasing the minor


00:30:05.540 --> 00:30:07.420
or patch versions right now.


00:30:07.420 --> 00:30:11.020
And because the core has to be tested thoroughly,


00:30:11.020 --> 00:30:13.500
even if there's a small bug in one of the providers,


00:30:13.500 --> 00:30:17.180
let's say a Google GCS bucket operator or something,


00:30:17.180 --> 00:30:19.380
it has to wait until the entire core


00:30:19.380 --> 00:30:21.300
has been tested and released.


00:30:21.300 --> 00:30:26.820
So the cycle can be large, whereas what everyone was thinking, at least the committees and


00:30:26.820 --> 00:30:32.740
PMC members, that providers can be released more frequently, even if it means it can be


00:30:32.740 --> 00:30:34.060
released.


00:30:34.060 --> 00:30:39.180
If we find a bug right now, we should fix it, go with the normal ASF releasing way,


00:30:39.180 --> 00:30:42.020
which is like three days of voting and release it.


00:30:42.020 --> 00:30:46.540
So it is quicker release rather than waiting for the next month to club it into the core


00:30:46.540 --> 00:30:47.700
for Airflow release.


00:30:47.700 --> 00:30:51.540
Plus that way it's easier to also check the changes


00:30:51.540 --> 00:30:53.980
that happens because imagine checking the change log


00:30:53.980 --> 00:30:57.500
for 70 odd providers, including the Airflow core


00:30:57.500 --> 00:31:00.540
in a single page, it will be a nightmare.


00:31:00.540 --> 00:31:01.380
- Yeah.


00:31:01.380 --> 00:31:02.420
- Yeah, I don't understand.


00:31:02.420 --> 00:31:03.260
- Yeah, I bet.


00:31:03.260 --> 00:31:04.900
I just think of all the coordination of,


00:31:04.900 --> 00:31:06.740
well, there's some people working on


00:31:06.740 --> 00:31:09.440
the Discord integration and someone's working


00:31:09.440 --> 00:31:12.460
on the Samba integration and we're gonna do a new release.


00:31:12.460 --> 00:31:15.220
You've got to kind of feature freeze all that stuff.


00:31:15.220 --> 00:31:18.340
So yeah, it makes a ton of sense to separate everything.


00:31:18.340 --> 00:31:21.220
- Actually, this is super cool that,


00:31:21.220 --> 00:31:24.540
I'm under release manager for providers so far.


00:31:24.540 --> 00:31:25.700
So I was releasing, I don't know,


00:31:25.700 --> 00:31:29.780
maybe six, seven releases over the last year.


00:31:29.780 --> 00:31:33.860
And actually I do it myself in like two, three hours.


00:31:33.860 --> 00:31:36.580
I'm able to bring all the changes


00:31:36.580 --> 00:31:38.900
and put the release notes for all the 70 providers.


00:31:38.900 --> 00:31:41.620
It's all fully automated and we can manage and release that


00:31:41.620 --> 00:31:44.340
without worry that it will break something


00:31:44.340 --> 00:31:46.840
because if one of those releases goes wrong,


00:31:46.840 --> 00:31:50.580
providers go wrong, we can simply yank this release.


00:31:50.580 --> 00:31:53.240
This is this fantastic feature of PyPy,


00:31:53.240 --> 00:31:54.740
we can yank the release.


00:31:54.740 --> 00:31:56.340
This actually happened yesterday.


00:31:56.340 --> 00:31:59.800
We discovered that the Postgres SQL we released,


00:31:59.800 --> 00:32:03.220
210 version, had an incompatibility bug


00:32:03.220 --> 00:32:04.780
with previous version of Airflow.


00:32:04.780 --> 00:32:06.820
We haven't discovered that during our testing.


00:32:06.820 --> 00:32:07.820
We test a lot of things,


00:32:07.820 --> 00:32:09.420
but this one slipped through.


00:32:09.420 --> 00:32:12.220
But what we've done, just yanked this release.


00:32:12.220 --> 00:32:13.700
Anyone can use this one.


00:32:13.700 --> 00:32:17.160
when they install Airflow and Postgres operators,


00:32:17.160 --> 00:32:18.860
they will install the latest version.


00:32:18.860 --> 00:32:21.240
And in the meantime, we can just fix the Postgres SQL


00:32:21.240 --> 00:32:22.420
and release a new version.


00:32:22.420 --> 00:32:26.460
And that's super cool actually for maintenance release


00:32:26.460 --> 00:32:31.460
and usability and stability of your installation.


00:32:31.460 --> 00:32:34.700
- Yeah, that's really good that you can change that.


00:32:34.700 --> 00:32:37.260
All right, so I wanna talk about,


00:32:37.260 --> 00:32:39.940
first of all, let's talk about installing.


00:32:39.940 --> 00:32:44.640
So what is, you know, how do I get Airflow onto my computer?


00:32:44.640 --> 00:32:50.580
- It depends on if you want a hosted managed version


00:32:50.580 --> 00:32:53.220
like Cloud Composer, which I work on,


00:32:53.220 --> 00:32:55.660
or there is one for Amazon, MWAA,


00:32:55.660 --> 00:32:59.140
and there's also Astronomer, which is where Caxel works.


00:32:59.140 --> 00:33:00.740
Or if you want to do it yourself.


00:33:00.740 --> 00:33:02.860
- Yes, yeah.


00:33:02.860 --> 00:33:03.700
Okay.


00:33:03.700 --> 00:33:06.660
- Yeah, in general though, we at least say


00:33:06.660 --> 00:33:10.180
that use the constraints, constraints file.


00:33:10.180 --> 00:33:13.020
So every time when we release an Airflow version,


00:33:13.020 --> 00:33:17.220
we also tag in GitHub the constraints file


00:33:17.220 --> 00:33:18.700
for each of the release.


00:33:18.700 --> 00:33:22.460
A constraints file contains the set of known dependencies


00:33:22.460 --> 00:33:25.660
that we have tested it, tested Airflow with on the CI


00:33:25.660 --> 00:33:28.740
because Airflow has a lot of dependencies.


00:33:28.740 --> 00:33:33.740
And last, before we started using constraints,


00:33:33.740 --> 00:33:35.500
there were a lot of instances


00:33:35.500 --> 00:33:36.980
where we just released Airflow


00:33:36.980 --> 00:33:39.060
and then one of the dependencies released


00:33:39.060 --> 00:33:41.220
a breaking change in a minor or a patch version,


00:33:41.220 --> 00:33:43.660
which means users couldn't install Airflow.


00:33:43.660 --> 00:33:47.460
And to get over it, we came up with this idea


00:33:47.460 --> 00:33:51.780
of using constraints file because Airflow is a library


00:33:51.780 --> 00:33:53.140
as well as an application.


00:33:53.140 --> 00:33:57.220
So for library users who want latest versions,


00:33:57.220 --> 00:33:58.260
whereas for application,


00:33:58.260 --> 00:34:00.780
you want the stable versions of everything.


00:34:00.780 --> 00:34:02.660
So we came up with this balance


00:34:02.660 --> 00:34:04.020
of using the constraints file.


00:34:04.020 --> 00:34:07.660
So if you check that Airflow version 2.1.2,


00:34:07.660 --> 00:34:08.980
we get the Python version


00:34:08.980 --> 00:34:11.860
and then we fetch that constraints file from GitHub


00:34:11.860 --> 00:34:14.000
and use that constraint file.


00:34:14.000 --> 00:34:18.060
Because that way we can guarantee that it is reproducible


00:34:18.060 --> 00:34:19.560
and it will work for sure.


00:34:19.560 --> 00:34:21.980
- Yeah, very cool.


00:34:21.980 --> 00:34:24.140
So if I go to the documentation,


00:34:24.140 --> 00:34:25.120
there's a couple of options.


00:34:25.120 --> 00:34:28.220
I can run it locally, I can run it in Docker,


00:34:28.220 --> 00:34:30.740
can run it in Astronomer.


00:34:30.740 --> 00:34:33.860
But looking through the script to set things up here,


00:34:33.860 --> 00:34:37.700
It looks like there's a couple of steps.


00:34:37.700 --> 00:34:40.820
So there's a database that does something.


00:34:40.820 --> 00:34:44.660
There's some users who execute the task


00:34:44.660 --> 00:34:47.760
or you don't wanna run as root most likely.


00:34:47.760 --> 00:34:50.520
I suspect that's something you all discourage.


00:34:50.520 --> 00:34:51.580
Probably well.


00:34:51.580 --> 00:34:53.980
And there's a web server and there's a scheduler.


00:34:53.980 --> 00:34:57.140
So who wants, maybe tell us about that.


00:34:57.140 --> 00:34:57.980
Whoever wants to take this.


00:34:57.980 --> 00:35:00.140
- Yeah, so I'll take it.


00:35:00.140 --> 00:35:06.640
So Airflow is pretty complex in setup because it has multiple components.


00:35:06.640 --> 00:35:10.380
Depending on the setup, you can talk to a Kubernetes cluster, you can execute to a work


00:35:10.380 --> 00:35:16.320
flow there, or you can have a Celery queue system processing your tasks and executing


00:35:16.320 --> 00:35:23.740
them on distributed workers because the scalability part, which was one of those features of Airflow.


00:35:23.740 --> 00:35:28.240
So you can have multiple workers, multiple machines,


00:35:28.240 --> 00:35:31.200
and even several hundreds of them if you want.


00:35:31.200 --> 00:35:36.100
And Airflow can be installed using all those capacity.


00:35:36.100 --> 00:35:38.840
So we have cellular workers, we have Kubernetes worker,


00:35:38.840 --> 00:35:41.140
we have scheduler, we have web server.


00:35:41.140 --> 00:35:44.040
And putting it together is not as simple


00:35:44.040 --> 00:35:45.800
as you would think.


00:35:45.800 --> 00:35:49.800
Or actually you can think that it's complex and it is.


00:35:49.800 --> 00:35:52.320
However, we've made like recently,


00:35:52.320 --> 00:35:55.960
especially with a lot of effort to make the kind of


00:35:55.960 --> 00:35:57.960
very simple way of installing Airflow.


00:35:57.960 --> 00:36:00.960
Like, you know, like you just install it and it works.


00:36:00.960 --> 00:36:04.120
And also if you want to scale it to like a very complex one,


00:36:04.120 --> 00:36:07.080
you can also turn on all the knobs,


00:36:07.080 --> 00:36:09.840
put as many components you want in a way


00:36:09.840 --> 00:36:11.600
that fits you best.


00:36:11.600 --> 00:36:14.800
So coming back a little bit to this installation manage,


00:36:14.800 --> 00:36:16.720
we have a Docker image.


00:36:16.720 --> 00:36:19.800
So that's something I also worked for quite some time


00:36:19.800 --> 00:36:22.080
together with Caxial and the other maintainers


00:36:22.080 --> 00:36:23.960
we iterated and perfected it.


00:36:23.960 --> 00:36:26.780
So we have a very nice Docker image that can be used


00:36:26.780 --> 00:36:29.740
to both run Airflow as it is,


00:36:29.740 --> 00:36:31.580
or build your own custom image,


00:36:31.580 --> 00:36:33.960
which contains all the new dependencies you want,


00:36:33.960 --> 00:36:37.180
or all the special packages that you want to install,


00:36:37.180 --> 00:36:38.880
which are needed for you.


00:36:38.880 --> 00:36:41.180
And then from that, we have Docker Compose,


00:36:41.180 --> 00:36:43.100
which is a kind of a quick start.


00:36:43.100 --> 00:36:45.700
So you can just, and this is this running Airflow


00:36:45.700 --> 00:36:47.440
in Docker this part.


00:36:47.440 --> 00:36:50.000
- When you run in Docker, does say the web server


00:36:50.000 --> 00:36:51.720
running on one container and the scheduler


00:36:51.720 --> 00:36:53.320
and another or something like that.


00:36:53.320 --> 00:36:54.160
- That's exactly what this Docker--


00:36:54.160 --> 00:36:56.520
- And then Kripo orchestrates it, yeah, okay.


00:36:56.520 --> 00:37:00.360
- Yes, but it's super easy, like it's really quick start.


00:37:00.360 --> 00:37:04.060
So you just download the Docker compose file,


00:37:04.060 --> 00:37:06.860
you just run two commands, if you go a little bit down,


00:37:06.860 --> 00:37:08.740
then there's like few commands to run.


00:37:08.740 --> 00:37:12.680
And then off you go, you have all these components


00:37:12.680 --> 00:37:16.080
talking together to each other and processing the DAGs.


00:37:16.080 --> 00:37:17.940
And you can start playing with that.


00:37:17.940 --> 00:37:20.600
It's not production ready, the Docker one,


00:37:20.600 --> 00:37:23.480
but then there is the next step.


00:37:23.480 --> 00:37:26.420
So like, so you have a local installation,


00:37:26.420 --> 00:37:28.320
you can do Docker compose, and then,


00:37:28.320 --> 00:37:31.460
and I will transfer it to Kaxi


00:37:31.460 --> 00:37:33.680
because he was working mostly on that.


00:37:33.680 --> 00:37:34.520
So.


00:37:34.520 --> 00:37:37.820
- Yep, so we also have the Helm chart that we,


00:37:37.820 --> 00:37:39.120
the first version of Helm chart


00:37:39.120 --> 00:37:41.400
we released in March of this year.


00:37:41.400 --> 00:37:44.440
So that's what we recommend for production uses.


00:37:44.440 --> 00:37:46.840
That uses the official Docker image.


00:37:46.840 --> 00:37:51.480
So we release a lot of artifacts for Airflow.


00:37:51.480 --> 00:37:53.240
And again, the documentation for Helmchart,


00:37:53.240 --> 00:37:56.800
if you click on documentation, again, at the top,


00:37:56.800 --> 00:37:58.360
and scroll all the way down,


00:37:58.360 --> 00:38:00.760
you will see a separate documentation for Helmchart.


00:38:00.760 --> 00:38:03.120
- Go to Helmchart, yeah, okay, got it.


00:38:03.120 --> 00:38:07.300
- Yeah, so we have worked on all this documentation


00:38:07.300 --> 00:38:09.840
separately because they are different artifacts


00:38:09.840 --> 00:38:12.340
and all of them have different release cadence


00:38:12.340 --> 00:38:14.400
and are released separately.


00:38:14.400 --> 00:38:17.120
And HelmChat is what we recommend for users


00:38:17.120 --> 00:38:19.280
because it comes with all the configurations


00:38:19.280 --> 00:38:22.080
that we have tested it in production environments.


00:38:22.080 --> 00:38:24.240
Astronomer donated the HelmChat last year


00:38:24.240 --> 00:38:29.240
and we traded on it a lot of time before we released it.


00:38:29.240 --> 00:38:33.080
We also, me and Jarek had a presentation


00:38:33.080 --> 00:38:35.000
in a recently concluded Airflow Summit.


00:38:35.000 --> 00:38:37.600
So if users are interested in it,


00:38:37.600 --> 00:38:42.160
we can probably drop a link at the end of this session,


00:38:42.160 --> 00:38:43.000
I guess.


00:38:44.240 --> 00:38:46.500
Yeah, you all just had the Airflow Summit, right?


00:38:46.500 --> 00:38:47.620
- Yeah. - Yep.


00:38:47.620 --> 00:38:50.420
- Yeah, oh, I have a lot to say about this.


00:38:50.420 --> 00:38:51.780
- All right, well, tell us.


00:38:51.780 --> 00:38:54.620
- Yeah, community is definitely where the majority


00:38:54.620 --> 00:38:57.060
of my contributions to Airflow come in.


00:38:57.060 --> 00:39:01.300
So this was our second ever Airflow Summit.


00:39:01.300 --> 00:39:03.260
So far it's been an annual thing,


00:39:03.260 --> 00:39:05.140
but I'm always nervous to say annual


00:39:05.140 --> 00:39:07.020
because I don't wanna make promises,


00:39:07.020 --> 00:39:09.260
but it's looking good, like we'll have it again.


00:39:09.260 --> 00:39:11.980
So we had our first summit in 2020.


00:39:11.980 --> 00:39:16.260
We had originally planned to have it be this 500 person


00:39:16.260 --> 00:39:19.260
in-person event, it was going to be in Mountain View.


00:39:19.260 --> 00:39:22.340
That's how I got involved because we were looking to host it


00:39:22.340 --> 00:39:23.720
at the Computer History Museum.


00:39:23.720 --> 00:39:26.180
When I said, "Oh, that's really close to where I work.


00:39:26.180 --> 00:39:29.380
I can like be your liaison to the location."


00:39:29.380 --> 00:39:31.340
And then, you know, there was a whole pandemic


00:39:31.340 --> 00:39:33.340
and everything, and we ended up pivoting


00:39:33.340 --> 00:39:35.740
to a totally virtual event.


00:39:35.740 --> 00:39:37.460
And it was a great success.


00:39:37.460 --> 00:39:40.500
We did it in partnership with Software Guru.


00:39:40.500 --> 00:39:43.020
they helped us run the summit last year.


00:39:43.020 --> 00:39:46.280
And we felt that it was such a good success


00:39:46.280 --> 00:39:47.960
that we did it again this year.


00:39:47.960 --> 00:39:50.940
And it just finished up in July.


00:39:50.940 --> 00:39:54.500
We had 10,000, I think more than 10,000 at this point,


00:39:54.500 --> 00:39:56.340
registered attendees from all over the world.


00:39:56.340 --> 00:39:59.040
- That's really good for an online conference.


00:39:59.040 --> 00:40:02.060
- For only the second edition too, we're pretty proud.


00:40:02.060 --> 00:40:04.900
And we had it live streamed


00:40:04.900 --> 00:40:06.580
in a bunch of different time zones.


00:40:06.580 --> 00:40:08.300
So sometimes it is more America's friendly,


00:40:08.300 --> 00:40:09.780
sometimes it is more EMEA friendly,


00:40:09.780 --> 00:40:11.820
sometimes it was more APAC friendly.


00:40:11.820 --> 00:40:14.240
And we had all variations of talks.


00:40:14.240 --> 00:40:18.060
We had ones that were customer use cases.


00:40:18.060 --> 00:40:21.640
So people who are running Airflow in production


00:40:21.640 --> 00:40:25.260
or running one of the hosted managed versions of Airflow


00:40:25.260 --> 00:40:27.100
and what they're using it for.


00:40:27.100 --> 00:40:28.860
We had people who are contributors


00:40:28.860 --> 00:40:32.520
talking about their first time contribution experience


00:40:32.520 --> 00:40:35.020
and why you shouldn't be scared to contribute to Airflow


00:40:35.020 --> 00:40:37.560
because we're really nice, I promise we are,


00:40:37.560 --> 00:40:38.700
or at least we try to be.


00:40:38.700 --> 00:40:44.220
And we had more experienced contributors like Yarek and Kaxil talk about some of the more


00:40:44.220 --> 00:40:48.700
complex things that they've been working on over the past year and everything in between.


00:40:48.700 --> 00:40:56.140
And there are so many talks and you have the Summit page up right now. Actually,


00:40:56.140 --> 00:41:01.660
all of the recordings and slides for those presentations that had slides available


00:41:01.660 --> 00:41:08.140
are up there for you to watch if you go to airflowsummit.org. There's many, many, many


00:41:08.140 --> 00:41:14.540
hours of content, I highly encourage you to watch whatever sounds interesting for you.


00:41:14.540 --> 00:41:20.860
Yeah, I think this is great. Like I said, congratulations on having 10,000 registered.


00:41:20.860 --> 00:41:21.660
Thank you.


00:41:21.660 --> 00:41:28.940
Yeah, that's pretty amazing. I think there's obviously a big group of people who know that


00:41:28.940 --> 00:41:33.100
this is like the right tool. I think there's a lot of people who necessarily don't know for sure.


00:41:33.100 --> 00:41:40.220
Like for example, there's on the Airflow GitHub page, it's 23,000 stars. That's big time.


00:41:40.220 --> 00:41:41.760
Yeah.


00:41:41.760 --> 00:41:48.460
You can go in Classica, 50K. So I mean, that's a lot of people using this and interested in this and so on.


00:41:48.460 --> 00:41:56.220
I think the best part about Airflow is the community. And that's like why we have those stars,


00:41:56.220 --> 00:41:59.580
but also like why we had such a summit. And Caxl, you were going to say something.


00:42:00.140 --> 00:42:18.900
Yeah, I was just going to say that if you go by the PyPy stats, we have like 3 million downloads a month or something like that, which is insane. I know a good number of those come from CI and automated processes. But hey, all the other packages also have the same thing. So you can at least compare them between packages.


00:42:18.900 --> 00:42:21.220
Yeah, it's a relative statement, at least right.


00:42:21.260 --> 00:42:28.300
Exactly and likely I mentioned like the biggest part about or biggest greatest thing about Airflow


00:42:28.300 --> 00:42:35.180
is its community. If you check the new contributors I think we are more than 1600 contributors to


00:42:35.180 --> 00:42:41.580
the Airflow project which is great and every day we at least get few new contributors trying to


00:42:41.580 --> 00:42:47.980
contribute to the project with whatever they can and they must and I again through your medium I


00:42:47.980 --> 00:42:51.780
I would encourage people to go to Airflow website.


00:42:51.780 --> 00:42:55.260
If you find anything, contribute it, fix it.


00:42:55.260 --> 00:42:58.700
If you have some ideas about hooks, operators, anything,


00:42:58.700 --> 00:43:02.100
contribute it and we are there to help you.


00:43:02.100 --> 00:43:04.380
Not only three of us,


00:43:04.380 --> 00:43:08.780
there are more than 30, 40 commuters and PMC members


00:43:08.780 --> 00:43:10.500
and there are users helping users


00:43:10.500 --> 00:43:11.620
in the Airflow Slack channel.


00:43:11.620 --> 00:43:15.180
We have more than 16, 17,000 members


00:43:15.180 --> 00:43:17.620
in the Airflow Slack workspace as well.


00:43:17.620 --> 00:43:18.700
- Wow, that's cool.


00:43:18.700 --> 00:43:20.700
- So I actually wanna give a quick plug


00:43:20.700 --> 00:43:23.500
for an Airflow Summit talk I gave this year


00:43:23.500 --> 00:43:25.700
that was authored by me and a colleague.


00:43:25.700 --> 00:43:27.140
It's called, "You Don't Have to Wait


00:43:27.140 --> 00:43:29.300
"for Someone to Fix it for You."


00:43:29.300 --> 00:43:32.540
And it is about the kinds of contributions


00:43:32.540 --> 00:43:33.940
that you can make to Airflow.


00:43:33.940 --> 00:43:36.060
'Cause there's all those things that Caxel mentioned,


00:43:36.060 --> 00:43:40.140
but my personal opinion is that one of the best


00:43:40.140 --> 00:43:42.540
and easiest ways to contribute to Airflow


00:43:42.540 --> 00:43:44.620
or any open source project really


00:43:44.620 --> 00:43:47.540
is to find something that is driving you nuts.


00:43:47.540 --> 00:43:51.820
and to fix it, or at least to articulate really well


00:43:51.820 --> 00:43:54.860
what's driving you nuts and what needs to change


00:43:54.860 --> 00:43:57.900
because a really good issue can be just as good


00:43:57.900 --> 00:44:01.220
of a contribution as a PR, 'cause you may have just


00:44:01.220 --> 00:44:05.100
made the foundation for someone else to write a fabulous PR


00:44:05.100 --> 00:44:07.340
with a really detailed issue.


00:44:07.340 --> 00:44:09.340
- Yeah, and let me add to that as well,


00:44:09.340 --> 00:44:13.300
because the community is definitely the thing


00:44:13.300 --> 00:44:15.820
that I love most about Airflow.


00:44:15.820 --> 00:44:18.860
The people are fantastic here.


00:44:18.860 --> 00:44:22.500
And we are, all of us, all the committors,


00:44:22.500 --> 00:44:25.540
we are so much into, you know,


00:44:25.540 --> 00:44:30.100
making, like inviting people to come and to join us


00:44:30.100 --> 00:44:33.940
or to give back for whatever they got from Airflow.


00:44:33.940 --> 00:44:36.740
Like it's a free software, anyone can use it for free.


00:44:36.740 --> 00:44:39.980
So giving back is just super nice.


00:44:39.980 --> 00:44:42.540
But we don't stop talking, only talking about them.


00:44:42.540 --> 00:44:45.700
Because if you see, scroll down a little bit above,


00:44:45.700 --> 00:44:49.860
you would see that we also run a workshop


00:44:49.860 --> 00:44:52.420
during the Aero Summit.


00:44:52.420 --> 00:44:55.500
And this workshop is about contributing to Apache Airflow.


00:44:55.500 --> 00:44:58.580
This year we had like 20 attendees coming


00:44:58.580 --> 00:45:02.220
and learning in three hours how to make your first VR,


00:45:02.220 --> 00:45:06.580
how to communicate, how to be present in the community,


00:45:06.580 --> 00:45:07.820
how to make the most of it,


00:45:07.820 --> 00:45:11.500
how to be super helpful to others as well.


00:45:11.500 --> 00:45:13.060
And then we were just,


00:45:13.060 --> 00:45:15.960
It was like part of it was about coping,


00:45:15.960 --> 00:45:18.400
but all the rest was all about communication,


00:45:18.400 --> 00:45:19.720
about speaking to people,


00:45:19.720 --> 00:45:22.240
about being able to express yourself


00:45:22.240 --> 00:45:23.640
and all the stuff that we just needed.


00:45:23.640 --> 00:45:24.480
And it is--


00:45:24.480 --> 00:45:27.400
- Who should I ask about this and things like that?


00:45:27.400 --> 00:45:29.520
- I know exactly who you should ask.


00:45:29.520 --> 00:45:32.400
So actually one of my favorite stories


00:45:32.400 --> 00:45:35.080
about this year's Airflow Summit is we had a speaker,


00:45:35.080 --> 00:45:37.480
I forget her last name, her first name is Tatiana,


00:45:37.480 --> 00:45:40.640
and she's like a principal data engineer at the BBC,


00:45:40.640 --> 00:45:43.500
And she went to the workshop last year.


00:45:43.500 --> 00:45:45.700
And this year she was a speaker at the summit


00:45:45.700 --> 00:45:49.440
and her talk about how to basically like how to debug


00:45:49.440 --> 00:45:53.920
when crazy stuff is going wrong in Airflow was fabulous.


00:45:53.920 --> 00:45:54.840
- Oh, super.


00:45:54.840 --> 00:45:55.680
Okay, yeah.


00:45:55.680 --> 00:45:58.040
And you can, that's, people can live stream


00:45:58.040 --> 00:45:58.960
that off the sessions.


00:45:58.960 --> 00:46:00.240
That's really cool.


00:46:00.240 --> 00:46:02.920
Clearing Airflow obstructions.


00:46:02.920 --> 00:46:03.760
Awesome.


00:46:03.760 --> 00:46:06.260
- So that is an example of that workshop working.


00:46:06.260 --> 00:46:07.100
- Yeah, yeah.


00:46:07.100 --> 00:46:07.920
Very cool.


00:46:08.720 --> 00:46:11.140
I feel like-- go ahead, Geksel.


00:46:11.140 --> 00:46:13.400
Yeah, I was just saying, Airflow Summit is also


00:46:13.400 --> 00:46:14.840
one-of-a-kind conference.


00:46:14.840 --> 00:46:17.080
It's not like the normal conferences,


00:46:17.080 --> 00:46:22.560
mainly because we had the local meetup groups hosting


00:46:22.560 --> 00:46:23.720
that day of the event.


00:46:23.720 --> 00:46:25.680
So we had the London meetup group.


00:46:25.680 --> 00:46:28.640
We had the Bangalore meetup group, Melbourne, Warsaw


00:46:28.640 --> 00:46:29.480
meetup group.


00:46:29.480 --> 00:46:32.720
And we were bringing the community together.


00:46:32.720 --> 00:46:35.520
So let's say the first day was hosted by the London meetup


00:46:35.520 --> 00:46:38.160
group, which was me, Ash, and other folks.


00:46:38.160 --> 00:46:42.160
We were hosting that event just for the Monday slot.


00:46:42.160 --> 00:46:45.080
And then on the Tuesday, there were other PMC members,


00:46:45.080 --> 00:46:48.600
other community members from Japan hosting that,


00:46:48.600 --> 00:46:50.200
some from Melbourne hosting that.


00:46:50.200 --> 00:46:52.400
Similarly, those were the slots.


00:46:52.400 --> 00:46:55.840
And someday, even we had some sort of overlap


00:46:55.840 --> 00:46:59.000
because we were trying to cover the Pacific time zone


00:46:59.000 --> 00:47:01.960
and the Asian time zones, which was incredible


00:47:01.960 --> 00:47:04.360
because now you have tons of content


00:47:04.360 --> 00:47:06.480
for the Airflow users to watch out.


00:47:06.480 --> 00:47:08.400
Also, we had two community days.


00:47:08.400 --> 00:47:09.840
We started from Thursdays.


00:47:09.840 --> 00:47:13.960
So we had Thursday, the talks about community,


00:47:13.960 --> 00:47:16.800
how you can make the contributions and stuff like that.


00:47:16.800 --> 00:47:18.640
Friday, we had that workshop.


00:47:18.640 --> 00:47:20.400
And then from Monday to Friday,


00:47:20.400 --> 00:47:22.920
there were more about the Airflow use cases


00:47:22.920 --> 00:47:27.520
and why Airflow 2.0 was the big milestone for the project


00:47:27.520 --> 00:47:30.920
and what we are planning ahead for Airflow


00:47:30.920 --> 00:47:32.360
and the stuff like that.


00:47:32.360 --> 00:47:35.600
- Oh yeah, there's a ton of stuff here.


00:47:35.600 --> 00:47:38.160
I think people could watch for the rest of the year


00:47:38.160 --> 00:47:39.920
and study this and get a lot out of it.


00:47:39.920 --> 00:47:40.760
- It's true.


00:47:40.760 --> 00:47:41.960
- I do think so.


00:47:41.960 --> 00:47:45.600
- And we actually even had a networking event there


00:47:45.600 --> 00:47:46.760
Friday night. - We did.


00:47:46.760 --> 00:47:48.360
- And that was a blast actually.


00:47:48.360 --> 00:47:49.480
- It was, yeah.


00:47:49.480 --> 00:47:51.680
- The networking this year was like,


00:47:51.680 --> 00:47:53.440
people learn how to use it online.


00:47:53.440 --> 00:47:55.840
And that was like, well,


00:47:55.840 --> 00:47:58.760
not maybe as good as physical conferences.


00:47:58.760 --> 00:48:00.240
So I'm looking forward to next year,


00:48:00.240 --> 00:48:02.840
which hopefully we're gonna be partially


00:48:02.840 --> 00:48:04.700
at least physical event.


00:48:04.700 --> 00:48:05.700
But it was good enough.


00:48:05.700 --> 00:48:09.940
And I think that was really cool to talk to those people


00:48:09.940 --> 00:48:12.200
about all the different things, not only Airflow.


00:48:12.200 --> 00:48:15.660
So we are not only Airflow and not only Python


00:48:15.660 --> 00:48:19.260
and not only programming, but also people.


00:48:19.260 --> 00:48:22.580
- Yeah, I feel like this is a project


00:48:22.580 --> 00:48:24.320
that would be easy to contribute to


00:48:24.320 --> 00:48:26.540
in the sense that if I'm gonna say contribute


00:48:26.540 --> 00:48:29.900
as a newcomer to Django, that's gonna be hard


00:48:29.900 --> 00:48:33.260
because that's a highly polished single piece of software.


00:48:33.260 --> 00:48:34.660
And if you're gonna make a change,


00:48:34.660 --> 00:48:37.680
that affects millions of people and it's not easy.


00:48:37.680 --> 00:48:40.360
Whereas here, if you wanna add some kind of integration


00:48:40.360 --> 00:48:42.220
and it didn't exist before,


00:48:42.220 --> 00:48:43.800
you're not gonna break anybody's code,


00:48:43.800 --> 00:48:45.620
you don't wanna work with a bunch of legacy code.


00:48:45.620 --> 00:48:48.580
There's a bunch of sort of broad but shallow places


00:48:48.580 --> 00:48:51.480
people could jump in and participate.


00:48:51.480 --> 00:48:54.280
- Well, and if people, if a newcomer does wanna come in


00:48:54.280 --> 00:48:56.680
and like really jump into the deep end,


00:48:56.680 --> 00:49:00.680
we do have this concept called AIP,


00:49:00.680 --> 00:49:03.600
which stands for Airflow Improvement Proposal.


00:49:03.600 --> 00:49:08.000
And it kind of sets you up to not run into a heartbreak


00:49:08.000 --> 00:49:10.680
if you open this, what you think is an amazing PR


00:49:10.680 --> 00:49:12.560
and we're like, oh no, no, no, hold on.


00:49:12.560 --> 00:49:13.760
We're not ready for that.


00:49:13.760 --> 00:49:15.960
'Cause it's almost like writing that outline


00:49:15.960 --> 00:49:17.440
before you write your essay.


00:49:17.440 --> 00:49:18.960
I know it sounds kind of dry,


00:49:18.960 --> 00:49:21.000
but what it really is, is it's an opportunity


00:49:21.000 --> 00:49:24.080
to fully flesh out this amazing idea you have


00:49:24.080 --> 00:49:25.820
and share it with the community


00:49:25.820 --> 00:49:28.000
and the community will give you feedback


00:49:28.000 --> 00:49:30.120
and they will be productive about it


00:49:30.120 --> 00:49:31.200
because if they're not,


00:49:31.200 --> 00:49:34.200
they're not abiding by community code of conduct.


00:49:34.200 --> 00:49:38.960
- Yeah, I find it very unfortunate.


00:49:38.960 --> 00:49:41.880
I feel really bad if people come and do a PR


00:49:41.880 --> 00:49:43.260
to some project that I have,


00:49:43.260 --> 00:49:45.420
and granted, these are all very small open source projects,


00:49:45.420 --> 00:49:47.520
but if they come and they actually do the work,


00:49:47.520 --> 00:49:50.200
and the first I know about it is, boom, here's a PR.


00:49:50.200 --> 00:49:51.040
- Yeah.


00:49:51.040 --> 00:49:52.260
- That is not in the,


00:49:52.260 --> 00:49:55.420
that's just not in the same zen


00:49:55.420 --> 00:49:57.020
of what I'm trying to accomplish with this.


00:49:57.020 --> 00:50:00.260
And it's gonna break the thing that makes it special.


00:50:00.260 --> 00:50:02.620
So I have to reject it, right? But you don't want to.


00:50:02.620 --> 00:50:04.800
It would be much better to say, "I have this idea.


00:50:04.800 --> 00:50:06.800
If I built this, would you want it?"


00:50:06.800 --> 00:50:09.300
Do you want the puppy? Here's a puppy for Christmas.


00:50:09.300 --> 00:50:10.800
Yes, exactly.


00:50:10.800 --> 00:50:15.040
This is precisely what we are teaching people at those workshops


00:50:15.040 --> 00:50:16.540
because it's not obvious.


00:50:16.540 --> 00:50:18.740
If you come from outside, you don't understand that.


00:50:18.740 --> 00:50:22.820
So we're not only teaching people about contributing the code,


00:50:22.820 --> 00:50:25.280
but also how to find yourself there,


00:50:25.280 --> 00:50:27.280
like how to be empathetic,


00:50:27.280 --> 00:50:34.160
how to think about, put yourself in our shoes and on the other hand, how to tell what you want to


00:50:34.160 --> 00:50:38.960
tell in the way that we will understand it. Because it's sometimes really different worlds,


00:50:38.960 --> 00:50:43.600
different people, different backgrounds, different expectations and assumptions. So all this is the


00:50:43.600 --> 00:50:48.880
communication is that I think, I'm a software engineer, I love to do software engineering,


00:50:49.680 --> 00:50:54.680
But like 30, 40, 50% of my time is communication.


00:50:54.680 --> 00:50:57.280
It's not actually coding.


00:50:57.280 --> 00:50:59.080
And this is cool.


00:50:59.080 --> 00:51:03.840
Related to this, I actually want to call out a really important Apache value


00:51:03.840 --> 00:51:06.160
that I think that Airflow embodies,


00:51:06.160 --> 00:51:10.480
which is the concept of the importance of community over code.


00:51:10.480 --> 00:51:15.600
And I really feel that the Airflow project lives that value.


00:51:15.600 --> 00:51:29.360
And folks in the community really are trying to foster a positive community because they understand that if the airflow community is not healthy, then the airflow code will not live on.


00:51:29.360 --> 00:51:30.360
It doesn't matter.


00:51:30.360 --> 00:51:31.840
Yeah, it doesn't matter.


00:51:31.840 --> 00:51:40.840
And if folks have questions about that, I do want to acknowledge that I am the one woman in the room.


00:51:40.840 --> 00:51:44.040
I am often the one woman in the room


00:51:44.040 --> 00:51:45.480
when it comes to Airflow,


00:51:45.480 --> 00:51:47.840
and I would love to see that change


00:51:47.840 --> 00:51:51.320
and have more gender diverse folks come join.


00:51:51.320 --> 00:51:54.200
And so if you are someone who identifies with that


00:51:54.200 --> 00:51:58.680
and wants to hear Leah's unfiltered views on the community,


00:51:58.680 --> 00:52:00.920
feel free to reach out to me in the Airflow Slack.


00:52:00.920 --> 00:52:03.800
- Yeah, that's fantastic. - Or on my Twitter, yeah.


00:52:03.800 --> 00:52:05.560
- And like I said, I do think this is a project


00:52:05.560 --> 00:52:07.040
that if you wanna get into open source,


00:52:07.040 --> 00:52:09.640
it's one that has relatively low barriers,


00:52:09.640 --> 00:52:10.640
technically speaking.


00:52:10.640 --> 00:52:11.640
Yes.


00:52:11.640 --> 00:52:12.640
Oh, yeah.


00:52:12.640 --> 00:52:18.320
The keynote talk I gave in the Airflow Summit on Thursday, the first talk, so if you go


00:52:18.320 --> 00:52:24.160
to the Airflow Summit page, the very first talk, where I talk about my journey as well,


00:52:24.160 --> 00:52:27.840
because I was very afraid of contributing to open source because it feels intimidating


00:52:27.840 --> 00:52:31.200
at first on like everything will be public.


00:52:31.200 --> 00:52:34.120
Who knows if I screw something up?


00:52:34.120 --> 00:52:35.120
What would people say?


00:52:35.120 --> 00:52:38.120
So be on my permanent record.


00:52:38.120 --> 00:52:39.120
Yeah.


00:52:39.120 --> 00:52:44.440
So, and I didn't know Python or didn't know it proficiently.


00:52:44.440 --> 00:52:49.640
So I talk about my journey of how I did it, talk about 10 minutes about that, and then


00:52:49.640 --> 00:52:52.920
how a new user can start contributing to the project.


00:52:52.920 --> 00:52:56.240
Because Airflow is a relatively still a larger code base.


00:52:56.240 --> 00:52:59.120
And there are a lot of areas that people can target.


00:52:59.120 --> 00:53:02.320
Because if you try to learn everything at once, it is going to be very difficult.


00:53:02.320 --> 00:53:07.360
We have Helm charts, we have Docker images, we have scheduler, which is core to Airflow,


00:53:07.360 --> 00:53:10.160
We have executors, we have the CLI, REST API,


00:53:10.160 --> 00:53:12.560
and a lot of things like that.


00:53:12.560 --> 00:53:14.400
So there are a lot of room for people


00:53:14.400 --> 00:53:16.720
to get expertise in a certain area.


00:53:16.720 --> 00:53:19.920
And then if you start including all the integrations,


00:53:19.920 --> 00:53:21.560
then it's a whole piece, right?


00:53:21.560 --> 00:53:23.200
You can just add your own integration


00:53:23.200 --> 00:53:26.760
and be an expert at that and become a contributor,


00:53:26.760 --> 00:53:30.440
committer, PMC member, just with that contributions.


00:53:30.440 --> 00:53:32.560
- Well, and in the interest of empathy,


00:53:32.560 --> 00:53:35.980
I would like to share that I do not know all of these parts.


00:53:35.980 --> 00:53:40.540
I think the part I'm most familiar with is the Google provider and I have never touched


00:53:40.540 --> 00:53:44.060
the helm chart and it scares me because I haven't taken the time to learn what it's


00:53:44.060 --> 00:53:45.060
all about.


00:53:45.060 --> 00:53:49.940
But the good news is that community members, other community members know and I know that


00:53:49.940 --> 00:53:54.980
I can look to them for help when I do need to mess around with it.


00:53:54.980 --> 00:53:57.460
That's the beauty of the project, right?


00:53:57.460 --> 00:54:01.180
If everyone knows everything, then why are we all here?


00:54:01.180 --> 00:54:05.060
Each one of us knows their part, then that's the community.


00:54:05.060 --> 00:54:07.820
Otherwise it's not a community project, yeah.


00:54:07.820 --> 00:54:08.740
- Yeah. - Yeah.


00:54:08.740 --> 00:54:09.820
- We're getting short on time.


00:54:09.820 --> 00:54:12.060
I do wanna touch on a couple of things that I think


00:54:12.060 --> 00:54:13.460
we haven't got a chance to touch on


00:54:13.460 --> 00:54:14.900
that are really important.


00:54:14.900 --> 00:54:21.380
One, yeah, let's talk about the user interface


00:54:21.380 --> 00:54:25.620
because one of the ways you all positioned to this


00:54:25.620 --> 00:54:28.560
is you don't wanna do this all with just cron jobs


00:54:28.560 --> 00:54:31.360
and like sort of little scripts that are put together


00:54:31.360 --> 00:54:34.500
and run on weird random triggers.


00:54:34.500 --> 00:54:38.100
And one of the real big benefits is you have this really beautiful UI


00:54:38.100 --> 00:54:42.140
for all sorts of visualization of running workflows and all kinds of stuff.


00:54:42.140 --> 00:54:44.140
You want to tell us about that?


00:54:44.140 --> 00:54:47.680
I'll do the simple version


00:54:47.680 --> 00:54:50.820
because I think that Kaxel and Jarek know more about it than me,


00:54:50.820 --> 00:54:53.280
but I'll tell you the two things I'm most excited about.


00:54:53.280 --> 00:54:56.520
One of them is that it just got a huge makeover with Airflow 2.


00:54:56.520 --> 00:54:59.920
So if you're an Airflow user and you haven't upgraded to Airflow 2,


00:54:59.920 --> 00:55:06.560
If you need one reason alone, it is that the UI is so much prettier and it is much more responsive.


00:55:06.560 --> 00:55:15.680
And as a former Cron user, I'll say that the best, easiest benefit you get from this is you can just see what's failing.


00:55:15.680 --> 00:55:20.440
If you don't have to dig around and try to figure out what's missing, like, you know that something went wrong.


00:55:20.440 --> 00:55:23.520
All right, Yarek and Kaxil, that's my, I'm off my soapbox now.


00:55:24.160 --> 00:55:28.040
Yeah, basically, you have all the information you need all the historical view in front of you.


00:55:28.040 --> 00:55:32.280
Like if you want to see which task failed historically, you could just check the preview.


00:55:32.280 --> 00:55:35.920
And then this is the graph view where you can see how your task is proceeding.


00:55:35.920 --> 00:55:45.640
Plus, we now have auto refresh like Leah mentioned from 2.0, which is like you don't need to press the refresh button, which was a bit annoying for the Airflow 110x version, which is very good.


00:55:45.640 --> 00:55:46.880
Your task will continue.


00:55:46.880 --> 00:55:50.320
You can see the progress that Airflow is continuing this task.


00:55:50.320 --> 00:55:53.920
If you click on that task, it will show you the logs of that task.


00:55:53.920 --> 00:55:58.920
So everything is very intuitive and easy to monitor.


00:55:58.920 --> 00:56:00.520
- For people who are listening


00:56:00.520 --> 00:56:02.160
and are not watching the live stream,


00:56:02.160 --> 00:56:05.200
you can go and for example, in the graph,


00:56:05.200 --> 00:56:08.040
it'll show you all your tasks that you would do,


00:56:08.040 --> 00:56:11.600
like download this file or run this bash script or whatever.


00:56:11.600 --> 00:56:13.400
And then it actually shows you how they're working together.


00:56:13.400 --> 00:56:15.200
And then they're colored as you progress


00:56:15.200 --> 00:56:18.160
through this DAG of tasks, right?


00:56:18.160 --> 00:56:21.200
So you can actually visually see, was this one skipped?


00:56:21.200 --> 00:56:22.360
Was this one successful?


00:56:22.360 --> 00:56:23.180
Which one failed?


00:56:23.180 --> 00:56:26.100
how far are you visually as a graph, which I think is awesome.


00:56:26.100 --> 00:56:30.420
Yeah. And one of the interesting thing over there is to understand the dependencies,


00:56:30.420 --> 00:56:34.180
which was very interesting when I initially started with Airflow,


00:56:34.180 --> 00:56:39.220
that for a user or for a company to understand what all the tasks they are working on


00:56:39.220 --> 00:56:42.700
and in a single flow, how does that dependency graph work on?


00:56:42.700 --> 00:56:47.420
If you are depending on data from a single client, how does that go to a dashboard?


00:56:47.420 --> 00:56:52.940
So that end to end view, like, it's an actual pipeline of sorts that you can see.


00:56:52.940 --> 00:56:59.740
Yes, and just to add on that, so the visualization of the data flow is like super important,


00:56:59.740 --> 00:57:02.740
because then you can, with a glance, you can see what's going on,


00:57:02.740 --> 00:57:09.240
and you can go to any part of it and focus on that and understand what's going on.


00:57:09.240 --> 00:57:13.840
However, I will come back to kind of the roots,


00:57:13.840 --> 00:57:22.880
because Airflow doesn't have a way by default to define those flows visually.


00:57:22.880 --> 00:57:27.200
You can see them visually, but they are all defined as Python code.


00:57:27.200 --> 00:57:31.760
And this is like the beauty of it, and that was a very, very deliberate choice.


00:57:31.760 --> 00:57:34.640
And this is the reason why we are at the Python talks today,


00:57:34.640 --> 00:57:37.440
because Airflow is all about Python.


00:57:37.440 --> 00:57:42.560
So this visualization that you see here are really reflection of the code


00:57:42.560 --> 00:57:47.680
that you wrote as a writer. And it means also that the common language between people


00:57:47.680 --> 00:57:54.160
using Airflow, different parts of it, is Python. And this is the common language that we're using.


00:57:54.160 --> 00:58:03.040
And this makes it so powerful. And the visual part is pretty much an addition. And it's necessary,


00:58:03.040 --> 00:58:06.720
but it's more kind of result of the Python code which is being written.


00:58:06.720 --> 00:58:11.360
A lot of workflow systems try to go in reverse, right? They're like, here's your draggy-droppy


00:58:11.360 --> 00:58:14.960
set of tasks and options, you drag it all together, then you press go.


00:58:14.960 --> 00:58:19.600
Right? This lets you at the code level.


00:58:19.600 --> 00:58:25.440
- This all breaks at the very moment when you want to have to do some custom work.


00:58:25.440 --> 00:58:31.040
Because if you are used to the drag and dropping, you will not do coding. You will not code the


00:58:31.040 --> 00:58:36.160
kind of customization that you want to do. You will ask someone else to do that. In Airflow,


00:58:36.160 --> 00:58:39.520
this is quite reverse. I mean, everything is Python. Everything.


00:58:39.520 --> 00:58:44.320
The dependencies are Python. The code itself is Python. The blocks are Python.


00:58:44.320 --> 00:58:47.760
But you can also write your own... In the same place where you define your


00:58:47.760 --> 00:58:50.960
DAG, you can write your own custom operator


00:58:50.960 --> 00:58:55.360
without having to use a black box operator of sorts.


00:58:55.360 --> 00:58:59.120
And you don't have to leave the box of working on Python


00:58:59.120 --> 00:59:02.960
while doing that. And this is so powerful, I think. This is the


00:59:02.960 --> 00:59:08.640
way how why it is so popular between data engineers all over the world. I think it's like one of the


00:59:08.640 --> 00:59:16.640
most popular workflow orchestration engine in the world right now. I wouldn't, I think that's


00:59:16.640 --> 00:59:22.320
it. I don't have hard data on that so you know like it's just a feeling but I think it's


00:59:22.320 --> 00:59:30.080
that's the case. I mean we can have four thousand people at the summit, Jark. Yeah for sure. And while it is


00:59:30.080 --> 00:59:35.240
is written in Python, you can use the bash operator to run like your Java code, for example,


00:59:35.240 --> 00:59:41.800
or Scala or whatever. So while everything is in Python, you can use it to run any other


00:59:41.800 --> 00:59:42.800
languages too.


00:59:42.800 --> 00:59:45.920
Yeah, you've got, you've got, go ahead.


00:59:45.920 --> 00:59:50.000
You can run Docker image, Kubernetes does, because a lot of those workflows are also


00:59:50.000 --> 00:59:55.800
okay, we have Kubernetes. So we run everything in Kubernetes, we run them as Docker containers.


00:59:55.800 --> 00:59:58.320
And that's the only way you can do that.


00:59:58.320 --> 00:59:59.440
Airflow can do that as well.


00:59:59.440 --> 01:00:00.800
No problem whatsoever.


01:00:00.800 --> 01:00:02.160
There is Kubernetes pod operator.


01:00:02.160 --> 01:00:06.000
You can spin off a new Kubernetes pod to run your tasks,


01:00:06.000 --> 01:00:07.640
but you can also have a Python code,


01:00:07.640 --> 01:00:10.300
which is very easy to like put together and play with


01:00:10.300 --> 01:00:12.680
and run locally without all the overhead


01:00:12.680 --> 01:00:14.460
of building the Docker images


01:00:14.460 --> 01:00:17.960
and making them available to run you as a task.


01:00:17.960 --> 01:00:22.080
It's so much more extensible and powerful in this way.


01:00:22.080 --> 01:00:24.880
- Yeah, that's a very good point.


01:00:24.880 --> 01:00:26.560
There's a lot of escape hatches


01:00:26.560 --> 01:00:27.800
to bring in other technologies.


01:00:27.800 --> 01:00:28.640
That's cool.


01:00:28.640 --> 01:00:30.580
Let me give people just a super quick sense


01:00:30.580 --> 01:00:34.000
of what it's like to write code for this, this Python code.


01:00:34.000 --> 01:00:37.800
So you would say with dag, with directed acyclic graph,


01:00:37.800 --> 01:00:39.560
and you give it some details,


01:00:39.560 --> 01:00:41.880
and then you create these various tasks,


01:00:41.880 --> 01:00:46.480
like a task might be a bash operator or something like that,


01:00:46.480 --> 01:00:48.920
or like you said, a Kubernetes pod or whatever,


01:00:48.920 --> 01:00:50.840
and then you just run them.


01:00:50.840 --> 01:00:52.040
One thing I did want to ask you all about,


01:00:52.040 --> 01:00:56.280
Like what is this T1 double arrows


01:00:56.280 --> 01:00:59.440
into list of T2, T3 for the task?


01:00:59.440 --> 01:01:00.320
- Good question.


01:01:00.320 --> 01:01:03.040
So you have those tasks mapped to variables


01:01:03.040 --> 01:01:05.240
called T1, T2, and T3.


01:01:05.240 --> 01:01:07.920
And this is how that visualization is defined


01:01:07.920 --> 01:01:11.160
using those like the bit shift operators in Python.


01:01:11.160 --> 01:01:16.160
So this one would say that T2 and T3 run after T1.


01:01:16.160 --> 01:01:18.360
- Interesting.


01:01:18.360 --> 01:01:19.440
- And they run in parallel.


01:01:19.440 --> 01:01:21.560
- And there are different ways of setting dependency.


01:01:21.560 --> 01:01:25.840
you scroll down or just search for setting up dependencies on the right side of your


01:01:25.840 --> 01:01:31.240
on the right. Yeah, setting up dependencies. Yeah, there you go. There are different ways


01:01:31.240 --> 01:01:34.880
you can set those dependencies between tasks you could do.


01:01:34.880 --> 01:01:35.880
I got you.


01:01:35.880 --> 01:01:42.080
Or even like right shift, you can left shift, you can double bit shift as a transitive type


01:01:42.080 --> 01:01:43.840
thing and set upstream.


01:01:43.840 --> 01:01:44.840
Okay.


01:01:44.840 --> 01:01:50.920
And the beauty of that, again, is that you can, this is all Python code. So those are


01:01:50.920 --> 01:01:55.240
custom operators, the left shift and right shift, they are just custom Python operators


01:01:55.240 --> 01:01:56.240
overridden.


01:01:56.240 --> 01:02:01.280
Right. And you can override them in the task, right? Just like pathlib overrides forward


01:02:01.280 --> 01:02:04.480
slash to mean like combined parts of the path, right?


01:02:04.480 --> 01:02:09.880
We wouldn't probably recommend that if you don't know Airflow that much. But the better


01:02:09.880 --> 01:02:16.160
thing there is that you can actually programmatically build the tasks and build the relationships.


01:02:16.160 --> 01:02:20.560
So this is not something that is predefined in one file in the declarative way, like it


01:02:20.560 --> 01:02:22.440
is an XML file or JSON.


01:02:22.440 --> 01:02:23.680
So this is a Python code.


01:02:23.680 --> 01:02:27.600
So you can pretty much dynamically build the DAG.


01:02:27.600 --> 01:02:30.660
So very complex, like if you saw like, you know,


01:02:30.660 --> 01:02:34.240
the DAGs which were like thousands, thousands of notes


01:02:34.240 --> 01:02:37.480
built with like 200 lines of code,


01:02:37.480 --> 01:02:39.560
because you could build those tasks.


01:02:39.560 --> 01:02:42.360
You know which relationships you want to build.


01:02:42.360 --> 01:02:45.760
- It's very hard to have a conditional in a JSON file


01:02:45.760 --> 01:02:47.520
or an XML file. - Exactly.


01:02:47.520 --> 01:02:48.840
That's the thing.


01:02:48.840 --> 01:02:52.560
Or loop, actually loop in JSON file is like no.


01:02:52.560 --> 01:02:54.240
I mean, there is no way to do that.


01:02:54.240 --> 01:02:55.760
- I mean, we do have XSLT.


01:02:55.760 --> 01:02:57.360
You can go crazy with them all.


01:02:57.360 --> 01:03:00.280
- Yeah, yeah, yeah.


01:03:00.280 --> 01:03:01.120
- Please no.


01:03:01.120 --> 01:03:05.720
- And also from Airflow 2.0 and onwards,


01:03:05.720 --> 01:03:08.320
this is an explicit way of setting dependencies,


01:03:08.320 --> 01:03:10.160
but from Airflow 2.0 and onwards,


01:03:10.160 --> 01:03:12.600
there's also an implicit way of having dependencies,


01:03:12.600 --> 01:03:15.480
which is like, if you say that your bash operator


01:03:15.480 --> 01:03:17.980
takes an input from another task,


01:03:17.980 --> 01:03:21.840
then Airflow sets dependencies between them implicitly


01:03:21.840 --> 01:03:25.120
because you are depending on an output of another task.


01:03:25.120 --> 01:03:26.640
So it knows.


01:03:26.640 --> 01:03:28.680
- Yeah, that makes a lot of sense.


01:03:28.680 --> 01:03:29.520
Cool, all right.


01:03:29.520 --> 01:03:31.960
So I think just two really quick things


01:03:31.960 --> 01:03:34.000
before we wrap it up,


01:03:34.000 --> 01:03:36.160
'cause we are short on time here.


01:03:36.160 --> 01:03:40.660
One is, we talked about the web UI


01:03:40.660 --> 01:03:41.760
for the stuff we're looking at,


01:03:41.760 --> 01:03:43.660
but there's also, you all describe,


01:03:43.660 --> 01:03:48.660
a rich command line utility to perform complex surgeries


01:03:48.660 --> 01:03:50.260
on bags.


01:03:50.260 --> 01:03:52.820
Okay, why would you perform a surgery


01:03:52.820 --> 01:03:53.700
on one of these things?


01:03:53.700 --> 01:03:55.060
And what is this all about?


01:03:55.060 --> 01:04:00.260
Who wants to take that one?


01:04:00.260 --> 01:04:04.300
- I don't know that I've done surgery with the CLI,


01:04:04.300 --> 01:04:07.300
but I have used the CLI to give me information


01:04:07.300 --> 01:04:09.020
about my environment to figure out


01:04:09.020 --> 01:04:11.020
when things are misbehaving.


01:04:11.020 --> 01:04:11.940
- Yeah, okay.


01:04:11.940 --> 01:04:12.100
Cool.


01:04:12.100 --> 01:04:13.940
It's like for diagnosis and stuff like that.


01:04:13.940 --> 01:04:14.460
Yeah.


01:04:14.460 --> 01:04:19.100
Like, cause we have this one command list DAGs and it also shows you how


01:04:19.100 --> 01:04:20.980
long the DAGs are taking to load.


01:04:20.980 --> 01:04:25.220
So you can kind of see if one of them is your problem DAG, if it's taking


01:04:25.220 --> 01:04:30.180
way longer to load than the rest, that usually means that I've made a mistake.


01:04:30.180 --> 01:04:30.620
Yeah.


01:04:30.620 --> 01:04:30.860
Yeah.


01:04:30.860 --> 01:04:34.200
That command also gives you the passing time and everything like that.


01:04:34.200 --> 01:04:37.740
So it can tell you that it took five seconds to pass your DAG file, which


01:04:37.740 --> 01:04:39.660
means something is wrong in your DAG file.


01:04:39.660 --> 01:04:41.780
you are probably importing a lot of things


01:04:41.780 --> 01:04:45.460
or doing some database calls on the top of your file,


01:04:45.460 --> 01:04:47.140
not inside the objects.


01:04:47.140 --> 01:04:49.460
So you can find those sort of issues.


01:04:49.460 --> 01:04:52.420
Also, you could use the Airflow backfill CLI command


01:04:52.420 --> 01:04:54.660
to run all the backfilling of data


01:04:54.660 --> 01:04:56.180
if you've got the data today


01:04:56.180 --> 01:04:59.100
and if you want to run it for last one year or so.


01:04:59.100 --> 01:05:03.740
- Yeah, but also it's what is not mentioned in the document.


01:05:03.740 --> 01:05:07.660
There is this, well, it is mentioned in the documentation.


01:05:07.660 --> 01:05:11.080
we have also a very, very powerful and rich


01:05:11.080 --> 01:05:13.480
and very well written API.


01:05:13.480 --> 01:05:16.020
So we have a stable Azure Airflow 2,


01:05:16.020 --> 01:05:18.380
that was one of the improvements implemented.


01:05:18.380 --> 01:05:20.160
So if you go to Apache Airflow, yeah,


01:05:20.160 --> 01:05:23.240
and scroll down on the left, yes, not this one.


01:05:23.240 --> 01:05:24.660
- All the way down here.


01:05:24.660 --> 01:05:28.680
- The, even below, there was like stable REST API.


01:05:28.680 --> 01:05:30.120
- Yeah, there you go. - Oh, yeah, yeah, gotcha.


01:05:30.120 --> 01:05:31.200
Okay. - Yeah.


01:05:31.200 --> 01:05:36.060
So this API is like written in open API standards,


01:05:36.060 --> 01:05:39.780
which means that all the tools which you can imagine


01:05:39.780 --> 01:05:43.220
for like managing access, for trying out things,


01:05:43.220 --> 01:05:45.380
for testing the API calls,


01:05:45.380 --> 01:05:47.740
all the beautiful documentation that you see here


01:05:47.740 --> 01:05:50.420
with examples, this is all automatically generated


01:05:50.420 --> 01:05:51.820
from our API.


01:05:51.820 --> 01:05:55.660
And this is super cool because you can actually,


01:05:55.660 --> 01:05:57.060
and this is surprising.


01:05:57.060 --> 01:06:00.860
You said that the UI is fantastic and yeah, it is.


01:06:00.860 --> 01:06:03.580
But there are some companies who have their own UI,


01:06:03.580 --> 01:06:06.640
their own ways of looking at the processing pipelines.


01:06:06.640 --> 01:06:09.300
And many, many, we've learned during the Airflow Summit,


01:06:09.300 --> 01:06:12.020
many of those companies, they actually build their own UI.


01:06:12.020 --> 01:06:14.020
They don't use Airflow UI at all.


01:06:14.020 --> 01:06:15.060
They just use the engine.


01:06:15.060 --> 01:06:16.140
- Maybe you want to integrate it


01:06:16.140 --> 01:06:18.260
into some larger thing they already have or something.


01:06:18.260 --> 01:06:19.260
- Exactly, exactly.


01:06:19.260 --> 01:06:21.300
And this API makes it possible.


01:06:21.300 --> 01:06:23.260
So you can just query which DAX you have,


01:06:23.260 --> 01:06:26.140
which are the relationships, how this all works,


01:06:26.140 --> 01:06:28.020
which is successful, which not.


01:06:28.020 --> 01:06:31.020
And then you can build beautiful UI


01:06:31.020 --> 01:06:33.380
or even ugly UI if you want.


01:06:33.380 --> 01:06:36.620
but the UI that is something that you're used to


01:06:36.620 --> 01:06:38.780
without looking even at the Airflow UI.


01:06:38.780 --> 01:06:41.540
And this is also a super powerful part of Airflow.


01:06:41.540 --> 01:06:46.300
- Yeah, and this is straight up REST API.


01:06:46.300 --> 01:06:48.720
So while Python is awesome,


01:06:48.720 --> 01:06:50.060
if you're not a Python person,


01:06:50.060 --> 01:06:51.260
but you still wanna adopt this,


01:06:51.260 --> 01:06:53.940
like here's a way to integrate with it, right?


01:06:53.940 --> 01:06:54.860
- Absolutely. - Absolutely.


01:06:54.860 --> 01:06:58.180
- And we have already started creating clients


01:06:58.180 --> 01:06:59.700
in different languages.


01:06:59.700 --> 01:07:03.260
like we have a Java client for Airflow


01:07:03.260 --> 01:07:05.260
built on this API spec.


01:07:05.260 --> 01:07:07.580
And users could use that


01:07:07.580 --> 01:07:11.580
and users can create their own APIs for specific language


01:07:11.580 --> 01:07:13.700
because it, under the hood uses open API


01:07:13.700 --> 01:07:18.620
so you can auto generate clients for different languages.


01:07:18.620 --> 01:07:20.100
- Yeah, fantastic.


01:07:20.100 --> 01:07:23.420
All right, I think that is about time for us.


01:07:23.420 --> 01:07:26.700
I did wanna point out that Astronomer and AWS,


01:07:26.700 --> 01:07:29.840
But Astronomer, where you work, Hexel, is a sponsor.


01:07:29.840 --> 01:07:34.840
So if you wanna run sort of Airflow as a service,


01:07:34.840 --> 01:07:38.520
that's kind of your job, right?


01:07:38.520 --> 01:07:39.360
- 100%.


01:07:39.360 --> 01:07:42.380
And also we, Astronomer has their own registry.


01:07:42.380 --> 01:07:45.720
So if you do open registry.astronomer.io,


01:07:45.720 --> 01:07:49.520
it makes it very easy to search for built-in providers


01:07:49.520 --> 01:07:51.040
that are baked inside Airflow,


01:07:51.040 --> 01:07:55.200
or if users create and maintain their own providers,


01:07:55.200 --> 01:07:57.920
it is very easy to search that as well.


01:07:57.920 --> 01:08:01.920
I just posted the link if you want to check out.


01:08:01.920 --> 01:08:02.760
- Yeah.


01:08:02.760 --> 01:08:05.960
- And just one comment on that,


01:08:05.960 --> 01:08:07.900
because we also have Google Cloud Composer.


01:08:07.900 --> 01:08:11.600
So we have Astronomer, AWS, and Cloud Composer.


01:08:11.600 --> 01:08:15.200
These are like big embrace of Airflow as a service.


01:08:15.200 --> 01:08:18.020
And for us, it's like, you can choose either.


01:08:18.020 --> 01:08:21.440
You run it on your own, you run it using Astronomer,


01:08:21.440 --> 01:08:23.760
which have like great expertise in everything,


01:08:23.760 --> 01:08:25.640
because we have lots of people from Astronomer,


01:08:25.640 --> 01:08:28.720
our commuters, then there are Amazon people,


01:08:28.720 --> 01:08:30.880
then there are Google or Amazon offering


01:08:30.880 --> 01:08:33.900
and Google offering, and you are free to choose


01:08:33.900 --> 01:08:37.040
whatever you want, like how you want to run Airflow.


01:08:37.040 --> 01:08:39.400
- And you can move probably if you decide you need to move.


01:08:39.400 --> 01:08:40.240
- Yeah, yeah.


01:08:40.240 --> 01:08:42.640
- Yeah, that means you have the infrastructure.


01:08:42.640 --> 01:08:45.520
The DAGs will be the same no matter where you take them.


01:08:45.520 --> 01:08:47.160
You'll might have to do a few changes


01:08:47.160 --> 01:08:49.480
when it comes to like off and making sure


01:08:49.480 --> 01:08:51.840
your keys are up to date, but that's it.


01:08:51.840 --> 01:08:52.660
- Cool.


01:08:52.660 --> 01:08:55.980
Let's wrap this up with a little bit of future looking.


01:08:55.980 --> 01:08:59.700
Just whoever has the right visibility in our group here.


01:08:59.700 --> 01:09:01.140
Just where are things going in the future?


01:09:01.140 --> 01:09:02.740
People are excited about Airflow.


01:09:02.740 --> 01:09:04.820
Like what can they look forward to?


01:09:04.820 --> 01:09:08.100
- Oh, there's a really good talk from the Airflow Summit


01:09:08.100 --> 01:09:11.340
that's called "Looking Ahead Beyond Airflow 2.0."


01:09:11.340 --> 01:09:16.180
It is with Ash from Astronomer and Ajamal from Google.


01:09:16.180 --> 01:09:19.620
And I think the thing that Ash said over and over again is,


01:09:19.620 --> 01:09:23.500
Well, there is no roadmap, but we do always have things going on.


01:09:23.500 --> 01:09:24.760
[LAUGHTER]


01:09:24.760 --> 01:09:26.020
No promises.


01:09:26.020 --> 01:09:29.460
No promises, but there are lots of that.


01:09:29.460 --> 01:09:33.460
So yeah, we pretty much know the direction we are heading to.


01:09:33.460 --> 01:09:36.700
So we want Airflow to be the orchestrator


01:09:36.700 --> 01:09:39.540
you want to use for whatever workflows you want to run.


01:09:39.540 --> 01:09:40.980
That's it.


01:09:40.980 --> 01:09:44.940
And there are lots of things to happen in order to get there,


01:09:44.940 --> 01:09:48.180
because we are so specialized on one hand


01:09:48.180 --> 01:09:51.860
on what we are opening up, but we are on the road


01:09:51.860 --> 01:09:54.860
to really make it easy to accommodate more use cases,


01:09:54.860 --> 01:09:58.200
make it easier to run, make it faster,


01:09:58.200 --> 01:10:02.260
make it serve those cases which currently cannot be served


01:10:02.260 --> 01:10:05.780
because of some reasons, historical reasons mainly.


01:10:05.780 --> 01:10:09.480
And this is definitely some direction we are heading


01:10:09.480 --> 01:10:11.380
to open up to even more cases


01:10:11.380 --> 01:10:14.560
without losing the single focus.


01:10:14.560 --> 01:10:17.620
Like we want to be great at scheduling tasks


01:10:17.620 --> 01:10:21.780
orchestration. That's it. We don't want to do processing. We don't want to go into this


01:10:21.780 --> 01:10:25.780
direction. That doesn't make sense for us. We want others to do processing and we will


01:10:25.780 --> 01:10:30.020
do orchestration the best way it's possible.


01:10:30.020 --> 01:10:34.660
And the two immediate things that we are already working on and we are almost close to merging


01:10:34.660 --> 01:10:40.780
it on the main branches, making the Airflows schedule more powerful. That is, the users


01:10:40.780 --> 01:10:48.180
will have more power than just like expressing it in cron users will also be able to say


01:10:48.180 --> 01:10:53.780
run it on the third of the third trading day of the month or something like that, like


01:10:53.780 --> 01:11:01.700
that level of powerful timetable we want to provide to the users we call it timetables


01:11:01.700 --> 01:11:06.580
we have we will have cron timetable will have time delta timetable of sorts we are figuring


01:11:06.580 --> 01:11:12.860
is that but we'll have that plus something called deferred operators. Leah mentioned


01:11:12.860 --> 01:11:19.140
about the sensors which are currently we poke for the API call and see until it succeeds.


01:11:19.140 --> 01:11:25.100
We are going to have a new component called trigger that will use Python's async library


01:11:25.100 --> 01:11:29.860
to use resources in a more optimized manner.


01:11:29.860 --> 01:11:34.340
- Instead of polling, you just await for it to happen and then boom, off it goes. Yeah.


01:11:34.340 --> 01:11:36.220
- That sounds cool.


01:11:36.220 --> 01:11:37.660
- Yeah. - Fantastic.


01:11:37.660 --> 01:11:39.200
- Just one comment to the scheduling


01:11:39.200 --> 01:11:42.080
because those great examples of one of the cases


01:11:42.080 --> 01:11:43.940
we want to serve, there is a real astronomer,


01:11:43.940 --> 01:11:46.380
not the company, real astronomer using Airflow.


01:11:46.380 --> 01:11:49.760
And he wanted to start DAX when there is a sunset


01:11:49.760 --> 01:11:51.700
and sunrise and you know, when you are astronomer


01:11:51.700 --> 01:11:55.740
and planning around Earth, that's a little bit complex.


01:11:55.740 --> 01:11:58.940
So the whole scheduling is gonna be there


01:11:58.940 --> 01:12:01.600
to implement this astronomer request, basically.


01:12:01.600 --> 01:12:02.500
- Yeah, fantastic.


01:12:02.500 --> 01:12:04.100
It sounds really useful.


01:12:04.100 --> 01:12:07.100
All right, well, I think that's it for covering Airflow,


01:12:07.100 --> 01:12:09.540
but let's quickly wrap up with,


01:12:09.540 --> 01:12:12.580
I guess just one of the questions


01:12:12.580 --> 01:12:13.900
since we're a little bit over time


01:12:13.900 --> 01:12:14.820
that I usually ask at the end.


01:12:14.820 --> 01:12:16.500
So I'll ask you about your editor.


01:12:16.500 --> 01:12:18.580
Jarek, if you're gonna work on Airflow and other stuff,


01:12:18.580 --> 01:12:20.760
but what editor do you use for Python?


01:12:20.760 --> 01:12:26.920
- Well, on a daily basis, I use IntelliJ Ultimate.


01:12:26.920 --> 01:12:29.900
That's my favorite editor.


01:12:29.900 --> 01:12:34.420
However, very, very frequently, my favorite editor is VI.


01:12:34.420 --> 01:12:38.180
I mean, I'm old type guy and VI is always


01:12:38.180 --> 01:12:40.220
when I have to do something quick.


01:12:40.220 --> 01:12:43.340
Somewhere where I don't have the editor started,


01:12:43.340 --> 01:12:47.500
VI is there and I have it in my fingers.


01:12:47.500 --> 01:12:49.820
I know how to quit VI, it's easy.


01:12:49.820 --> 01:12:51.500
I can learn you, I can teach you.


01:12:51.500 --> 01:12:56.340
- Fantastic, yeah, I love that joke about random strings.


01:12:56.340 --> 01:12:57.380
Kaxel?


01:12:57.380 --> 01:13:00.340
- For me it's PyCharm, I love PyCharm.


01:13:00.340 --> 01:13:03.100
It's debugging, it's going to the source code


01:13:03.100 --> 01:13:06.460
and those intelligent help it gives away.


01:13:06.460 --> 01:13:09.260
Just a big fan of PyCharm.


01:13:09.260 --> 01:13:10.260
- Right on, Leah?


01:13:10.260 --> 01:13:13.380
- I use a combination of VS Code


01:13:13.380 --> 01:13:15.740
and I also have SoftSpot for Vim.


01:13:15.740 --> 01:13:16.580
- Okay, very cool.


01:13:16.580 --> 01:13:21.060
- So Vim is gonna be fast, VS Code if it's not.


01:13:21.060 --> 01:13:22.340
- Yeah, we're gonna be here for a while,


01:13:22.340 --> 01:13:23.980
let's get down to it.


01:13:23.980 --> 01:13:24.820
- Yeah.


01:13:25.660 --> 01:13:27.980
- Right on, well, thank you all for being here.


01:13:27.980 --> 01:13:29.340
It's been really great.


01:13:29.340 --> 01:13:31.860
Final call to action, people wanna get started


01:13:31.860 --> 01:13:35.260
either using Airflow or contributing to Airflow.


01:13:35.260 --> 01:13:36.660
What do you tell them?


01:13:36.660 --> 01:13:39.360
- Oh, I tell them to go to the community page


01:13:39.360 --> 01:13:42.180
on the Airflow website and I tell them to sign up


01:13:42.180 --> 01:13:45.420
for the dev list and to join the Airflow Slack.


01:13:45.420 --> 01:13:47.700
- Yep. - Fantastic.


01:13:47.700 --> 01:13:50.180
All right, well, thanks again.


01:13:50.180 --> 01:13:51.580
Thanks for being here. - Thank you.


01:13:51.580 --> 01:13:54.100
- Thank you for inviting us, it was a great time.


01:13:54.100 --> 01:13:55.620
- Yeah, yeah, it was great. - Thank you.


01:13:55.620 --> 01:13:56.960
bye Kirby.


01:13:56.960 --> 01:13:57.800
- Yep.