Transcript

No one likes working on Rails upgrades, and who can blame them? They can be thankless, time-consuming, and—worst of all—unpredictable. How long will they take? When can we get back to feature work? What’ll break when they’re deployed? It’s tempting to say, “Let’s do it later,” but later is usually never. Instead of living in fear of the big, scary Rails upgrade, your team can set up systems for incremental change that let you:

  • continue feature work
  • deliver predictable results
  • seamlessly jump into the next upgrade (because there’s always a next upgrade)

And I’m gonna show you how.

In the years since I’ve worked on upgrading Rails at GitHub, I’ve refined an approach to Rails upgrades and successfully implemented it at several organizations. In this talk, you’ll learn everything you need to burn down uncertainty and boldly go into your next Rails upgrade.

📝 From the Test Double blog

See the rails-upgrades tag for more posts about Rails upgrades!

🧰 Open source tools

00:00
(electronic music)
00:04
- Hi, everybody.
00:05
Hi.
00:07
- I'm Ali, and I'm here today
00:10
to talk about everyone's favorite subject.
00:12
Drum roll, please.
00:14
(Ali imitates drum roll)
00:19
Rails upgrades.
00:20
(audience cheers)
00:23
So there are dozens of us, okay, cool.
00:30
All right, so I'm surprised.
00:32
I thought more people were not going to like Rails upgrades.
00:35
That's fine.
00:36
But when I look past, through past RailsConfs,
00:39
I see a lot of talks about Rails upgrades.
00:41
So there was one in 2017,
00:44
one in 2018, excuse me, two in 2018,
00:48
one in 2019.
00:50
There was one last year, and just
00:52
yesterday, there was a workshop on upgrading Rails.
00:55
Great workshop, by the way.
00:57
And since we all love working on Rails upgrades,
01:01
why are we spending so much time talking about them?
01:04
When I think about this question,
01:05
I think about all the different upgrades I've worked
01:07
on over the course of my career at Test Double.
01:10
And one thing stands out.
01:13
These organizations aren't coming to us
01:15
to work on these upgrades 'cause they're easy.
01:18
They're coming to us 'cause they're hard.
01:21
But what is it about these upgrades that makes them so hard?
01:25
The first thing that might come to mind
01:27
are all the technical challenges involved
01:29
with working on an upgrade.
01:31
For example, the first thing
01:33
you might want to do in an upgrade,
01:34
bundle update rails.
01:36
Might be a multi-day journey through pain and suffering,
01:40
trying to get your gem versions to work with Bundler,
01:43
and you find out, uh, this gem's not maintained anymore,
01:46
so what do you do about that?
01:48
Or there's some weird fork and you're just like, really?
01:50
I just want to run bundle update Rails.
01:52
A standard command, right?
01:54
Or sometimes there's changes in the public API.
01:57
For example,
01:58
how many people here remember the strong params migration
02:01
from Rails 3 to Rails 4, right?
02:03
Pretty big migration.
02:04
And if you had a big app with a lot of controllers,
02:06
this is a big change.
02:07
But no matter what the technical challenges are,
02:11
all of us here, we can solve these technical problems.
02:14
And I say that pretty confidently, because if we couldn't,
02:18
there wouldn't be any upgrades ever.
02:20
And if there were no upgrades,
02:22
none of us would be here
02:24
'cause who would be building stuff in Rails, right?
02:28
So maybe we need to reframe this question
02:31
about why is it so hard for organizations
02:34
to work on these upgrades.
02:36
And instead, it's why is it so hard for organizations
02:39
to invest the time and money to do these upgrades?
02:43
And to think about that, let's shift gears a little bit
02:46
and talk about something that organizations do every day,
02:50
and that's feature work.
02:51
And let's see how this compares to Rails upgrades.
02:54
So when your team embarks on building feature X, Y, Z,
02:58
all the people involved, all the stakeholders,
03:00
whether it's your product managers, UI/UX, engineering, QA,
03:05
whoever needs to be involved in making that feature happen,
03:08
all of y'all have some context,
03:11
some kind of shared understanding
03:12
of this is the app that we're working in,
03:15
or this is our business domain, et cetera, right?
03:18
All of y'all know something about the app,
03:20
so you can talk about it.
03:22
And because you have this shared understanding,
03:24
it's easier to break up the work.
03:26
And this might mean identifying the scope,
03:29
but what's our appetite here?
03:30
How much of this do we want to do right now?
03:32
Or maybe it's breaking it up into small discrete steps
03:35
that you can work on bit by bit by bit,
03:38
shipping stuff incrementally.
03:40
And because we can break up the work,
03:43
it's easier, then, to share progress.
03:45
You might say this feature,
03:46
we broke it up into 10 cards and a couple cards in,
03:49
we kind of know, are we on target for our deadline
03:53
or do we need to have another conversation and say, hey,
03:55
like, we found some new information out,
03:58
we either need to adjust scope or our deadline here.
04:00
Something like that, right?
04:01
And all this is easier because everyone involved
04:04
has some kind of shared understanding,
04:06
and you can break that work up into smaller pieces.
04:10
So how does this compare then to Rails upgrades?
04:13
Well, for one, Rails has a lot of code in it, right?
04:19
I've gotten deep in the weeds
04:21
of ActiveRecord a couple times,
04:23
and I'll say I know maybe 5% of the code in Rails.
04:27
And if you saw Eileen's keynote yesterday,
04:29
she said there's no core committer that knows all of Rails.
04:33
So it's a really big code base
04:35
that no one really understands, right?
04:39
And Rails is crucial, crucial infrastructure
04:42
in our application, because without it,
04:45
we don't really have an app to talk about, right?
04:47
So now we're talking about changing something
04:51
that we don't know really well,
04:54
and we want to change it, potentially, a lot.
04:58
Who is going to sign up for this project
05:01
that has so many unknowns?
05:03
What could go wrong?
05:04
How long is this going to take?
05:05
When will it be done?
05:08
Nah, no one signing up to work
05:10
on this risky Rails upgrade project.
05:13
But what if we could change
05:15
the way we think about Rails upgrades
05:17
and think of them more like feature work
05:20
where we take a big black box of a project
05:25
and we break it up into smaller pieces
05:28
that we can work through incrementally,
05:30
shipping stuff bit by bit
05:32
and using the information that we learn as we ship stuff
05:35
to help inform the next step we're going to do.
05:38
If we could do that,
05:40
we'd have upgrades that are more predictable,
05:43
and if they were more predictable,
05:46
they'd be more repeatable, because spoiler alert,
05:49
there's going to be another Rails upgrade, right?
05:52
So how do we get here?
05:55
How do we get to having Rails upgrades
05:57
that are more predictable and more repeatable?
06:02
Well, first things first,
06:04
we have to figure out what version
06:05
of Rails are we even upgrading to,
06:07
which might seem like a pretty obvious question.
06:10
We just want to get to whatever
06:11
the latest version of Rails is, right?
06:14
Well, let's imagine we have an app running
06:16
on Rails 5.2,
06:17
and right now, Rails 7 is the latest version.
06:20
So here's our big black box.
06:22
We want to upgrade to Rails 7.
06:25
Is there anything we can do to break this up,
06:28
break this up into smaller pieces?
06:30
Well, since we're coming from Rails 5,
06:32
we can say Rails 7 is made up of two major versions,
06:35
Rails 6 and Rails 7.
06:38
And if we want to upgrade to Rails 7,
06:41
we have to upgrade to Rails 6 no matter what.
06:43
So we could try to do it in one big giant PR
06:46
that changes everything,
06:47
or we could just focus on upgrading to Rails 6.
06:51
And now, our black box is looking a little smaller.
06:56
Is there anything else we could do
06:57
to break this down even further?
06:59
Well, Rails 6 is made up of two minor versions,
07:02
Rails 6.0 and Rails 6.1.
07:05
And just like talking about going from 6 to 7,
07:06
you have to do 6 no matter what,
07:08
if you want to go to 6.1,
07:10
you're going to have to upgrade to 6.0.
07:11
So instead of trying to do that all at once,
07:15
let's just upgrade to 6.0.
07:17
And now, our big black box is looking a lot smaller.
07:21
So what we're trying to do here is,
07:24
when we're trying to figure out what version
07:25
we want to upgrade to,
07:26
is first target that next minor version.
07:29
So if you're on Rails 5.0,
07:31
next minor version is 5.1,
07:33
that's what you're going to upgrade to.
07:35
And once you get to 5.1,
07:37
your next minor version is 5.2,
07:38
so that's your next upgrade.
07:40
And once you've exhausted all the minor versions,
07:43
you're going to upgrade then to the next major version.
07:46
So if you're now on Rails 5.2, there is no Rails 5.3,
07:51
you're ready to go on to that next major version,
07:53
Rails 6.0.
07:55
And in doing this,
07:57
we're making our upgrades smaller.
07:59
Instead of trying to take years and years
08:00
and years of changes to Rails
08:01
and apply it to our application all at once,
08:04
we're just going to focus on a narrow chunk
08:06
that's right in front of us,
08:09
and this will help drive down the risks of our upgrade,
08:11
because again, we're not going to try to make a bunch
08:13
of changes in one big go.
08:17
So now we're starting to get a little more clarity, right,
08:20
of what our target version we want to upgrade to.
08:24
So now we're ready to run bundle update rails, right?
08:28
Well, before we get ahead of ourselves,
08:32
we need to talk about everyone's second favorite subject.
08:36
We're already talking about Rails upgrades,
08:37
which everyone loves,
08:39
but we have to talk about everyone's second favorite subject
08:42
and that's deprecation warnings.
08:44
I know, I've got all the hits today, right?
08:46
Okay, so I'm going to need some audience participation here.
08:49
Can I see a raise of hands
08:51
if you've ignored deprecations in your app recently?
08:55
Yeah, right?
08:56
Me too.
08:56
That's what I do every day,
08:58
ignoring deprecations because they're just warnings.
09:01
They're not, our app's still working
09:03
and nothing for us to do about 'em, right?
09:05
But when you're starting on a Rails upgrade,
09:07
it's really important that you look at them
09:08
to see what they're saying.
09:10
For example, imagine now that we're on an app
09:13
that's running in Rails 6.0,
09:14
and we see this deprecation warning
09:15
flying through our logs.
09:18
It's saying, accessing hashes from config_for
09:22
by non-symbol keys will be removed in Rails 6.1.
09:27
We're on Rails 6.0,
09:29
so this thing that we're doing,
09:31
accessing the hashes with non-symbol keys,
09:33
still works fine, right?
09:35
But when we go to Rails 6.1, the next minor version,
09:39
it's not going to work anymore.
09:41
And the last thing this deprecation warning is telling us
09:43
is use symbols for access instead.
09:46
It's not saying upgrade to Rails 6.1
09:48
and open that giant PR that's changing
09:50
a ton of things, right?
09:52
It's saying use symbols for access instead,
09:54
and this is something you can do in your application today.
09:58
And we can do that for all
09:59
the deprecation warnings, right?
10:00
We don't have to wait for a huge big effort
10:03
to get our app one step closer
10:06
to working on that newer version of Rails.
10:09
And the good news is,
10:11
we don't have to fix all these deprecations at once.
10:14
For example, imagine you have a deprecation warning
10:17
that's popping up across hundreds
10:18
of files in your application.
10:19
It's a huge mess.
10:21
You could try to open it in one giant PR
10:24
and end up changing a ton of files,
10:25
but no one on your team is going to want to review them.
10:28
And eventually, you find somebody who's like, look,
10:30
don't worry, it's just deprecations,
10:31
just gimme a check mark, please.
10:33
So, okay, you get your check mark.
10:34
But when it's time to hit that merge button
10:36
and deploy this thing to production,
10:38
you're going to be pretty nervous
10:39
because you're changing a lot of stuff in your app.
10:42
Instead, you could fix that deprecation warning in one file.
10:48
If you change it in one file,
10:49
it's going to be easier to talk about,
10:50
easier to get your team to review it,
10:52
because okay, it's a smaller change,
10:53
we can see what's going on here, right?
10:56
It's going to be easier to test,
10:57
and it's going to be easier to verify that,
10:59
yeah, we made this change,
11:00
and it's working on production the way we expected it to.
11:03
And with that information, the stuff that you're learning,
11:06
it's going to be easier then for you to come back
11:08
and confidently make that change
11:10
to all the other files in your application.
11:14
So now you take your deprecation warnings
11:15
and you divvy them up across the people on your team.
11:17
Say, Hey, everyone,
11:18
pick one up and fix it here, fix it there.
11:21
And you're working really hard, working diligently,
11:24
getting your app, again, one step closer
11:26
to running on a new version of Rails.
11:28
And lo and behold, someone comes behind you
11:31
and brings back a deprecation that you just fixed.
11:33
And this is a super, super frustrating experience
11:36
'cause you're working so hard,
11:38
it's like, okay, I'll fix the deprecations.
11:39
This is like a good thing to do.
11:41
And why would a developer do this?
11:42
Why are they not paying attention
11:44
to all the work that we're doing?
11:45
But we just said a couple minutes ago,
11:48
everyone's ignoring the deprecation warnings.
11:50
So can we really blame a developer
11:52
for doing something that we all do every day?
11:56
No.
11:57
Instead, we can make it easier for them to adopt
12:00
the patterns that we're going to need to use going forward.
12:03
So here's a blog post I wrote up about one strategy I like
12:06
to use when dealing with deprecations,
12:09
and if you Google,
12:10
you'll find a bunch of different ways to do it,
12:12
but they all kind of have the same core idea.
12:16
Once you've fixed a deprecation,
12:18
you'll set it up in your app so it errors in dev and test.
12:21
That way that developer, like me,
12:24
who's used to writing code a certain way,
12:26
when they're writing tests
12:27
or they're trying to boot up the app or whatever,
12:29
and they write deprecated code,
12:31
the computer's going to yell at them
12:32
and say, nope, you can't do this anymore.
12:34
And because nobody likes the computer yelling at them,
12:37
eventually, they're going to figure out, okay,
12:39
I need to adopt this new pattern going forward.
12:42
And we'll couple that with logging in production
12:45
because tests aren't perfect,
12:47
we're never going to catch 100% of everything,
12:49
so we'll log these on production
12:51
to give us one last fail-safe
12:52
to make sure there's no deprecations slipping
12:55
through the cracks.
12:58
And in doing this, we're shipping code.
13:00
We're shipping code bit by bit by bit.
13:03
Again, that's getting us one step closer
13:05
to that new version of Rails.
13:07
Wouldn't it be nice if we could stay in this mode
13:10
for all of our Rails upgrade?
13:12
Well, let's see what we can do.
13:14
So now that you fixed all your deprecation warnings,
13:17
you're ready to run bundle update rails
13:19
and you check out a new branch.
13:22
Nothing new there, right?
13:23
We always check out a new branch
13:23
when we do something new in our application,
13:26
but for as long as this branch is living,
13:30
you're driving up the risks of your upgrade.
13:33
So let's imagine you're on your branch down here
13:35
and you spend a couple days doing the Bundler battle
13:38
to get the new version of Rails to install,
13:41
and eventually, you get it to work.
13:45
And in all that time you keep
13:46
on working away on this branch,
13:48
development's been cruising along on main.
13:51
And not a big deal, it's only been maybe a day or two,
13:53
so you pull those changes in from main to your branch.
13:57
And development keeps happening on main,
13:59
and there was some security patch
14:01
for another gem that got released in that time,
14:03
so they went ahead and made that change
14:04
to the Gemfile, pushed it up.
14:06
So now, when you come back another couple days later,
14:09
after you've been trying to fix tests or get the app to boot,
14:12
you rebase and you run into some conflicts
14:14
because now the Gemfile's changed in two places,
14:16
it's like, ugh, this is kind of frustrating,
14:18
but maybe it's not too bad.
14:20
You figure it out in an hour or something like that.
14:23
And then your boss comes through,
14:24
and says, whoa, whoa, whoa, whoa, whoa, whoa, whoa.
14:27
We have a really important urgent thing we need
14:29
to work on right now, so please,
14:31
get to a stopping point on this upgrade,
14:33
and we'll get back to it when we have time.
14:34
You say, okay, I'll push up my changes to my branch,
14:36
and we'll come back to it, no big deal.
14:39
And development cruises along on main.
14:43
Time passes.
14:44
Could be weeks, months, years.
14:48
I've seen it.
14:50
And you come back and now your branch is so far behind main.
14:54
You try to pull the changes in,
14:56
but now, the Gemfile's changed again.
14:57
It's like, ugh, this is frustrating,
14:59
I have to do this whole Bundler thing again,
15:01
or those tests that you were working so hard to fix,
15:03
now they've changed on main,
15:05
so the changes that you made on your branch
15:06
don't make sense anymore,
15:08
and you're scratching your head like, what do I do?
15:09
How do I get this to work?
15:10
And in the end, you're like, you know what?
15:13
I'm just going to start this upgrade over from scratch.
15:16
And that is a huge risk 'cause what we're saying there is,
15:20
if at any point we have to stop our upgrade
15:23
because life changes, things happen,
15:26
we could lose all of the work
15:27
that we've done up to that point,
15:30
and that's a huge bummer,
15:31
and no one wants to be in that situation.
15:33
So what can we do to get out of this?
15:36
Well, we can turn to a strategy called dual booting.
15:39
And what you'll do with dual booting
15:41
is you'll set up your application on the main branch
15:44
so it can run on either the production version of Rails
15:46
or the target version of Rails you're upgrading to,
15:49
and you'll have some kind of switch,
15:50
and by default, that switch will usually be off.
15:52
And when it's off, your app's running
15:55
on the production version of Rails.
15:57
Nothing out of the ordinary there, right?
15:59
It's just the same thing that's always been happening.
16:01
But when the switch is turned on,
16:04
we'll be running the target version of Rails
16:06
on our app on the main branch.
16:10
There are a lot of different strategies
16:12
out there for dual booting.
16:13
I like using Bootboot.
16:15
It's a Bundler plugin made by Shopify,
16:17
and I like using it so much that I'll bend over backwards
16:21
to get it working on Rails 4.
16:23
Story for another day.
16:26
So now, when you go back to your branch
16:28
and you run bundle update rails,
16:31
you're not just going to be doing the Bundler dance
16:33
to get the new version of gem,
16:34
the new version of Rails installed,
16:36
you'll also be adding that dual booting functionality,
16:39
so you can toggle the app back
16:40
and forth between Rails versions.
16:42
And as soon as you get those two things working,
16:45
you can bundle the new version of Rails,
16:47
and you have dual booting enabled
16:48
so you can toggle Rails versions.
16:51
You ship that code.
16:52
You're not going to wait to say,
16:53
oh, let me get these tests, I can get these tests to work.
16:55
Mm, mm.
16:56
Or, the app's not booting, let me make sure that, no.
16:59
As soon as you get dual booting working
17:02
and you can bundle a new version of Rails,
17:03
you ship that off to production,
17:05
because from that point forward,
17:07
any changes that you need to make
17:09
for your Rails upgrade will be on main,
17:11
and after today, I never want to hear again
17:14
somebody maintaining a long-lived branch
17:16
that they're rebasing every day
17:17
for months for their upgrade.
17:19
You don't have to do that anymore.
17:20
Dual booting is here, please use it.
17:23
And if you do that, then, you're shipping code.
17:26
You're shipping code every day to main,
17:28
and you're driving down the risk of your upgrade.
17:32
But if you're following along with me,
17:34
you might be asking yourself some questions
17:37
of like how is this going to work exactly,
17:39
because there's got to be new stuff
17:42
in the new version of Rails
17:44
that isn't in the old version of Rails,
17:46
and somehow we're going to deploy that code to main,
17:48
but how's it going to work without the new version of Rails?
17:52
I'm just confused saying that.
17:54
Like, how is this going to work, right?
17:57
Well, so once you go down this strategy
17:59
where you have dual booting in place,
18:01
you're going to see conditionals
18:02
like this pop up in your application.
18:03
And here, let's imagine we have an app running on Rails 5.2.
18:07
We'll say, if our app currently is running on 5.2,
18:11
here's the code that we want to run on production.
18:14
Otherwise, here's the code that we'll be using
18:17
in the future version of Rails.
18:19
And now, let's just ship the code
18:21
for both version of Rails to main.
18:23
And you might see a few dozen
18:26
of these or whatever, you know.
18:28
Some of these pop up in your code base, not a big deal.
18:30
Sometimes there's method signatures,
18:32
change in Rails versions, that kind of stuff.
18:35
But if you start to see the same pattern of if-elses
18:38
repeat over and over and over again,
18:43
that's going to drive up the risk of your upgrade
18:46
because that's an indicator that there's some pattern
18:49
that your team's used to that's not going to work going forward
18:54
and you're going to work hard adding those if-elses,
18:57
if-else, if-else, right,
18:58
every time somebody comes back and writes code
19:00
that the way they're used to writing it,
19:03
and you'll be stuck in a game of Whack-a-mole
19:05
that's going to be really hard to win
19:07
because all these things are always going to pop back
19:09
up in your application,
19:10
and you're never going to know, like,
19:11
did I catch all these cases?
19:13
What's failing now, right?
19:15
Not where we want to be.
19:17
So what can we do here?
19:20
Well, we can use a strategy called backporting.
19:22
And what you'll do with backporting
19:24
is you'll take whatever that new thing is in Rails,
19:27
whether it be a new feature or a new way of calling code,
19:30
whatever it is,
19:31
and you backport it to the old version of Rails
19:35
that your app is using
19:36
so then your team can start writing code using
19:40
that new feature before we finish our upgrade.
19:44
And there's an excellent open source example
19:47
of this with Strong Params.
19:49
We talked about that briefly at the beginning of this talk.
19:51
Strong Params was released in Rails 4,
19:53
and again, it was a big change in the way you have
19:55
to dealt with your parameters.
19:58
At the time that Rails released Strong Params in Rails 4,
20:02
they also released the Strong Parameters gem
20:05
and this gem targeted Rails 3.
20:08
So there's your backport.
20:09
This is Strong Parameters that you could add
20:12
to your app running on Rails 3.
20:15
And the way it worked was is,
20:16
if you have a model in your app, let's say it's a Post,
20:20
you add the gem to your Gemfile,
20:23
and then you have this module available to you
20:26
that you can include to opt in to the new behavior,
20:29
and you would include it only in the one model
20:32
and work on migrating that model
20:34
so it uses params the way we need to use it for Rails 4,
20:37
the way we need to use 'em going forward.
20:39
So you work on doing that migration,
20:41
you test everything out, verify it, ship it to production,
20:44
and once you ship it to production,
20:47
no one can use params the old way anymore
20:50
because this module will error and tell people,
20:52
nope, you can't do that anymore, right?
20:55
So as soon as you do that, you do it for the one PR,
20:57
make sure it's working, everything's good.
21:00
You then do the same process for the next model
21:03
and the model after that,
21:05
shipping small PRs bit by bit,
21:07
migrating towards how things are going to work going
21:10
forward in a new version of Rails.
21:12
And once you've done it for all
21:14
of your models in your application,
21:16
you can include it globally
21:17
for all of the ActiveRecord models in your app.
21:20
So from that point forward, someone adds a new model,
21:23
and they have to use params the new way.
21:26
And again, this is an example of a backport
21:28
that the Rails core team made for us,
21:31
and it's really helpful, right?
21:33
But we can do this ourselves.
21:36
We don't have to depend on Rails
21:38
to make all our backports for us.
21:40
So here's a blog post that I worked on talking
21:43
about a similar situation I ran into in an upgrade
21:45
and how we implemented the backports
21:47
and helped the team adopt a new pattern,
21:51
because that's really what we're trying
21:52
to do with these backports.
21:53
We're trying to coach our teams to adopt a new patterns
21:57
that we're going to need going forward.
22:00
And depending on when you bring your backport in,
22:02
they might be writing code for that new version
22:05
of Rails for months.
22:07
So, when you do make that switch on production
22:09
to running the new version of Rails,
22:11
they're not going to be surprised that,
22:12
oh, I can't write code like this anymore, what happened?
22:14
No, they might have been doing
22:15
it for a long time already, right?
22:18
And again, we're shipping code,
22:20
always shipping code bit by bit by bit,
22:23
getting us closer to that new version of Rails.
22:27
So things are going good,
22:28
we feel pretty confident about our Rails upgrade,
22:31
but we have one more big black box to talk about,
22:35
and that's test failures.
22:37
On all of the upgrades I've worked on,
22:39
I've seen teams spend the majority
22:41
of their time fixing failing tests.
22:44
And what can I say about test suites?
22:47
They're big, slow, complicated, brittle, hard to understand.
22:53
I could keep going.
22:54
(Ali laughing)
22:56
So what do you do when you're working on a Rails upgrade
22:59
and you see hundreds or thousands
23:01
and thousands of test failures
23:03
in these big complicated test suites?
23:06
We've been talking today about breaking things
23:07
up into smaller pieces
23:09
that we can work through incrementally,
23:11
so if you're seeing thousands
23:13
and thousands of test failures,
23:15
you could break up the work that way and say,
23:17
everyone, take one of these failures,
23:19
figure out how to fix it and open your PRs,
23:22
and we'll ship little tiny PRs bit by bit by bit, right?
23:26
But personally, I don't want to open thousands
23:29
and thousands of PRs,
23:31
or even worse, write thousands and thousands
23:33
and thousands of Jira tickets, right?
23:35
No one wants to do that.
23:36
So there's got to be some middle ground here
23:38
between having a giant PR that's fixing a bunch
23:41
of unrelated tests
23:43
and having little tiny PRs that are narrowly focused
23:46
where we're doing the same kind of things, potentially,
23:48
over and over again.
23:50
What can we do?
23:52
Well, we could try to group our failures.
23:55
So let's say we look at our tests,
23:56
we look at all of our failures,
23:59
and we notice a bunch of tests failing for,
24:01
expected 200, got 400.
24:04
Well, we could pull those off as a group, and say,
24:07
two people, go look at those tests
24:09
and see why are they failing?
24:11
Are they failing for the same reason or not?
24:13
And after a couple days, they look at it,
24:14
and they say, oh, it looks like there's a bunch
24:16
of tests failing for reason A,
24:18
and then the rest of them are failing for reason B.
24:21
Well, you can break that down then
24:22
into two other groups of tests, right?
24:25
And you can say, person one, go ahead
24:27
and work on that first group of tests
24:29
and the other person can then tackle
24:31
the other group of tests.
24:33
Or maybe you see a bunch of tests failing
24:35
for ActiveRecords::StatementInvalid
24:38
you can't do this anymore.
24:40
You can pull those apart as a group,
24:42
and they're probably also failing for the same reason.
24:45
So give somebody that piece of work to work on,
24:48
and they can work through a solution.
24:50
And you'll do this for all the failures in the test
24:52
where you're trying to identify,
24:54
trying to identify all these different groups.
24:57
And you'll start to see they're just finding chunks of work
25:00
that are just the right size.
25:03
They're not too big where you have a bunch
25:06
of unrelated changes going on,
25:08
and they're not too small where you're just opening
25:10
the same PR kind of over and over again, right?
25:13
Kind of a Goldilocks situation.
25:17
And the grouping you do is going to depend on your application
25:20
and the failures you're seeing.
25:22
So for your upgrade,
25:24
it might make sense to group the failures by file.
25:27
Totally valid option.
25:29
Or maybe you run your tests on CI across 40 containers
25:33
and you have 15 failing.
25:34
You say, okay, everyone take a container
25:36
and make that container green.
25:38
Again, valid option.
25:39
Depends on your application, the failures you're seeing,
25:42
and you know, the upgrades you're working on.
25:45
And what we're trying to do here, again,
25:46
is find chunks of work that are the right size
25:49
that keep us in a state where we are building momentum
25:52
and we're shipping code,
25:54
always shipping code bit by bit by bit
25:57
to get us closer to running on a new version of Rails.
26:02
So now we've fixed all of our deprecations,
26:04
we've got dual booting working,
26:06
and we fixed all of our tests.
26:08
So we're ready to make that switch from production,
26:10
from the production version of Rails
26:12
to the new version that we're upgrading to.
26:16
Are we though?
26:18
Even after you've done all this hard work,
26:21
getting your app, again, closer to the new version of Rails,
26:25
it can be really hard to assess
26:27
what's the risk left of making this switch?
26:30
So at this late stage of our upgrade,
26:33
is there anything we can do to drive
26:35
down the risks involved in making the switch?
26:40
Well, we can try small experiments
26:42
in production environment, excuse me.
26:44
We can try small experiments in production-like environments
26:48
to see if the upgrade is behaving the way we expected it.
26:51
So you can imagine, you have your app running in production,
26:54
and you decide we're going to make a deployment
26:56
that has the upgrade that's for internal use only.
26:59
And we ask all of our employees
27:00
to please use this upgrade version of the app,
27:03
and we'll be monitoring our Datadog dashboards
27:06
or Rollbar errors,
27:07
whatever observability you have to see,
27:10
are we seeing new errors?
27:11
Is the app working the way we expected it to or not?
27:14
Might call this dog filter, right?
27:17
Or maybe you want to try something like a canary deploy
27:20
where you say, we'll ship the upgrade
27:24
to 5% of our traffic for two minutes.
27:28
Might have some impact on your users,
27:30
but it's a really small experiment.
27:31
We're only going to deploy this out for 5%
27:33
of our traffic for two minutes.
27:35
And after that's done, we're going to pull it back.
27:37
And in those two minutes, again,
27:38
you have all your dashboards,
27:40
and you're monitoring everything up to see,
27:42
are new errors coming through?
27:43
Is this working the way we expected?
27:44
All that stuff.
27:46
And after a couple minutes, you're like, oh yeah,
27:48
there are a bunch of errors that we did not expect to see.
27:50
So you go back, fix those errors,
27:53
and you say, okay, when we come back
27:54
and do this experiment again,
27:57
we expect there to be less errors, right?
27:59
Because if we fix them, the error should go down.
28:01
So you do that, you see less errors,
28:04
and you say, okay, let's go ahead
28:06
and bump this up then to 10% of our traffic for two minutes.
28:09
And you do the same kind of dance,
28:11
maybe you find some new errors, go back and fix it,
28:12
try to experiment again.
28:14
And after a while, it starts looking really good,
28:17
so you say, let's bump this up to 10%
28:18
of our traffic for five minutes,
28:21
and this time, whoa, wow, it's looking really good.
28:24
And then finally, you're like, you know,
28:26
let's bump this up to 40% for 10 minutes,
28:28
because we're feeling really good about the upgrade.
28:31
We're not seeing new errors pop through.
28:33
It seems to be that the app is working really well,
28:35
and we're starting to feel more confident
28:38
about this upgrade that we've been working on.
28:41
And that's really what we've been trying to do,
28:44
talking through all these different strategies today.
28:48
We're trying to drive down the risks of our upgrades,
28:52
gaining new information to help us feel more confident
28:55
that we're not going to cause an incident
28:57
by shipping out a Rails upgrade.
28:59
And if we can do that,
29:01
we'll all be shipping zero downtime Rails upgrades.
29:06
All right, so the first upgrade that you do like this,
29:11
it's going to be the hardest.
29:13
It doesn't really matter what version
29:15
of Rails you're on or anything like that.
29:17
It's going to be the hardest because your team,
29:19
your organization, probably has never done
29:21
an upgrade like this before.
29:23
So you have to figure out, what tooling do we like?
29:25
How do we want to divvy up the test failures.
29:27
All that stuff, you have to sort through that, right?
29:30
And in that process, you're going to be learning a ton.
29:34
You're going to be learning, oh,
29:35
we really like using this tool for deprecations,
29:37
or we were really happy with the way
29:39
the deploy went out in doing it this way.
29:41
Whatever it is, you figure that out,
29:43
document it, reflect on it,
29:45
and that's going to help you build a playbook, right,
29:47
a process where this is how we're going to do
29:50
the upgrade next time.
29:52
And if you do that,
29:53
when it's time to do the second upgrade,
29:55
might be a little bumpy,
29:56
but it's going to be a little bit easier.
29:59
And by the time you get to the third upgrade,
30:02
you're going to be cruising, and like, okay,
30:03
we've done this before,
30:04
we know the in and outs of this thing,
30:06
no problem, we can do it.
30:09
And maybe one day, you'll get to a place
30:12
like where GitHub is,
30:13
where they're upgrading Rails weekly, right?
30:17
I don't know if anyone wants to live
30:18
on the cutting edge like that,
30:19
always running the latest version of Rails on edge, right?
30:24
But it goes to show,
30:25
once you refine your process and figure it out,
30:28
these are the tools that we like,
30:29
this is the way we want to do it,
30:30
you can get to this point where it's every week,
30:32
not a big ceremony to do another Rails upgrade.
30:36
And as you go through this journey,
30:38
learning these new things
30:39
and applying different things to your upgrades,
30:43
share what you've learned.
30:44
As I showed you at the beginning of this talk,
30:46
every year there seems to be another talk
30:48
about Rails upgrades.
30:49
So there's always going to be something new to share,
30:52
so please come back, share what you learn,
30:55
and you can help everyone else
30:57
also ship zero downtime Rails upgrades.
30:59
Or if you don't like public speaking,
31:01
you can write a blog post.
31:02
I have a bunch of those too.
31:05
So that's all I have today.
31:08
If you have any questions about Rails upgrades
31:10
or anything, really, feel free to email me,
31:14
aliwritesback@gmail.com.
31:16
And I'm also on the Slack for however long
31:18
that's going to be around, at my full name.
31:20
And again, I mentioned a few blog posts in this talk.
31:23
You can find them all on the Test Double blog
31:25
and other blog posts about upgrades, too,
31:27
and a bunch of other good stuff.
31:30
And there's a lot of us here from Test Double.
31:33
I really enjoyed Daniel's talk yesterday talking
31:35
about legacy code and being more curious
31:37
and how can we make it better.
31:39
I'm looking forward to seeing Landon speak, soon,
31:43
in a few minutes about machine learning and Ruby.
31:45
I think that's going to be really cool.
31:47
And Megan and Justin going to be having
31:50
a little town hall tomorrow,
31:51
making standard happen, even better for all of us, right?
31:55
So that's all I have.
31:57
Thank you.
31:58
(audience applauding)

Ali Ibrahim

Person An icon of a human figure Status
Double Agent
Hash An icon of a hash sign Code Name
Agent 0022
Location An icon of a map marker Location
Baltimore, MD