Zero downtime Rails upgrades

No one likes working on Rails upgrades, and who can blame them? They can be thankless, time-consuming, and—worst of all—unpredictable. How long will they take? When can we get back to feature work? What’ll break when they’re deployed? It’s tempting to say, “Let’s do it later,” but later is usually never. Instead of living in fear of the big, scary Rails upgrade, your team can set up systems for incremental change that let you:

• continue feature work
• deliver predictable results
• seamlessly jump into the next upgrade (because there’s always a next upgrade)

And I’m gonna show you how.

In the years since I’ve worked on upgrading Rails at GitHub, I’ve refined an approach to Rails upgrades and successfully implemented it at several organizations. In this talk, you’ll learn everything you need to burn down uncertainty and boldly go into your next Rails upgrade.

Links from the talk

📝 From the Test Double blog

• Stop ignoring your Rails (and Ruby) deprecations!
• Get Bootboot to work on Rails 4.x in deployment environments with Docker
• Working strategically through Rails upgrades

Check out more content about rails-upgrades!

🧰 Open source tools

• Bootboot—Bundler plugin for dual booting multiple Rails versions on your app’s main branch
• Strong Parameters (Rails 3.x backport)—an example of backporting new features to older versions of Rails

00:00

(electronic music)

00:04

- Hi, everybody.

00:05

Hi.

00:07

- I'm Ali, and I'm here today

00:10

to talk about everyone's favorite subject.

00:12

Drum roll, please.

00:14

(Ali imitates drum roll)

00:19

Rails upgrades.

00:20

(audience cheers)

00:23

So there are dozens of us, okay, cool.

00:30

All right, so I'm surprised.

00:32

I thought more people were not going to like Rails upgrades.

00:35

That's fine.

00:36

But when I look past, through past RailsConfs,

00:39

I see a lot of talks about Rails upgrades.

00:41

So there was one in 2017,

00:44

one in 2018, excuse me, two in 2018,

00:48

one in 2019.

00:50

There was one last year, and just

00:52

yesterday, there was a workshop on upgrading Rails.

00:55

Great workshop, by the way.

00:57

And since we all love working on Rails upgrades,

01:01

why are we spending so much time talking about them?

01:04

When I think about this question,

01:05

I think about all the different upgrades I've worked

01:07

on over the course of my career at Test Double.

01:10

And one thing stands out.

01:13

These organizations aren't coming to us

01:15

to work on these upgrades 'cause they're easy.

01:18

They're coming to us 'cause they're hard.

01:21

But what is it about these upgrades that makes them so hard?

01:25

The first thing that might come to mind

01:27

are all the technical challenges involved

01:29

with working on an upgrade.

01:31

For example, the first thing

01:33

you might want to do in an upgrade,

01:34

bundle update rails.

01:36

Might be a multi-day journey through pain and suffering,

01:40

trying to get your gem versions to work with Bundler,

01:43

and you find out, uh, this gem's not maintained anymore,

01:46

so what do you do about that?

01:48

Or there's some weird fork and you're just like, really?

01:50

I just want to run bundle update Rails.

01:52

A standard command, right?

01:54

Or sometimes there's changes in the public API.

01:57

For example,

01:58

how many people here remember the strong params migration

02:01

from Rails 3 to Rails 4, right?

02:03

Pretty big migration.

02:04

And if you had a big app with a lot of controllers,

02:06

this is a big change.

02:07

But no matter what the technical challenges are,

02:11

all of us here, we can solve these technical problems.

02:14

And I say that pretty confidently, because if we couldn't,

02:18

there wouldn't be any upgrades ever.

02:20

And if there were no upgrades,

02:22

none of us would be here

02:24

'cause who would be building stuff in Rails, right?

02:28

So maybe we need to reframe this question

02:31

about why is it so hard for organizations

02:34

to work on these upgrades.

02:36

And instead, it's why is it so hard for organizations

02:39

to invest the time and money to do these upgrades?

02:43

And to think about that, let's shift gears a little bit

02:46

and talk about something that organizations do every day,

02:50

and that's feature work.

02:51

And let's see how this compares to Rails upgrades.

02:54

So when your team embarks on building feature X, Y, Z,

02:58

all the people involved, all the stakeholders,

03:00

whether it's your product managers, UI/UX, engineering, QA,

03:05

whoever needs to be involved in making that feature happen,

03:08

all of y'all have some context,

03:11

some kind of shared understanding

03:12

of this is the app that we're working in,

03:15

or this is our business domain, et cetera, right?

03:18

All of y'all know something about the app,

03:20

so you can talk about it.

03:22

And because you have this shared understanding,

03:24

it's easier to break up the work.

03:26

And this might mean identifying the scope,

03:29

but what's our appetite here?

03:30

How much of this do we want to do right now?

03:32

Or maybe it's breaking it up into small discrete steps

03:35

that you can work on bit by bit by bit,

03:38

shipping stuff incrementally.

03:40

And because we can break up the work,

03:43

it's easier, then, to share progress.

03:45

You might say this feature,

03:46

we broke it up into 10 cards and a couple cards in,

03:49

we kind of know, are we on target for our deadline

03:53

or do we need to have another conversation and say, hey,

03:55

like, we found some new information out,

03:58

we either need to adjust scope or our deadline here.

04:00

Something like that, right?

04:01

And all this is easier because everyone involved

04:04

has some kind of shared understanding,

04:06

and you can break that work up into smaller pieces.

04:10

So how does this compare then to Rails upgrades?

04:13

Well, for one, Rails has a lot of code in it, right?

04:19

I've gotten deep in the weeds

04:21

of ActiveRecord a couple times,

04:23

and I'll say I know maybe 5% of the code in Rails.

04:27

And if you saw Eileen's keynote yesterday,

04:29

she said there's no core committer that knows all of Rails.

04:33

So it's a really big code base

04:35

that no one really understands, right?

04:39

And Rails is crucial, crucial infrastructure

04:42

in our application, because without it,

04:45

we don't really have an app to talk about, right?

04:47

So now we're talking about changing something

04:51

that we don't know really well,

04:54

and we want to change it, potentially, a lot.

04:58

Who is going to sign up for this project

05:01

that has so many unknowns?

05:03

What could go wrong?

05:04

How long is this going to take?

05:05

When will it be done?

05:08

Nah, no one signing up to work

05:10

on this risky Rails upgrade project.

05:13

But what if we could change

05:15

the way we think about Rails upgrades

05:17

and think of them more like feature work

05:20

where we take a big black box of a project

05:25

and we break it up into smaller pieces

05:28

that we can work through incrementally,

05:30

shipping stuff bit by bit

05:32

and using the information that we learn as we ship stuff

05:35

to help inform the next step we're going to do.

05:38

If we could do that,

05:40

we'd have upgrades that are more predictable,

05:43

and if they were more predictable,

05:46

they'd be more repeatable, because spoiler alert,

05:49

there's going to be another Rails upgrade, right?

05:52

So how do we get here?

05:55

How do we get to having Rails upgrades

05:57

that are more predictable and more repeatable?

06:02

Well, first things first,

06:04

we have to figure out what version

06:05

of Rails are we even upgrading to,

06:07

which might seem like a pretty obvious question.

06:10

We just want to get to whatever

06:11

the latest version of Rails is, right?

06:14

Well, let's imagine we have an app running

06:16

on Rails 5.2,

06:17

and right now, Rails 7 is the latest version.

06:20

So here's our big black box.

06:22

We want to upgrade to Rails 7.

06:25

Is there anything we can do to break this up,

06:28

break this up into smaller pieces?

06:30

Well, since we're coming from Rails 5,

06:32

we can say Rails 7 is made up of two major versions,

06:35

Rails 6 and Rails 7.

06:38

And if we want to upgrade to Rails 7,

06:41

we have to upgrade to Rails 6 no matter what.

06:43

So we could try to do it in one big giant PR

06:46

that changes everything,

06:47

or we could just focus on upgrading to Rails 6.

06:51

And now, our black box is looking a little smaller.

06:56

Is there anything else we could do

06:57

to break this down even further?

06:59

Well, Rails 6 is made up of two minor versions,

07:02

Rails 6.0 and Rails 6.1.

07:05

And just like talking about going from 6 to 7,

07:06

you have to do 6 no matter what,

07:08

if you want to go to 6.1,

07:10

you're going to have to upgrade to 6.0.

07:11

So instead of trying to do that all at once,

07:15

let's just upgrade to 6.0.

07:17

And now, our big black box is looking a lot smaller.

07:21

So what we're trying to do here is,

07:24

when we're trying to figure out what version

07:25

we want to upgrade to,

07:26

is first target that next minor version.

07:29

So if you're on Rails 5.0,

07:31

next minor version is 5.1,

07:33

that's what you're going to upgrade to.

07:35

And once you get to 5.1,

07:37

your next minor version is 5.2,

07:38

so that's your next upgrade.

07:40

And once you've exhausted all the minor versions,

07:43

you're going to upgrade then to the next major version.

07:46

So if you're now on Rails 5.2, there is no Rails 5.3,

07:51

you're ready to go on to that next major version,

07:53

Rails 6.0.

07:55

And in doing this,

07:57

we're making our upgrades smaller.

07:59

Instead of trying to take years and years

08:00

and years of changes to Rails

08:01

and apply it to our application all at once,

08:04

we're just going to focus on a narrow chunk

08:06

that's right in front of us,

08:09

and this will help drive down the risks of our upgrade,

08:11

because again, we're not going to try to make a bunch

08:13

of changes in one big go.

08:17

So now we're starting to get a little more clarity, right,

08:20

of what our target version we want to upgrade to.

08:24

So now we're ready to run bundle update rails, right?

08:28

Well, before we get ahead of ourselves,

08:32

we need to talk about everyone's second favorite subject.

08:36

We're already talking about Rails upgrades,

08:37

which everyone loves,

08:39

but we have to talk about everyone's second favorite subject

08:42

and that's deprecation warnings.

08:44

I know, I've got all the hits today, right?

08:46

Okay, so I'm going to need some audience participation here.

08:49

Can I see a raise of hands

08:51

if you've ignored deprecations in your app recently?

08:55

Yeah, right?

08:56

Me too.

08:56

That's what I do every day,

08:58

ignoring deprecations because they're just warnings.

09:01

They're not, our app's still working

09:03

and nothing for us to do about 'em, right?

09:05

But when you're starting on a Rails upgrade,

09:07

it's really important that you look at them

09:08

to see what they're saying.

09:10

For example, imagine now that we're on an app

09:13

that's running in Rails 6.0,

09:14

and we see this deprecation warning

09:15

flying through our logs.

09:18

It's saying, accessing hashes from config_for

09:22

by non-symbol keys will be removed in Rails 6.1.

09:27

We're on Rails 6.0,

09:29

so this thing that we're doing,

09:31

accessing the hashes with non-symbol keys,

09:33

still works fine, right?

09:35

But when we go to Rails 6.1, the next minor version,

09:39

it's not going to work anymore.

09:41

And the last thing this deprecation warning is telling us

09:43

is use symbols for access instead.

09:46

It's not saying upgrade to Rails 6.1

09:48

and open that giant PR that's changing

09:50

a ton of things, right?

09:52

It's saying use symbols for access instead,

09:54

and this is something you can do in your application today.

09:58

And we can do that for all

09:59

the deprecation warnings, right?

10:00

We don't have to wait for a huge big effort

10:03

to get our app one step closer

10:06

to working on that newer version of Rails.

10:09

And the good news is,

10:11

we don't have to fix all these deprecations at once.

10:14

For example, imagine you have a deprecation warning

10:17

that's popping up across hundreds

10:18

of files in your application.

10:19

It's a huge mess.

10:21

You could try to open it in one giant PR

10:24

and end up changing a ton of files,

10:25

but no one on your team is going to want to review them.

10:28

And eventually, you find somebody who's like, look,

10:30

don't worry, it's just deprecations,

10:31

just gimme a check mark, please.

10:33

So, okay, you get your check mark.

10:34

But when it's time to hit that merge button

10:36

and deploy this thing to production,

10:38

you're going to be pretty nervous

10:39

because you're changing a lot of stuff in your app.

10:42

Instead, you could fix that deprecation warning in one file.

10:48

If you change it in one file,

10:49

it's going to be easier to talk about,

10:50

easier to get your team to review it,

10:52

because okay, it's a smaller change,

10:53

we can see what's going on here, right?

10:56

It's going to be easier to test,

10:57

and it's going to be easier to verify that,

10:59

yeah, we made this change,

11:00

and it's working on production the way we expected it to.

11:03

And with that information, the stuff that you're learning,

11:06

it's going to be easier then for you to come back

11:08

and confidently make that change

11:10

to all the other files in your application.

11:14

So now you take your deprecation warnings

11:15

and you divvy them up across the people on your team.

11:17

Say, Hey, everyone,

11:18

pick one up and fix it here, fix it there.

11:21

And you're working really hard, working diligently,

11:24

getting your app, again, one step closer

11:26

to running on a new version of Rails.

11:28

And lo and behold, someone comes behind you

11:31

and brings back a deprecation that you just fixed.

11:33

And this is a super, super frustrating experience

11:36

'cause you're working so hard,

11:38

it's like, okay, I'll fix the deprecations.

11:39

This is like a good thing to do.

11:41

And why would a developer do this?

11:42

Why are they not paying attention

11:44

to all the work that we're doing?

11:45

But we just said a couple minutes ago,

11:48

everyone's ignoring the deprecation warnings.

11:50

So can we really blame a developer

11:52

for doing something that we all do every day?

11:56

No.

11:57

Instead, we can make it easier for them to adopt

12:00

the patterns that we're going to need to use going forward.

12:03

So here's a blog post I wrote up about one strategy I like

12:06

to use when dealing with deprecations,

12:09

and if you Google,

12:10

you'll find a bunch of different ways to do it,

12:12

but they all kind of have the same core idea.

12:16

Once you've fixed a deprecation,

12:18

you'll set it up in your app so it errors in dev and test.

12:21

That way that developer, like me,

12:24

who's used to writing code a certain way,

12:26

when they're writing tests

12:27

or they're trying to boot up the app or whatever,

12:29

and they write deprecated code,

12:31

the computer's going to yell at them

12:32

and say, nope, you can't do this anymore.

12:34

And because nobody likes the computer yelling at them,

12:37

eventually, they're going to figure out, okay,

12:39

I need to adopt this new pattern going forward.

12:42

And we'll couple that with logging in production

12:45

because tests aren't perfect,

12:47

we're never going to catch 100% of everything,

12:49

so we'll log these on production

12:51

to give us one last fail-safe

12:52

to make sure there's no deprecations slipping

12:55

through the cracks.

12:58

And in doing this, we're shipping code.

13:00

We're shipping code bit by bit by bit.

13:03

Again, that's getting us one step closer

13:05

to that new version of Rails.

13:07

Wouldn't it be nice if we could stay in this mode

13:10

for all of our Rails upgrade?

13:12

Well, let's see what we can do.

13:14

So now that you fixed all your deprecation warnings,

13:17

you're ready to run bundle update rails

13:19

and you check out a new branch.

13:22

Nothing new there, right?

13:23

We always check out a new branch

13:23

when we do something new in our application,

13:26

but for as long as this branch is living,

13:30

you're driving up the risks of your upgrade.

13:33

So let's imagine you're on your branch down here

13:35

and you spend a couple days doing the Bundler battle

13:38

to get the new version of Rails to install,

13:41

and eventually, you get it to work.

13:45

And in all that time you keep

13:46

on working away on this branch,

13:48

development's been cruising along on main.

13:51

And not a big deal, it's only been maybe a day or two,

13:53

so you pull those changes in from main to your branch.

13:57

And development keeps happening on main,

13:59

and there was some security patch

14:01

for another gem that got released in that time,

14:03

so they went ahead and made that change

14:04

to the Gemfile, pushed it up.

14:06

So now, when you come back another couple days later,

14:09

after you've been trying to fix tests or get the app to boot,

14:12

you rebase and you run into some conflicts

14:14

because now the Gemfile's changed in two places,

14:16

it's like, ugh, this is kind of frustrating,

14:18

but maybe it's not too bad.

14:20

You figure it out in an hour or something like that.

14:23

And then your boss comes through,

14:24

and says, whoa, whoa, whoa, whoa, whoa, whoa, whoa.

14:27

We have a really important urgent thing we need

14:29

to work on right now, so please,

14:31

get to a stopping point on this upgrade,

14:33

and we'll get back to it when we have time.

14:34

You say, okay, I'll push up my changes to my branch,

14:36

and we'll come back to it, no big deal.

14:39

And development cruises along on main.

14:43

Time passes.

14:44

Could be weeks, months, years.

14:48

I've seen it.

14:50

And you come back and now your branch is so far behind main.

14:54

You try to pull the changes in,

14:56

but now, the Gemfile's changed again.

14:57

It's like, ugh, this is frustrating,

14:59

I have to do this whole Bundler thing again,

15:01

or those tests that you were working so hard to fix,

15:03

now they've changed on main,

15:05

so the changes that you made on your branch

15:06

don't make sense anymore,

15:08

and you're scratching your head like, what do I do?

15:09

How do I get this to work?

15:10

And in the end, you're like, you know what?

15:13

I'm just going to start this upgrade over from scratch.

15:16

And that is a huge risk 'cause what we're saying there is,

15:20

if at any point we have to stop our upgrade

15:23

because life changes, things happen,

15:26

we could lose all of the work

15:27

that we've done up to that point,

15:30

and that's a huge bummer,

15:31

and no one wants to be in that situation.

15:33

So what can we do to get out of this?

15:36

Well, we can turn to a strategy called dual booting.

15:39

And what you'll do with dual booting

15:41

is you'll set up your application on the main branch

15:44

so it can run on either the production version of Rails

15:46

or the target version of Rails you're upgrading to,

15:49

and you'll have some kind of switch,

15:50

and by default, that switch will usually be off.

15:52

And when it's off, your app's running

15:55

on the production version of Rails.

15:57

Nothing out of the ordinary there, right?

15:59

It's just the same thing that's always been happening.

16:01

But when the switch is turned on,

16:04

we'll be running the target version of Rails

16:06

on our app on the main branch.

16:10

There are a lot of different strategies

16:12

out there for dual booting.

16:13

I like using Bootboot.

16:15

It's a Bundler plugin made by Shopify,

16:17

and I like using it so much that I'll bend over backwards

16:21

to get it working on Rails 4.

16:23

Story for another day.

16:26

So now, when you go back to your branch

16:28

and you run bundle update rails,

16:31

you're not just going to be doing the Bundler dance

16:33

to get the new version of gem,

16:34

the new version of Rails installed,

16:36

you'll also be adding that dual booting functionality,

16:39

so you can toggle the app back

16:40

and forth between Rails versions.

16:42

And as soon as you get those two things working,

16:45

you can bundle the new version of Rails,

16:47

and you have dual booting enabled

16:48

so you can toggle Rails versions.

16:51

You ship that code.

16:52

You're not going to wait to say,

16:53

oh, let me get these tests, I can get these tests to work.

16:55

Mm, mm.

16:56

Or, the app's not booting, let me make sure that, no.

16:59

As soon as you get dual booting working

17:02

and you can bundle a new version of Rails,

17:03

you ship that off to production,

17:05

because from that point forward,

17:07

any changes that you need to make

17:09

for your Rails upgrade will be on main,

17:11

and after today, I never want to hear again

17:14

somebody maintaining a long-lived branch

17:16

that they're rebasing every day

17:17

for months for their upgrade.

17:19

You don't have to do that anymore.

17:20

Dual booting is here, please use it.

17:23

And if you do that, then, you're shipping code.

17:26

You're shipping code every day to main,

17:28

and you're driving down the risk of your upgrade.

17:32

But if you're following along with me,

17:34

you might be asking yourself some questions

17:37

of like how is this going to work exactly,

17:39

because there's got to be new stuff

17:42

in the new version of Rails

17:44

that isn't in the old version of Rails,

17:46

and somehow we're going to deploy that code to main,

17:48

but how's it going to work without the new version of Rails?

17:52

I'm just confused saying that.

17:54

Like, how is this going to work, right?

17:57

Well, so once you go down this strategy

17:59

where you have dual booting in place,

18:01

you're going to see conditionals

18:02

like this pop up in your application.

18:03

And here, let's imagine we have an app running on Rails 5.2.

18:07

We'll say, if our app currently is running on 5.2,

18:11

here's the code that we want to run on production.

18:14

Otherwise, here's the code that we'll be using

18:17

in the future version of Rails.

18:19

And now, let's just ship the code

18:21

for both version of Rails to main.

18:23

And you might see a few dozen

18:26

of these or whatever, you know.

18:28

Some of these pop up in your code base, not a big deal.

18:30

Sometimes there's method signatures,

18:32

change in Rails versions, that kind of stuff.

18:35

But if you start to see the same pattern of if-elses

18:38

repeat over and over and over again,

18:43

that's going to drive up the risk of your upgrade

18:46

because that's an indicator that there's some pattern

18:49

that your team's used to that's not going to work going forward

18:54

and you're going to work hard adding those if-elses,

18:57

if-else, if-else, right,

18:58

every time somebody comes back and writes code

19:00

that the way they're used to writing it,

19:03

and you'll be stuck in a game of Whack-a-mole

19:05

that's going to be really hard to win

19:07

because all these things are always going to pop back

19:09

up in your application,

19:10

and you're never going to know, like,

19:11

did I catch all these cases?

19:13

What's failing now, right?

19:15

Not where we want to be.

19:17

So what can we do here?

19:20

Well, we can use a strategy called backporting.

19:22

And what you'll do with backporting

19:24

is you'll take whatever that new thing is in Rails,

19:27

whether it be a new feature or a new way of calling code,

19:30

whatever it is,

19:31

and you backport it to the old version of Rails

19:35

that your app is using

19:36

so then your team can start writing code using

19:40

that new feature before we finish our upgrade.

19:44

And there's an excellent open source example

19:47

of this with Strong Params.

19:49

We talked about that briefly at the beginning of this talk.

19:51

Strong Params was released in Rails 4,

19:53

and again, it was a big change in the way you have

19:55

to dealt with your parameters.

19:58

At the time that Rails released Strong Params in Rails 4,

20:02

they also released the Strong Parameters gem

20:05

and this gem targeted Rails 3.

20:08

So there's your backport.

20:09

This is Strong Parameters that you could add

20:12

to your app running on Rails 3.

20:15

And the way it worked was is,

20:16

if you have a model in your app, let's say it's a Post,

20:20

you add the gem to your Gemfile,

20:23

and then you have this module available to you

20:26

that you can include to opt in to the new behavior,

20:29

and you would include it only in the one model

20:32

and work on migrating that model

20:34

so it uses params the way we need to use it for Rails 4,

20:37

the way we need to use 'em going forward.

20:39

So you work on doing that migration,

20:41

you test everything out, verify it, ship it to production,

20:44

and once you ship it to production,

20:47

no one can use params the old way anymore

20:50

because this module will error and tell people,

20:52

nope, you can't do that anymore, right?

20:55

So as soon as you do that, you do it for the one PR,

20:57

make sure it's working, everything's good.

21:00

You then do the same process for the next model

21:03

and the model after that,

21:05

shipping small PRs bit by bit,

21:07

migrating towards how things are going to work going

21:10

forward in a new version of Rails.

21:12

And once you've done it for all

21:14

of your models in your application,

21:16

you can include it globally

21:17

for all of the ActiveRecord models in your app.

21:20

So from that point forward, someone adds a new model,

21:23

and they have to use params the new way.

21:26

And again, this is an example of a backport

21:28

that the Rails core team made for us,

21:31

and it's really helpful, right?

21:33

But we can do this ourselves.

21:36

We don't have to depend on Rails

21:38

to make all our backports for us.

21:40

So here's a blog post that I worked on talking

21:43

about a similar situation I ran into in an upgrade

21:45

and how we implemented the backports

21:47

and helped the team adopt a new pattern,

21:51

because that's really what we're trying

21:52

to do with these backports.

21:53

We're trying to coach our teams to adopt a new patterns

21:57

that we're going to need going forward.

22:00

And depending on when you bring your backport in,

22:02

they might be writing code for that new version

22:05

of Rails for months.

22:07

So, when you do make that switch on production

22:09

to running the new version of Rails,

22:11

they're not going to be surprised that,

22:12

oh, I can't write code like this anymore, what happened?

22:14

No, they might have been doing

22:15

it for a long time already, right?

22:18

And again, we're shipping code,

22:20

always shipping code bit by bit by bit,

22:23

getting us closer to that new version of Rails.

22:27

So things are going good,

22:28

we feel pretty confident about our Rails upgrade,

22:31

but we have one more big black box to talk about,

22:35

and that's test failures.

22:37

On all of the upgrades I've worked on,

22:39

I've seen teams spend the majority

22:41

of their time fixing failing tests.

22:44

And what can I say about test suites?

22:47

They're big, slow, complicated, brittle, hard to understand.

22:53

I could keep going.

22:54

(Ali laughing)

22:56

So what do you do when you're working on a Rails upgrade

22:59

and you see hundreds or thousands

23:01

and thousands of test failures

23:03

in these big complicated test suites?

23:06

We've been talking today about breaking things

23:07

up into smaller pieces

23:09

that we can work through incrementally,

23:11

so if you're seeing thousands

23:13

and thousands of test failures,

23:15

you could break up the work that way and say,

23:17

everyone, take one of these failures,

23:19

figure out how to fix it and open your PRs,

23:22

and we'll ship little tiny PRs bit by bit by bit, right?

23:26

But personally, I don't want to open thousands

23:29

and thousands of PRs,

23:31

or even worse, write thousands and thousands

23:33

and thousands of Jira tickets, right?

23:35

No one wants to do that.

23:36

So there's got to be some middle ground here

23:38

between having a giant PR that's fixing a bunch

23:41

of unrelated tests

23:43

and having little tiny PRs that are narrowly focused

23:46

where we're doing the same kind of things, potentially,

23:48

over and over again.

23:50

What can we do?

23:52

Well, we could try to group our failures.

23:55

So let's say we look at our tests,

23:56

we look at all of our failures,

23:59

and we notice a bunch of tests failing for,

24:01

expected 200, got 400.

24:04

Well, we could pull those off as a group, and say,

24:07

two people, go look at those tests

24:09

and see why are they failing?

24:11

Are they failing for the same reason or not?

24:13

And after a couple days, they look at it,

24:14

and they say, oh, it looks like there's a bunch

24:16

of tests failing for reason A,

24:18

and then the rest of them are failing for reason B.

24:21

Well, you can break that down then

24:22

into two other groups of tests, right?

24:25

And you can say, person one, go ahead

24:27

and work on that first group of tests

24:29

and the other person can then tackle

24:31

the other group of tests.

24:33

Or maybe you see a bunch of tests failing

24:35

for ActiveRecords::StatementInvalid

24:38

you can't do this anymore.

24:40

You can pull those apart as a group,

24:42

and they're probably also failing for the same reason.

24:45

So give somebody that piece of work to work on,

24:48

and they can work through a solution.

24:50

And you'll do this for all the failures in the test

24:52

where you're trying to identify,

24:54

trying to identify all these different groups.

24:57

And you'll start to see they're just finding chunks of work

25:00

that are just the right size.

25:03

They're not too big where you have a bunch

25:06

of unrelated changes going on,

25:08

and they're not too small where you're just opening

25:10

the same PR kind of over and over again, right?

25:13

Kind of a Goldilocks situation.

25:17

And the grouping you do is going to depend on your application

25:20

and the failures you're seeing.

25:22

So for your upgrade,

25:24

it might make sense to group the failures by file.

25:27

Totally valid option.

25:29

Or maybe you run your tests on CI across 40 containers

25:33

and you have 15 failing.

25:34

You say, okay, everyone take a container

25:36

and make that container green.

25:38

Again, valid option.

25:39

Depends on your application, the failures you're seeing,

25:42

and you know, the upgrades you're working on.

25:45

And what we're trying to do here, again,

25:46

is find chunks of work that are the right size

25:49

that keep us in a state where we are building momentum

25:52

and we're shipping code,

25:54

always shipping code bit by bit by bit

25:57

to get us closer to running on a new version of Rails.

26:02

So now we've fixed all of our deprecations,

26:04

we've got dual booting working,

26:06

and we fixed all of our tests.

26:08

So we're ready to make that switch from production,

26:10

from the production version of Rails

26:12

to the new version that we're upgrading to.

26:16

Are we though?

26:18

Even after you've done all this hard work,

26:21

getting your app, again, closer to the new version of Rails,

26:25

it can be really hard to assess

26:27

what's the risk left of making this switch?

26:30

So at this late stage of our upgrade,

26:33

is there anything we can do to drive

26:35

down the risks involved in making the switch?

26:40

Well, we can try small experiments

26:42

in production environment, excuse me.

26:44

We can try small experiments in production-like environments

26:48

to see if the upgrade is behaving the way we expected it.

26:51

So you can imagine, you have your app running in production,

26:54

and you decide we're going to make a deployment

26:56

that has the upgrade that's for internal use only.

26:59

And we ask all of our employees

27:00

to please use this upgrade version of the app,

27:03

and we'll be monitoring our Datadog dashboards

27:06

or Rollbar errors,

27:07

whatever observability you have to see,

27:10

are we seeing new errors?

27:11

Is the app working the way we expected it to or not?

27:14

Might call this dog filter, right?

27:17

Or maybe you want to try something like a canary deploy

27:20

where you say, we'll ship the upgrade

27:24

to 5% of our traffic for two minutes.

27:28

Might have some impact on your users,

27:30

but it's a really small experiment.

27:31

We're only going to deploy this out for 5%

27:33

of our traffic for two minutes.

27:35

And after that's done, we're going to pull it back.

27:37

And in those two minutes, again,

27:38

you have all your dashboards,

27:40

and you're monitoring everything up to see,

27:42

are new errors coming through?

27:43

Is this working the way we expected?

27:44

All that stuff.

27:46

And after a couple minutes, you're like, oh yeah,

27:48

there are a bunch of errors that we did not expect to see.

27:50

So you go back, fix those errors,

27:53

and you say, okay, when we come back

27:54

and do this experiment again,

27:57

we expect there to be less errors, right?

27:59

Because if we fix them, the error should go down.

28:01

So you do that, you see less errors,

28:04

and you say, okay, let's go ahead

28:06

and bump this up then to 10% of our traffic for two minutes.

28:09

And you do the same kind of dance,

28:11

maybe you find some new errors, go back and fix it,

28:12

try to experiment again.

28:14

And after a while, it starts looking really good,

28:17

so you say, let's bump this up to 10%

28:18

of our traffic for five minutes,

28:21

and this time, whoa, wow, it's looking really good.

28:24

And then finally, you're like, you know,

28:26

let's bump this up to 40% for 10 minutes,

28:28

because we're feeling really good about the upgrade.

28:31

We're not seeing new errors pop through.

28:33

It seems to be that the app is working really well,

28:35

and we're starting to feel more confident

28:38

about this upgrade that we've been working on.

28:41

And that's really what we've been trying to do,

28:44

talking through all these different strategies today.

28:48

We're trying to drive down the risks of our upgrades,

28:52

gaining new information to help us feel more confident

28:55

that we're not going to cause an incident

28:57

by shipping out a Rails upgrade.

28:59

And if we can do that,

29:01

we'll all be shipping zero downtime Rails upgrades.

29:06

All right, so the first upgrade that you do like this,

29:11

it's going to be the hardest.

29:13

It doesn't really matter what version

29:15

of Rails you're on or anything like that.

29:17

It's going to be the hardest because your team,

29:19

your organization, probably has never done

29:21

an upgrade like this before.

29:23

So you have to figure out, what tooling do we like?

29:25

How do we want to divvy up the test failures.

29:27

All that stuff, you have to sort through that, right?

29:30

And in that process, you're going to be learning a ton.

29:34

You're going to be learning, oh,

29:35

we really like using this tool for deprecations,

29:37

or we were really happy with the way

29:39

the deploy went out in doing it this way.

29:41

Whatever it is, you figure that out,

29:43

document it, reflect on it,

29:45

and that's going to help you build a playbook, right,

29:47

a process where this is how we're going to do

29:50

the upgrade next time.

29:52

And if you do that,

29:53

when it's time to do the second upgrade,

29:55

might be a little bumpy,

29:56

but it's going to be a little bit easier.

29:59

And by the time you get to the third upgrade,

30:02

you're going to be cruising, and like, okay,

30:03

we've done this before,

30:04

we know the in and outs of this thing,

30:06

no problem, we can do it.

30:09

And maybe one day, you'll get to a place

30:12

like where GitHub is,

30:13

where they're upgrading Rails weekly, right?

30:17

I don't know if anyone wants to live

30:18

on the cutting edge like that,

30:19

always running the latest version of Rails on edge, right?

30:24

But it goes to show,

30:25

once you refine your process and figure it out,

30:28

these are the tools that we like,

30:29

this is the way we want to do it,

30:30

you can get to this point where it's every week,

30:32

not a big ceremony to do another Rails upgrade.

30:36

And as you go through this journey,

30:38

learning these new things

30:39

and applying different things to your upgrades,

30:43

share what you've learned.

30:44

As I showed you at the beginning of this talk,

30:46

every year there seems to be another talk

30:48

about Rails upgrades.

30:49

So there's always going to be something new to share,

30:52

so please come back, share what you learn,

30:55

and you can help everyone else

30:57

also ship zero downtime Rails upgrades.

30:59

Or if you don't like public speaking,

31:01

you can write a blog post.

31:02

I have a bunch of those too.

31:05

So that's all I have today.

31:08

If you have any questions about Rails upgrades

31:10

or anything, really, feel free to email me,

31:14

aliwritesback@gmail.com.

31:16

And I'm also on the Slack for however long

31:18

that's going to be around, at my full name.

31:20

And again, I mentioned a few blog posts in this talk.

31:23

You can find them all on the Test Double blog

31:25

and other blog posts about upgrades, too,

31:27

and a bunch of other good stuff.

31:30

And there's a lot of us here from Test Double.

31:33

I really enjoyed Daniel's talk yesterday talking

31:35

about legacy code and being more curious

31:37

and how can we make it better.

31:39

I'm looking forward to seeing Landon speak, soon,

31:43

in a few minutes about machine learning and Ruby.

31:45

I think that's going to be really cool.

31:47

And Megan and Justin going to be having

31:50

a little town hall tomorrow,

31:51

making standard happen, even better for all of us, right?

31:55

So that's all I have.

31:57

Thank you.

31:58

(audience applauding)