Looking for failure

Failures are interesting. They tend to teach lasting personal lessons, but we are loath to share them. These lessons have profound impacts on our habits and viewpoints, but they’re difficult to recount. We need to get comfortable talking about failure. Unfortunately, our brains are wired to screw this up. When confronted with a failure, we are conditioned to ignore it, make ourselves blameless, or create elaborate strategies to avoid the possibility of failure altogether. I’d like to present another alternative: embrace failure. It will happen. That’s ok.

How can we challenge our natural human bias? I’ve spent the last couple years digging into “pop” psychology and thinking about how books like “Thinking Fast and Slow”, “Antifragile”, “Drive”, “Blink”, and “Outliers” apply in a software development environment. I’m most excited about the idea that if we can make failure “safe” then we can use it to learn things that we would otherwise be tempted to avoid and ignore because it might go badly. A focus on failure, both deliberate and accidental, has been beneficial in moving my projects beyond the no defects, 100% test coverage type focus and towards finding innovative ways to detect and quickly reverse failure.

The video above was recorded at Agile and Beyond on May 5, 2017.

00:00

The title of this talk is

00:01

Looking for Failure

00:03

My name is Steve Jackson

00:04

I'm a double agent at Test Double

00:06

Test Double's mission is

00:07

to make software better

00:09

so that naturally means we have to get

00:10

pretty comfortable with failure

00:12

If you have any trouble with failure

00:14

and want to get in touch with us

00:16

you can email us at hello@testdouble.com

00:18

We might be able to help

00:19

This is my personal twitter, @stevejxsn

00:21

and it's spelled weird

00:22

because it's really common, right?

00:24

And then my email, steve@testdouble.com

00:26

I'm interested in this topic of failure

00:28

and seeing if it's resonating with people.

00:32

I'd like to get a conversation started

00:33

around with this

00:34

If any of this makes sense to you

00:35

and you want to share your story

00:37

I'd love to hear it

00:39

I love this quote from Thomas Edison

00:41

It really embodies that

00:44

ideal of innovation to me

00:46

That if you just keep pushing

00:47

if you just keep trying

00:49

you'll get past all your failures

00:50

and eventually be a success

00:52

That's not what this talk is about

00:55

This is not a talk about perseverance

00:58

This is a talk about identifying failure early

01:00

looking it straight in the face

01:02

and stepping into it anyway

01:06

What sort of things happen when you look to fail?

01:10

So I'm a big believer in the Law of Two Feet.

01:11

I want everyone to get as much value

01:13

as they can out of this conference.

01:14

So, what this talk is about

01:16

is we're going to talk about failure

01:18

as a learning aid and how we can use it effectively

01:21

or how it could be used effectively

01:24

I'm going to talk about some of the reasons

01:25

why it's difficult to use failure effectively

01:28

and all the different biases we have in place

01:30

that make that really hard

01:32

I'm going to talk about some hypotheses

01:34

that I'm actively testing to get myself

01:36

better at dealing with failure

01:38

And then, I wanna kind of close talking

01:40

about how we can make it safe

01:41

for other people to fail and

01:42

some things maybe we should think about

01:46

So, we're kinda raised from a young age

01:48

to believe that if you're getting up early

01:50

you're doing all your homework

01:51

you're helping the old ladies across the street

01:54

eventually you'll be successful

01:56

But, if you're watching too much TV

01:58

and doing drugs

01:59

and hanging out with the wrong crowd

02:01

You're doomed. No chance

02:03

We see success as this linear thing

02:05

We assume that anyone that was successful

02:07

obviously was successful their entire life

02:09

and it's just kind of this

02:10

wonderful golden path

02:12

But it completely

02:14

contradicts our own experiences

02:15

We know we have ups and downs

02:17

and good days and bad days

02:18

and that's how the world works

02:22

Objectively, success and failure

02:24

are just the effects

02:25

of an action or a non-action that we take

02:29

If the effect is positive, we call that success

02:32

If the effect is negative, we call that failure

02:35

It's just feedback

02:38

I got this from Jurgen Appelo

02:40

This is his Celebration Grid

02:42

And what I really like about it

02:44

is it takes these two binary outcomes

02:45

of success and failure and points out

02:47

that your mindset and your intent do matter

02:50

So up at the top here, we have

02:51

when you're successful by mistake

02:54

You're "Wow, I managed to get lucky

02:56

and pull that off anyway"

02:57

And this is pointing out that there's really

02:59

not much to celebrate there

03:01

It wasn't through any particular amazing skill

03:03

that you did that

03:04

And then underneath we have your

03:05

garden-variety failures that we regret

03:07

and obsess about for the rest of our lives

03:11

In the middle, if we're in an

03:13

experimental mindset here

03:14

then either success or failure are fine

03:16

We're in a mindset where we can learn

03:18

So, either way we can find something

03:20

to celebrate in that

03:22

We didn't do exactly what we thought

03:23

but we learned from it and it was good

03:26

In this last column we have this idea of practices

03:29

If you succeed by doing the right things

03:31

then you should celebrate that

03:32

That's great!

03:33

But if you don't succeed while doing the

03:35

right things, well that's just the opposite

03:37

of that other good luck

03:38

That's just bad luck

03:40

And maybe we shouldn't be

03:41

so down about that

03:42

Now where I would differ a little bit

03:43

from the Celebration Grid is

03:44

I think there is value in these other

03:46

two types of failure

03:47

So I'm going to talk about that

03:49

a little bit more

03:50

But if you're interested

03:51

in this sort of thing

03:52

this comes from Workout

03:53

This is a great book

03:55

lots of nice visualizations about working

03:56

with teams and improving

03:59

the way that you work

04:03

Failures are interesting to me

04:05

They have a certain amount of stickiness

04:09

that success doesn't seem to have

04:11

When I succeed at something

04:12

I immediately look to the next thing

04:14

I have to do

04:15

It's a very temporary feeling

04:16

But a failure, man

04:17

I can sit there and think about that for awhile

04:20

Think about that stupid thing

04:21

I did back in high school. Right?

04:23

I'll see someone that's successful

04:26

and you might try to copy their success

04:28

You want to emulate them

04:30

But we often miss something in the middle

04:32

Because again, our brains just assume

04:33

It's a linear path

04:34

So we miss the struggle and the growth

04:36

that they had to go through to get there

04:39

But, we're really, really good

04:41

at pointing out other people's mistakes

04:43

Everyone's like, yes, obviously Mr. Stormtrooper

04:46

You should check the wash for the red sock

04:48

We all know that

04:49

We've all learned that lesson before

04:52

Successes don't teach us a whole lot

04:54

except that success is possible

04:56

But what happens when you fail?

04:57

You reduce your overconfidence

05:00

You start rethinking your strategy

05:02

You question your assumptions

05:04

You want to redeem yourself

05:05

You engage

05:07

You start paying attention

05:09

Video games have an interesting

05:11

relationship with failure

05:14

It's not until you fail that you

05:15

understand how to play the game

05:17

I have to go and touch that turtle

05:18

to find out he's not my friend

05:20

and I shouldn't do that in this game

05:22

But what's more interesting to me

05:24

is that video games have different

05:25

difficultly levels

05:26

Now if the point of playing the game

05:28

is to win

05:28

Why would you play

05:29

at anything but the easiest level?

05:32

And I can imagine, well ok

05:33

maybe it's not very much fun

05:34

so we need different levels to

05:35

challenge you a little bit more.

05:37

Maybe that makes sense

05:39

But what I find interesting is

05:40

a lot of players who play a lot of games

05:42

start on the hardest level

05:44

The hardest difficulty level

05:45

Why is that?

05:47

What I think it is, is that at easier stages

05:50

you can succeed by mistake more often

05:53

You just kinda get by and don't know why

05:55

But when you play on the harder levels

05:57

your mistakes are punished more harshly

05:59

And it's through these mistakes

06:01

it's through these failures

06:02

that the right way to play the game

06:04

is revealed to you.

06:06

And you learn it much sooner

06:07

than at the end of the game

06:10

Now you might say, well that's

06:12

kind of interesting

06:13

Maybe I want to look at treating my life

06:16

like a video game

06:17

Maybe I want to try to fail fast

06:18

and get that feedback

06:19

and use that effectively

06:21

That's sounds great

06:22

But unfortunately, we have this problem

06:24

Our psychology has evolved over the years

06:27

over the eons to really

06:29

really dislike failure

06:31

Because failure used to mean that

06:33

you were going to get eaten by a tiger

06:35

So we want to stay

06:36

very, very, very far away from that

06:38

We try to stay far away from failure

06:40

We don't want to deal with it.

06:42

So there are basically four ways

06:44

that we treat failure

06:46

The first one is that we ignore it

06:50

This is considered a positive thing

06:52

in basketball, we call this

06:53

the shooter's mentality

06:55

Every shot I take is going in

06:57

I don't ever think about

06:58

the ones that don't go in

06:59

We all want to be like Swaggy P here

07:01

Of course! Money. Every time.

07:05

And it's common

07:06

When someone asks you about something and

07:08

you felt like it was a success

07:09

and you recount it:

07:10

Like if you asked me about that

07:11

angular project, I'll be like -

07:13

Yeah, we did these cool filterings

07:14

and our users were delighted about the way

07:16

we could re-arrange these things

07:17

We wrote these cool custom directives.

07:20

And I'll just kind of forget to mention

07:22

the days I spent

07:23

debugging lifecycle problems

07:25

and camel casing attributes

07:26

and all these other things

07:28

Our brain just kind of lets those

07:29

slip from your mind

07:33

The second thing we'll do

07:34

is we'll reframe our failure

07:36

so that it's actually a success

07:38

We'll lower the bar

07:40

So as an example

07:42

say I decide to go running this morning

07:44

I get my shoes on, I get out the door

07:46

Get down the road

07:47

get about 10 minutes in

07:48

And I hurt. I'm tired. It is cold.

07:53

I'm going to turn around and go home

07:55

you know?

07:56

Then I get to the end of the day

07:57

I get to my calendar and I have this

07:59

successive days worked out.

08:00

I'm like on 70

08:02

And I'm just going to check today off

08:04

I got outside, I exercised, I tried

08:07

And I do this like subconsciously

08:09

I don't really reflect

08:10

If I stopped and thought about it

08:12

I would be like, well my

08:13

intent was to run five miles today

08:15

I barely made it one

08:19

The third thing we'll do

08:20

if we fail, is deflect

08:22

We'll find somebody else to blame it on

08:23

That's usually convenient

08:25

Or, just as commonly

08:27

our ego will step in and protect us

08:29

And then we'll say

08:30

other people might be able to do that thing

08:32

but I can't do that

08:33

I'm not an art person, you can't

08:35

I can't design that

08:36

Of course it didn't go well

08:38

I'm not a people person

08:40

Why am I talking to customers?

08:42

That makes no sense

08:43

I'm not a morning person

08:44

Why am I up at 7 o'clock in the morning?

08:46

That's crazy

08:49

The fourth thing we'll do

08:50

is we'll try to avoid the failure

08:52

This gets kind of positive

08:54

This is were we actually start

08:56

getting constructive with our failure

08:58

So we'll come up with strategies

09:00

When my son was in kindergarten

09:02

he had to be at school at 7:30 every day

09:04

It was on my way to work

09:06

and it was about 20 minutes away

09:07

so it made sense that I'd get him ready

09:09

and we'd go

09:10

So, you know, I've got to get up

09:11

I got to get ready, got to get him ready

09:13

got to have breakfast, got to have lunches

09:14

got to go shovel the driveway

09:15

All that stuff to make sure

09:17

we can get to school on time

09:19

So naturally, I start thinking about ways

09:20

Well I don't want to be late so

09:21

I have to get up early

09:22

So I'll set an alarm here

09:24

And I'll put my phone across the room

09:26

And I'll have another alarm there

09:28

Maybe I'll get stuff ready

09:29

I've got all these plans to make sure

09:31

I don't hit that failure

09:34

I'd like to suggest that maybe

09:36

there is a 5th option

09:38

I will fail

09:40

At least some of the time

09:43

I don't like that

09:44

I don't want to say that

09:47

But, if I start to accept that a little bit

09:49

an interesting thing starts to happen

09:51

I start thinking about contingencies

09:53

Ok, if I fail to get up in the morning

09:56

What do I do?

09:57

Am I going to call my kid off of school?

09:59

Or, maybe I'll decide to drop everything

10:01

I get him to school, and then

10:03

I'll come back and get myself ready

10:05

Or maybe more interestingly

10:07

I'll decide to change the rules of the game

10:09

If I can't win

10:10

I don't want to play that game

10:12

So instead, maybe I'll find somebody

10:14

to drive my kid to school

10:15

Or maybe the problem is getting a

10:17

five year old ready in the morning

10:18

I'll hire a nanny to come help with that

10:20

Maybe I'll decide that we should move!

10:24

Or switch schools. I mean, 7:30?

10:26

That's not reasonable!

10:28

Maybe I'll decide to switch jobs

10:30

so I don't have to be work until noon.

10:31

And now it doesn't matter anymore

10:33

It's no big deal

10:34

All these options and opportunities

10:36

don't really occur to me until

10:38

I accept that I can fail

10:41

Superhuman effort won't just get me

10:43

through every time

10:46

It's just this changing of perspective

10:49

Switching the lens a little bit

10:51

opens up new opportunities

10:54

I learned this technique from Gary Klein

10:56

This is the premortem

10:57

How this works is

10:58

He suggests that if you've had a hard time

11:00

coming to a group decision

11:01

You finally come to something and you have

11:04

a plan in place, here's what he suggests:

11:07

Imagine it's a year in the future

11:09

We have executed the plan

11:10

exactly as we intended

11:12

And it's gone terribly

11:15

We've pushed United Airlines

11:16

out of the spotlight

11:18

We are the front page of the New York Times

11:21

What does that newspaper article look like?

11:26

That's when you start thinking

11:28

about these things

11:29

How can it actually go wrong?

11:31

Our default as humans

11:32

is to always be optimistic

11:34

If I'm involved, it's going to go well

11:37

And this changes that a little bit

11:39

It also gives you an opportunity

11:40

There are probably some

11:41

misgivings about this plan

11:42

Now they will emerge naturally

11:44

Ah, that was the thing I was worried about

11:46

But this can be hard for teams

11:48

People don't like to be negative

11:49

especially out of the box

11:51

So another useful tip is Six Thinking Hats

11:53

You will come at the problem from a

11:55

number of different perspectives

11:57

and kind of try out different things

11:59

But again, it's just really about changing

12:01

your perspective a little bit to let in

12:03

some more options and opportunities

12:07

So, I don't like failure

12:08

but I think there's some value

12:10

in getting better at it

12:12

So I'm working through some hypotheses

12:14

And I'm trying to experiment on these

12:16

and see how they work

12:17

and how they feel for me

12:19

The first one is around this

12:20

idea of Regret Avoidance

12:22

I learned about regret avoidance

12:24

from Daniel Kahneman's book

12:25

Thinking Fast and Slow

12:27

This is an excellent book

12:28

if you're interested

12:29

in finding out all the different ways that

12:31

your brain is trying to trick you

12:32

into thinking you're a rational person

12:35

Very excellent

12:36

What I realized is that regret avoidance

12:38

is probably the number one thing

12:39

that holds me back

12:41

And here's how I think

12:42

about regret avoidance

12:43

So imagine I have

12:45

10,000 shares of Luigi stock

12:47

But, I've decided, you know

12:49

I want some Mario stock

12:50

And it happens to be

12:51

trading at the same price

12:52

Has been for awhile

12:53

But, you know, I want something different

12:55

I want to find some Mario

12:56

And I find out, hey Todd!

12:58

Todd has some Mario stock.

13:00

Hey Todd, would you like to trade with me?

13:02

And Todd goes, Nah, you know I've had this

13:04

I'm pretty happy with it

13:05

It's cool. It's fine

13:06

Oh Ok, Justin!

13:08

I hear you have some Mario stock

13:10

Would you be willing to trade with me?

13:12

He says, "Sure"

13:13

Great. Got what I wanted

13:15

Everything's great

13:17

Fast forward 2 weeks

13:18

Luigi stock has gone through the roof

13:21

It's google + Facebook, greatest IPO ever

13:23

Justin is a billionaire!

13:27

It's pretty obvious

13:29

How does Justin feel right now?

13:31

Two Thumbs Up

13:32

He's a pretty happy guy

13:33

How do Todd and I feel?

13:36

Well, objectively we'd say we

13:37

both should probably feel kinda sad

13:40

We had an opportunity to be a billionaire

13:42

And we're not

13:44

But, almost everyone would agree

13:46

that I should feel worse

13:49

Because I took an action

13:51

that put me into this situation

13:54

Todd just stood pat

13:57

And that's what regret avoidance is about

13:59

Regret avoidance is when my aunt calls

14:01

And I don't know if I want to answer it

14:03

And I love my aunt!

14:05

We have great conversations

14:06

she's a wonderful person

14:07

But every once in awhile

14:09

she wants me to fix her computer

14:11

And so...ummm...

14:13

maybe I'll let it go to voicemail

14:15

We'll see

14:17

So how do I get past this?

14:19

What I'm trying here is

14:22

getting ready for failure

14:24

Priming myself for this failure

14:26

I've found that if I think something's

14:28

going to really suck

14:29

and then I do it anyway

14:30

it's usually not so bad

14:32

I can get used to it.

14:34

Shortly after my son was born

14:36

my wife and I had one of those

14:37

rare opportunities to get out of the house

14:39

for a few hours

14:40

So we started driving around

14:42

We come up to the movie theater

14:44

GI Joe is playing

14:46

Ok, we'll go see GI Joe

14:49

I was a little trepidatious about this, maybe

14:53

Like I knew this was going to be a bad movie

14:56

And I really really loved GI Joe

14:58

That's the highlight of my childhood

15:00

was going to my friend's houses

15:02

and playing GI Joe

15:04

So we went and saw the movie

15:06

And I was right!

15:08

It was awful

15:10

But we had such a good time!

15:12

We cheezed it up, we laughed at

15:13

all the terrible things they were doing

15:15

And it was amazing

15:17

And If I think about this objectively

15:19

I should have a lot of regret around this

15:21

This is the one movie we saw all year

15:23

I wasted it on GI Joe

15:24

Not Iron Man or any other good movie

15:26

that came out that year

15:29

The other place I turn for this

15:31

is to science

15:33

So we've all seen

15:34

the scientific method before

15:36

You make some observations

15:37

form a hypothesis

15:39

to explain those observations

15:40

You experiment to test the hypothesis

15:42

Then from those results you

15:44

change your understanding

15:45

maybe do more experimentation

15:48

Lean Startup has really jumped

15:49

all over this idea of doing

15:51

hypotheses and experiments

15:52

We're talking about it all the time

15:54

So we'll have a hypothesis like:

15:56

If we add feedback to the application

15:58

it will help with user re-engagement

16:01

So we run an experiment

16:03

we'll add 5-star ratings

16:05

We expect that 50% users will re-engage

16:08

with the app in the next couple weeks

16:10

And I've helped a number of startups

16:12

work up and think through these experiments

16:14

What I've found, almost universally

16:16

is that we always craft the experiment

16:18

to prove the hypothesis correct

16:21

Which is the exact opposite of how

16:23

science runs it's initial experiments

16:26

You always try to prove the hypothesis wrong

16:29

Because you don't want to spend

16:31

the next several months, or years

16:33

testing and re-testing and refining

16:35

that hypothesis that has a flaw

16:37

You want to find those faster

16:39

You want to fail sooner

16:40

And the place I'm most comfortable

16:42

with this is in my development habits

16:44

I like to do TDD

16:45

I like to start with that failing test

16:48

And I think about that as my

16:49

control in the experiment

16:51

The universe works the way I think it does

16:54

And then

16:55

using my powers of software mastery

16:57

I change the universe

17:00

and make my test pass

17:02

And that feels really good

17:04

And I'm perfectly

17:05

comfortable with this failure

17:07

I love these failures

17:08

In fact, if these tests pass, I get scared

17:10

The universe doesn't work the way

17:12

I think it does

17:13

and it's a weird and strange place

17:15

I like these failures

17:16

I'm fine with them

17:17

Another place I think we could really

17:20

embrace failure is around meetings

17:23

Ok. so how many people in the room

17:24

have attended a meeting where the

17:27

key decision maker didn't show up?

17:30

They're very busy people

17:33

And in how many cases, did you go ahead

17:35

and have the meeting anyway?

17:38

This is common!

17:39

We make excuses for this:

17:40

We're going to help

17:42

We'll get through some of the discussion now

17:44

We might come to a consensus

17:45

We won't have to do all this

17:48

But it never works

17:49

We end up rehashing the same

17:50

exact conversation once we get

17:51

the person in the room

17:53

Or we rob them of the context they

17:55

need to make a good decision

17:56

She can't make a good decision

17:58

because she doesn't have all the

17:59

criteria in front of her

18:01

So I'd like to suggest

18:02

that if you're in charge of

18:03

setting up meeting agendas

18:04

maybe think about defining some

18:06

failure criteria

18:07

If you hit the failure criteria the

18:09

meeting is over

18:10

Now you can create success criteria too

18:11

Nobody cares as long as the

18:13

meeting ends sooner

18:14

This is all good

18:17

My next hypothesis is around this

18:19

problem of reframing

18:21

It happens so easily

18:23

It's subconscious

18:24

How do we keep this from happening?

18:26

So my thought here is

18:27

if I can get my goals out there

18:29

I might be able to do something about it

18:31

And so for this I steal from Adam Savage

18:34

He had a great job with Mythbusters

18:36

essentially blowing stuff up

18:38

for our education

18:40

And he says, the difference

18:41

between screwing around

18:42

and science is writing it down

18:44

So, if I write down my goal

18:46

it's harder for me to subconsciously

18:48

change what the goal is

18:50

Now I can still consciously do that

18:52

That's fine

18:53

I just don't want my brain

18:54

doing weird stuff behind my back

18:56

I want -

18:56

Let's get up front about this, brain

18:58

What we're trying to do here

19:01

My third hypothesis was influenced by

19:04

Nassim Taleb's book, Antifragile

19:07

The way I think about that is like this

19:10

So, for nine years

19:13

I was a defense contractor

19:15

and it was very important to us that

19:17

we had robust systems

19:18

They needed to handle all kinds of

19:20

terrible conditions and pressure

19:22

and everything else and work fine

19:23

It was very important to us

19:25

So we wrote in java

19:27

And what we would do is catch exceptions

19:29

and try to recover from them

19:30

try to handle them

19:31

So you'd be in situation where it's like

19:33

Oh, I lost connection to the database

19:36

Tis but a scratch!

19:38

Oh, my navigation system

19:40

doesn't have a GPS feed.

19:42

It's just a flesh wound.

19:44

Oh, had enough, eh?

19:46

We spent a lot of time jumping

19:47

through all kinds of hoops to keep

19:49

this thing running no matter what

19:52

We didn't want to let our users down

19:55

Antifragile has this idea that the

19:57

opposite of a fragile thing is not

19:59

actually a robust thing

20:01

It's an antifragile thing

20:02

So a fragile thing is something

20:04

like a wine glass

20:05

Doesn't take a lot of pressure

20:06

Doesn't take a lot of change

20:07

Just a little tiny chip

20:09

and it's worthless as an object now

20:12

A robust thing is something like a stone

20:14

You can throw a lot of things at it

20:16

It's still stays a stone

20:18

it's as useful as it ever was

20:21

It's resilient to disorder

20:23

But an antifragile thing actually

20:24

gets stronger when disorder is applied

20:27

And probably my favorite example

20:29

from the book

20:29

was the idea of building muscle

20:31

When you work out

20:32

you actually tear your muscle

20:34

Small micro-tears

20:36

You cause it to fail

20:37

And it's from that failure that it

20:39

grows back stronger than it was before

20:42

And what's more is

20:43

if you do not cause your muscles to fail

20:44

they will atrophy and die

20:47

This is a system that needs disorder

20:49

it needs failure, to move forward

20:52

And when I think about software systems

20:54

I think about that system, where you fix

20:57

this bug and this other one pops up

20:59

And you fix this bug

21:00

and now this one is over here

21:01

And you're holding this thing

21:03

and every time you touch it

21:04

it seems to crumble in your hands

21:05

And it's very fragile and terrible

21:07

So naturally, we want to keep us from

21:09

changing it

21:10

So we create these things with

21:12

change control boards and all this other

21:13

stuff and that will make it robust, right?

21:15

Because now we can't let any of the

21:16

randomness and chaos and bad things

21:18

into our system

21:19

Everything will be fine

21:21

But I feel like this way of thinking

21:23

is doomed

21:24

Software has to change

21:27

Even if you manage to write bug-free

21:30

perfect software the first time

21:32

And I'm sure most of you have done that

21:34

Even if you pull that off

21:36

the world will change

21:37

The expectations for your software

21:40

will change

21:41

You have to be able to change with it

21:44

Software maintenance does not mean

21:46

that your software works the exact same

21:48

as it did last month

21:50

If it did, it would be free

21:52

Software maintenance means that it's

21:54

at least as useful as it was last month

21:57

And that requires change

21:59

So it seems like we need to get better

22:01

at this sort of thing

22:03

But, it's easy for me

22:04

to stand up here and say

22:05

You know what guys

22:06

We should fall all over the place

22:08

We should add more chaos

22:09

It's going to be great

22:11

Trust me

22:11

Who's behind me? Let's go!

22:14

There are problems with this

22:15

There are a lot of situations where it's hard

22:18

to take on more negative things

22:20

We have debt

22:21

We feel very vulnerable

22:24

If I'm barely making ends meet

22:25

I don't want to talk to you about

22:27

your startup idea

22:28

That's terrifying

22:30

If my system is one of those fragile systems

22:32

and you want to introduce more failure to it?

22:34

No. We're not doing that

22:36

That doesn't sound good to me at all

22:38

If I'm new to a team

22:39

or If I've failed recently

22:41

I'm not going to want to add

22:42

more failure to my resume or my reputation

22:46

It's not a good place to be

22:48

So my thinking starts to shift

22:50

How do we create safer failures?

22:52

That don't feel as bad when we're

22:54

feeling vulnerable?

22:56

The obvious one here, for me

22:58

seems to be that if you're gonna fail

23:00

you would prefer to fail in the small

23:02

Maybe get good at that

23:04

and get to larger failures

23:07

The engineering organization that

23:09

I feel has really taken this to heart

23:11

is Netflix

23:12

They have these tools

23:14

called the Simian Army

23:15

Essentially what it does, there are

23:17

various tools that will take out the

23:19

things that your application

23:20

or service depends on

23:22

So they have things like

23:23

Chaos Monkey that will take away

23:25

your dependent services

23:26

And you're supposed to stay running

23:28

And by the way

23:29

they run all this stuff in production

23:30

They have Latency Monkey

23:32

which will add time to the request

23:34

and you still have to give a

23:35

good user experience

23:36

And they've built this into their products

23:38

You know this is going to happen

23:40

You expect this

23:40

So they've gotten good enough with this

23:42

that they have things like Chaos Kong now

23:44

Which will take down your entire AWS region

23:48

And Netflix is pretty reliable

23:49

It works in a lot of places with not very

23:51

good network infrastructure behind it

23:55

For me personally

23:56

This is about pushing out my comfort zone

23:59

a little bit

24:00

Like if I can screw up in front of my friends

24:02

Maybe I can make a mistake in front of

24:04

my coworkers

24:06

Maybe I can mess up something

24:07

for my client

24:09

Eventually, maybe I can

24:10

fall flat on my face

24:11

in front of an audience

24:14

And yes, as we add and invite

24:17

more disorder and bad things

24:18

into our lives bad things will happen

24:21

But there's also the flip side of this

24:23

There's good luck that comes with the bad

24:25

And so, as we open ourselves up to these

24:27

different opportunities and interactions

24:29

we get good things

24:31

I answer the phone

24:32

and my aunt has playoff tickets

24:34

Great!

24:35

I'm glad I took that risk

24:38

My next hypothesis is that if we can

24:42

cause a failure to happen

24:43

and then reverse it

24:44

like we expect this to happen

24:46

then maybe we can use those same techniques

24:48

when the random failures

24:49

that we don't expect happen

24:51

Really what I'm talking about here is

24:53

building undo into everything

24:54

It's great to be able to just get back

24:57

to a good state when you're in a bad place

24:59

Either through undo buttons or

25:00

git reset all the things

25:02

Sounds pretty good

25:03

And I feel like the place where

25:05

we've really been pushing this forward

25:06

is in the areas of continuous deployment

25:08

and continuous delivery

25:10

Where we're getting pretty good

25:11

at putting small little experiments

25:13

out there and trying them

25:14

And seeing what happens

25:15

And if they fail they're still kind of

25:17

small and easy to reason about

25:18

Rather than the huge mega-releases where

25:20

we never knew what was going on

25:22

And again, I'll pick on Netflix

25:24

I learned about their deployment process

25:26

a couple years ago

25:27

Essentially you push a commit

25:29

That causes a cluster to spin up

25:31

in production with the new code

25:33

Then the load balancer starts pushing

25:34

some amount of traffic to the new code

25:36

and if it goes well

25:37

that becomes the new production

25:39

That's great

25:39

But if it goes badly

25:40

it gets dropped out of the load balancer

25:42

So it feels pretty safe to fail

25:44

in this environment

25:45

Your opportunity for screwing something up

25:48

is pretty small and your infrastructure

25:50

will save you

25:52

The place where my brain really started to

25:54

get twisted on this is when I learned

25:55

about Erlang and OTP

25:57

The Open Telecom Platform

26:00

So the thing about Erlang is that it

26:01

can run lots of little tiny processes

26:04

Like millions of them on a tiny laptop

26:07

And they startup super fast

26:09

So Erlang has evolved a very different

26:11

error handling strategy than C-based languages

26:14

In C, we always try to deal with our errors

26:16

In Erlang, you just let it go

26:18

You let it fail

26:19

You let it die

26:20

And OTP immediately starts the same

26:22

exact thing again

26:24

So again

26:25

it's very safe to fail in this environment

26:27

And this twisted my brain

26:29

because I've spent so much time

26:31

thinking about recovering

26:32

from these bad things

26:33

and How do we handle this?

26:35

What do we do now?

26:36

Just let it fail

26:37

OK

26:39

We're really trying to get to the point

26:41

where failure is a non-event

26:43

It's not, we don't get all the

26:44

psychological baggage and damage

26:47

from screwing something up

26:49

It's no big deal

26:50

We just revert to where we were

26:52

Now I wonder about this a little bit

26:54

I talked earlier about all those

26:56

benefits from failure

26:57

We'll slow down

26:58

We'll reduce our overconfidence

27:00

Are we still going to do that

27:02

if we don't have all the baggage

27:03

that comes with it?

27:05

And so far, I think it works

27:07

Because, even though

27:08

I don't feel devastated

27:10

I still don't like being wrong

27:13

And I get a little annoyed that I'm wrong

27:15

We're finding that a little bit of anger

27:17

is actually a beneficial thing

27:19

It's energizing

27:20

You want to change it

27:21

I got to fix that

27:23

it's driving me crazy

27:24

And if you can just get a

27:26

little caremad about stuff

27:27

you can get a lot of things done

27:29

90% of my twitter feed appears

27:30

to be powered by this magical energy

27:35

So this kind of gives me a framework

27:37

for exploring things

27:39

First I set a goal

27:41

And then, I write it down

27:43

So I can't reframe

27:44

And the next thing I do...

27:46

is I think about failing at it

27:49

I don't like that

27:50

Kind of get that taste in your mouth

27:52

Ok, so what I do at this point is

27:54

I don't give up!

27:55

This seemed achievable 10 seconds ago

27:57

I can do this

27:59

So I start thinking about the failure

28:01

a little bit

28:02

Are there things I can do to make

28:04

this safer?

28:05

Can I make it smaller?

28:06

Can I come up with a way to

28:08

reset out of it?

28:09

Maybe if I take some friends with me

28:11

it won't be so bad

28:12

And it may be that this failure

28:14

is just too big

28:15

It will hurt my friends

28:16

I'll end up crying in the corner

28:18

Ok, this isn't a suicide pact

28:20

This is fine. You can avoid failure

28:23

But, if it doesn't seem like that

28:25

kind of failure

28:25

I'll resolve to try

28:28

But first, I write down

28:30

what I think might happen here

28:32

And I tend to record failure criteria

28:34

only because I have problems with

28:35

sunk cost fallacy type problems

28:38

Where I'll get 80% into the quicksand

28:40

well I better keep going

28:41

That's what I should do now.

28:44

So I try to write down what I think might happen

28:46

and how I should get out of it

28:48

And then I'll try

28:50

So I'll try and maybe it will go badly

28:53

I'll hit my failure condition

28:54

and I'll pull myself out of it

28:56

Ok, I've got to go around this one

28:57

And that's fine

28:58

I learned something about

29:00

in this context, this isn't a good idea

29:01

I need to do something else

29:03

Maybe I'll try a different idea next time

29:05

But usually, I'm done. I tried

29:07

Let's keep going

29:09

The second thing that might happen

29:11

is that against all odds

29:12

I will succeed!

29:14

That feels amazing

29:15

When you know you're going to

29:17

screw something up and you don't!

29:19

Man I am awesome!

29:20

I'm so good at this!

29:21

This is great!

29:24

The third thing that might happen

29:25

is that I will try and I will fail

29:28

and then I'll realize this failure

29:30

isn't really that big a deal

29:32

And maybe I'll come up with a way to fix it

29:34

In a lot of ways

29:35

this is what I'm really striving for

29:37

Because I want to build up this

29:38

toolset of things to do

29:39

when things go wrong

29:41

And so, it helps me feel

29:42

more comfortable in those situations

29:44

For instance

29:45

if a deploy to production goes badly

29:47

I don't freak out

29:49

Because, I've broken very many

29:50

deploys to production

29:52

And I have a whole set of ideas

29:53

and tools to get things back to good

29:56

without freaking out

29:59

The fourth thing that might happen

30:01

is I will try

30:03

and I'll end up someplace

30:05

completely unexpected

30:07

This is really interesting to me

30:09

I still kinda want to go this way

30:10

I want to get to my goal

30:12

But I didn't even know this existed

30:14

as an option

30:15

So now I'm over here

30:17

Maybe I should keep going this way

30:19

Let me see what's this way

30:20

I have different things

30:21

But again, if I had just saw the failure

30:23

and tried to stay away from it

30:25

I would have never really explored

30:27

this as an opportunity

30:32

It's easy to talk about failing

30:35

when it's just you

30:36

Nobody notices, it's no big deal

30:37

Failure in modern society

30:39

has nothing to do with tigers

30:41

It has everything to do with people

30:43

looking at you and saying you screwed up

30:45

you're the mistake

30:47

Why did you do that?

30:48

That's the sort of thing we want to avoid

30:50

Because, when that happens we naturally

30:52

want to shell up and hide

30:53

And not try anything very risky at all

30:57

I gave this talk at a meetup

30:59

and a man came up to me afterwards

31:00

And he said he had one of those situations

31:03

where he screwed up something in production

31:05

and everybody freaked out

31:07

But he and his boss got together

31:09

and kind of figured it out

31:10

and got it back to good

31:12

And they realized that there

31:13

were some things they could have

31:15

done differently

31:15

There were some ways they could

31:17

improve this and make it safer

31:18

And so he left for the day feeling OK

31:21

Not the best day in the world

31:23

Definitely not the way he wanted it to go

31:25

But he's going to keep his job

31:27

and they had a chance to improve things

31:29

Things were looking good

31:30

So objectively

31:31

this is the exact kind of failure I want

31:33

It was fairly safe

31:34

It was a learning opportunity

31:36

No real psychological damage done

31:39

He comes in the next day

31:40

and he's kind of excited

31:42

He's going to make this better

31:43

This is a good opportunity

31:45

And a coworker comes up to the cubicle

31:48

Saddles up and was like

31:49

That was a boneheaded thing you did yesterday

31:53

He said: I just froze up

31:55

I was afraid to do anything for two weeks

31:57

because I might screw it up

31:59

All of that imposter syndrome

32:01

comes roaring back

32:02

I can't do this thing

32:03

I can't believe they pay me to do this

32:07

So how do we get past that?

32:09

How do we get into a place where it's OK

32:11

to screw up and we can learn from these things?

32:15

This is hard

32:16

We all naturally build up these personas

32:19

this certainty and confidence that we have

32:22

And it's very useful for us

32:23

in an uncertain world to appear certain

32:26

Doctors understand this

32:27

If you go to a doctor and he's like

32:29

I don't really know what's wrong with you

32:31

We're going to run some tests

32:33

maybe we'll try this thing...

32:35

You don't want that doctor!

32:37

You already feel vulnerable

32:38

You don't want to talk to this guy

32:40

You want this guy, who's like:

32:41

It's going to be fine

32:42

We're going to run these tests

32:43

Don't worry we'll figure this out

32:45

We're in this together

32:48

I feel this way sometimes as a consultant

32:50

A client will bring a weird strange problem

32:52

I've never seen before, and I'm like

32:54

I don't really know how to handle that

32:56

But that's not the way I say it

32:58

I'll be like, well ok -

32:59

we'll do this to reduce risk

33:01

we'll try these things to get feedback

33:02

and don't worry we'll figure this out

33:04

Because they're feeling vulnerable

33:06

And I feel like I should help reassure them

33:08

Everything is going to be ok

33:10

I realize that I'm speaking from a

33:12

place of great privilege

33:14

If I screw up, it might reflect badly on me

33:18

Maybe it would take my company down

33:21

That's probably worst case scenario.

33:23

But certainly nobody would think that

33:25

some random white guy in Iowa

33:26

is a screw-up because I messed up

33:29

But if you feel like you're representing

33:31

an entire religion

33:32

an entire race

33:34

an entire gender

33:35

Then you're going to have a much

33:37

harder time being OK with failure

33:39

You're not going to want to let your

33:41

shields down

33:42

You know how hard it is to build

33:43

this respect and reputation

33:45

and you don't want to lose any of it

33:48

I feel like my existing framework of

33:51

experimentation isn't going to work here

33:54

I'm ok with doing things to myself and

33:56

seeing how it feels, and that's fine

33:58

But I don't want to experiment with

33:59

other people's feelings

34:01

So instead, I'm considering some questions

34:03

and I'd like to invite you to consider

34:05

them as well

34:08

So the first question is:

34:09

What do you celebrate?

34:12

It seems pretty obvious to me

34:14

that the thing we're really good at

34:17

celebrating are those random successes

34:19

that we're not supposed to celebrate

34:21

Justin's a billionaire!

34:22

You won the lottery!

34:24

We're going to write newspaper articles

34:26

We're all going to talk about that

34:27

That's the thing we want to talk about

34:29

Imagine you had a coworker

34:31

Just had a successful launch

34:32

And you found out that she had been

34:35

working 80-90 hours a week

34:36

for the last month

34:37

to pull this off

34:38

And so naturally, you want to celebrate

34:40

You want to take her out

34:41

You want to celebrate all that

34:43

hard work and effort

34:44

Wow! Way to go!

34:46

But imagine this same exact situation

34:48

the launch had been botched

34:50

Our natural tendency is to go to correction

34:53

Well, if you hadn't been so sleep-deprived

34:55

Maybe you should have asked for help

34:58

We say that we are celebrating the

34:59

effort and the action

35:01

But we have a hard time taking it

35:04

out of the outcome

35:06

There's a great story from IBM about

35:08

Thomas Watson Sr

35:10

IBM was in a bad place

35:12

They needed to make some sales

35:14

Watson was going to lose his job

35:16

And one of his salesmen

35:17

blew a million dollar deal

35:19

So that salesman comes into the office

35:21

zhe has resignation in hand

35:22

And Thomas Watson's like

35:24

Well, what happened?

35:26

And he kind of explains

35:27

where the deal went wrong

35:28

what he should have maybe tried differently

35:30

And he turns to leave

35:32

And Watson stops him at the door and says

35:34

I can't accept this.

35:36

I just spent

35:37

a million dollars on your education

35:41

It's hard for me to imagine being

35:44

Thomas Watson in that scenario

35:46

I needed that deal, my job is on the line

35:49

But I can imagine what it felt like

35:51

to be that salesman

35:53

How empowered I would feel

35:54

How supported I would feel

35:56

How I would want to redeem myself

35:58

How determined I would be

36:00

to prove Watson correct

36:02

Can we find real, genuine, things to celebrate

36:06

when things go wrong?

36:07

Can we celebrate the effort

36:09

independent of the outcome?

36:13

What sorts of failure do you acknowledge?

36:16

it's very easy for us to put stuff

36:18

under the rug

36:20

We don't like to talk about failure

36:22

I've been part of a number of initiatives

36:24

Where we have the big kickoff

36:26

We get everybody together

36:27

This is going to be great

36:28

We're going to change everything

36:29

We're going to do it this way

36:30

And then two months later

36:31

Nobody is talking about it

36:34

What happened?

36:36

Nobody knows

36:38

So my brain naturally starts

36:40

filling in details and they're not good

36:41

The stories I'm telling aren't really great

36:44

And I feel like we're missing

36:45

such an opportunity here

36:47

To talk about how our company works

36:49

and what we value and what we believe in

36:51

And maybe that we weren't ready

36:53

for this particular thing

36:54

OK

36:56

And maybe that 90% of the employees

36:58

really hated the idea

36:59

So we're not going to try

37:00

to shove it down anyone's throat

37:03

Maybe we discovered this is going to

37:04

be a lot of work and we didn't want

37:07

everybody working 60-70 hours to

37:08

pull this off

37:10

All these things tell useful stories

37:11

And I think, by having to sort of

37:14

acknowledge them in the same place

37:15

that you did the big announcement

37:17

You have an opportunity to use it

37:19

as a positive experience

37:21

And if nothing else, it helps inform

37:23

our next initiative idea

37:25

Maybe we should check on these things first

37:29

Likewise, what sort of things do you

37:31

discuss in the open?

37:32

Which ones do you try to hide?

37:34

I was really impressed with the way that

37:36

GitLab handled their recent data outage

37:38

They were open and transparent

37:40

about the triaging they were doing

37:41

the entire way through

37:43

And it felt pretty blameless

37:44

So much so, the person who did the

37:46

typing mistake raised their hand

37:48

and said, yeah it was me

37:49

That's pretty remarkable

37:51

Most of us don't want to be singled out

37:53

in that way

37:54

And I can contrast this to S3

37:56

which had an outage not long after

37:58

and nobody knew what was going on

38:00

Is everything ok?

38:01

Yeah, everything fine

38:02

Are you sure?

38:03

I'm having a lot of trouble here

38:04

It took a number of hours

38:05

for things to get resolved

38:07

And we would all like for our dependencies

38:09

And the people we work with to be 100%

38:11

awesome all the time

38:12

But that's not reality

38:14

And I'd really rather work with people

38:16

who are open and transparent

38:17

So I know things are being done

38:19

And maybe I can help

38:23

What kind of exceptions do you make?

38:26

And which ones do you never allow?

38:28

If the product owner can be 10 minutes late

38:31

to every show and tell

38:32

But I get slammed if I'm 30 seconds late

38:34

to a standup

38:35

I'm going to notice that

38:36

I'm going to key on that

38:38

And I'm not saying that everything has

38:40

to be equal and fair

38:41

Equitable is a nice goal

38:44

But, implicit arbitrary rules destroy trust

38:48

and they destroy any ability to

38:49

make a place safe

38:52

So if you're going to have exceptions

38:53

be explicit about them

38:54

Be open about the sort of things that you allow

38:59

What is your own personal tolerance for failure?

39:03

When I think back on early management

39:06

and mentorship opportunities I had,

39:07

I screwed this up pretty bad

39:10

I was trying to help

39:12

I would see someone kind of going down

39:14

the wrong path

39:15

Don't do that, you want to try this instead

39:17

No, that seems risky, let's try this instead

39:21

I'm trying to help

39:21

I don't want you to feel the pain of failure

39:24

So I will help you - stay away from that!

39:26

I robbed them of a learning opportunity

39:31

Now to be clear, I think you can learn

39:33

from other people's failures

39:35

But it requires a lot of context

39:37

It requires a lot of "Why?"

39:38

You almost have to - practically

39:40

step into the problem to understand the

39:42

lesson to learn there

39:45

Secondly, I inadvertently made them

39:48

more dependent on me

39:50

This was not my intention

39:51

But if every time you start something

39:52

someones like

39:53

No, no, no, don't do it that way

39:55

You're naturally going to

39:56

want to clear it with them first

39:57

So you don't waste your effort

39:59

Not my intention

40:01

I wanted them to be independent thinkers

40:03

I had no interest in

40:04

command and controlling everything

40:06

But that was the natural response

40:07

Because of the way I was acting

40:10

But the third thing was perhaps worst of all

40:13

I potentially ruined an opportunity

40:16

for innovation

40:17

The amazing thing about novices

40:19

is they don't know what's impossible

40:23

I've seen high school students do things

40:25

that I didn't think could be done

40:27

All my experience, all my baggage

40:28

All my context

40:29

Says you can't get here from there

40:31

And they did it anyway

40:33

So not only did I ruin

40:34

opportunities for their learning

40:36

I ruined opportunities for my learning

40:38

And I potentially cheated the world

40:40

out of an innovative way to look at a problem

40:42

All because I was focused on keeping people safe

40:45

and on the nice straight path

40:46

Where it's ok to go here

40:51

This is the last question

40:53

What do you say about people

40:55

outside your team

40:57

during challenging situations?

40:59

I have said some very not nice things

41:02

about external partners in bad places

41:05

When everything is on fire

41:08

And I'll get on the phone with them

41:10

I'll be very supportive

41:11

I'm kind - I want to help

41:13

I just want to get past this thing

41:15

But when I go back with my team

41:16

I might tell a joke

41:18

I might vent a little bit

41:21

Right, it's safe

41:22

It's cool, we're all together

41:24

Think about the kind of message

41:25

I'm sending though

41:27

Hey it's ok, that you screwed up

41:28

We're going to learn from this

41:30

It's fine

41:31

What do you imagine I'm saying

41:32

as soon as I turn my back?

41:34

We've already established

41:35

that I have this kind of behavior

41:37

So, maybe it's ok that S3 goes down

41:41

Everyone is faliable, it's fine

41:43

We'll get through this

41:47

The point of all this

41:49

is that change and disorder and chaos

41:51

are the laws of the universe

41:53

We can't get past them

41:54

So I feel like we ought to join them

41:56

and get better at it

42:00

Change has lasted even longer than taxes

42:03

It's been around much longer

42:04

And it will win. Over all other things.

42:06

And there are opportunities here

42:08

We can do really amazing things

42:10

It's is viscerally satisfying

42:13

to see your vision come out of this chaos

42:16

into reality

42:17

It feels so great to do these things

42:21

If I look at something like wikipedia

42:23

Conceptually, think of all the noise

42:25

that goes into wikipedia

42:26

We don't who you are, what you know

42:29

what your experience is

42:30

Everyone is welcome to add their own

42:31

particular two cents

42:33

And somehow a tremendous amount

42:35

of signal comes out of this thing

42:37

Wikipedia has changed the world

42:39

it's hard to imagine getting by

42:41

without wikipedia nowadays

42:45

Likewise I look at a place like Venice

42:49

I'm just amazed

42:50

Venice has been around for millenia

42:52

It's built on the world's worst

42:54

possible imaginable building material

42:57

Water is chaotic and fickle and ever-changing

43:00

and they have made it work for centuries

43:04

And then I think about some of our

43:06

master planned communities

43:07

that we have out there

43:08

that got just a little bit of disorder

43:10

and completely fell apart

43:15

It is our nature to just want to

43:17

put things into binary buckets

43:19

when we think about them

43:20

Success - Failure

43:21

Fragile - Robust

43:24

Republican - Democrat

43:25

Man - Woman

43:26

White - Black

43:27

Waterfall - Agile

43:30

And we're missing all this interesting

43:31

stuff that's in the middle

43:33

because we hang out on the extremes

43:35

So I'd like to suggest that if you've been

43:37

a hard-line successist your entire life

43:40

maybe try on a little failure

43:43

and see what it's like

43:44

Because, let's face it

43:45

if you're not trying new things

43:47

you're not growing, you're not learning

43:49

and you're probably not doing anything interesting

43:52

So I'm really interested in hearing

43:54

more about this

43:55

If you have any thoughts or feedback

43:57

please, come talk to me or send me an email

43:59

I'd love to hear from you

44:01

Thank you so much for your time and attention

44:03

I really appreciate it