Failures are interesting. They tend to teach lasting personal lessons, but we are loath to share them. These lessons have profound impacts on our habits and viewpoints, but they're difficult to recount. We need to get comfortable talking about failure. Unfortunately, our brains are wired to screw this up. When confronted with a failure, we are conditioned to ignore it, make ourselves blameless, or create elaborate strategies to avoid the possibility of failure altogether. I’d like to present another alternative: embrace failure. It will happen. That’s ok.
How can we challenge our natural human bias? I've spent the last couple years digging into "pop" psychology and thinking about how books like "Thinking Fast and Slow", "Antifragile", "Drive", "Blink", and "Outliers" apply in a software development environment. I'm most excited about the idea that if we can make failure "safe" then we can use it to learn things that we would otherwise be tempted to avoid and ignore because it might go badly. A focus on failure, both deliberate and accidental, has been beneficial in moving my projects beyond the no defects, 100% test coverage type focus and towards finding innovative ways to detect and quickly reverse failure.
A transcript of the presentation follows:
[00:00] The title of this talk is [00:01] Looking for Failure [00:03] My name is Steve Jackson [00:04] I'm a Double Agent at Test Double [00:06] Test Double's mission is [00:07] to make software better [00:09] so that naturally means we have to get [00:10] pretty comfortable with failure [00:12] If you have any trouble with failure [00:14] and want to get in touch with us [00:16] you can email us at email@example.com [00:18] We might be able to help [00:19] This is my personal twitter, @stevejxsn [00:21] and it's spelled weird [00:22] because it's really common, right? [00:24] And then my email, firstname.lastname@example.org [00:26] I'm interested in this topic of failure [00:28] and seeing if it's resonating with people. [00:32] I'd like to get a conversation started [00:33] around with this [00:34] If any of this makes sense to you [00:35] and you want to share your story [00:37] I'd love to hear it [00:39] I love this quote from Thomas Edison [00:41] It really embodies that [00:44] ideal of innovation to me [00:46] That if you just keep pushing [00:47] if you just keep trying [00:49] you'll get past all your failures [00:50] and eventually be a success [00:52] That's not what this talk is about [00:55] This is not a talk about perseverance [00:58] This is a talk about identifying failure early [01:00] looking it straight in the face [01:02] and stepping into it anyway [01:06] What sort of things happen when you look to fail? [01:10] So I'm a big believer in the Law of Two Feet. [01:11] I want everyone to get as much value [01:13] as they can out of this conference. [01:14] So, what this talk is about [01:16] is we're going to talk about failure [01:18] as a learning aid and how we can use it effectively [01:21] or how it could be used effectively [01:24] I'm going to talk about some of the reasons [01:25] why it's difficult to use failure effectively [01:28] and all the different biases we have in place [01:30] that make that really hard [01:32] I'm going to talk about some hypotheses [01:34] that I'm actively testing to get myself [01:36] better at dealing with failure [01:38] And then, I wanna kind of close talking [01:40] about how we can make it safe [01:41] for other people to fail and [01:42] some things maybe we should think about [01:46] So, we're kinda raised from a young age [01:48] to believe that if you're getting up early [01:50] you're doing all your homework [01:51] you're helping the old ladies across the street [01:54] eventually you'll be successful [01:56] But, if you're watching too much TV [01:58] and doing drugs [01:59] and hanging out with the wrong crowd [02:01] You're doomed. No chance [02:03] We see success as this linear thing [02:05] We assume that anyone that was successful [02:07] obviously was successful their entire life [02:09] and it's just kind of this [02:10] wonderful golden path [02:12] But it completely [02:14] contradicts our own experiences [02:15] We know we have ups and downs [02:17] and good days and bad days [02:18] and that's how the world works [02:22] Objectively, success and failure [02:24] are just the effects [02:25] of an action or a non-action that we take [02:29] If the effect is positive, we call that success [02:32] If the effect is negative, we call that failure [02:35] It's just feedback [02:38] I got this from Jurgen Appelo [02:40] This is his Celebration Grid [02:42] And what I really like about it [02:44] is it takes these two binary outcomes [02:45] of success and failure and points out [02:47] that your mindset and your intent do matter [02:50] So up at the top here, we have [02:51] when you're successful by mistake [02:54] You're "Wow, I managed to get lucky [02:56] and pull that off anyway" [02:57] And this is pointing out that there's really [02:59] not much to celebrate there [03:01] It wasn't through any particular amazing skill [03:03] that you did that [03:04] And then underneath we have your [03:05] garden-variety failures that we regret [03:07] and obsess about for the rest of our lives [03:11] In the middle, if we're in an [03:13] experimental mindset here [03:14] then either success or failure are fine [03:16] We're in a mindset where we can learn [03:18] So, either way we can find something [03:20] to celebrate in that [03:22] We didn't do exactly what we thought [03:23] but we learned from it and it was good [03:26] In this last column we have this idea of practices [03:29] If you succeed by doing the right things [03:31] then you should celebrate that [03:32] That's great! [03:33] But if you don't succeed while doing the [03:35] right things, well that's just the opposite [03:37] of that other good luck [03:38] That's just bad luck [03:40] And maybe we shouldn't be [03:41] so down about that [03:42] Now where I would differ a little bit [03:43] from the Celebration Grid is [03:44] I think there is value in these other [03:46] two types of failure [03:47] So I'm going to talk about that [03:49] a little bit more [03:50] But if you're interested [03:51] in this sort of thing [03:52] this comes from Workout [03:53] This is a great book [03:55] lots of nice visualizations about working [03:56] with teams and improving [03:59] the way that you work [04:03] Failures are interesting to me [04:05] They have a certain amount of stickiness [04:09] that success doesn't seem to have [04:11] When I succeed at something [04:12] I immediately look to the next thing [04:14] I have to do [04:15] It's a very temporary feeling [04:16] But a failure, man [04:17] I can sit there and think about that for awhile [04:20] Think about that stupid thing [04:21] I did back in high school. Right? [04:23] I'll see someone that's successful [04:26] and you might try to copy their success [04:28] You want to emulate them [04:30] But we often miss something in the middle [04:32] Because again, our brains just assume [04:33] It's a linear path [04:34] So we miss the struggle and the growth [04:36] that they had to go through to get there [04:39] But, we're really, really good [04:41] at pointing out other people's mistakes [04:43] Everyone's like, yes, obviously Mr. Stormtrooper [04:46] You should check the wash for the red sock [04:48] We all know that [04:49] We've all learned that lesson before [04:52] Successes don't teach us a whole lot [04:54] except that success is possible [04:56] But what happens when you fail? [04:57] You reduce your overconfidence [05:00] You start rethinking your strategy [05:02] You question your assumptions [05:04] You want to redeem yourself [05:05] You engage [05:07] You start paying attention [05:09] Video games have an interesting [05:11] relationship with failure [05:14] It's not until you fail that you [05:15] understand how to play the game [05:17] I have to go and touch that turtle [05:18] to find out he's not my friend [05:20] and I shouldn't do that in this game [05:22] But what's more interesting to me [05:24] is that video games have different [05:25] difficultly levels [05:26] Now if the point of playing the game [05:28] is to win [05:28] Why would you play [05:29] at anything but the easiest level? [05:32] And I can imagine, well ok [05:33] maybe it's not very much fun [05:34] so we need different levels to [05:35] challenge you a little bit more. [05:37] Maybe that makes sense [05:39] But what I find interesting is [05:40] a lot of players who play a lot of games [05:42] start on the hardest level [05:44] The hardest difficulty level [05:45] Why is that? [05:47] What I think it is, is that at easier stages [05:50] you can succeed by mistake more often [05:53] You just kinda get by and don't know why [05:55] But when you play on the harder levels [05:57] your mistakes are punished more harshly [05:59] And it's through these mistakes [06:01] it's through these failures [06:02] that the right way to play the game [06:04] is revealed to you. [06:06] And you learn it much sooner [06:07] than at the end of the game [06:10] Now you might say, well that's [06:12] kind of interesting [06:13] Maybe I want to look at treating my life [06:16] like a video game [06:17] Maybe I want to try to fail fast [06:18] and get that feedback [06:19] and use that effectively [06:21] That's sounds great [06:22] But unfortunately, we have this problem [06:24] Our psychology has evolved over the years [06:27] over the eons to really [06:29] really dislike failure [06:31] Because failure used to mean that [06:33] you were going to get eaten by a tiger [06:35] So we want to stay [06:36] very, very, very far away from that [06:38] We try to stay far away from failure [06:40] We don't want to deal with it. [06:42] So there are basically four ways [06:44] that we treat failure [06:46] The first one is that we ignore it [06:50] This is considered a positive thing [06:52] in basketball, we call this [06:53] the shooter's mentality [06:55] Every shot I take is going in [06:57] I don't ever think about [06:58] the ones that don't go in [06:59] We all want to be like Swaggy P here [07:01] Of course! Money. Every time. [07:05] And it's common [07:06] When someone asks you about something and [07:08] you felt like it was a success [07:09] and you recount it: [07:10] Like if you asked me about that [07:11] angular project, I'll be like - [07:13] Yeah, we did these cool filterings [07:14] and our users were delighted about the way [07:16] we could re-arrange these things [07:17] We wrote these cool custom directives. [07:20] And I'll just kind of forget to mention [07:22] the days I spent [07:23] debugging lifecycle problems [07:25] and camel casing attributes [07:26] and all these other things [07:28] Our brain just kind of lets those [07:29] slip from your mind [07:33] The second thing we'll do [07:34] is we'll reframe our failure [07:36] so that it's actually a success [07:38] We'll lower the bar [07:40] So as an example [07:42] say I decide to go running this morning [07:44] I get my shoes on, I get out the door [07:46] Get down the road [07:47] get about 10 minutes in [07:48] And I hurt. I'm tired. It is cold. [07:53] I'm going to turn around and go home [07:55] you know? [07:56] Then I get to the end of the day [07:57] I get to my calendar and I have this [07:59] successive days worked out. [08:00] I'm like on 70 [08:02] And I'm just going to check today off [08:04] I got outside, I exercised, I tried [08:07] And I do this like subconsciously [08:09] I don't really reflect [08:10] If I stopped and thought about it [08:12] I would be like, well my [08:13] intent was to run five miles today [08:15] I barely made it one [08:19] The third thing we'll do [08:20] if we fail, is deflect [08:22] We'll find somebody else to blame it on [08:23] That's usually convenient [08:25] Or, just as commonly [08:27] our ego will step in and protect us [08:29] And then we'll say [08:30] other people might be able to do that thing [08:32] but I can't do that [08:33] I'm not an art person, you can't [08:35] I can't design that [08:36] Of course it didn't go well [08:38] I'm not a people person [08:40] Why am I talking to customers? [08:42] That makes no sense [08:43] I'm not a morning person [08:44] Why am I up at 7 o'clock in the morning? [08:46] That's crazy [08:49] The fourth thing we'll do [08:50] is we'll try to avoid the failure [08:52] This gets kind of positive [08:54] This is were we actually start [08:56] getting constructive with our failure [08:58] So we'll come up with strategies [09:00] When my son was in kindergarten [09:02] he had to be at school at 7:30 every day [09:04] It was on my way to work [09:06] and it was about 20 minutes away [09:07] so it made sense that I'd get him ready [09:09] and we'd go [09:10] So, you know, I've got to get up [09:11] I got to get ready, got to get him ready [09:13] got to have breakfast, got to have lunches [09:14] got to go shovel the driveway [09:15] All that stuff to make sure [09:17] we can get to school on time [09:19] So naturally, I start thinking about ways [09:20] Well I don't want to be late so [09:21] I have to get up early [09:22] So I'll set an alarm here [09:24] And I'll put my phone across the room [09:26] And I'll have another alarm there [09:28] Maybe I'll get stuff ready [09:29] I've got all these plans to make sure [09:31] I don't hit that failure [09:34] I'd like to suggest that maybe [09:36] there is a 5th option [09:38] I will fail [09:40] At least some of the time [09:43] I don't like that [09:44] I don't want to say that [09:47] But, if I start to accept that a little bit [09:49] an interesting thing starts to happen [09:51] I start thinking about contingencies [09:53] Ok, if I fail to get up in the morning [09:56] What do I do? [09:57] Am I going to call my kid off of school? [09:59] Or, maybe I'll decide to drop everything [10:01] I get him to school, and then [10:03] I'll come back and get myself ready [10:05] Or maybe more interestingly [10:07] I'll decide to change the rules of the game [10:09] If I can't win [10:10] I don't want to play that game [10:12] So instead, maybe I'll find somebody [10:14] to drive my kid to school [10:15] Or maybe the problem is getting a [10:17] five year old ready in the morning [10:18] I'll hire a nanny to come help with that [10:20] Maybe I'll decide that we should move! [10:24] Or switch schools. I mean, 7:30? [10:26] That's not reasonable! [10:28] Maybe I'll decide to switch jobs [10:30] so I don't have to be work until noon. [10:31] And now it doesn't matter anymore [10:33] It's no big deal [10:34] All these options and opportunities [10:36] don't really occur to me until [10:38] I accept that I can fail [10:41] Superhuman effort won't just get me [10:43] through every time [10:46] It's just this changing of perspective [10:49] Switching the lens a little bit [10:51] opens up new opportunities [10:54] I learned this technique from Gary Klein [10:56] This is the premortem [10:57] How this works is [10:58] He suggests that if you've had a hard time [11:00] coming to a group decision [11:01] You finally come to something and you have [11:04] a plan in place, here's what he suggests: [11:07] Imagine it's a year in the future [11:09] We have executed the plan [11:10] exactly as we intended [11:12] And it's gone terribly [11:15] We've pushed United Airlines [11:16] out of the spotlight [11:18] We are the front page of the New York Times [11:21] What does that newspaper article look like? [11:26] That's when you start thinking [11:28] about these things [11:29] How can it actually go wrong? [11:31] Our default as humans [11:32] is to always be optimistic [11:34] If I'm involved, it's going to go well [11:37] And this changes that a little bit [11:39] It also gives you an opportunity [11:40] There are probably some [11:41] misgivings about this plan [11:42] Now they will emerge naturally [11:44] Ah, that was the thing I was worried about [11:46] But this can be hard for teams [11:48] People don't like to be negative [11:49] especially out of the box [11:51] So another useful tip is Six Thinking Hats [11:53] You will come at the problem from a [11:55] number of different perspectives [11:57] and kind of try out different things [11:59] But again, it's just really about changing [12:01] your perspective a little bit to let in [12:03] some more options and opportunities [12:07] So, I don't like failure [12:08] but I think there's some value [12:10] in getting better at it [12:12] So I'm working through some hypotheses [12:14] And I'm trying to experiment on these [12:16] and see how they work [12:17] and how they feel for me [12:19] The first one is around this [12:20] idea of Regret Avoidance [12:22] I learned about regret avoidance [12:24] from Daniel Kahneman's book [12:25] Thinking Fast and Slow [12:27] This is an excellent book [12:28] if you're interested [12:29] in finding out all the different ways that [12:31] your brain is trying to trick you [12:32] into thinking you're a rational person [12:35] Very excellent [12:36] What I realized is that regret avoidance [12:38] is probably the number one thing [12:39] that holds me back [12:41] And here's how I think [12:42] about regret avoidance [12:43] So imagine I have [12:45] 10,000 shares of Luigi stock [12:47] But, I've decided, you know [12:49] I want some Mario stock [12:50] And it happens to be [12:51] trading at the same price [12:52] Has been for awhile [12:53] But, you know, I want something different [12:55] I want to find some Mario [12:56] And I find out, hey Todd! [12:58] Todd has some Mario stock. [13:00] Hey Todd, would you like to trade with me? [13:02] And Todd goes, Nah, you know I've had this [13:04] I'm pretty happy with it [13:05] It's cool. It's fine [13:06] Oh Ok, Justin! [13:08] I hear you have some Mario stock [13:10] Would you be willing to trade with me? [13:12] He says, "Sure" [13:13] Great. Got what I wanted [13:15] Everything's great [13:17] Fast forward 2 weeks [13:18] Luigi stock has gone through the roof [13:21] It's google + Facebook, greatest IPO ever [13:23] Justin is a billionaire! [13:27] It's pretty obvious [13:29] How does Justin feel right now? [13:31] Two Thumbs Up [13:32] He's a pretty happy guy [13:33] How do Todd and I feel? [13:36] Well, objectively we'd say we [13:37] both should probably feel kinda sad [13:40] We had an opportunity to be a billionaire [13:42] And we're not [13:44] But, almost everyone would agree [13:46] that I should feel worse [13:49] Because I took an action [13:51] that put me into this situation [13:54] Todd just stood pat [13:57] And that's what regret avoidance is about [13:59] Regret avoidance is when my aunt calls [14:01] And I don't know if I want to answer it [14:03] And I love my aunt! [14:05] We have great conversations [14:06] she's a wonderful person [14:07] But every once in awhile [14:09] she wants me to fix her computer [14:11] And so...ummm... [14:13] maybe I'll let it go to voicemail [14:15] We'll see [14:17] So how do I get past this? [14:19] What I'm trying here is [14:22] getting ready for failure [14:24] Priming myself for this failure [14:26] I've found that if I think something's [14:28] going to really suck [14:29] and then I do it anyway [14:30] it's usually not so bad [14:32] I can get used to it. [14:34] Shortly after my son was born [14:36] my wife and I had one of those [14:37] rare opportunities to get out of the house [14:39] for a few hours [14:40] So we started driving around [14:42] We come up to the movie theater [14:44] GI Joe is playing [14:46] Ok, we'll go see GI Joe [14:49] I was a little trepidatious about this, maybe [14:53] Like I knew this was going to be a bad movie [14:56] And I really really loved GI Joe [14:58] That's the highlight of my childhood [15:00] was going to my friend's houses [15:02] and playing GI Joe [15:04] So we went and saw the movie [15:06] And I was right! [15:08] It was awful [15:10] But we had such a good time! [15:12] We cheezed it up, we laughed at [15:13] all the terrible things they were doing [15:15] And it was amazing [15:17] And If I think about this objectively [15:19] I should have a lot of regret around this [15:21] This is the one movie we saw all year [15:23] I wasted it on GI Joe [15:24] Not Iron Man or any other good movie [15:26] that came out that year [15:29] The other place I turn for this [15:31] is to science [15:33] So we've all seen [15:34] the scientific method before [15:36] You make some observations [15:37] form a hypothesis [15:39] to explain those observations [15:40] You experiment to test the hypothesis [15:42] Then from those results you [15:44] change your understanding [15:45] maybe do more experimentation [15:48] Lean Startup has really jumped [15:49] all over this idea of doing [15:51] hypotheses and experiments [15:52] We're talking about it all the time [15:54] So we'll have a hypothesis like: [15:56] If we add feedback to the application [15:58] it will help with user re-engagement [16:01] So we run an experiment [16:03] we'll add 5-star ratings [16:05] We expect that 50% users will re-engage [16:08] with the app in the next couple weeks [16:10] And I've helped a number of startups [16:12] work up and think through these experiments [16:14] What I've found, almost universally [16:16] is that we always craft the experiment [16:18] to prove the hypothesis correct [16:21] Which is the exact opposite of how [16:23] science runs it's initial experiments [16:26] You always try to prove the hypothesis wrong [16:29] Because you don't want to spend [16:31] the next several months, or years [16:33] testing and re-testing and refining [16:35] that hypothesis that has a flaw [16:37] You want to find those faster [16:39] You want to fail sooner [16:40] And the place I'm most comfortable [16:42] with this is in my development habits [16:44] I like to do TDD [16:45] I like to start with that failing test [16:48] And I think about that as my [16:49] control in the experiment [16:51] The universe works the way I think it does [16:54] And then [16:55] using my powers of software mastery [16:57] I change the universe [17:00] and make my test pass [17:02] And that feels really good [17:04] And I'm perfectly [17:05] comfortable with this failure [17:07] I love these failures [17:08] In fact, if these tests pass, I get scared [17:10] The universe doesn't work the way [17:12] I think it does [17:13] and it's a weird and strange place [17:15] I like these failures [17:16] I'm fine with them [17:17] Another place I think we could really [17:20] embrace failure is around meetings [17:23] Ok. so how many people in the room [17:24] have attended a meeting where the [17:27] key decision maker didn't show up? [17:30] They're very busy people [17:33] And in how many cases, did you go ahead [17:35] and have the meeting anyway? [17:38] This is common! [17:39] We make excuses for this: [17:40] We're going to help [17:42] We'll get through some of the discussion now [17:44] We might come to a consensus [17:45] We won't have to do all this [17:48] But it never works [17:49] We end up rehashing the same [17:50] exact conversation once we get [17:51] the person in the room [17:53] Or we rob them of the context they [17:55] need to make a good decision [17:56] She can't make a good decision [17:58] because she doesn't have all the [17:59] criteria in front of her [18:01] So I'd like to suggest [18:02] that if you're in charge of [18:03] setting up meeting agendas [18:04] maybe think about defining some [18:06] failure criteria [18:07] If you hit the failure criteria the [18:09] meeting is over [18:10] Now you can create success criteria too [18:11] Nobody cares as long as the [18:13] meeting ends sooner [18:14] This is all good [18:17] My next hypothesis is around this [18:19] problem of reframing [18:21] It happens so easily [18:23] It's subconscious [18:24] How do we keep this from happening? [18:26] So my thought here is [18:27] if I can get my goals out there [18:29] I might be able to do something about it [18:31] And so for this I steal from Adam Savage [18:34] He had a great job with Mythbusters [18:36] essentially blowing stuff up [18:38] for our education [18:40] And he says, the difference [18:41] between screwing around [18:42] and science is writing it down [18:44] So, if I write down my goal [18:46] it's harder for me to subconsciously [18:48] change what the goal is [18:50] Now I can still consciously do that [18:52] That's fine [18:53] I just don't want my brain [18:54] doing weird stuff behind my back [18:56] I want - [18:56] Let's get up front about this, brain [18:58] What we're trying to do here [19:01] My third hypothesis was influenced by [19:04] Nassim Taleb's book, Antifragile [19:07] The way I think about that is like this [19:10] So, for nine years [19:13] I was a defense contractor [19:15] and it was very important to us that [19:17] we had robust systems [19:18] They needed to handle all kinds of [19:20] terrible conditions and pressure [19:22] and everything else and work fine [19:23] It was very important to us [19:25] So we wrote in java [19:27] And what we would do is catch exceptions [19:29] and try to recover from them [19:30] try to handle them [19:31] So you'd be in situation where it's like [19:33] Oh, I lost connection to the database [19:36] Tis but a scratch! [19:38] Oh, my navigation system [19:40] doesn't have a GPS feed. [19:42] It's just a flesh wound. [19:44] Oh, had enough, eh? [19:46] We spent a lot of time jumping [19:47] through all kinds of hoops to keep [19:49] this thing running no matter what [19:52] We didn't want to let our users down [19:55] Antifragile has this idea that the [19:57] opposite of a fragile thing is not [19:59] actually a robust thing [20:01] It's an antifragile thing [20:02] So a fragile thing is something [20:04] like a wine glass [20:05] Doesn't take a lot of pressure [20:06] Doesn't take a lot of change [20:07] Just a little tiny chip [20:09] and it's worthless as an object now [20:12] A robust thing is something like a stone [20:14] You can throw a lot of things at it [20:16] It's still stays a stone [20:18] it's as useful as it ever was [20:21] It's resilient to disorder [20:23] But an antifragile thing actually [20:24] gets stronger when disorder is applied [20:27] And probably my favorite example [20:29] from the book [20:29] was the idea of building muscle [20:31] When you work out [20:32] you actually tear your muscle [20:34] Small micro-tears [20:36] You cause it to fail [20:37] And it's from that failure that it [20:39] grows back stronger than it was before [20:42] And what's more is [20:43] if you do not cause your muscles to fail [20:44] they will atrophy and die [20:47] This is a system that needs disorder [20:49] it needs failure, to move forward [20:52] And when I think about software systems [20:54] I think about that system, where you fix [20:57] this bug and this other one pops up [20:59] And you fix this bug [21:00] and now this one is over here [21:01] And you're holding this thing [21:03] and every time you touch it [21:04] it seems to crumble in your hands [21:05] And it's very fragile and terrible [21:07] So naturally, we want to keep us from [21:09] changing it [21:10] So we create these things with [21:12] change control boards and all this other [21:13] stuff and that will make it robust, right? [21:15] Because now we can't let any of the [21:16] randomness and chaos and bad things [21:18] into our system [21:19] Everything will be fine [21:21] But I feel like this way of thinking [21:23] is doomed [21:24] Software has to change [21:27] Even if you manage to write bug-free [21:30] perfect software the first time [21:32] And I'm sure most of you have done that [21:34] Even if you pull that off [21:36] the world will change [21:37] The expectations for your software [21:40] will change [21:41] You have to be able to change with it [21:44] Software maintenance does not mean [21:46] that your software works the exact same [21:48] as it did last month [21:50] If it did, it would be free [21:52] Software maintenance means that it's [21:54] at least as useful as it was last month [21:57] And that requires change [21:59] So it seems like we need to get better [22:01] at this sort of thing [22:03] But, it's easy for me [22:04] to stand up here and say [22:05] You know what guys [22:06] We should fall all over the place [22:08] We should add more chaos [22:09] It's going to be great [22:11] Trust me [22:11] Who's behind me? Let's go! [22:14] There are problems with this [22:15] There are a lot of situations where it's hard [22:18] to take on more negative things [22:20] We have debt [22:21] We feel very vulnerable [22:24] If I'm barely making ends meet [22:25] I don't want to talk to you about [22:27] your startup idea [22:28] That's terrifying [22:30] If my system is one of those fragile systems [22:32] and you want to introduce more failure to it? [22:34] No. We're not doing that [22:36] That doesn't sound good to me at all [22:38] If I'm new to a team [22:39] or If I've failed recently [22:41] I'm not going to want to add [22:42] more failure to my resume or my reputation [22:46] It's not a good place to be [22:48] So my thinking starts to shift [22:50] How do we create safer failures? [22:52] That don't feel as bad when we're [22:54] feeling vulnerable? [22:56] The obvious one here, for me [22:58] seems to be that if you're gonna fail [23:00] you would prefer to fail in the small [23:02] Maybe get good at that [23:04] and get to larger failures [23:07] The engineering organization that [23:09] I feel has really taken this to heart [23:11] is Netflix [23:12] They have these tools [23:14] called the Simian Army [23:15] Essentially what it does, there are [23:17] various tools that will take out the [23:19] things that your application [23:20] or service depends on [23:22] So they have things like [23:23] Chaos Monkey that will take away [23:25] your dependent services [23:26] And you're supposed to stay running [23:28] And by the way [23:29] they run all this stuff in production [23:30] They have Latency Monkey [23:32] which will add time to the request [23:34] and you still have to give a [23:35] good user experience [23:36] And they've built this into their products [23:38] You know this is going to happen [23:40] You expect this [23:40] So they've gotten good enough with this [23:42] that they have things like Chaos Kong now [23:44] Which will take down your entire AWS region [23:48] And Netflix is pretty reliable [23:49] It works in a lot of places with not very [23:51] good network infrastructure behind it [23:55] For me personally [23:56] This is about pushing out my comfort zone [23:59] a little bit [24:00] Like if I can screw up in front of my friends [24:02] Maybe I can make a mistake in front of [24:04] my coworkers [24:06] Maybe I can mess up something [24:07] for my client [24:09] Eventually, maybe I can [24:10] fall flat on my face [24:11] in front of an audience [24:14] And yes, as we add and invite [24:17] more disorder and bad things [24:18] into our lives bad things will happen [24:21] But there's also the flip side of this [24:23] There's good luck that comes with the bad [24:25] And so, as we open ourselves up to these [24:27] different opportunities and interactions [24:29] we get good things [24:31] I answer the phone [24:32] and my aunt has playoff tickets [24:34] Great! [24:35] I'm glad I took that risk [24:38] My next hypothesis is that if we can [24:42] cause a failure to happen [24:43] and then reverse it [24:44] like we expect this to happen [24:46] then maybe we can use those same techniques [24:48] when the random failures [24:49] that we don't expect happen [24:51] Really what I'm talking about here is [24:53] building undo into everything [24:54] It's great to be able to just get back [24:57] to a good state when you're in a bad place [24:59] Either through undo buttons or [25:00] git reset all the things [25:02] Sounds pretty good [25:03] And I feel like the place where [25:05] we've really been pushing this forward [25:06] is in the areas of continuous deployment [25:08] and continuous delivery [25:10] Where we're getting pretty good [25:11] at putting small little experiments [25:13] out there and trying them [25:14] And seeing what happens [25:15] And if they fail they're still kind of [25:17] small and easy to reason about [25:18] Rather than the huge mega-releases where [25:20] we never knew what was going on [25:22] And again, I'll pick on Netflix [25:24] I learned about their deployment process [25:26] a couple years ago [25:27] Essentially you push a commit [25:29] That causes a cluster to spin up [25:31] in production with the new code [25:33] Then the load balancer starts pushing [25:34] some amount of traffic to the new code [25:36] and if it goes well [25:37] that becomes the new production [25:39] That's great [25:39] But if it goes badly [25:40] it gets dropped out of the load balancer [25:42] So it feels pretty safe to fail [25:44] in this environment [25:45] Your opportunity for screwing something up [25:48] is pretty small and your infrastructure [25:50] will save you [25:52] The place where my brain really started to [25:54] get twisted on this is when I learned [25:55] about Erlang and OTP [25:57] The Open Telecom Platform [26:00] So the thing about Erlang is that it [26:01] can run lots of little tiny processes [26:04] Like millions of them on a tiny laptop [26:07] And they startup super fast [26:09] So Erlang has evolved a very different [26:11] error handling strategy than C-based languages [26:14] In C, we always try to deal with our errors [26:16] In Erlang, you just let it go [26:18] You let it fail [26:19] You let it die [26:20] And OTP immediately starts the same [26:22] exact thing again [26:24] So again [26:25] it's very safe to fail in this environment [26:27] And this twisted my brain [26:29] because I've spent so much time [26:31] thinking about recovering [26:32] from these bad things [26:33] and How do we handle this? [26:35] What do we do now? [26:36] Just let it fail [26:37] OK [26:39] We're really trying to get to the point [26:41] where failure is a non-event [26:43] It's not, we don't get all the [26:44] psychological baggage and damage [26:47] from screwing something up [26:49] It's no big deal [26:50] We just revert to where we were [26:52] Now I wonder about this a little bit [26:54] I talked earlier about all those [26:56] benefits from failure [26:57] We'll slow down [26:58] We'll reduce our overconfidence [27:00] Are we still going to do that [27:02] if we don't have all the baggage [27:03] that comes with it? [27:05] And so far, I think it works [27:07] Because, even though [27:08] I don't feel devastated [27:10] I still don't like being wrong [27:13] And I get a little annoyed that I'm wrong [27:15] We're finding that a little bit of anger [27:17] is actually a beneficial thing [27:19] It's energizing [27:20] You want to change it [27:21] I got to fix that [27:23] it's driving me crazy [27:24] And if you can just get a [27:26] little caremad about stuff [27:27] you can get a lot of things done [27:29] 90% of my twitter feed appears [27:30] to be powered by this magical energy [27:35] So this kind of gives me a framework [27:37] for exploring things [27:39] First I set a goal [27:41] And then, I write it down [27:43] So I can't reframe [27:44] And the next thing I do... [27:46] is I think about failing at it [27:49] I don't like that [27:50] Kind of get that taste in your mouth [27:52] Ok, so what I do at this point is [27:54] I don't give up! [27:55] This seemed achievable 10 seconds ago [27:57] I can do this [27:59] So I start thinking about the failure [28:01] a little bit [28:02] Are there things I can do to make [28:04] this safer? [28:05] Can I make it smaller? [28:06] Can I come up with a way to [28:08] reset out of it? [28:09] Maybe if I take some friends with me [28:11] it won't be so bad [28:12] And it may be that this failure [28:14] is just too big [28:15] It will hurt my friends [28:16] I'll end up crying in the corner [28:18] Ok, this isn't a suicide pact [28:20] This is fine. You can avoid failure [28:23] But, if it doesn't seem like that [28:25] kind of failure [28:25] I'll resolve to try [28:28] But first, I write down [28:30] what I think might happen here [28:32] And I tend to record failure criteria [28:34] only because I have problems with [28:35] sunk cost fallacy type problems [28:38] Where I'll get 80% into the quicksand [28:40] well I better keep going [28:41] That's what I should do now. [28:44] So I try to write down what I think might happen [28:46] and how I should get out of it [28:48] And then I'll try [28:50] So I'll try and maybe it will go badly [28:53] I'll hit my failure condition [28:54] and I'll pull myself out of it [28:56] Ok, I've got to go around this one [28:57] And that's fine [28:58] I learned something about [29:00] in this context, this isn't a good idea [29:01] I need to do something else [29:03] Maybe I'll try a different idea next time [29:05] But usually, I'm done. I tried [29:07] Let's keep going [29:09] The second thing that might happen [29:11] is that against all odds [29:12] I will succeed! [29:14] That feels amazing [29:15] When you know you're going to [29:17] screw something up and you don't! [29:19] Man I am awesome! [29:20] I'm so good at this! [29:21] This is great! [29:24] The third thing that might happen [29:25] is that I will try and I will fail [29:28] and then I'll realize this failure [29:30] isn't really that big a deal [29:32] And maybe I'll come up with a way to fix it [29:34] In a lot of ways [29:35] this is what I'm really striving for [29:37] Because I want to build up this [29:38] toolset of things to do [29:39] when things go wrong [29:41] And so, it helps me feel [29:42] more comfortable in those situations [29:44] For instance [29:45] if a deploy to production goes badly [29:47] I don't freak out [29:49] Because, I've broken very many [29:50] deploys to production [29:52] And I have a whole set of ideas [29:53] and tools to get things back to good [29:56] without freaking out [29:59] The fourth thing that might happen [30:01] is I will try [30:03] and I'll end up someplace [30:05] completely unexpected [30:07] This is really interesting to me [30:09] I still kinda want to go this way [30:10] I want to get to my goal [30:12] But I didn't even know this existed [30:14] as an option [30:15] So now I'm over here [30:17] Maybe I should keep going this way [30:19] Let me see what's this way [30:20] I have different things [30:21] But again, if I had just saw the failure [30:23] and tried to stay away from it [30:25] I would have never really explored [30:27] this as an opportunity [30:32] It's easy to talk about failing [30:35] when it's just you [30:36] Nobody notices, it's no big deal [30:37] Failure in modern society [30:39] has nothing to do with tigers [30:41] It has everything to do with people [30:43] looking at you and saying you screwed up [30:45] you're the mistake [30:47] Why did you do that? [30:48] That's the sort of thing we want to avoid [30:50] Because, when that happens we naturally [30:52] want to shell up and hide [30:53] And not try anything very risky at all [30:57] I gave this talk at a meetup [30:59] and a man came up to me afterwards [31:00] And he said he had one of those situations [31:03] where he screwed up something in production [31:05] and everybody freaked out [31:07] But he and his boss got together [31:09] and kind of figured it out [31:10] and got it back to good [31:12] And they realized that there [31:13] were some things they could have [31:15] done differently [31:15] There were some ways they could [31:17] improve this and make it safer [31:18] And so he left for the day feeling OK [31:21] Not the best day in the world [31:23] Definitely not the way he wanted it to go [31:25] But he's going to keep his job [31:27] and they had a chance to improve things [31:29] Things were looking good [31:30] So objectively [31:31] this is the exact kind of failure I want [31:33] It was fairly safe [31:34] It was a learning opportunity [31:36] No real psychological damage done [31:39] He comes in the next day [31:40] and he's kind of excited [31:42] He's going to make this better [31:43] This is a good opportunity [31:45] And a coworker comes up to the cubicle [31:48] Saddles up and was like [31:49] That was a boneheaded thing you did yesterday [31:53] He said: I just froze up [31:55] I was afraid to do anything for two weeks [31:57] because I might screw it up [31:59] All of that imposter syndrome [32:01] comes roaring back [32:02] I can't do this thing [32:03] I can't believe they pay me to do this [32:07] So how do we get past that? [32:09] How do we get into a place where it's OK [32:11] to screw up and we can learn from these things? [32:15] This is hard [32:16] We all naturally build up these personas [32:19] this certainty and confidence that we have [32:22] And it's very useful for us [32:23] in an uncertain world to appear certain [32:26] Doctors understand this [32:27] If you go to a doctor and he's like [32:29] I don't really know what's wrong with you [32:31] We're going to run some tests [32:33] maybe we'll try this thing... [32:35] You don't want that doctor! [32:37] You already feel vulnerable [32:38] You don't want to talk to this guy [32:40] You want this guy, who's like: [32:41] It's going to be fine [32:42] We're going to run these tests [32:43] Don't worry we'll figure this out [32:45] We're in this together [32:48] I feel this way sometimes as a consultant [32:50] A client will bring a weird strange problem [32:52] I've never seen before, and I'm like [32:54] I don't really know how to handle that [32:56] But that's not the way I say it [32:58] I'll be like, well ok - [32:59] we'll do this to reduce risk [33:01] we'll try these things to get feedback [33:02] and don't worry we'll figure this out [33:04] Because they're feeling vulnerable [33:06] And I feel like I should help reassure them [33:08] Everything is going to be ok [33:10] I realize that I'm speaking from a [33:12] place of great privilege [33:14] If I screw up, it might reflect badly on me [33:18] Maybe it would take my company down [33:21] That's probably worst case scenario. [33:23] But certainly nobody would think that [33:25] some random white guy in Iowa [33:26] is a screw-up because I messed up [33:29] But if you feel like you're representing [33:31] an entire religion [33:32] an entire race [33:34] an entire gender [33:35] Then you're going to have a much [33:37] harder time being OK with failure [33:39] You're not going to want to let your [33:41] shields down [33:42] You know how hard it is to build [33:43] this respect and reputation [33:45] and you don't want to lose any of it [33:48] I feel like my existing framework of [33:51] experimentation isn't going to work here [33:54] I'm ok with doing things to myself and [33:56] seeing how it feels, and that's fine [33:58] But I don't want to experiment with [33:59] other people's feelings [34:01] So instead, I'm considering some questions [34:03] and I'd like to invite you to consider [34:05] them as well [34:08] So the first question is: [34:09] What do you celebrate? [34:12] It seems pretty obvious to me [34:14] that the thing we're really good at [34:17] celebrating are those random successes [34:19] that we're not supposed to celebrate [34:21] Justin's a billionaire! [34:22] You won the lottery! [34:24] We're going to write newspaper articles [34:26] We're all going to talk about that [34:27] That's the thing we want to talk about [34:29] Imagine you had a coworker [34:31] Just had a successful launch [34:32] And you found out that she had been [34:35] working 80-90 hours a week [34:36] for the last month [34:37] to pull this off [34:38] And so naturally, you want to celebrate [34:40] You want to take her out [34:41] You want to celebrate all that [34:43] hard work and effort [34:44] Wow! Way to go! [34:46] But imagine this same exact situation [34:48] the launch had been botched [34:50] Our natural tendency is to go to correction [34:53] Well, if you hadn't been so sleep-deprived [34:55] Maybe you should have asked for help [34:58] We say that we are celebrating the [34:59] effort and the action [35:01] But we have a hard time taking it [35:04] out of the outcome [35:06] There's a great story from IBM about [35:08] Thomas Watson Sr [35:10] IBM was in a bad place [35:12] They needed to make some sales [35:14] Watson was going to lose his job [35:16] And one of his salesmen [35:17] blew a million dollar deal [35:19] So that salesman comes into the office [35:21] zhe has resignation in hand [35:22] And Thomas Watson's like [35:24] Well, what happened? [35:26] And he kind of explains [35:27] where the deal went wrong [35:28] what he should have maybe tried differently [35:30] And he turns to leave [35:32] And Watson stops him at the door and says [35:34] I can't accept this. [35:36] I just spent [35:37] a million dollars on your education [35:41] It's hard for me to imagine being [35:44] Thomas Watson in that scenario [35:46] I needed that deal, my job is on the line [35:49] But I can imagine what it felt like [35:51] to be that salesman [35:53] How empowered I would feel [35:54] How supported I would feel [35:56] How I would want to redeem myself [35:58] How determined I would be [36:00] to prove Watson correct [36:02] Can we find real, genuine, things to celebrate [36:06] when things go wrong? [36:07] Can we celebrate the effort [36:09] independent of the outcome? [36:13] What sorts of failure do you acknowledge? [36:16] it's very easy for us to put stuff [36:18] under the rug [36:20] We don't like to talk about failure [36:22] I've been part of a number of initiatives [36:24] Where we have the big kickoff [36:26] We get everybody together [36:27] This is going to be great [36:28] We're going to change everything [36:29] We're going to do it this way [36:30] And then two months later [36:31] Nobody is talking about it [36:34] What happened? [36:36] Nobody knows [36:38] So my brain naturally starts [36:40] filling in details and they're not good [36:41] The stories I'm telling aren't really great [36:44] And I feel like we're missing [36:45] such an opportunity here [36:47] To talk about how our company works [36:49] and what we value and what we believe in [36:51] And maybe that we weren't ready [36:53] for this particular thing [36:54] OK [36:56] And maybe that 90% of the employees [36:58] really hated the idea [36:59] So we're not going to try [37:00] to shove it down anyone's throat [37:03] Maybe we discovered this is going to [37:04] be a lot of work and we didn't want [37:07] everybody working 60-70 hours to [37:08] pull this off [37:10] All these things tell useful stories [37:11] And I think, by having to sort of [37:14] acknowledge them in the same place [37:15] that you did the big announcement [37:17] You have an opportunity to use it [37:19] as a positive experience [37:21] And if nothing else, it helps inform [37:23] our next initiative idea [37:25] Maybe we should check on these things first [37:29] Likewise, what sort of things do you [37:31] discuss in the open? [37:32] Which ones do you try to hide? [37:34] I was really impressed with the way that [37:36] GitLab handled their recent data outage [37:38] They were open and transparent [37:40] about the triaging they were doing [37:41] the entire way through [37:43] And it felt pretty blameless [37:44] So much so, the person who did the [37:46] typing mistake raised their hand [37:48] and said, yeah it was me [37:49] That's pretty remarkable [37:51] Most of us don't want to be singled out [37:53] in that way [37:54] And I can contrast this to S3 [37:56] which had an outage not long after [37:58] and nobody knew what was going on [38:00] Is everything ok? [38:01] Yeah, everything fine [38:02] Are you sure? [38:03] I'm having a lot of trouble here [38:04] It took a number of hours [38:05] for things to get resolved [38:07] And we would all like for our dependencies [38:09] And the people we work with to be 100% [38:11] awesome all the time [38:12] But that's not reality [38:14] And I'd really rather work with people [38:16] who are open and transparent [38:17] So I know things are being done [38:19] And maybe I can help [38:23] What kind of exceptions do you make? [38:26] And which ones do you never allow? [38:28] If the product owner can be 10 minutes late [38:31] to every show and tell [38:32] But I get slammed if I'm 30 seconds late [38:34] to a standup [38:35] I'm going to notice that [38:36] I'm going to key on that [38:38] And I'm not saying that everything has [38:40] to be equal and fair [38:41] Equitable is a nice goal [38:44] But, implicit arbitrary rules destroy trust [38:48] and they destroy any ability to [38:49] make a place safe [38:52] So if you're going to have exceptions [38:53] be explicit about them [38:54] Be open about the sort of things that you allow [38:59] What is your own personal tolerance for failure? [39:03] When I think back on early management [39:06] and mentorship opportunities I had, [39:07] I screwed this up pretty bad [39:10] I was trying to help [39:12] I would see someone kind of going down [39:14] the wrong path [39:15] Don't do that, you want to try this instead [39:17] No, that seems risky, let's try this instead [39:21] I'm trying to help [39:21] I don't want you to feel the pain of failure [39:24] So I will help you - stay away from that! [39:26] I robbed them of a learning opportunity [39:31] Now to be clear, I think you can learn [39:33] from other people's failures [39:35] But it requires a lot of context [39:37] It requires a lot of "Why?" [39:38] You almost have to - practically [39:40] step into the problem to understand the [39:42] lesson to learn there [39:45] Secondly, I inadvertently made them [39:48] more dependent on me [39:50] This was not my intention [39:51] But if every time you start something [39:52] someones like [39:53] No, no, no, don't do it that way [39:55] You're naturally going to [39:56] want to clear it with them first [39:57] So you don't waste your effort [39:59] Not my intention [40:01] I wanted them to be independent thinkers [40:03] I had no interest in [40:04] command and controlling everything [40:06] But that was the natural response [40:07] Because of the way I was acting [40:10] But the third thing was perhaps worst of all [40:13] I potentially ruined an opportunity [40:16] for innovation [40:17] The amazing thing about novices [40:19] is they don't know what's impossible [40:23] I've seen high school students do things [40:25] that I didn't think could be done [40:27] All my experience, all my baggage [40:28] All my context [40:29] Says you can't get here from there [40:31] And they did it anyway [40:33] So not only did I ruin [40:34] opportunities for their learning [40:36] I ruined opportunities for my learning [40:38] And I potentially cheated the world [40:40] out of an innovative way to look at a problem [40:42] All because I was focused on keeping people safe [40:45] and on the nice straight path [40:46] Where it's ok to go here [40:51] This is the last question [40:53] What do you say about people [40:55] outside your team [40:57] during challenging situations? [40:59] I have said some very not nice things [41:02] about external partners in bad places [41:05] When everything is on fire [41:08] And I'll get on the phone with them [41:10] I'll be very supportive [41:11] I'm kind - I want to help [41:13] I just want to get past this thing [41:15] But when I go back with my team [41:16] I might tell a joke [41:18] I might vent a little bit [41:21] Right, it's safe [41:22] It's cool, we're all together [41:24] Think about the kind of message [41:25] I'm sending though [41:27] Hey it's ok, that you screwed up [41:28] We're going to learn from this [41:30] It's fine [41:31] What do you imagine I'm saying [41:32] as soon as I turn my back? [41:34] We've already established [41:35] that I have this kind of behavior [41:37] So, maybe it's ok that S3 goes down [41:41] Everyone is faliable, it's fine [41:43] We'll get through this [41:47] The point of all this [41:49] is that change and disorder and chaos [41:51] are the laws of the universe [41:53] We can't get past them [41:54] So I feel like we ought to join them [41:56] and get better at it [42:00] Change has lasted even longer than taxes [42:03] It's been around much longer [42:04] And it will win. Over all other things. [42:06] And there are opportunities here [42:08] We can do really amazing things [42:10] It's is viscerally satisfying [42:13] to see your vision come out of this chaos [42:16] into reality [42:17] It feels so great to do these things [42:21] If I look at something like wikipedia [42:23] Conceptually, think of all the noise [42:25] that goes into wikipedia [42:26] We don't who you are, what you know [42:29] what your experience is [42:30] Everyone is welcome to add their own [42:31] particular two cents [42:33] And somehow a tremendous amount [42:35] of signal comes out of this thing [42:37] Wikipedia has changed the world [42:39] it's hard to imagine getting by [42:41] without wikipedia nowadays [42:45] Likewise I look at a place like Venice [42:49] I'm just amazed [42:50] Venice has been around for millenia [42:52] It's built on the world's worst [42:54] possible imaginable building material [42:57] Water is chaotic and fickle and ever-changing [43:00] and they have made it work for centuries [43:04] And then I think about some of our [43:06] master planned communities [43:07] that we have out there [43:08] that got just a little bit of disorder [43:10] and completely fell apart [43:15] It is our nature to just want to [43:17] put things into binary buckets [43:19] when we think about them [43:20] Success - Failure [43:21] Fragile - Robust [43:24] Republican - Democrat [43:25] Man - Woman [43:26] White - Black [43:27] Waterfall - Agile [43:30] And we're missing all this interesting [43:31] stuff that's in the middle [43:33] because we hang out on the extremes [43:35] So I'd like to suggest that if you've been [43:37] a hard-line successist your entire life [43:40] maybe try on a little failure [43:43] and see what it's like [43:44] Because, let's face it [43:45] if you're not trying new things [43:47] you're not growing, you're not learning [43:49] and you're probably not doing anything interesting [43:52] So I'm really interested in hearing [43:54] more about this [43:55] If you have any thoughts or feedback [43:57] please, come talk to me or send me an email [43:59] I'd love to hear from you [44:01] Thank you so much for your time and attention [44:03] I really appreciate it