OKRs โดย Google Ventures

Good morning. My name is Rick Klau. I’m a partner at Google Ventures. For those of you I haven’t met before, thank you for coming this morning. Thank you to those of you watching online. We’re here to talk about objectives and key results. This is one of the critical elements of how Google managed its early execution, and really grew the company, and remains a pretty critical element of its DNA today.

I came to Google a little over five years ago. I was part of a company, as some of you know, called Feed Burner. Feed Burner was acquired in June of ’07. The very first day that I joined Google was the first day of a quarter, which meant that Google was already about a week and a half, two weeks into that quarter’s OKR process. I didn’t know what “OKRs” meant, I was asked to draft my OKRs within my new team. I really didn’t have any idea what was expected of me.

Certainly, at startups prior to Google, I’d never had a formal process of setting my objectives or measuring goals. But in the last five years, I’ve set OKRs as a bus. dev. [business development] lead, as a project manager on a couple of different products, and then more recently, in my time as a partner at Google Ventures.

The process had a real clarifying effect for me personally and for the teams that I was a part of. It helped me understand what mattered, and just as importantly, helped me understand what I wasn’t going to be working on.

Today, we’re going to talk about: what are OKRs? How do we implement them at Google? How to adopt this process at your own company – I think one of the things startups often struggle with, and one of the things I spend a lot of time working with you all on, is fighting the urge to just say, “Well, that’s Google.” Right? “You guys – you’re different. We’re not Google.” Well, as you’ll see in this deck, Google wasn’t Google when this was first adopted at Google. Google was under a year old when John Doerr came to Google and pitched Larry, Sergey, Marissa, and others on how to adopt this process at a very young and ambitious company.

I’m going to apologize in advance because I’ve committed at least two cardinal sins of – in this case, Key Note, not PowerPoint – so much so it wouldn’t surprise me if someone from Cupertino drove over and wrestled control of this remote from me. Not only have I used the slides that are almost entirely text, which is bad enough on its own, but in addition, I’ve also embedded an all-text presentation inside of an all-text presentation, which is a new low for me as a presenter. But I think as you’ll see, there’s a reason for having done that.

Quick show of hands: how many of you read the book, In the Plex, by Steven Levy? It’s a book about Google’s origins and growth up until about two years ago. Okay, maybe 20 percent or so. This really is a great book. Steven was given, really, incredibly direct access to not only our founders and early employees, but a number of people across Google that gave Steven a very broad sense of who we are and how we worked. And, he talked about how in 1999, John Doerr, who is a partner as some of you may know at Kleiner Perkins, remains a board member for Google to this day. And he came to Google to talk about a process that he’d witnessed at Intel called OKRs, objectives and key results. And Steven, in his book, talks about (and I know that those of you in the live stream can’t see this, so I won’t read the whole pages here) how Doerr came into Google and OKRs were an immediate hit. Unlike what you might expect, where this might be seen as some kind of “dilbertization” of a tech company, instead, it was immediately apparent to Googlers that this was data. Right? That what this gave company access to was a way to quantify – instead of just qualify – what it was they were working on and how well they were doing towards those goals. And, periodically reviewing: did they meet expectations for what it was that they were trying to do?

In a highlighted element here, a couple of things that are critical to how we think about OKRs, and I think you’ll see themes resonate throughout this discussion: “It was essential that OKRs be measurable. An employee didn’t say, ‘I will make Gmail a success’ but, ‘I will launch Gmail in September and have a million users by November.’” Real critical there are two different points: in the key result, “launch by September,” and “a million users by November.” Those are both measurable. The key here is that not only is the objective clear, but the means by which you would grade whether or not you had met that objective, is really what’s key here. And Marissa goes on to say, “It’s not a key result unless it has a number.”

“The OKR embodied ambition and sanctioned the ability to take risks,” says Doerr. It goes on to talk a little bit more about what else makes OKRs key, and it’s important to note that everyone’s OKRs, from Larry and Sergey on down, are public within the company. Everyone. It’s part of your directory listing and what’s called “Moma,” Steven references it by name here in the book.  If you go into Moma and look up an employee, right next to their cell phone, their email address, their title, is a link to their OKRs. That means that not only can you see what they’re working on this quarter, but you can see what they were working on last quarter, the quarter before, and you can see their grades. And we’ll talk more about the process of how to grade them and why different approaches may be beneficial in your companies.

So, I got a hold of John Doerr’s actual deck. This is the embedded all-text presentation I was talking about a few minutes ago… but I actually got the original deck from 1999, and I think it helps. I’m not including the entire deck. I tried to take what I felt were the most critical elements that illustrated both how we think about them at Google, and how I think you should put them into context for understanding how they might apply in your company.

So, objective of this deck – imagine that I’m John Doerr for a moment, presenting this to Larry, Sergey, and Marissa. He started his deck with his OKR for that meeting, and the objective was, “I’m going to develop a workable model for planning, as measured by…” And it established three key results. Note how each of these three are inherently measureable. One: finishing the presentation on time. Now, I haven’t told any of you how long this workshop is going to take, so I think I have a pretty good chance of finishing on time, since only I know what “on time” is. [Key result #2] Completing a sample set of three months objectives and key results. So at the end of the meeting, they were going to have a set of their OKRs. Three: having management agree to institute a trial system for a three-month period. Pretty straightforward.

So John goes on to give examples of an OKR in the context of a football team. Now I’m a big 49er fan, so I’m going to use the 49ers as my illustration. I think the key here is one of the things – that when I’ve talked with many of you individually, you’ve struggled with – is how to connect the individual objectives with the team objectives, with the company objectives. And I think John’s example of the football team actually goes a long way to help clarify what that looks like.

So – general manager, I’ll say, team owner Jed York, his job is he’s going to win the Superbowl, and fill the stands to 88%. Now, the 49s would tell you, “We’re going to have sell-outs all season,” but these are measurable key results with an objective that, in this case, is defined as the GM making money for the owners. You can get a sense of what that looks like.

Now, filter that down. The head coach – notice that he doesn’t care about filling the stands. All he cares about is winning the game. Right? Winning the Superbowl. And the way he’s going to win the Superbowl is, he’s going to have a passing attack. Now, in case the 49ers – eh, bit of a crapshoot if Alex Smith is going to throw for 200 yards in a game, but maybe it’s more important that we get Frank Gore the ball, and so we might say that we’re going to cumulatively run for more than 200 yards. Or, number three in defensive stats, given the 49ers defense, you might say things like, “We’ll have a net positive takeaway differential where, you know, we’ll have more interceptions or fumble recoveries than the other team.” Right? These are measurable things, that at a high level – in this case – the head coach is focused on.

Now, these roll up into the prior comment, which was, you know, “The GM is going to win the Superbowl, fill the stands.” But obviously, there are some things that the GM is not worried about that the head coach is intensely focused on.

Pivot from the head coach to public relations: their responsibility is – they own filling the stands. Right? [Key result #1] “We’re going to hire two colorful players.” I’m not sure that if I were in charge of PR for the 49ers, that would be my focus… but if I think back to ’99, I think that’s when they had hired Deion Sanders, so maybe that exactly was the strategy back at that time. [Key results #2 and #3] “Get media coverage, highlight key players…” Obviously, this is a way that you generate enthusiasm amongst your fans. That may be a byproduct of what the coach focuses on, but it is absolutely 100% of what the PR function is focused on here.

Now, filter that further down. Defensive coordinator… all the defensive coordinator is focused on is, obviously, the defensive stats: holding the other team under 100 yards passing, intercepting at least twice in the game, recording at least three sacks, whatever you want to extend that analogy… whatever might be important given the players that that defensive coordinator has at his disposal.

Offense, on the other hand, of course doesn’t care about interceptions… except for not throwing them. But they’re going to set an objective of 75% completion rate. Now, I should note here that for those of you who follow football, a 75% completion rate would be pretty astounding, particularly in a Superbowl. Part of the goal of an objective – and we’ll talk about this again as we get into the mechanics of how you define OKRs – is that it be a little uncomfortable. You don’t want to always get (on a grading scale of zero to one), you don’t always want to get a “one.” In fact, the target ought to be a .6, .7. So, you should always have goals that are slightly out of reach, that feel a little uncomfortable, so that you are always striving to do more, to do better than perhaps you did the quarter before, or you think is reasonable in the context of the current quarter.

Think about – now, we’ve talked about defense and offense – there is a special teams unit. Right? They’re not focused on interceptions. They’re not focused on how well the quarterback does. They’re focused specifically on running back kicks, trying to block the punt, trying to block the field goal. They have very specific goals that are completely distinct from the other guys. Same with the news staff, same with the scouts. Right? They may not even be on the field. They may be off watching spring practice for a bunch of colleges.

The idea here – and what John Doerr was trying to reinforce to the Google team at the time – was that if you roll all of the things up from this foundation level, in their objectives and key results are reflected the ultimate company’s priorities. But not every single one of them is going to be considered a company priority. Right? Here the news staff has a key result of three Sunday features articles, or it might be an interview in ESPN, or it might be something along those lines. That’s not going to be a company OKR. Right? That’s not going to be the focus of what the 49ers are thinking about. The GM or the owner of the team isn’t going to be waking up every day saying, you know, “How are we doing on those Sunday features?” But you’d better believe the head of the news team, PR team, within the 49ers organization is absolutely thinking only about those things. Now, what is that further to the company objective? Well, it ties back up into filling the stands, right? So you can start to see how you can pretty clearly connect these dots, even if not all of the foundational objectives and key results ultimately show up in the company OKRs.

Now, I want to talk about why you use objectives and key results. Now, this was John trying to convince Google. Google now has lived this approach for 13 years. For those of you that have not yet made the commitment to embracing this method of measuring both your objectives looking forward and grading yourselves looking backwards: let me talk about some of the things that, less to John’s bullets here, where he was trying to make the case, let me tell you about as a Googler who’s now lived this for five of Google’s 13 years of using OKRs.

What does it do? It is absolutely amazing for imposing discipline on the organization. It helps me individually understand what I’m working on and why. It also helps me make conscious decisions about what I’m not working on, which is often just as important. Make sure that by making these OKRs public, everyone on my team, anyone in the company that cares, can see exactly what I’m working on, can see what my priorities are. Why is that helpful? Well, to give you one example, when I get dinged by a team at Google who wants… let’s say when I was the PM for YouTube’s home page. Somebody pings me and says, “Hey, we would love to promo this new product on the YouTube home page.” Well everyone would love to promo things on the YouTube home page. The person who’s going to make that pitch – it might help them to know what my priorities were for the quarter in which we were working, and they can immediately know, even before they talk to me, what my answer is likely to be. If the thing that they’re promoting is somehow connected to the things that I care about for that quarter, we’ve got a very easy case to make. They’re helping me deliver on my objectives.

If, on the other hand, it’s a distraction, they at least know going into the conversation they’re unlikely to get much traction in that quarter. Now, it might help to plant the seed with me, let’s say, in late October, so that as I start thinking about Q1’s OKRs coming up, that I have them on my radar.

By writing objectives and key results – specifically, the key results that are inherently measureable – you’re establishing how you’re going to measure what it is you’re working on. And, in the process of drafting those OKRs – and we’ll talk about how particularly individuals set their OKRs – you come up with, really, consensus, from the individual to the manager, manager to the team, to the company, depending on how big you are. That might be individual to CEO. Where everyone understands, at a pretty basic level, what is it we’re trying to do, and more importantly, how to measure whether we’re there or not, and how close we are. And then finally, focusing effort… I think we’ll talk more about that in a little bit.

The thing I really like about how this process ends up getting embraced is that there’s a really virtuous cycle that happens. Company objectives are generally set by the CEO, your board, the leadership team, who should have a pretty strong sense in any given quarter of what the things are that matter most to the company. Right? Take my FeedBurner example back in the day. Dick Costolo was our CEO. He had a somewhat comical way of establishing it, but the mission was: get all the feeds. Right? If we get all the feeds syndicating all these RSS feeds through FeedBurner’s platform, where we could both measure all the traffic that was happening and give publishers a way to monetize that traffic.

It was a very clear statement of what it was we were trying to do. That meant that when I would go off to New York and come back and say, “Well, such-and-such company wants us to do X,” the question that always followed was, “Does that get us closer to getting all the feeds?” And if it was a distraction – might be an interesting revenue opportunity – but it was not delivering on the main core focus of what it was we were trying to achieve; then the answer was no.

So, these corporate objectives are set, and generally, communicated down throughout the organization. Individual objectives, by contrast, are often what the individual wants to work on, and what the manager wants that person working on. Instead of this being an all top-down discussion, it often ends up being a bit of a cycle. We know generally what the company wants to be accomplishing; as an individual, I want to work on important things, so I’m going to find the things that are within my sphere that I can do that are most beneficial to the company. And I’ll come up with a list, and communicate that to my manager, or the CEO, or whomever it is I report to, and we can have a bit of a give-and-take.

Now, what’s interesting in this process is that occasionally, things that individuals want to be working on might end up opening up new discussions for what the company should be doing. Good example of that at Google: engineer named Paul Buchheit was really annoyed with the state of mail clients in Lenox, so he started… he thought, “Maybe I could make mail searchable, as sort of the core interface.” And he started what became Gmail. Well, over time Gmail became its own practice group at Google effectively, and became one of the core strategic goals at Google, which then had a way of creating its own objectives and key results for the people who were working on Gmail.

So, let’s talk about the communication. One-on-one meetings, where, as an individual contributor, I talk to my manager, my boss, the CEO, whomever it is, about what it is I want to be working on, and what I think is the best use of my time. The one-on-ones, at the very end of the quarter and the very beginning of the quarter, best to… John talks here about negotiating what those key results might be. That’s, I think, actually a pretty good term for it, because you are ultimately offering “what it is I want to work on,” and what I’m hearing back is “what we want you to be working on,” and somewhere in the middle, we find the best combination of the two.

At a company-wide meeting, the company OKRs are communicated, and the team owners identify what it is they’re going to be working on. There’s also a grading process that happens here that we’ll talk about in a little bit.

So typical cycle: I’m not going to dwell on this time line, only because I drafted one specifically for time frame we’re in, about the end of October, that I think will help clarify what the general rhythm of, you know, when you should be thinking about which steps in the process. So we’ll come back to that.

Some basic hygiene: you don’t want to have twelve objectives. It’s too much for any one person to be working on. Really, if you think about a quarter being effectively 12 weeks, if you had, in total, only a week to give to any given objective, it’d be unlikely that you could move the needle much on any of them. So you really want to limit the number of objectives you have, and you want to make sure that you’re really clear about what the key results of that objective are. Don’t – in the interest of limiting your objectives – don’t stuff every objective with 12 key results. You want to be very clear – it’s fine to be ambitious, but I’ll tell you personally, there was one quarter when I had seven objectives. I have never been more exhausted, and just mentally drained, from the amount of intellectual juggling it took to keep all of those different objectives sort of clear in my head, and keeping track on where I was with each, and who I was relying on, and what I needed to coordinate. It was too much and in the end, it ultimately took its toll. I mean, it was clear at the end of that quarter that I had been stretched a little too thin.

John here has a number – I don’t know that the number here matters as much as the overall intent, which is: 60% of the objectives ought to come from the bottom up. I think I would phrase this a little differently. I’d say more than half of the company’s objectives need to be coming from the individuals up through the organization. If there is too much top-down dictation, it’s going to be hard to inspire a lot of those people to be working on things, because they’re going to be told what to do instead of telling you what they think is the best use of their time and talent.

There’s a comment here about: “Everyone must mutually agree, no dictating.” I think that’s really just another way of saying what I said a minute ago. Interesting that they say one page is best, we do this all electronically now so there’s no issue of page length.

Here’s a really important point: this surprised a few of you when I talked ahead of giving this workshop. OKRs are not a performance evaluation tool. What that means is, in the annual performance review cycle that Google goes through, OKRs are not a standard element of that evaluation. Let me be clear: when I was reviewed each of the last five years, when I would sit down and write up my summary of what I did in the year prior, what I think that means for the company, whether I wanted to go up for promotion or not, at no point in time were my quarterly OKR bullets factored into that evaluation. Now, it’s always impossible to just include your OKRs as part of that packet. I found, individually, they were always tremendously useful, because it gave me, really, at a summary glance, what I’d been up to over the last year. I could look at the last four quarters of OKRs, I could look at the grades… Again, we’ll talk about the grading process in a little bit… And, I knew really within a few minutes, what were the biggest things I’ve worked on, what were the most dramatic results as a byproduct of that work, and I could then summarize my contribution to the company, my impact, in a way that previously, when I thought about, you know, at prior start-ups, prior to coming to Google, was much harder to recall back, “What did I do six, nine, 12 months ago?”

I mentioned this in passing, at the outset: your target should be a .6 to .7 on a zero to one scale. If you consistently get ones, you’re not crushing at your sand bag. Anything below 40% is definitely in the red. That means you missed your key result entirely, and that should be a forcing function for reevaluating whether or not that is an objective you want to continue to work on.

Last slide of the embedded slide; I promise we’ll move on after this. John here talks about [how] this has an alignment function in that it ensures everybody in the organization is working towards the same set of results. I agree with that; that was certainly my experience at Google. It was very easy to see amongst my team members where we were working on complimentary things, where we were working on dependent things, where there was something I was doing that I needed three other people to deliver something, and so, by keeping track of where they were in their progress helped me understand how likely I was to deliver on my OKRs.

It fairly organically ensures the organization is tuned in to what everybody is up to, and that virtuous cycle I talked about a few slides ago, there’s a wonderful – I used the word “organic” a minute ago, it’s the best that I can think of here – that just ensures that you’re working on things that support what the company is focused on, as do the people sitting next to you, and across from you, and down the hall. And, there is a confidence, and candidly, a comfort, that comes from knowing that you’re not treading water… You’re not walking through molasses, you know everyone that shares those commitments. Bizarrely, John even says here in the last bullet that OKRs are fun. That, I think is open to some interpretation. But I do think that there are elements of how to make this part of the team process that I’ll talk about in a little bit.

Okay. Keys to good OKRs. Just in terms of rhythm and process, OKRs are both quarterly and annually set. Annual OKRs are not written in stone. If you set them in December and in May you’ve assessed and you say, you know, the things that we thought were true in November, December, just aren’t proving to be true, don’t obviously feel beholden to see them out. Revise them. They are directional aids, not blinders.

As I’ve said, and I’ll continue to say, I think this is one of the critical elements of OKRs: they’re measurable. The fact that I know whether or not we shipped in September like I said we would, well, that’s measurable. We shipped in September, great. Give it a one, if in fact that was the entirety of the key result.

We’ll talk a little bit more, but important to note here that there are OKRs for each individual, there are OKRs at the team level that summarize what the collection of people are working on, and then there are OKRs up at the company level, which tend to reflect the most important three, four, no more than five things that the company cares about.

And, of course, OKRs are publicly available to the entire company. These cannot be siloed. It is critical for everyone in the organization to know what everyone else is working on. That’s true from the CEO on down.

Grading is also fundamental to this exercise.

Okay, before diving into my… I pulled some of my personal OKRs from my days when I was a PM on Blogger, I’ve got two more comments here, two more slides. I want to talk about differentiating between the objective and the key results, and then talk a little bit about differentiating personal versus team versus company.

Objectives should feel uncomfortable. Going into a quarter, you should be uncertain as to whether or not you can deliver on the thing you just said you will deliver on. If you are certain you will nail it, you’re not pushing hard enough. You’re not thinking broadly enough. You’re not thinking big. The key results are the things that will ensure that you deliver on the objective. They have to be quantifiable, they have to lead to you being able to objectively grade them. All right, don’t set an OKR that says, “I will make Gmail better.” Because in your opinion, you’ve made it better because you made it prettier. But if that degrades usage, it leads to user attrition, it means people send fewer messages per minute, hour, whatever the metric is, then you have not made Gmail better. And it’s impossible to objectively assess that. Don’t say make it better, make it… Say that you’ll be able to help users respond to messages more quickly. Right? Well, that’s easy. And in fact, the better way to say that is, “I will reduce the average time to response sent from whatever it is now, by 30%.”

Okay, personal versus team versus company. Personal OKRs are pretty straightforward. What are you working on? What is most important to you? What are the things, that at the end of the quarter, you’ll be able to show you did? Team OKRs are the priorities for the team, not just the bucket of all of the personal OKRs. Think about our example from John Doerr’s slides of the 49ers. Personal OKRs: that’s Alex Smith, Frank Gore, David Akers, Vernon Davis, Crabtree, Moss, you name it… It is each individual player, has their own OKRs, and obviously those are going to be quite a bit different. The team OKRs: offensive coordinator, defensive coordinator, PR head, ticket sales, the new stadium person who’s coordinating, you know, the season tickets… Those are the team OKRs. And then the company OKRs… big picture, top level focus. Right? Win the Superbowl. Might have a few others there. Secure home field advantage through the playoffs might be another company-level OKR. Things that help clarify for each individual what the most critical things are to the organization as a whole.

All right, as I said, we’re going to look at a couple of OKRs from my Blogger days. Some helpful context before we dive into this: at the time that I took on Blogger, Blogger was the eighth-largest web property in the world. Still, incidentally, in the top ten worldwide. So that’s a good trivia question. Next time you’re at a bar and somebody asks, “What is Google’s third-largest web property by page-view traffic?”  Not by visitors anymore – you know, first is obviously Google. Second, obviously YouTube. Almost no one knows that Blogger’s the third. I take a little bit of pride in this as you can tell. But, again, for context, Blogger was a fairly ignored product internally, fairly forgotten, at least certainly within the U.S. tech echo chamber even though it remained this amazing, huge engine of traffic worldwide.

It was the middle of the recession. Google was looking at different products that it owned and trying to figure out whether any of them were revenue-generating or could be revenue-generating, and Blogger was making a little bit of money at the time, but not a lot. So, I was asked to come in and turn Blogger into more of a business.

So, in these personal OKRs that I’m going to share, I’ve removed some of the specific details because, for illustrative purposes, they’re not relative, and I don’t want to put you in possession of any material information about some of our products. But my first objective was always: accelerate revenue growth. Every quarter that I managed Blogger, my first objective – the thing that I needed to think about every week that I was running that product – make more money.

Key results… First element. So this was the first quarter that I was on Blogger. At the time, it was – I think I had counted 14 clicks for any Blogger owner who wanted to put AdSense on their site. Which is ridiculous… Google owned Blogger, Google owned AdSense. I wanted to launch a monetized tab that would make that a one-click operation. So, first key result: launch. Launch a tab… And, by the way, I had other key results as part of this that quantified the number of publishers that we would bring into AdSense as a result of this.

Another very specifically-written key result: Implement AdSense host channel placement targeting (that’s a bunch of words I’ll explain in a moment) to increase RPMs by xx percent. Note here – this key result is written in a way that the core goal is increasing RPMs by a set percentage. If I had implemented and launched this host channel placement targeting, which was just a way of bundling a bunch of similarly-themed blogs into a targetable channel for advertisers… If I had launched that and had no impact on increasing revenue on those blogs, then I’d get a zero, or maybe a .2 on this key result. The key for nailing this key result is in ensuring that it had measurable uplift impact on revenues.

And then finally, this was something that really was disciplined from Joe Kraus, who hired me to run Blogger, he’s who brought me over to Google Ventures: launch three revenue-specific experiments to learn what drives revenue growth. In late ’08, early ’09, there was a lot we didn’t know what we didn’t know about making money on Blogger. So Joe’s thing was: don’t assume you have the answers; do lots of experiments. Get into a normal rhythm of launching something to test whether it has an impact. And if it does, great, do more of that. If it doesn’t, well, file that away. Know that it doesn’t have an impact, and don’t do more like that.

And then finally, and this was a couple of quarters after I had been on Blogger: finalize the PRD (products requirement doc) for the Blogger ad network, secure eng. allocation to build it out the following quarter. So this is, again, a measurable OKR, or key result rather, in that it showed preparation work that I’d be doing, in this case in Q4, so that something would get launched in Q1.

Now, if I wrote the PRD, entirely under my own control whether I did or I didn’t, but failed to ensure that the engineers would be on the hook for building it out in Q1, you know, at best I might get a .5 on this OKR. I would argue that far more important to get the eng. allocation than it is to just write the PRD, so again, in grading this, that will come into play when we look at grading this. And I will be grading them in just a sec.

Next objective is to grow Blogger traffic by a certain percentage over our organic growth. Now, Blogger was this wonderful gift in that it just – it had hundreds of millions of visitors every month, and would continue to have hundreds of millions of visitors a month, whether I got out of bed or not. And they would continue to read blog posts, and they would continue to visit through search, and it was just this engine. At that point, it was turning ten years old that year – it was inevitable that it would continue to get traffic. And as the internet continued to grow, and more people came online, almost certain, the traffic would go up over what it had been the year before.

So my objective here was to not sandbag and say, “Hey, grow traffic,” but to ensure that I was the one moving the lever up. So the key here is: growing traffic over and above what it would do normally. So, key results to define this: launch three features that will have a measurable impact on Blogger traffic. Improve our 404 handling – very tactical. But if you think about a site that gets hundreds of millions of visitors, not all of them land on pages that still exist. People delete posts, blogs go away… If we had wanted to create a worse 404 experience for users, I don’t think it would have been possible. At the time, we were, you know, blogs had their own style, had their own themes, we were serving an error page that was not using the default style. So if you were familiar with a blog but landed on a 404, it looked like a system error page, not an error page served up by that blog. We weren’t giving you access to archives on that blog. We weren’t giving you a search box. We weren’t even telling you that this belonged to a blog. It wasn’t even a link to the blog on which the 404 was being served.

So we looked at our blogs and we realized… Good Lord, we’re dumb as a box of rocks. That’s a key area where we could dramatically improve the user experience and probably increase our traffic while we’re at it.

Improving Blogger’s reputation was another objective I had. Again, Blogger was about to turn ten years old that year, largely forgotten in the U.S., even though it continued to have a lot of use. And we had not been proactive in communicating with press, with users, with partners, and I saw part of my job as reestablishing Blogger as a member of a community, that we were committed to being engaged. So that meant going out and speaking at events. Getting Blogger back out in front of people who pay attention to these things. Again, measureable key result: I said I’d speak at three industry events. Coordinate Blogger’s tenth birthday… PR efforts. Figured out that Blogger would turn ten in August of that year, so it was important to say, all right, what are we going to do to draw attention to the fact that Blogger has reached this milestone? And, how can we use that to further our other objectives: increasing traffic, increasing revenue? So some of the partnerships that I launched in conjunction with this tenth birthday were done specifically to drive revenue. Others were done specifically to drive traffic. We partnered with a company that had built an iPhone app so that our mobile usage would grow. Each of these ultimately served a couple of different objectives.

I pulled a list of the top end users as measured by traffic, and personally sent each of them an email. Many of them had been on the platform for years and had never heard from anybody on our team. And I felt like it was critical, that, you know… In some cases, we were making considerable money, as they were. That was one of the really great things about working in that product. Any amount of money I made, they were making more, because Google would get a percentage of whatever ads were running on their site, if they were using AdSense, and they would get the majority. But I gave them my email address, my cell phone number, my Twitter account, so that they knew how to get in touch if they needed help. And they no longer felt like they were using this nameless, faceless product anymore.

Very tactical issue, for those that understand in U.S. law, the DMCA, as a host of content, we are legally obligated to take content down if it is alleged to be infringing the rights of another. Well, a lot of music blogs on Blogger – the RAAA, and its international counterpart are fans of sending DMCA notices, and the DMCA process if you were a Blogger user was pretty broken. And I felt this was hurting our reputation with some of our core constituents. Not to mention, it was a degraded user experience overall, whether you were the blogger, whether you were the reader who wanted to read the post.

So, I added as a key result, fixing this process and eliminating mis – and there should be another word in here – mistaken take downs of entire blogs, when the goal was to just take down a post.

And then finally, improving our reputation, getting us active on Twitter. That was where a lot of our core demographic were actively chatting with their readers, with each other, with the world, and if Blogger had a problem, they would often rant on Twitter about us, and you know it was sort of into a void. So I wanted to get us active on Twitter, as @Blogger, it’s convenient when the founders of Blogger are the founders of Twitter; they can help get you started on that path, and so we did that.

So let’s look at the grades. As I said, accelerating revenue growth was always a key for me. And these grades are written to be measureable. Launching the “monetize” tab – well, we did. We launched; it was a fairly significant change to the overall UI, so that got a one. Here, you know, as I said earlier, the goal is a .6 or a .7, but if there’s a very tangible binary key result, you don’t give yourself a .7 because you never get a 1.0. In this case, I did what I said we were going to do, it was considered critical, so you get a one. [Key result #2] Implement AdSense host channel placement targeting to increase RPMs by xx%… Well, we implemented a version of this, and it didn’t have much of an impact on RPMs. So, this one was a fail. [Key result #3] Launch three revenue-specific experiments… We launched three experiments. Only two of them really told us anything about driving revenue growth. That was largely because the third experiment that we launched that quarter that we thought would tell us something interesting about revenue really, in the end, didn’t. So that was as much our error – my error – in not crafting the experiment more effectively.

And then finally, the PRD for the ad network… [I] wrote the PRD, was still at the end of this quarter doing some horse trading with my eng. manager on which engineers would be building it out, but we were fairly agreed at the point at which we were grading these that it would get built. There were still a few open questions, so I didn’t consider this a one, but overall it was a .8.

Now, quick note on scoring: you’ll note that the .7 in white at the top here next to the objective is just an average of the four key results below. I’ve seen some get really cute in how they grade their OKRs by trying to determine by weight each of the key results, and say, “Oh, well, you know that – that’s twice as important as this other one, so I’m going to give it a .7.” Or, “Of the objectives, this is the most important objective, so that’s 50…” It’s ridiculous. The goal of this is to provide feedback – not only to the individual who owns the OKR – but to the group, and the company ultimately. It is far less critical to get these down to the second, third, fourth decimal point. I always felt, and continue to feel, grades don’t matter except as directional indicators of how you’re doing. If you’re spending more than a few minutes at the end of a quarter summarizing your grades, you’re doing something wrong. The work should go into doing – and delivering on – the OKRs, not grading them.

On growing traffic, we said we’d launch three features that have a measurable impact on Blogger traffic. We launched two, each of which had a measureable impact, so that’s good. But we said we were going to do three, and candidly, to be totally honest, we probably should have been able to do four or five. So, this could have been quite a bit better.

On improving Blogger’s 404 handling: by the end of this quarter in which this one was graded, we had not proceeded past the mock stage. [I] was pretty sure we knew how we were going to do it – I just hadn’t prioritized getting engineers to build it out, so that was a big mess.

On improving Blogger’s reputation: not least of which, because so much of this was solely on my shoulders, this was a much easier one to deliver on.

So let’s talk a little bit more about grading. As I alluded to on the prior slide, simplicity is key. Grade each key result, then average them up to give yourself an objective grade. And then average your objective grades to give yourself an overall grade for the quarter if you like. I don’t ultimately think that much matters, but it is helpful to get a sense of, you know, where you are. I think in my best quarter, I hit a point… it was north of .7, I don’t think I ever had .8 averaged across all of my objectives. I’m pretty sure that’s right… I think it might have been a .78 – somewhere thereabouts. That was a phenomenal quarter, and the grade reflected that. I had a bunch of quarters between a .6 and .7, not because it was overly engineered to get there, that just was… I was in the, you know, sort of sweet spot of… I was delivering on more than I was missing, but there was still a lot of room for improvement. And, there were probably two or three quarters where I was below a .6, where I was absolutely not delivering to a standard I felt was good.

High scores are possible, right? .8 – 1.0… again, if it’s binary and you did it, you get a one. High scores are possible; they’re unlikely. You know, I was going to say they’re as unlikely as Mike Arrington rejoining TechCrunch, but then he did, last night… So, sort of a… maybe a bad example, I guess. When it happens, it’s exceptional. Be proud of the accomplishment; be absolutely eager to share with colleagues, your manager, your team, what have you. But I think the converse is probably more important to focus on, which is: the low scores, right? .4 and below… It’s not failure, right? One of the reasons when Steven Levy talked about how this became so embraced so early at Google was that this was data. The .4, the .3, the .2, zero… it’s not a failure; it’s… figure out what to stop doing. Or, what did you learn? What did you not know when you said you would do that thing that prevented you from delivering it? Well, do you have any control over removing that obstacle in the quarter ahead? Or, is it immovable? If it’s immovable, go around it. Figure out… either don’t focus on that objective any more, or figure out different ways of achieving it. The scores ultimately benefit everyone in the process more by what they help you know what not to do, what to change, what to continue to do more of. That is ultimately where they’re most helpful.

Company-wide scoring reinforces the commitment here. You… Your commitment individually, your commitment as a CEO to repeatedly grade at the company level, at the individual level, helps everyone understand that this really does matter. And once you get into that rhythm, it’s just a byproduct of doing your job. It doesn’t take a ton of time. But over the quarters and years, they become extremely useful.

So, the company-wide process. Unless the company is only four or five people, you’re not going to be grading everyone’s OKRs in front of the entire company. That’s just… that’s impractical. But it is practical that at least once every quarter, sometime after the beginning of the next quarter, that the company’s going to get together, and they’re going to discuss how they did. And the owners of those individual OKRs – sorry, of the team OKRs – so, in a larger company, that’s the SVP… In a small, growing company, that might be, you’ve got the head of product, the head of engineering, head of marketing, maybe your sales… are each going to get up and say, “Okay, this was the product OKR for last quarter. Here’s how we did: we got a .6. Here’s why we got a .6.”

It’s important to explain to everyone in the company why you got the grade you did, and what you’re going to do differently in the quarter ahead. This calibration ends up being extremely useful for keeping everybody honest, ensuring the company is as transparent as it can be, but then also saying, okay, what did we learn? For me, that was always the most important part. And then at this company meeting, the last part is setting the upcoming quarter’s OKRs. Now, often, there’s continuity from the prior quarter to the next quarter. Sometimes, it’s, you know, we put that project to bed. We either finished it, shipped it, and it’s done, or we learned that that’s not what we want to be focused on moving forward. Now, we’re shifting our attention to this other thing.

Let’s talk about how to implement, for a moment. At Google, we eventually built our own system. It’s not a complex system. It’s a bunch of text fields, with text boxes next to the text fields where you can enter in your grades. And then, the most sophisticated that it gets is there’s a button you can push that says “average my scores.” And it does an immediate roll-up of your key results and gives that objective a score, and if you want to get even fancier, then you can average your objective scores to get your overall score for the quarter.

That’s not something we make available to the public, so that’s obviously not a tool you have at your disposal, but it occurs to me that, I think, some relatively low-tech options are just as valuable. Google Sites – you can create a sites page for the quarter, and each person can add themselves to that page, where their name becomes their quarter’s objectives and key results, just a bulleted list, very simple. And then at the end of the quarter, they can go back, they can edit their page, and they can add their grades next to it. Pretty low-tech.

Slightly more flexible might be a wiki. You could use Google Docs or Google Spreadsheets, I think that starts to get a little unwieldy, particularly as the company grows, but it is potentially a way in which you could do it. The one benefit of using something like Google Spreadsheets, of course, is that if you’ve got a lot of people, the ability to do calculations across all of them to get, sort of directional indications by team, by department, or what have you, might be really useful.

At least as important as the tool you use is the communication you do about the commitment itself. What I mean by that is: this needs to not be a half-hearted effort. It needs to be clear to everyone in the company that for at least a three-month trial period, ideally longer, that everyone in the company is behind this idea.

At Google, it was always clear – not only because you could see Larry and Sergey’s OKRs – it was always clear that from senior management all the way down, OKRs mattered. When I was in the product team… To tell you how much of an impact this next email had, I can tell you that when I was putting these slides together over the last week or so, I knew when I got to this point in the presentation, I was going to have a screenshot of this email, because over two years later, I still remembered the email.

And the email is one from Jonathan Rosenberg, who used to be the head of product at Google, still very close to Google and advises a bunch of folks in the company on a number of different things. But every quarter, he would send out a version of this email. And this was the one that was particularly bad, which is why I remembered it. But the email that he would send out every quarter within the first few weeks was a public shaming of whoever, on his team, had not updated their OKRs for the quarter.

In this particular quarter, he didn’t send it out until the beginning of November, so it was already about five weeks into the beginning of the quarter. And I’ll just read to you a little bit:

“Our performance on OKRs this quarter was poor. As of today, nine PMs have still not posted their Q4 OKRs, out of several hundred. That’s a full month into the quarter. The offenders are listed below.”

And then, right below, you can see right here, he has publicly shamed my colleagues… Two friends of mine are on this list, as not having taken the time to tell the company what it was they were working on.

He goes on to say:

“I don’t understand why it’s so hard for so many of you to take just a few minutes and list your priorities for the quarter.”

So, last paragraph:

“For now, I recommend we all torture the offenders below by asking them in the halls why the heck they’re late on their OKRs. For those of you who find the previous statement offensive, I’ll be happy for you to chastise me as long as your OKRs are posted and current.”

So, the key here is: when you’re on a team and you receive an email like this, from the person ultimately responsible for your – in this case, the product team, but this could easily have come from the CEO, you know, if you’re an engineer, the head of engineering – I know this matters. And as an individual, I never wanted to be on this list. I never wanted the person who was responsible for my team to identify me as holding the rest of the group back. Pretty powerful, and simple.

So here’s a timeline. We’re coming into the beginning of November. A couple of thoughts on things that are really critical, I think: In the next month, I think there are two things that matter most for a company thinking about embracing OKRs for a three-month trial. The first is, you’ve got to be thinking about what your Q1 objectives are going to be. You probably also want to be thinking about annual OKRs, but tactically, thinking about: what are the immediate things that, you know, we’re going to be worried about. If I’m the CEO, I’m thinking just about: what are the three or four most important things for 2013?

The other thing that’s really critical to be working on right now is deciding how you are going to capture what the OKRs are, and how they’re going to be shareable inside the company. This process will fail unless everyone in the company has a way to see what everybody else is working on. So I think it’s critical over the next four or five weeks that you figure out whether you’re going to use a Wiki, whether you’re going to use Google Sites, whether some engineer is going to build a really simple rails app that’s just going to have a bunch of text fields and, you know, connect to their username, and whatever. Keep it simple, don’t over-engineer it initially. It’s always easy to migrate to something more powerful down the road. I think the key here is: just know what it’s going to be, so that if there’s any kinks to work out, you get them worked out well ahead of the end of the year.

Now, into December, I think the company-wide objectives have got to be communicated for Q1 and for the year. That way, everyone else in the company (you also, by the way, have to communicate what the hell OKRs are, so that there’s a general understanding within the company. And by the way, this video is being recorded, is available for anyone in your company to watch. So if anybody has questions about what it is, you can just as easily share this video with them). Communicate annual and quarterly Q1 objectives for the company. Now, it’s up to the individuals in the company, at the end of December, by the very beginning of January, to have their personal OKRs drafted. And that should be at least one draft with a meeting with the manager, and at least in iteration to get to consensus. This is that negotiation that John Doerr was talking about.

Company-wide meeting: probably no later than about the second week of January, where the company-wide objectives are presented and, if appropriate, depending on the size of your team, the team objectives within that company are presented. Now, throughout the rest of Q1, [it’s] important that there’s individual meetings happening between individual contributors and their manager at least once in a quarter. It doesn’t need to be overdone, it doesn’t need to be weekly check-ins, but at least once or twice so that there’s a general point in the process where everyone in the company has gone through that exercise of saying, “You know, this is what I said at the beginning of the quarter I would be doing.” By February 15th, am I in the ballpark? If I am, fantastic; keep doing what I’m doing. If I’m not, I need to now recalibrate. Do I need to stop doing certain things so that I can deliver on the objectives I set? Or, where am I going to miss, and what can I stop focusing on entirely, so that I can at least deliver on a few that are really important.

And then finally, you repeat this whole process, so that towards the end of March, now it’s up to each individual to start thinking about grading their OKRs. Team owners start coming up with the team scores, grades, and then the CEO leadership team are going to grade the company’s OKRs. And in the beginning of April, there’s going to be a company-wide meeting where you grade Q1’s OKRs, explain why you missed or hit, and what you’ve learned about what’s going to happen in Q2. And this process then just happens on a rolling, quarterly basis.

All right, believe it or not, we are at the end of my formal presentation here. So I am going to pull up the Q&A that have come in from you all online and in the room. So thank you, for that.

Let’s see, “What kind of tools do you use,” Luke asked, “for managing OKRs?” Well, we talked a little bit about that. I’ve seen it go through at least three revs at Google. Right? Today we have a very simple tool that is really no more than: I log in with my user name, RKlau, and I click on the quarter, and I add, type in an objective, and then under each objective I have a button to add a key result. And I add each individual key result. I’d show it to you, but again, it’s internal and it’s proprietary so it’s not the sort of thing we’d share in a workshop here. But you get the idea of what it is. Very, very simple. At the end of the quarter, I go in and I click the score button. And that’s when I am now just assigning a number in the text box next to each key result. And as I mentioned earlier, then I have the ability to average the really sophisticated math going on there.

I think in most of the cases for the folks that are here, for the folks that are watching online, I think something as simple as a Wiki – if you use a Wiki internally – I think Google Sites would probably be enough. You know, I think this is the sort of thing you could over-engineer pretty quickly. The keys, are not, in my opinion, are not the tool… It’s the commitment to being measurable, the transparency that everything is available internally to everyone in the company, and the grading process itself. I think beyond that, I think that the tool matters less.

What’s the cadence for the planning portion of an OKR period? For example, is it all in one week, or spread over multiple? How many meetings do you have with whom in attendance? For reference, at Puppet Labs, it’s one exec retrospective meeting, and then an exec planning meeting, and then team meetings, and then exec review meeting all in one week.

I think each company will likely be a little bit difference. In my own assessment, if I were implementing this at my own company tomorrow, I want this to be a relatively lightweight process. I don’t think it necessarily needs to be in a meeting. I think this can be this sort of collaborative drafting that  can happen where everyone can look at a doc, make some suggestion changes, then have a meeting to wrestle over or negotiate over key points so that there can be some consensus. I think you run the risk of meeting… over meeting the process, which I think could be problematic because then that’s reinforcing the wrong thing. The point here is to be doing the work, not meeting about doing the work.

So, what would I say there? I think in terms of grading, I think it’s important that people have that mid-quarter check-in so they have a sense of where they’re at, I think there’s probably some, I think, diligence – maybe an hour towards the end of a quarter, and then a final meeting at the very beginning of the following quarter to come up with a grade.

In terms of planning for the quarter ahead, I think often that’s a natural byproduct of the grading process, where I think about: what did I get done last quarter? Are there things that are rolling from last quarter into the next? In which case, that’s sort of a copy and paste operation. Are there things that I wanted to be working on that I deliberately chose not to do in prior quarters for prioritization purposes and add them in? I think cramming it all into one week is probably a little too tight. I would say at least bound it from a week before the end of the quarter to a week after the end of the quarter; maybe a little bit of buffer on either side. I think that will ultimately give you enough room in which to give people the feeling that it’s aiding in the planning process, not removing time from your schedule.

So Jeff asked, “How do you ensure OKRs roll up, or decompose down the orid structure?” “Trickle down” might be better; “decompose” suggests that they go away, but I think I know what Jeff meant there.

So, how do we do this? I talked in the deck about the sort of virtuous cycle, and I really do – not just believe it, I’ve seen it work. The way we ensure is… It used to be Eric when I first joined Google, now Larry is the CEO… There is a quarterly meeting, and it is given by Larry, and Larry says, “These are the most important things the company is working on. These are the grades for what we did last quarter, and now I’m going to have Vic, and Urs, and Susan, and Jeff, and each individual group head get up and explain how they did last quarter, and what they’re working on next quarter.”

Now, that has a really important clarifying point. I think most people – particularly in startups – want to be working on the most important things. It was always very helpful for me, as I was going through the process of drafting my OKRs, that I would draft them at least in draft form, and then I would attend the company OKR meeting, and I would do a reality check against the company OKRs, and would want to know: Can I see in my OKRs, the company OKRs? You know, is what I’m focused on reflected in what the company is most focused on? Think about, you know, the punter, the quarterback, the running back: is what they’re focused on reflected in the GM or the team owner?

I can tell you, you know, one extreme… There was one quarter when I was running a home page at YouTube that I owned two of the company’s five… five or six OKRs. That was terrifying, because I didn’t… My OKRs weren’t just reflected in the company OKRs, they were the company OKRs. That’s exciting, but it’s also, for a company, of, at the time about 30,000 employees, a little scary when you’re running the team that is responsible for a significant proportion of what the company deems to be most important at the time. The reason that was a helpful process was: if I couldn’t see the company OKRs in my individual OKRs, that’s when you take the red pen to your personal OKRs and revise.

Jeff asked, “Cadence for checking progress against OKRs?” Different teams will do it differently. You might – it might be more natural to do it every few weeks. Particularly in larger teams, I think it’s probably realistic that each individual is going to have to own the discipline of managing themselves, and maybe having only one or two check-ins in a quarter with their manager specifically against OKRs.

So I was asked: was I using OKRs at Blogger before I joined Google… To clarify, I joined Google in ’07, Google acquired Blogger in ’03, and I was not, obviously, a part of the Blogger team until late ’08. By that point, Blogger was five, six years under Google’s ownership and had really embraced OKRs. So, sorry if I didn’t make that clear.

Last question, unless there’s any others here in the room. I see several nodding heads, that, you know, they’ve been jotting them down in the dock, so that’s great. At what point in a company’s evolution should the management team institute OKRs? Is it five people, ten people, 15, more? Easy answer: as soon as possible. The sooner this is part of the DNA of the company, the sooner this is accepted… As soon as this is part of the normal rhythm that the company gets in, the better off you’ll be. You are… There is always a point at which you can do this. Obviously, I’m hosting this workshop because I believe every one of you should have some version of this approach in place at your companies. So even if you’ve been around a year or two, it’s not too late. But the longer you wait, the longer you are going to have habits form within the company about how things are done. And, the more inertia you’re going to have to overcome. So, even if it feels a little artificial if you’re a company of five, the discipline that it brings is immeasurable.

I committed to doing OKRs, even though, at the Startup Lab right now, I’m a team of one. Now, technically, I’m part of Google Ventures, we’re a large and growing team, and I have a manager who I check in with periodically on progress. But for me, the discipline of looking ahead and saying, “These are the things I’m going to focus on this quarter, these are the things I’m not,” and then evaluating, how am I doing… That… It’s hard for me to imagine doing my job without this discipline. At the end of the quarter it makes it unbelievably easy to summarize what we’ve done, and whether we did well or not. And, we say “we,” I mean “me,” for the Startup Lab. So, how many workshops did we schedule? In this quarter, one of my objectives was launching the video archives, which we launched, you know, in the second week. It had been an objective in Q3. I had done most of the heavy lifting in Q3, couldn’t get the trigger pulled on getting the archives up, so it slipped into Q4. My grade for Q3 wasn’t as good, my grade for Q4 was less about launching, because I knew that was going to happen a week or two after the start of the quarter. It was more important to focus on, okay… Now we’re going to measure how much traffic are the archives getting? How many playbacks are there? How many people have asked for access to the archives? That’s the sort of thing that starts to give you a sense of how that mentality can really be embraced, and drive action and accomplishment.

So, with that, I’m going to wrap up. Let me again, thank you for taking the time. I teased you about not telling you how long this was going to go; I budgeted 90 minutes, I think I’m at about 88, so, you know, hey, I’ll score that a one on the OKRs. Please fill out the feedback form for those of you here in person or watching online. The data is critical. As you know, not only from this presentation but from others, we are very data-driven as a group. Not just the quantitative score that gives us a sense of how we’re doing on that one to seven scale, but also the qualitative: what takeaways did you get from this event? Knowing what resonates helps me when I think about planning future topics – really know what to focus on and how to steer presenters, too, so that’s really critical.

Topics, again, as I said at the beginning of this workshop… If you’ve got a topic that you would like to see us cover, I want to know what it is. I want to hear from you. My job is to find the expertise and experience at Google and make it available to you. It’s a great job, I love doing it; but I’d rather not be prescriptive and tell you what I think you should know. I’d rather know what you think you need to know so that I can make that available.

Stay tuned on the announcement list. We’ve got a lot of stuff coming up for the balance of Q4, and [we’re] starting to do a lot of planning for Q1, and really excited about some changes in the approach and format. So, looking forward to seeing you again online, in person. Until then, thank you again. See you soon, bye.