Episode 53 - Scaling GreenOps at Back Market with Dawn Baker

Dawn (00:00)
It is always important to be controlling costs when you're a CTO and infrastructure is always a significant component of the PNL. And if you can make sure people are focused on the environmental impact, it will have a cost impact. And so you can advocate for GreenOps because it is tied to the bottom line.

Gaël Duez (00:04)
Hello everyone, welcome to Green IO, the podcast for responsible technologists building a greener digital world, one byte at a time. Every two Tuesdays, our guests from across the globe share insights, tools and alternative approaches, enabling people within the tech sector and beyond to boost digital sustainability. Migrating to a new cloud provider because of the lack of sustainability commitment from the previous one is almost never seen in our IT industry. So I was really impressed with the talk delivered by two Back Market senior engineers at Green IO Paris last year. So impressed that I wanted to learn more about Back Market engineering culture. I discovered teams who have seriously embraced the GreenOps approach and whose feedback could greatly benefit product and tech teams, no matter the industry they work in. This is why I'm so excited to welcome Dawn Becker, Back Market CTO, on the show today. Dawn is a senior engineering leader who has managed infrastructure, operations, and software development teams in organizations ranging from 50 to more than 100,000 employees, from startups to Fortune 500 companies and her functional expertise spans across many layers of the technology stack, including data center, network systems and database operations, software engineering, course, you name it, I guess you understood that she is a very versatile expert. Welcome to the show Dawn and thanks for making room in your busy calendar for this interview.

Dawn (02:06)
Thank you for inviting me, Gaël. It's a pleasure to be here.

Gaël Duez (02:09)
Pleasure is mine. I'd to start asking you what is the achievement from your teams which you are the proudest in terms of sustainability.

Dawn (02:19)
So it is definitely the migration of our infrastructure to a new cloud provider. We did it in under six months. And we have up to 40,000 containers in our infrastructure. So it's a significant achievement. And it's been great results for us so far.

Gaël Duez (02:42)
Yeah, what was the result that you were looking such a massive project? just before you answer this question, for the sake of understanding, what size of the tech stack and the human power are we talking about at Back Market? How many engineers?

Dawn (03:00)
Yes, so we have about 260 engineers at Back Market. We operate up to 40,000 containers and we have about 4 million unique visitors a day to our site. We work really hard as a company to be conscious about our impact on the environment. And so when we were evaluating our next steps for infrastructure. were at a point where we needed to modernize our infrastructure. We're a 10 year old company and as happens in 10 year old companies, we had a bit of debt that we needed to pay down and it was going to require us to recast our infrastructure. And to do that, it gives you the opportunity, which doesn't come along very often to do a change in cloud providers.

Gaël Duez (04:00)
That was also an opportunistic move because it's not like you were scaling up the tech stack and suddenly in the morning you say, okay, enough is enough. I want to change my cloud provider. It was also connected with a reflection on the current state of your tech stack and the level of technical debt you have reached. And okay, enough is enough. We need to refactor a bit and maybe even that's a great timing to ask ourselves on which platforms do we want to be hosted? Am I correct to state it that way?

Dawn (04:31)
Exactly.

Gaël Duez (04:32)
And what were the results that you wanted to achieve in terms of sustainability by migrating from, I guess we can name them, it was made public in Green IO Paris that you migrated from AWS to Google Cloud Platform.

Dawn (04:48)
Yes, so the biggest factor in the decision in the end was the impact and our ability to track it. number one, the visibility we get from Google Cloud on our CO2 emissions and the ability to locations with renewable power was a big factor in our decision we're a B Corp and this becomes a very important aspect of what we do is to be able to report on that. So that became an important component of the decision-making process. And it's also that we can start to break down and start to invest in breaking down not only that information for the infrastructure, but each component of what we run and be able to align teams around the impact that they have. we're still at the early stages of this because we've just finished our migration in April. But we have ambitions to create greater and greater insights with more and more detail so that we can have a clear view of what it costs to serve our users specifically for certain functionality and be able to make decisions about engineering investment on ways to improve that over time.

Gaël Duez (06:11)
So it means that the ultimate goal is that each team, feature team, impact teams, I don't know how you name them, are accountable of the amount of greenhouse gas emissions they emit.

Dawn (06:27)
Yeah, exactly.

Gaël Duez (06:28)
And that was the number one reason for you to migrate into Google Cloud because they can provide this level of accuracy.

Dawn (06:38)
it was the biggest weight in the decision. There are of course other factors. There's a better visibility and higher impact we have on the roadmap and discussions that we can have with our, our account team, the level of partnership that's provided. were also big factors. We run on Kubernetes. So having the managed GKE offering with definite uplift for us.

Gaël Duez (07:06)
You mentioned Kubernetes. There are several projects around incorporating sustainability natively in K8 and also putting them in the CI-CD. Is it the sort of project that you're looking for

Dawn (07:22)
Well, first we want to be able to have the level of granularity, which doesn't really exist today, on knowing for what type of instance that we choose, what can be the carbon footprint for that. And we did this. One of the engineers on my team, Florian Valeye he wrote an open source contribution that would, for AWS, using data set provided by Teeds. They did some research in 2021. So he wrote an open source tool called Track Carbon. And that allowed anyone to track the energy consumption and calculate carbon emissions for their use in AWS. That research course was a point in time and it didn't continue to evolve. And so we didn't have the data for the evolving instance types. So it became obsolete. But we would love to be able to do that at this level of detail, not only for ourselves, but make it available to others as well so that we could track with greater accuracy the impact at the instance level.

Gaël Duez (08:22)
And on Google Cloud, because maybe I missed some recent updates, but I remember that Google shares some sustainability metrics for the different instances that are running. the feedback I had so far is that it's not that much actionable for engineering teams, mostly because of the lag that it is updated on a monthly basis.

Once again, if I remember that well, that most of the time a software engineering team, want to experiment things and see the next day or within the next few days if the change they made in the architecture, in the code, whatever the design has provided an impact. So maybe could you clarify a how teams are actually using this information from Google Cloud to reduce their carbon footprint?

Dawn (09:29)
So today, we're not actually using it to reduce our carbon footprint. We're using it to create visibility. And visibility is the first step for accountability. If people are aware of the impact that they're having, they are more likely to take action on their own. And you're right, the data doesn't come in fast enough, right?

Gaël Duez (09:36)
Okay.

Dawn (09:56)
Which is part of why we are looking at what different tooling we could have for ourselves to be able to take it a step further than gets provided from Google today. One of the things that we envision doing over time is having costs to serve through infrastructure costs, but also through carbon impact because how well optimized your code is uses more or less CPU and uses more or less energy. And it correlates of course to how much spend you have, but not only. And so being able to have that picture, think is really important. It's also, I think every organization at some point starts to scrutinize the amount that they're spending on infrastructure because you get to a size and scale. If things are going well, that it starts to become a significant part of your, of your P and L and every team then ends up having to put focus on cost cutting optimization. never fun, right? It's never a super motivating task for people. When you think about the other way you can look at this, that is a different cost cutting measure, far more motivating and highly correlated is thinking about how can you reduce your carbon emissions? How can you reduce your impact on the planet? And it's a very different way to think about how to optimize your code, how to optimize your infrastructure to have an impact on, yes, the bottom line from a cost perspective, but it also has a big impact on the cost on the planet.

Gaël Duez (11:53)
And so is it something that, I mean, do you have FinOps team today at Back Market?

Dawn (11:58)
There's, I would say it's a, we don't have a specific team. have a focus area on FinOps

Gaël Duez (12:04)
Okay, which means that each team has at least one person who's in charge of paying attention to the FinOps detail.

Dawn (12:13)
Right now this is concentrated in our platform teams looking at this to create the visibility across the feature teams.

Gaël Duez (12:24)
And is it the same people within the platform team who will have the same focus on carbon accounting and creating visibility on carbon or will there be different profiles?

Dawn (12:38)
It's the same people for us today. Again, we're at the beginning of this journey and there are not a lot of models out there to follow.

Gaël Duez (12:51)
I got it. actually going back to what you've said previously about the tooling and the fact that you're still in search of the tooling, what are the tools that you've considered using, whether they're open source or paid tools, or are you actually considering building your own?

Dawn (13:08)
Well, one of the reasons we are looking building our own tooling around this is because to do what we want, which is reproduce some of the work that was done by Florian on that open source project is that it doesn't exist today. There's not a tool that will do that for us there are some tools, I think, that show up that can give you some help but they're still very high level and a bit generic and maybe not as actionable as they could be with a little more detail. And so us trying a path forward that would allow us to give real insights to our engineers to be able to make everyday decisions that can have an impact they do their work to optimize for carbon emissions.

Gaël Duez (14:07)
Really interesting because there are several tools popping up here and there. There are obvious open source tools such as Cloud Carbon Footprint or Cloud Scanner from Boavizta There are also several paid solutions. and most of the time they take the, financial inputs from cloud provider, they cross it with an emission factor or information from providers such as electricity maps, for instance, to get the carbon intensity of the electricity grid at a certain time in a certain day. And I was wondering what other features do you find missing? Especially you mentioned the granularity that you want to develop your own tool.

Dawn (14:55)
All of those things you said are true and there is definitely... We have the ability to look at what type of power in which data center, what percentage is renewable. When are, when can you run your, your intense job so that it's not competing at the time where there's a heavy load on the grid? So all of that is true. And there are things certainly today that we could even do better with the tools that exist, sure we are able to scale to zero at times when everything's idle right, and to do that in an automated way. There's, think, more work to do there. And again, very tied to cost as well in those cases. But when it comes to what can, and I think this is the question that everyone has about their daily lives, is like, what can I do? Like, I'm just one person out of, you know, seven or eight billion on the planet. What does it matter what I do? And when we think about this, there's, I think there's so much that we're not. We don't pay attention to, such as, and I'm going to put this in terms of the everyday human experience, where people don't think about it. And then let's talk about how that translates into engineering. So if you think about how many email addresses do you have and how much storage are you using and how many photos do you have that are uploaded into the cloud? When was the last time you looked at all of them, right? And every bite that is out there, your digital footprint, it also has a carbon cost because it's costing for that. And so when we think about that from an engineering perspective, what are the things that you store and never access? You know, how much logging are you doing? And do you need to do that much logging? How long are you storing it for? And do you need to store it that long? And having some visibility into that, that is your daily dashboard would be hugely, hugely helpful to hold yourself accountable for, are you putting your effort and the cost that you're doing in the right way.

Gaël Duez (17:28)
That means that you're incorporating in this green ops dashboarding that you're building at Back Market also management approach. You want to focus on storage, not only the amount of time that you're doing computing or using servers at a certain time of the day and certain location, but also incorporating this data management principles that we store too much, we load too much, et cetera, et cetera. Am I right?

Dawn (17:58)
It's a, part of my vision for it. And I would say maybe a little too optimistic to say that we're in the process of building that functionality right now. But when you think about how people as individuals can have the impact. This is, you have to go to that level of detail because that's where it starts. Everything starts from just a small amount of impact that compounds over time and suddenly it's huge.

Gaël Duez (18:30)
and that you can scale this impact. And what other features do you envision for this environmental dashboard and tooling that you want your teams to be able to build now that you've migrated to Google. So obviously, we mentioned carbon aware and was also a bit of grid aware. We've just mentioned data management. Is there other aspect where you want to empower your engineering team to make better decision?

Dawn (18:57)
That's a big question. I think the perspective of software engineering work that we do for writing and executing code and hosting it in production. Because that is also really important. the run part of software development is where probably the highest carbon costs are because it's going all the time and it is the most obvious place to look. It's not only that, right? Because we're also doing monitoring. We also have a bunch of other tool sets we run to do our job. And all of those things combined. It's just one on top of another, on top of another that is consuming energy I think would be an advanced step here. Because the first most obvious place to look is absolutely on the run, on how what you're doing runs every day and what that impact is. And it's the biggest chunk. At least we think that right now. I don't know if it would turn out, it would, it turned out to be true if you add up all the costs of day-to-day operations that aren't really about what's running. It's probably not, not as insignificant as we think, or we don't think about it. So I don't know, but it's an interesting place. think that we should start to look and validate where do we have impact and where along the way can we optimize it? Because I find that when you start to think about this, start to make different choices.

Gaël Duez (20:42)
Could you give us an example of one of these choice that you've made differently thanks to sustainability inputs?

Dawn (20:51)
Okay, so we are reconsidering our business continuity disaster recovery planning and there's a non-zero cost to having a solid plan in place, right? And being able to make sure that we have the best possible RPO and RTO for the most reasonable impact, right? Because you can go… can do warm backup and you can do full redundancy. full redundancy obviously has the largest impact. It also costs the most, but it gives you the greatest peace of mind. You end up with so much just sitting and unused completely cold is a bit too long for recovery time at scale, right? So it can be a pretty long time. But especially in a cloud environment, having a warm, small footprint that you're able to manage closely can be a really nice compromise between two and gives you the flexibility with a much lower impact.

Gaël Duez (22:17)
Got it and that's interesting to see that the carbon price has become part of the equation as much as the financial price of recovery plans. And that leads me to another question which is how do you make GreenOps efficient and part of the discussion on almost every aspect of your engineering culture? Because we took the example of the cloud migration, of the example you just provided about the recovery plans. But I got it that you don't have yet all the tools that you want to provide the most accurate insights possible. But when I've discussed with your engineer, they were already very, very focused on green ops. I was wondering, how did you achieve this focus on green ops across your teams?

Dawn (23:12)
So we're at a slight advantage maybe in this area because we are a mission-driven company with an environmental mission. And one of our corporate values is that it's not green enough. So this is a mantra that shows up everywhere in the culture. And having that kind of mindset, it's not only about your job, it really is about your life and the mission and the values that the back market holds attracts people who are aligned with them and it becomes a part of everybody's thought process.

Gaël Duez (23:52)
That raises a very interesting point because do you believe that achieving such a green ops culture in focus would have been possible in, I would say, a regular Fortune 500 company? Or does it require first that the environmental culture is high enough? And, my question is for all the other CTO and CPO and head of engineering listening to you, like, do you have a chance to actually move the needle in the right direction to embrace green ops if you work in a more like, you know, business as usual, I would say company.

Dawn (24:31)
Yeah, it's an interesting question. when I first started talking about carbon PNL with my peers,

I met Patrick at an event for Google Cloud. And I was talking with him about Carbon P &L because we at Back Market have started on the journey of a Carbon P &L. And his company does it already. And they're a very big company. And they have a very active focus on this. And so it's possible. And I think that that is very inspiring that that's the case.

Gaël Duez (25:05)
It is.

Dawn (25:07)
I've also, so I've seen the power of the employee base in this way as well. spent some time at Google. The employees at Google are very environmentally oriented and they do make noise about this. And Google as a company does respond to the things that their employees value in this way. It's harder, I think, in companies that are perhaps not as successful as that to get it to be a focus. There was a lot of discussion during the time I was at Fitbit from the employee base about making the devices more recyclable, having a better sustainability mindset around it which was difficult to make a focus. when the company isn't hugely profitable or hugely growing, you know, when you're facing business headwinds, it is the first thing to fall to the back. is a hard to fight against.

Gaël Duez (26:06)
That's interesting because the way I framed the question was the wrong one. I expected that the mission of the company would influence how much a CTO has some leeway to implement GreenOps practices. And actually your answer points toward a slightly different directions, which is it's not really the mission in itself, but it's more the power of employee engagement and the financial situation of the company, which might be even for a B Corp, highly environmentally focused company, facing a lot of turmoil business-wise, we might actually experience a reduction in the to achieve some environmental goals. Very interesting point of view. Thank you.

Dawn (27:02)
Yeah. So on the question you were actually asking, I do believe that a CTO in any company can push green ops because today, younger generation of workers is far more aware of environmental impact than anyone was 20 years ago. And as I said earlier, it's always important to be controlling costs when you're a CTO and infrastructure is always a significant component of the PNL. And if you can make sure people are focused on the environmental impact, it will have a cost impact. And so you can advocate for GreenOps because it is tied to the bottom line. And I do believe that the business. We'll buy into it because the correlation is there.

Gaël Duez (28:00)
Which is a great enabler. And you were discussing about advocating and that's another question I was wondering how much you managed to have a discussion with your CPO or CMO. I don't know how you're structured regarding product and marketing, but basically people pushing for some features,

when it comes to Green Ops, like, do you manage to say, hey, well, maybe we shall not use video or maybe this feature is not that useful how much they are involved in this Green Ops process? Because we know that design set the tone for most of the music when it comes to Green Ops. So CTO, if they ask to provide more and more features or more and more products, that can completely offset all their efforts to reduce their environmental footprint. So how is the level of the discussion at back market?

Dawn (28:51)
Again, because we have the it's not green enough value, it is the topic that comes up. I think we can do better than we're doing and, and the engineering team has a greater responsibility and a bigger role to play in this than any other department, because we have that information. either have access to the information or we can get access to that information in a way that is a bit more difficult perhaps for the CMO and the CPO if engineering teams are very conscientious about it, and if we provide the carbon impact as part of the planning, it can have a different impact and change people's minds. We are also the ones that are best positioned to propose alternatives and say, you are asking for this, for an objective we understand as this, and we can achieve that same objective with a lower impact. One of the superpowers of engineering that we forget that we have. It was just to have this level of influence over the entire corporation because we have the information, we have access to it or we can get it. And with a little bit of effort, we can educate people and really influence the decisions that get made and costs

Gaël Duez (30:25)
And maybe just to close the episode, because you've already shared in full transparency and humility, I must say, it's not all shiny and perfect at Back Market, but it's really this focus and these first steps that you've already started and this culture that you managed to But we've briefly mentioned Back Market has being environmentally focused because of its very nature. How the discussion regarding e-waste, and maybe you can explain a bit what Back Market is doing to the audience, how this focus on e-waste, which is at the very core of Back Market mission, influences also the discussion within the engineering teams when it comes to green ops. I was wondering how much weight do you put on creating software that will not… Accelerate hardware obsolescence.

Dawn (31:22)
For me, that's a super interesting question. We have a fight against ways of working for the OEMs, which is to make their devices obsolete. They stop updating them. You can no longer get the latest OS update, the security updates, et cetera, beyond a certain model. And thanks to government regulations, these things are being extended more and more. And you see commitments from companies to provide longer term support for the devices that they make than they were 10 years ago. It's a good trend. So from a influence perspective, this is a focus at Back Market is to make sure that we are active in this discussion and advocating for every person out there who wants to keep what they have and they just want it to work longer because they're happy with what they have and through software development, that is what we do and what we advocate for and what we educate people about. Again, something interesting that I think we don't talk about very much is that we are, as engineers, probably almost worse than the general population when it comes to pushing for the latest technology on things. And I don't know how to change this, actually, because when you think about every time new cloud provider or our cloud provider puts out their newest version of instances. They're faster and they're more efficient. And you have a dilemma in that you want to get the speed gains and the efficiency gains you can get from the latest hardware refresh. But the very nature of that hardware refresh can create e-waste. And so we have opposing forces and the best that we can do to serve our business because there's a, have to stay competitive and responsive to fulfill the mission of getting people to buy through the circular economy this is where the carbon PNL comes in as well, to make sure that you're holding yourself accountable for what you put out, what you emit and what you try to save, what you can save. But can we extend, can you choose providers or hardware if you're doing on-prem that is upgradable in place? So maybe you don't have to replace entire chassis every time you're doing an upgrade. Can you choose a cloud provider that has documented and certified recycling practices because we haven't found a way today to get out of this cycle of the constant upgrades for faster, better compute power.

Gaël Duez (34:36)
So if I understood you right, that's a very open and transparent way that you put the issue on the table it's a day-to-day dilemma of running green ops based on carbon PNL. I think it could wrap up the entire episode thank you joining and sharing that openly. And it was a pleasure to listening to your team also on Green IO Paris stage. So thanks for all the support.

Dawn (35:05)
Thank you, Gaël It was a pleasure talking with you