How much is A/B testing driving app development at Netflix?

Recorded February 24th 2021 – watch the full episode on YouTube.

How much is A/B testing driving your plans to develop new features, versus relying on the gut instincts of team members?

Jordanna Kwok: Certainly both. I would say both because you might have a gut feeling about something, and to validate it you would do the A/B test. Often these A/B tests will fail, and we'll learn a lot from it. So this is another thing with the culture where making mistakes and failing is part of it and, yeah it might be my gut says something is going to be a great feature, and it turns out at the end of the day no one's using it.

"That's how we vet things out, and it's never like, "No, we don't want to do this", but it's more of, "Hey, let's try it and see what we learn from it."

So then you've learned something – maybe there's something that wasn't quite right, whether in the design or the engineering. And you could run it again; you can make tweaks and try it again and run the A/B test again, and when you get a positive signal, then great, we can roll this out. So, ideas come from everyone: engineers, designers, product managers, and more; it's not just the product managers.

Because typically people think of product managers as the keeper of the ideas, and they're the ones driving all these features. That's not necessarily the case. Lots of team members have pitched their own ideas and worked very closely with designers to flesh them out, and A/B test them, and a lot of times we're surprised when they come come back with a positive benefit for Netflix members using it. So that's how we vet things out, and it's never like, "No, we don't want to do this", but it's more of, "let's try it and see what we learn from it."

Sponsor Hacking with Swift and reach the world's largest Swift community!