How did your team respond to expanding?

Recorded February 24th 2021 – watch the full episode on YouTube.

Thinking about your iOS app building infrastructure, the people working there, the tools you have, and more – how have they changed or evolved over time to respond to the kind of expansion Netflix has gone through?

Jordanna Kwok: We've kind of grown massively on that front throughout my time. You might have metrics like how many requests per second you're getting on this particular endpoint that we're fetching data from. That's most basic or how many people are playing back, things like that. We've augmented a lot of those metrics to be able to capture more errors and things at scale like crashes, those are the most sensitive metrics we get. So, we will kind of monitor those metrics as we roll out now, and we do have a mobile productivity team now that is helping support us.

"Even with the 1%, we can sense that something is wrong and may need to pull back a feature or worse case scenario make a point release."

Before, there really wasn't a separate team even to help us out on that front, we would have to do this all on our own as engineers. You're building a client, but you're also potentially building out some tools, but now there's a separate team that builds out tools, dashboards, and whatnot for us to help monitor the health of the application and it's very sensitive, like you said. We actually use Apple's slow roll out capabilities, you know how it does 1% on one day, 2% on the next day.

And even with the 1%, we can sense that something is wrong and may need to pull back a feature or worse case scenario make a point release. And that's not great, because you're like, Hey, we need to go through the review process again, and potentially expedite if it's pretty bad.

Paul Hudson: I'm almost imagining a gigantic TV screen, it being Netflix and all, with very large numbers of graphs and numbers and stuff happening right now, almost like a live dashboard of what's going on.

Jordanna Kwok: In the office there are multiple TVs that actually have these dashboards. I'm sure people probably have multiple monitors at home now, probably, to monitor it.

"We do have a system where people are on call, and we rotate around the team and so we will have coverage in case anything happens. But, thankfully, it's very low incident rate."

Paul Hudson: Right, that makes sense. And, yeah, so the scale thing is fascinating because you are working on such large numbers that even with a phased rollout, as you said, you can get bugs appearing immediately. Hopefully it doesn't happen often, but if a little clanger sneaks through, suddenly you'll get feedback very quickly and hopefully be able to respond quickly. I'm not sure how much you can say, but I'm guessing you can get an app review from Apple fairly quickly because you're Netflix.

Jordanna Kwok: It totally depends on who's available, I think, on their end. During holiday season I'm sure the times are a little bit slower versus maybe during regular business hours. So, we do have a system where people are on call, and we rotate around the team and so we will have coverage in case anything happens. But, thankfully, it's very low incident rate, so we're pretty good with keeping things fairly stable.

Paul Hudson: Do you have any internal iOS apps? Because if you have all metrics coming in all the time, is your team thinking, we could build a macOS or tvOS or iOS app to actually make that nicer?

Jordanna Kwok: Not iOS specifically. A lot of these tools are written with JavaScript, it's more web based. We're also supporting Android, we're also supporting laptops and TVs, so a lot of these things with dashboards and whatnot, we want one tool that would be able to scale to all these different platforms. So that's one reason why we haven't just built an iOS or macOS app to kind of capture that.

Paul Hudson: Yes, that makes sense. With so much things going on at the same time, so many sort of cogs moving independently, it's going to be hard.

Sponsor Hacking with Swift and reach the world's largest Swift community!