Interview: Janie Clayton
We chat about community, Metal, Face ID, and more.
In the first of a series of interviews with prominent members of the Apple development community, we spoke to Janie Clayton – a regular on the speaking circuit, author of an upcoming book on Metal, and a pug owner addict.
Swift Developer News: For readers who haven’t heard of you before, could you give a brief outline of your history as an iOS developer?
Janie Clayton: I originally got a degree in broadcast journalism and English composition. My hope at the time was to work in radio news at places like NPR or the BBC. I didn't get the memo that journalism, especially radio journalism, was basically dying and when I got out of school I couldn't find a job.
“I realized I had made some grievous errors and that I needed to make better decisions while I still had time to pivot to something else.”
Rather than being smart and doing something practical, I switched my focus to TV and went back to school to learn video production and audio engineering. About three weeks after I finished my degree the Great Recession hit. All of the places I thought I was going to work at went out of business or drastically reduced staff. I realized I had made some grievous errors and that I needed to make better decisions while I still had time to pivot to something else.
SDN: That’s a pretty grim time to graduate! So what did you do?
JC: I was either going to go to law school or become a programmer. I looked at how much law school cost compared to learning programming at the local community college and what the starting salaries for both were and it was kind of a no-brainer. I also noticed my fellow liberal arts refugees were flocking to law school looking for safe haven and I really didn't want to keep competing with them anymore.
I settled on iOS programming because it was relatively new at the time. People were desperate to hire anyone with any iOS experience and they were less picky about having you pass algorithm screens or have 5+ years of experience for an entry level job.
While studying iOS, I found out about Core Audio and OpenGL. I realized that all of the software I used in my previous life as an AV geek were built on these technologies. I got incredibly excited about programming because I wanted to make the tools I had been using a few years earlier and it was amazing to me that these frameworks were being exposed so I could make my own synthesizers.
SDN: And how did that lead you to Metal?
JC: When Metal was announced in 2014, I felt like it was my framework. I had felt that I got started on OpenGL too late and I wanted to be the first one on Metal because it was a new thing. You could get in on the ground floor with a new graphics technology.
It helped if you knew previous ones, but there were so many books on how graphics concepts and math worked that it was possible to go in and learn Metal without having a lot of documentation or other books on the subject out.
I've worked as a consultant for the last two years. Prior to that, I worked for a year for Brad Larson. Brad is the creator of GPUImage and had one of the first 500 apps out on the store. He had taught advanced iOS at my community college and I was interested in his work with graphics. We got to be friends and he hired me to help rewrite the code base for his robotics software in Swift when it initially came out.
SDN: You also talk a lot at conferences, right? What got you into the conference scene?
JC: I got to attend my first conference when I was still a student. It was a local conference in Madison called Snow*Mobile. The organizers are a pair of wonderful people who gave students the opportunity to attend the conference by having them work as volunteers. I got to hand out badges to people and so everyone kind of knew who I was. I figured out how Twitter works at that conference because I could tweet at a speaker whose talk I liked and then introduce myself later and they kind of knew who I was.
“I won a ticket to CocoaConf Boston in 2013 and I had to beg, borrow, and steal all the money I needed to cover my expenses to get there.”
When I was a kid I was super socially awkward and my dad kept telling me when I went to college I would find my people. He said I would find people like me who would understand me and I would have a grand utopian experience where I realized I wasn't alone in the world. That did not work out at all and I found college to be quite disappointing. But when I went to my first conference I felt like that finally happened. I never thought I would find my people and when I did, it had a very powerful effect on me.
I decided I wanted to go to as many conferences as possible, but I didn't have any money. I was an unemployed college student. I won a ticket to CocoaConf Boston in 2013 and I had to beg, borrow, and steal all the money I needed to cover my expenses to get there. I took a train because it was $100 less than a flight, so I sat on a train for 24 hours to get to Boston.
SDN: What made you take the leap from attending conferences to speaking at them?
JC: While I was at CocoaConf Boston, one of the speakers asked me when I was going to start speaking. I was like, "I'm a broke unemployed college student! I don't have anything to say!" He took me to the organizer and asked him if he would invite me to speak their next conference in Chicago. He said sure and told me to just send him a description of my talk.
That trip really changed the way I saw myself as a person. Before that trip I was a dumb college student and after I was a professional developer.
I was given a lot of opportunities to speak at various conferences, which has been a real life line for me. I went through a divorce a few years ago and getting to attend these conferences gave me an escape from my crumbling marriage. I found a very welcoming friendly community of people that I dearly love.
SDN: Would you say that Metal is a subset of that iOS community, or is it more like a whole other world?
JC: Metal is like an intersection of a Venn diagram in terms of community. There is the greater iOS community that tends to have a few pet frameworks they stick to because those are the ones most used by iOS generalists, like NSURLSession and Core Data.
A lot of iOS developers don't see the value in learning Metal because most jobs don't require it. I've actually had a lot of frustration in trying to land general contracts because people see I wrote a book on Metal and they assume I don't know general iOS development and they pass on me. So that whole bit of advice to write a book on a niche topic to show how smart you are really backfired on me!
Then you have the graphics programming community. That community doesn't do much with Metal because it is proprietary to Apple. They tend to flock to things that are cross platform like OpenGL or Vulkan. If they do pick a proprietary framework, it is usually one that is PC only because more people in gaming do things on PCs.
SDN: So if those developers are preferring things like OpenGL, where does that leave Metal?
JC: The place where I see the most potential with Metal is in machine learning applications. CoreML and Vision are built on Metal. There are fewer cross platform solutions for compute operations than there are for graphics.
If you want to write a game that will go on all platforms, there are dozens of game engines. If you want to implement a convolutional neural network on an iPhone, there are only the tools that Apple has provided.
SDN: Your new book is called the Metal Programming Guide, which is a broad title – what kind of reader is it aimed at? Those neural network developers?
JC: My target audience is intermediate to advanced iOS developers. I am assuming an understanding of Swift and Cocoa design patterns. The format of all of the chapters is basically me explaining a concept, like neural networks, to the reader on a high level so they have a human readable explanation of what a neural network is and what it does.
“I am mostly focused on making sure the reader has a broad conceptual overview of what the framework actually does so they can understand why it was designed this way.”
I then go into the Metal framework and show the commands necessary to put one in place. I am hoping if people have a vocabulary of graphics and machine learning concepts that they will be able to look a the docs and find what they need when things inevitably change.
I am mostly focused on making sure the reader has a broad conceptual overview of what the framework actually does so they can understand why it was designed this way. I got frustrated trying to learn OpenGL because many people would assume you had a knowledge of these graphics concepts. Or that you came from a graphics background. I wanted a book for people who were interested in graphics but don't know where to start.
SDN: Right – so it’s more about the why rather than the how?
JC: Yes. Graphics is a multi-disciplinary thing.
There are graphics specific concepts like the rendering pipeline and various world and object spaces. It's also very math heavy because there is a lot of linear algebra and vector calculus to control lighting and animation.
There is also the need to have some familiarity with the underlying hardware. Lastly, you need to understand the framework/API that is being used to take advantage of all these disciplines.
Most books focus solely on the API because that's pretty easy to explain. Trying to get someone to understand an identity matrix or the difference between vertex and fragment shaders is a bit more daunting, but necessary for comprehension.
SDN: There are so many aspects to Metal, though; can you really cram them all into one book? Or do you perhaps just hope to give people enough of a cognitive framework so they have sufficient understanding to continue learning independently?
JC: My goal was to provide a solid foundation for people to build off of. I had wanted to write a book that explained everything, but I realized that would be like 3000 pages and no one would read it!
There are a few solid foundational concepts that you need to understand in order to use Metal that are incredibly difficult to grasp. I targeted this book on those foundational concepts. So there are a bunch of people who know more about Metal than I do who will probably be disappointed with the book because it tells them things they already know.
If you look at programming books, books that are targeted at beginners go over some really foundational concepts that we don't think about anymore because we've advanced to the point where those are simply given.
Metal is the same way. It has some foundational concepts that if you don't fully understand those, you aren't going to get anywhere. I tried to narrow the scope of the book to those concepts so that the reader would not be overwhelmed.
It is my hope that they can apply those concepts into whatever direction they are particularly interested in. Some people want to make game engines. Some want to filter live video streams. Some want to write neural networks. I hope that they get a solid enough foundation that they can explore those niches on their own.
SDN: Do you think people will own your book alongside the OpenGL Programming Guide and the Vulkan Programming Guide, or is this more aimed at Apple platform developers?
JC: One reason I wanted to target base concepts for the book is that I am hoping anyone who is interested in graphics will get something out of the book.
“My book is targeted at someone who is interested in graphics and knows some programming but doesn't really know graphics. Or neural networks.”
I have a chapter on the math concepts that I feel very proud of because I feel it does a good job of explaining how you build complexity using math you learned in high school and how all of those facts work together to show what you see on a screen. I am also proud of my neural networks overview.
I think that if you're already well versed in something like OpenGL you can probably figure out Metal without the help of my book. For those developers, they would need some familiarity with Swift and Cocoa in order to implement Metal.
So my book is targeted at someone who is interested in graphics and knows some programming but doesn't really know graphics. Or neural networks.
SDN: So the Cocoa side is unavoidable? I remember having a really hard time getting the hang of Cocoa Touch when I first started programming for iOS.
JC: Unfortunately yes. In order to render anything to the screen, there is a custom subclass of UIView that implements Metal. There are a lot of objects and structures you have to put together on the CPU side, which is in Swift and Cocoa, in order to connect to Metal shaders that are written in the Metal Shading Language.
One stumbling block I see from a lot of people is that they find those CPU side structures like render encoders and command buffers and argument tables to be boring. They want to dive right into writing shaders and assume the CPU side is just boilerplate they can copy and paste. Then they realized they don't understand how to get data into the shaders and how they connect to the CPU side and they get frustrated and quit.
Ironically, you can avoid a lot of the boilerplate Metal code and just write shaders, but only if you use and understand SceneKit to take care of all those pesky buffers and encoders for you. But that requires even more familiarity with Cocoa!
SDN: You’re not making Metal sound easy!
JC: Metal was not designed to be an easy implementation. It takes close to a hundred lines of code just to get a triangle to appear on the screen. If you want to have an abstracted graphics framework, you can use Core Graphics or SpriteKit or SceneKit. Metal is for people who enjoy nitpicking their bits and controlling every aspect of their pipeline.
SDN: It’s interesting that you mention SpriteKit and SceneKit – do you think there’s a benefit for games developers to get more familiar with Metal, or is it just easier to stick with those higher-level frameworks?
JC: I think that these high level frameworks deserve more credit than they get. I think when people think about graphics programming, SceneKit is more in line with what they think graphics programming should actually be. If you want a box, you can create one using the SCNBox() initializer.
“I think that being familiar with SceneKit helps tremendously with Metal.”
On the other hand, if you want one in Metal you either have to hard code the vertices or import a model file. You need to allocate memory for the vertex data. You have to set up a data structure to explain to the GPU what data type each bit it's being sent actually means. You need a correlating data structure in the shader file to match the one on the CPU side. Then you have to write a vertex and a fragment shader. There is no point in doing all that stuff when you can do the exact same thing with 3-4 lines of code.
I think that being familiar with SceneKit helps tremendously with Metal. I have been learning SceneKit after I finished the rough draft of the Metal book and I see all of the options for things like subsampling and mipmapping in the SceneKit editor. It really helps you get familiar with the concepts behind textures and materials without forcing you to also figure out how to cram those into a rendering pipeline. I think people would really set themselves up for success if they learn SceneKit.
SDN: Putting games to one side, I believe the latest iPhone X face scanning technology is also based on Metal. This goes back to what you were saying earlier about general purpose computing uses – presumably this opens up a whole new world for Metal developers?
JC: I was blown away by the facial mapping tech that the iPhone X uses. I am currently working on a contract where the developers had rolled their own facial mapping technology. The advances in machine vision and neural networks that open up possibilities for object detection and recognition are really astonishing.
We carry around super computers in our back pockets. I don't feel we're scratching the surface of what these devices can do. Right now a lot of this mapping and processing is being sent to servers because people want things to be cross platform, but these could totally be done on device.
We've had a lot of prominent data hacks recently and I think having the selling point that you own your own data because it can be processed on device is an underutilized selling point that will become more and more important as people become aware they must protect their privacy.
SDN: And the facial mapping is just one of the new features introduced in iOS 11 – Vision, Core ML, ARKit, and others must seem like a dream come true for Metal developers!
JC: I was incredibly grateful to see the introduction of things like Vision and Core ML at WWDC this year. Metal was introduced in 2014. I had some worries that when the book was finally released, it would land with a giant thud because no one would care anymore. I've seen things like TextKit get announced and then they're promptly forgotten.
Seeing how many sessions Apple dedicated to Metal and technologies that built on Metal really overwhelmed me and made me very happy. I think that Metal is going to become more and more important in the Apple ecosystem and I hope that my book helps open up that knowledge to more people so that we can have even more things in the Metal space.
SDN: Along with big changes to Metal and major new frameworks, your book is all written for Swift – a language not known for staying still very long. How are you finding Metal on Swift? Is it as much as a moving target as it is for UIKit and AppKit developers?
JC: I honestly don't think Swift is as big of a moving target as most people think that it is. I agree that it changes fairly rapidly, but if you're writing a book that is primarily about a framework, the underlying framework changes are far more of an issue.
“I honestly don't think Swift is as big of a moving target as most people think that it is.”
Swift is basically the glue that holds those framework calls together. If you try to write old Swift code, the compiler has generally been good about prompting you to fix the correct issue as it remembers what it used to be and that it has now changed.
Some people have tried getting around this by writing their books in Objective-C, but we now have a large crop of programmers who have never used Objective-C. I know many who won't buy a book written in it because they don't want to deal with trying to translate it in their heads. At this point, I don't either, to be honest.
The nature of our industry is that things change very quickly. I have a code repository of working projects that I intend to maintain and I am treating the code in my book as almost pseudocode with the intention that the reader will actually download and reference the project if they want to implement the code in a project.
SDN: That code repository sounds important. Could you let us know a little about your process for writing for such a complicated topic – clearly this is a difficult book to write! Where do you start trying to assemble such a repository?
JC: I replaced a previous author on the book who realized he couldn't complete it and dropped out without telling anyone. I inherited his book outline, but I wanted to replace it as I felt it didn't cover the right kinds of topics.
One thing that I am eternally grateful to the publisher for was that they insisted that I have an approved outline before I began writing the book. I was at something of a loss as to what I needed to write about because I didn't know what I didn't know.
I reached out to one of the technical reviewers for some help creating the outline and they basically wrote my entire initial outline along with sub-topics for most chapters. This was the greatest tool anyone ever gave me because it gave me a road map to find out all the things I didn't know about yet.
This outline was scary as hell because of the 20 chapters in the outline, I knew only about one or two of them. So instead of panicking, I focused on writing those chapters. Then I looked for the next easiest one. Then I looked for one that I was desperately passionate about writing.
I figured out that a few chapters were foundational chapters for more complex topics, so I didn't worry about the harder chapters until I finished the easier chapters. I made small amounts of progress every day, and it just built up over time.
SDN: And did that help make the task feel less huge?**
JC: It was easier to stay focused and motivated because I had a road map about all the things I needed to talk about. I could see progress and I could feel a sense of accomplishment every time I finished a chapter, or a quarter, or half of the book.
It also helped that it was a close-ended project where it would be "done" when I finished the last chapter. It was like running a marathon. If you don't have a plan about what you're going to accomplish, you can't pace yourself or be able to motivate yourself to finish because you don't know what finishing means.
Because of tight time constraints, I am just now working on the code for the repository. I am hoping that because I am only focused on the code I can complete it by the time the book is released.
SDN: Well, at least that means you can take advantage of any recent changes in Swift – you can even (gasp) refactor your Swift code thanks to Xcode 9!
JC: Huzzah! One nice thing about the GitHub repo is that you can update it with recent changes, and add new projects that you couldn't include in the book or weren't around when it was published.
SDN: Well, thanks for taking the time to answer all our questions. I have just one more to finish up: what’s your best advice for junior iOS developers?
JC: Don't get discouraged. You're learning something that is unfamiliar and complicated. You're not going to understand it the first time you hear it, or the second, and maybe not even the fifth. It doesn't mean you're stupid.
Our brains take a long time to learn a new concept. If you're interested in learning how to code, don't worry about what other people think. If you're tenacious and keep trying, you will get it – I promise.
Janie’s new book Metal Programming Guide: Tutorial and Reference via Swift is due for release in December, and is available for pre-order from Amazon.
TAKE YOUR SKILLS TO THE NEXT LEVEL If you like Hacking with Swift, you'll love Hacking with Swift+ – it's my premium service where you can learn advanced Swift and SwiftUI, functional programming, algorithms, and more. Plus it comes with stacks of benefits, including monthly live streams, downloadable projects, a 20% discount on all books, and free gifts!
Sponsor Hacking with Swift and reach the world's largest Swift community!
