Alum Runze Zhang Talks Expanding Creativity Working on TikTok’s Effect House

Runze Zhang currently works at TikTok as Technical Product Manager, leading the visual scripting and interaction product for the augmented reality platform, Effect House, a tool for public creators to make TikTok AR effects. Previously working as Senior Interactive Engineer at TikTok, Runze developed one of the most viral effects Time Warp Scan (~20m TikTok videos published), and worked as Creative Technologist at Kilograph, shipping multiple client-facing ARVR apps in architectural visualization. A SCI-Arc alum, he was the Gehry Prize winner in 2018.

We spoke to Runze to discover the depths of his design sensibility as it relates to the intensive engineering and management of a groundbreaking AR platform for a tech and social media giant.

First of all, how do you form your concepts and modalities when choosing what projects you'd like to work on?

In general, I would prefer products over projects. I am not interested in a specific set of operations designed to accomplish a singular goal, and I tend to be curious to see what it is able to achieve if as many as possible parallel paths have been explored at the same time, thus a platform, application, and/or system interest me most. Among products I would choose to be involved in those that help build the digital infrastructure for content creation. I tend to dig deeper to find what is the source or “engine” that creates driving forces to shape what we are looking at and interfacing with today. In order to create a larger impact, I am more willing to contribute to building or learn how to help build future “engines” that unleash human’s creativity through their own craftsmanship.

Can you talk about TikTok Effect House and what that’s like to put forth as a designer right now?

TikTok Effect House is a mobile AR creator tool integrating engine, algorithm, and editor capabilities. It enables external creators to publish TikTok effects so that everyone on the platform can use them. Effect House is in public beta right now, we have a Discord channel, we have gained from several private beta testers till now about 50k beta creators in four months, some of them have already become super successful effect creators. And we just released Asset Library in collaboration with Sketchfab where creators can download 3D Assets directly from third-party resources and use them in Effect House with just one click without exporting and importing. We prioritize visual scripting so that creators do not need to worry about the syntax barrier from different programming languages. Many other features are on the roadmap including shader/material graph, VFX graph, etc.

To be able to deliver a large product like Effect House, we need to manage all the XFN from design/prototyping to research, to engineering, UX writing, compliance review, QA, release note review, and finally to shipping, making sure everything is followed up within the release cycle. I would say organizing and managing skills matter more, but in terms of assurance of the final product, I still need to and am happy to put myself forth as a designer. Since I am the owner of all the graphs, I need to monitor three parts of the design, product design, technical design, and system design. Product design contains things for the user-oriented UIUX for using all the graphs; technical design including all the API design of all the atomic nodes ranging from beginner to advanced levels, say, how it bridges between the user-facing UIUX and lower-level engine and algorithm code bases; and system design is, for example, how we design a backend resources distribution system logic for hot reload or auto parsing system for node assets sync between the in-tool and online documentations as well as multi-language distribution etc. Since visual scripting is intersecting most of the features in Effect House, I always need to discuss with other product/algorithm/SDK feature owners for a better design strategy and solution holistically.

How did you originally develop your workflow?

One mindset I have is to think of the workflow as a system not a single solution. When I am facing a problem where I need to actually develop a workflow, usually it is not a simple question but more of a complex product/project that requires systematic designs. Then I need a system to tackle a system. So, I often ask myself a lot of questions in the beginning before I even start: What is the ultimate impact scope I am trying to achieve? What is still missing in terms of my support system? How do I document in a way I will be able to go back in time and re-communicate/represent based on some of the past information? What is the workflow and what should I avoid and test? How long do I plan to spend on each iteration? How often should I review my roadmap and milestones? What is the exit plan for sustainability? What tools am I going to use? These help shape my sense of product scope and boundary, it gives me directions on how to be resourceful while monitoring what is still missing along the way and move forward fast. And by answering all these questions, I am already starting to build my metric systems that can help me focus and know where to stop.

What are you the most excited about right now in terms of what you're making and both in terms of the techniques and the tools that you're using and also the ideas that you're working with?

The most exciting thing is definitely having a larger impact and creating a product that tens of thousands of creators/developers are using. I was mainly using existing tools such as Unity Unreal to create a specific AR/VR product while I was working at Kilograph, and then when I first joined TikTok, we were developing in a lot of immature in-house tools to deliver individual specific TikTok AR effects while making the tools easier to use, and now I am the one who defines and owns a big chunk of the tool for everyone who wants to become an AR effect creator. To me, it is shifting from an end-user of an existing tool to contributing plugins for internal tools and then to making public tools/platforms myself. Now you look at those ARVR hiring roles description where it says: “you need to have XX years of experience in using Effect House,” and I am proud to say that’s our product.

Currently as my role shifts more towards leading and managing, I use a lot of managing tools. Before when I was still an Interactive Engineer for developing AR effects I used a lot of in-house tools, and the languages we used were Javascript/Lua, and also GLSL for shaders. And before that when I was Creative Technologist for Kilograph, I was using Unity, so C# and HLSL. Shader is definitely fun to play with. Now I am more excited about some other generative AI capabilities like text-to-image, text-to-3D-object, etc.

We already have released an early simplified version effect using text-to-image, and I am looking forward to implementing those algorithms in Effect House at some point, so that the impact could even be more amplified and everyone can unleash their creativity with AI-aided tools (I believe AI will just bring new digital craftsmanship and not killing creativity, but that’s another topic that can expand for another time).

How would you say what you learned at SCI-Arc or what you studied at SCI-Arc informed how you've created your career and what you've been able to do after leaving SCI-Arc?

Workflow-oriented design thinking was intriguing and benefited me a lot for all the professional roles I have been taking after leaving SCI-Arc, from Creative Technologist to Interactive Engineer to Technical Product Manager. Workflow design is one of the systematic thoughts that is so fundamental that it provides multiple perspectives just from oneself. And I did not usually think of workflow as part of the design project to begin with because I used to love being free and relying on my own talent from an artist perspective. It was so underrated that even though I knew the importance, I was never actually trained to spend time playing with different workflow procedures and thus form a knowledge of combining different digital tools strategically and wisely that can even unleash your unexposed talent or insights on things that will not be obtained otherwise. Workflow consists of multiple steps of copying, generating, removing information from last step to next, the sequence of steps not only changes design effects and results, but also guides design thinking and awareness.

What are you most looking forward to in terms of what you're working on now in your career and what you project going forward?

Now Effect House has over 100,000 users in the current beta version, and I am looking forward to expanding my scope of impact by reaching more creators to using Effect House, and see how the younger generations will use this tool after we add more advanced capabilities into Effect House, especially those AI and CV capabilities. You will never know how creative users can get and are willing to be, I remember when I first delivered the Time Warp Scan effect I didn't expect it to be so viral and so creativity-inspiring. It generated so many trends and topics around this little blue line effect.

A single filter/effect is able to allow secondary creativity to this extent, and we are building a platform that allows everyone to empower everyone, the potential is just endless. I am pretty sure, to me, seeing the creator ecosystem grow will be most elated in the years to come.