NICKMU.COM
These webpages have not been modified for mobile experience. They are designed for wider displays.
CONNECT 
WITH NICK

When GenAI is replacing everything,
how can we identify the point at which human creativity intersects with generated content?

STACK

Revamping the Video Post-Production Process with LLM

Team

Yining Li
Nick Mu
Brian Seong
Yan Zhu

Timeline

Sep 2023 - Mar 2024

Contribution

UX/UI Design
User Research

Overview

In September 2023, Meta tasked us with researching and developing a tool to streamline the creative process for video content creators, within a 6-month timeline.

The result is STACK — a project designed to transform the video post-production process. Tailored specifically for amateurs and early-stage professionals, STACK introduces a newly designed workflow that reimagines how users approach post-production, making it more accessible and efficient for creators.

Credit

Created at UW, supported by Meta, with gratitude to our mentor Kristie Tsao.

STACK - Brief Introduction

Starting Point

The Era of Short Video
The Challenge

The Future of Video Creation
To research and develop systems and tooling -- utilizing emerging technologies (such as GenAI) -- to help Video Content creators more easily produce high-quality, engaging, content.

Our Approach

We conducted 2 months of research, both secondary and primary, followed by 4 months of designing, prototyping, and iterating to develop the final MVP.

Discovery — Research Phase

During the first two months, we explored various research methods to investigate the research question: What are the primary challenges users face in the video content creation process that have potential business value and are technically feasible?

After conducting secondary research, we found that the main challenges users face are in post-production. We identified amateurs and early-stage professionals as our primary user group and focused on improving the video post-production process.

The interviews and contextual inquiries uncovered additional workflow issues in real-world settings.

Key Findings

Diverse Workflows

Video creation processes differ significantly between amateurs and professionals, with amateurs facing more challenges in navigating the pre-production and post-production phases.

Common Barriers

Common barriers include scripting, selecting appropriate video assets, and editing, particularly for early-stage creators.

Disconnect in Tools

Amateurs often lack access to practical guidance and tools for crucial moments in the creative process, like scriptwriting and video asset management.

Innovation — Design Phase

During the 3 sprints, we design, evaluate, and iterate three times in response to the design question.

Prototyping

We delivered an MVP iOS app for testing and technical evaluation.

Solution & Impact

Vision and Value

We want to emphasize human creativity in our solution.  We are striving to find the optimal balance point between 100% human-made and 100% AI-generated on a scale, maximizing creators' creativity.

Our goal is to develop products beyond chatbots and copilots.

Solution
Impact

Natural Language
Video Searching

A more user-friendly searching method powered by LLM.

Block Editing

Make video editing more accessible through newly designed interactions.

AI Storyboarding

Incorporating GenAI into the creative process to enhance human creativity, rather than replacing it.

Research: What are the primary challenges users face in the video content creation process that have potential business value and are technically feasible?

We began by examining the overall research question, then narrowed it down to formulate a specific design question.

Exploring Video Creation Ecosystem

Understanding the Problem Space

During the first week, we conducted research to understand the current space we stand, assess our technological capabilities, analyze the competitive landscape, and identify key stakeholders' need. We also conducted research on user grouped by audience size, platforms, and genre to gain a comprehensive understanding.

Understanding
the Challenges in Video Creation

We categorized the frequent occurred challenges by dividing them into stages and steps.

Early-stage Findings

After exploring the video creation ecosystem, it is clear that most creators are amateurs or in the early-career professionals with limited audiences. Their primary need is to saves time and receive assistance in the post-production. This need is consistent across various content genres and platforms.

Other findings include similar average earnings per view on different platforms and varying purposes among creators of different audience sizes.

Next

After conducting secondary research, we plan to meet with users in person to observe their workflow and gather firsthand findings. This will help validate our previous research and test our new ideas.

Dive Deep into Real-world Workflow

Engage with stakeholders to address gaps, align ideas, validate findings, and gain insights.

Understanding of Creators and Their Work Processes

Interviews and Contextual Inquiries

After conducting secondary research, we noticed that several key elements were missing

Missing Elements

To what extent does the video production process differ among creators with varying levels of experience, genres, and audience sizes?
‍
What are the specific needs and challenges of creators based on their level of experience, genre, and audience size?
‍
What emotional and practical challenges do creators encounter during the video production process, and how can tools better support them?

To explore this, we conducted 1 contextual inquiry and 7 interviews with a diverse group of participants, including 2 amateurs, 3 early-stage creators, 2 professionals, and 1 subject matter expert (SME).

Mapping Findings

The areas where clusters emerged indicate the key pain points, the post-production stage, which we have identified as our focal points for further investigation.

Following the interviews, we organized the insights by mapping them into the video production process.

What Did We Find

1

When we met with creators at various levels to map the user journey, it became evident that video creation processes vary greatly between amateurs (early-stage professionals) and professionals. Creators don't follow a strict formula, and the process of creation is more like managing a messy situation than following a recipe.

2

Our research highlighted a key disconnect in the creative process for amateur creators: while they are often focused on the production phase (capturing content), they tend to struggle with the equally important stages of pre-production (scripting) and post-production (editing). This is primarily due to a lack of filming tips and insufficient support from existing tools at crucial moments.

3

The main blockers to starting are script writing, selecting appropriate video assets, and editing the video.

Can't Boil the Ocean

Quick Evaluation

Given the time constraints, we conducted 6 rapid semi-structured interviews and a survey to quickly verify our findings.

Brainstorming

Throughout the video production process, there are multiple blockers and pain points, including scripting, sorting video assets, video editing, and video searching. Our brainstorming session generated various ideas to address these challenges.

Focus

After assessing development capabilities and timelines, we decided to focus on video asset indexing and searching, guiding the design and development of an app tool aimed at enhancing the video creation experience for users.

Design: How can we help amateur video creators efficiently get relevant footage from their extensive video asset libraries during post-production?

Reframing the Challenge into a Design Question                                                                                                                                                                                                                                                                     

Building the fundations

Prototype 1.0

Building a Feasible Product

Backend Development

Given that this project was intended for launch in a short-term period of time, it was essential for the design and development to be in sync with both the business objectives and the technical capabilities of our team. We prioritized developing the core functionality (backend) while simultaneously working on a design prototype, ensuring that our solution was aligned with the timeline.

Building a Desirable Product

User Flow

In the first version of wireframe, we established the basic architecture of the app, focusing on its core functionality. We created user flow with four primary functions. Users could import, search for video assets using natural language, and export. The searching and video indexing functions are powered by integrated LLM API.

Core Feature Wireframe

Feedbacks from Meta Glasses Team

In the mean time, we interviewed Shreyas Sundararaman, Product Manager at Meta Reality Labs. His insights helped us refine our design direction by focusing on personal branding in video creation, addressing gaps between amateurs and professionals, and exploring the role of AI in the video creation process. We appreciate his time and valuable input.

Integrating into workflow                

                                                             

Ideation and Prototyping

Integrating Our Solution into Current Workflow

Our focus was on embedding the tool seamlessly into the existing workflow, rather than creating a standalone app. By pulling concepts of each process into a workflow concept board, we improved our design.

The challenge became to design a intuitive process combines scripting, searching, sorting, and rough editing.

Not Another Chatbot App

We start by focusing on the core feature of video content indexing and natural language searching. From there, we brainstorm new ways to interact with LLM without just replicating a chatbot design.

Ideation

We created wireframes to streamline the existing video creation tools by identifying common elements as a starting point.

Building Block
Design Concept

Taking inspiration from the concept of 'blocks' in Notion, we simplify the post-production process by making it as straightforward as assembling building blocks.

Iterations                                                            

Three Sprints

Prototype Iterations

We refine our design over three sprints by designing, evaluating, and iterating.

Homepage

We updated the homepage based on evaluations to improve user interactions.

v1: Draft

v2: Function-oriented

v3: User-centered

Library

The search function was relocated to the Library page, making it easier to find.

v1

v3: New Visual Branding

LLM-Powered Block Editing
Scripting + Searching + Storyboarding
Sorting

v3.1

v3.1

v3.2: New Visual Branding

Preview

v3.1

v3.2

Evaluations

We refine our design over three sprints by designing, evaluating, and iterating.

Evaluation: First Sprint

In the first sprint evaluation, we conducted a cognitive walkthrough to identify early usability issues,  These insights highlighted the need for clearer information architecture, improved iconography, and enhanced guidance for key features.

To be improved 1.0

Confusion around the app's structure
Unclear icons
Lack of clarity in feature functionality

Evaluation: Second Sprint

We conducted a usability test with video hobbyists to evaluate prototype improvements, focusing on task completion and clarity of interactions. The evaluation revealed challenges with ambiguous terminology (e.g., “save as a project” and “add new media”), unclear distinctions between video storage states (local vs. cloud), and inconsistencies in search functionality, leading to recommendations for clearer hierarchy, standardization, and improved keyword-based search options.

To be improved 2.0

Ambiguous terminology
Unclear distinctions
Inconsistencies in functionality

Evaluation: Third Sprint

We focused on usability testing and heuristic evaluation to refine the app’s interaction flow and terminology. The findings revealed user confusion with timeline navigation, unclear video selection operations, and a lack of guidance on adding videos, prompting redesigns for improved intuitiveness and onboarding support.

To be improved 3.0

Confusion with timeline navigation
Unclear video selection
Lack of guidance on adding videos

Visual Identity and
Design System

In response to the feedback from users, we revisited our visual language and established our draft design system. We use Apple Design Guidelines as the foundation of the design system. We took inspiration from the visual style of the Meta Glasses app, adopting a black color scheme to better emphasize the video assets.

An Example of Our User Testing Method

The second round of user evaluations was conducted with 6 participants.

Scenario 1

Scenario 2

Scenario 3

PROMPT

After returning from a trip to Las Vegas, where you captured stunning sunset videos at Bryce Canyon, you recalled filming New York sunsets in the past. Now, you are planning to create a compilation video of sunsets.

PROMPT

You have been working on your Seattle vlog for a while now. Recently, you captured two perfect videos to include at the end. You are ready to combine them with your earlier footage and export everything to CapCut for editing.

PROMPT

You have collected all the assets for your new vlog and are now ready to organize them for detailed editing in the editing software.

TASK

Create a project about 'sunset' and add videos to it.

TASK

Add videos from the cloud library page to your 'Seattle' project and preview it in its entirety. Then output to CapCut.

TASK

Move blocks in the project to roughly edit it. Share with a friend.

CHECKPOINTS

Create a video project showcasing sunsets, beginning with footage of the sunset at Bryce Canyon
‍
Incorporate additional sunset videos captured in New York to enhance the project

CHECKPOINTS

Add the cave and sunset videos from Nov 22, 2023 to the project Seattle
‍
Preview 'Seattle' project

Output to CapCut for further editing

CHECKPOINTS

In the sunset project, move the blocks

Take a look at the final video, and share it with a friend via message