Integrating AI-Driven AR Filters and Auto-Framing into a Smart TV Singing App

1. Context

This initiative focused on integrating AI-driven AR filters and auto-framing into a smart TV singing application to create a more engaging and dynamic experience for users. The project was delivered in collaboration with an external app partner, with responsibilities shared between the platform team and the app provider.

AR filters were designed to enhance the visual experience by allowing users to customise their appearance with:

• Background effects such as dance floors, themed environments, and seasonal settings
• Face and prop filters including sunglasses, wigs, beards, and hats

Given prior experience with AR filter development on the platform, the internal team took the lead on specific design and implementation aspects, while the app provider focused on additional filters. A shared quality control and feedback process ensured all filters met the necessary visual fidelity and performance standards, particularly for large-screen displays.

2. Challenge

Key challenges included:

• Coordinating responsibilities between teams: Ensuring a structured approach to filter creation, with contributions from both the internal design team and the app provider while maintaining a seamless user experience.
• Ensuring high-quality, realistic AR effects: Unlike basic laptop webcam filters, the filters needed to be optimised for large-screen viewing, requiring:
  • Realistic motion dynamics, ensuring wigs and props responded naturally to head movements
  • Depth effects and floating objects to enhance background immersion
  • Seamless fit and alignment for props like beards and sunglasses to maintain a natural look
• Filter positioning and usability: Instead of removing less-used filters, the focus was on prioritising the most popular ones, making them easier to access.
• Seasonal updates: Establishing a sustainable approach for introducing seasonal and event-based filters, such as those for holidays or special occasions, while maintaining a high-quality standard.

3. Approach

To ensure a successful integration and long-term viability, a structured and phased approach was followed:

• Filter Development and Quality Assurance:
  • Defined team responsibilities, leveraging internal AR expertise while enabling the app provider to contribute their own designs.
  • Implemented a joint quality control process to ensure filters were optimised for large-screen viewing (avoiding pixelation or unnatural scaling), designed with realistic motion effects for natural responsiveness, and integrated with depth perception elements to enhance immersion.
• Pre-Launch Testing and Validation:
  • Conducted internal testing under varied lighting conditions and movement scenarios to verify filter performance.
  • Collaborated with engineering and UX teams to ensure smooth filter selection, transitions, and application without affecting app speed.
• Launch and Post-Launch Expansion Strategy:
  • Rolled out an initial core set of AR filters, followed by seasonal and event-based updates.
  • Used a data-driven approach for filter positioning: popular filters appeared earlier in the selection menu, seasonal filters were highlighted at appropriate times, and user engagement data informed future updates and feature improvements.

4. Role and Achievements

As the Senior Product Lead for AR Filters for the singing app, I:

• Led the strategic development and integration of AI-powered AR filters, ensuring a high-quality, immersive experience.
• Defined and managed the collaborative framework between the internal team and the app provider, ensuring consistency in design and performance.
• Optimised UX and filter placement to improve discoverability while enabling regular content updates.
• Established a structured roadmap for continuous refinement and scalable feature expansion.

5. Results

• Successfully launched AI-powered AR filters, enhancing the singing app with high-quality, customizable visuals.
• Implemented a workflow for ongoing filter creation and updates, ensuring consistent quality standards.
• Developed a scalable approach for seasonal and demand-driven filter updates without additional overhead for product teams.
• Maintained optimal app performance, ensuring that new filters did not negatively impact usability or system responsiveness.