The Facebook insider building content moderation for the AI era

When Brett Levenson left Apple in 2019 to lead business integrity at Facebook, the social media giant was in the thick of the Cambridge Analytica fallout. At the time, he believed better technology could fix Facebook’s content moderation problem.

He quickly learned the issue ran deeper. Human reviewers were expected to memorize a 40-page policy document that had been machine-translated into their language. They had about 30 seconds per piece of flagged content to decide not only whether it violated the rules, but what action to take: block it, ban the user, or limit its spread. According to Levenson, those rapid decisions were only “slightly better than 50% accurate.”

“It was kind of like flipping a coin, whether the human reviewers could actually address policies correctly, and this was many days after the harm had already occurred anyway,” Levenson told TechCrunch.

That delayed, reactive model is increasingly unsustainable in a world of well-funded adversarial actors. The rise of AI chatbots has compounded the challenge, with high-profile incidents involving chatbots providing teens with self-harm guidance or AI-generated imagery evading safety filters.

From policy documents to “policy as code”

Levenson’s frustration led him to develop the idea of “policy as code,” a way to convert static policy documents into executable, updatable logic tightly coupled to enforcement. That concept became Moonbounce, which has raised $12 million in a funding round co-led by Amplify Partners and StepStone Group.

Moonbounce provides an additional safety layer wherever content is generated, whether by users or by AI systems. The company has trained its own large language model to analyze a customer’s policy documents, evaluate content at runtime, respond in 300 milliseconds or less, and take action.

Depending on customer preferences, that action may involve slowing distribution while content awaits human review or blocking high-risk material in real time.

Scaling moderation across AI platforms

Moonbounce focuses on three primary verticals:

• Platforms dealing with user-generated content, such as dating apps
• AI companies building characters or companions
• AI image generators

The company supports more than 40 million daily reviews and serves over 100 million daily active users across platforms, according to Levenson. Customers include AI companion startup Channel AI, image and video generation company Civitai, and character roleplay platforms Dippy AI and Moescape.

“Safety can actually be a product benefit,” Levenson said. “It just never has been because it’s always a thing that happens later, not a thing you can actually build into your product. And we see our customers are finding really interesting and innovative ways to use our technology to make safety a differentiator, and part of their product story.”

Tinder’s head of trust and safety recently described how the dating platform uses LLM-powered moderation services to achieve a 10x improvement in detection accuracy.

Lenny Pruss, general partner at Amplify Partners, said in a statement: “Content moderation has always been a problem that plagued large online platforms, but now with LLMs at the heart of every application, this challenge is even more daunting. We invested in Moonbounce because we envision a world where objective, real-time guardrails become the enabling backbone of every AI-mediated application.”

Rising pressure on AI companies

AI companies face growing legal and reputational pressure as chatbots are accused of pushing teenagers and vulnerable users toward suicide, and image generators such as xAI’s Grok have been used to create nonconsensual nude imagery. As internal guardrails falter, safety has become a liability issue.

Levenson said AI companies are increasingly seeking external partners to strengthen their safety infrastructure.

“We’re a third party sitting between the user and the chatbot, so our system isn’t inundated with context the way the chat itself is,” he said. “The chatbot itself has to remember, potentially, tens of thousands of tokens that have come before… We’re solely worried about enforcing rules at runtime.”

Next step: Iterative steering

Levenson runs the 12-person company with former Apple colleague Ash Bhardwaj, who previously built large-scale cloud and AI infrastructure across Apple’s core offerings.

Their next focus is a capability called “iterative steering,” developed in response to cases such as the 2024 suicide of a 14-year-old Florida boy who became obsessed with a Character AI chatbot. Instead of issuing a blunt refusal when harmful topics arise, Moonbounce’s system would intercept and redirect the conversation, modifying prompts in real time to guide the chatbot toward a more actively supportive response.

“We hope to be able to add to our actions toolkit the ability to steer the chatbot in a better direction to, essentially, take the user’s prompt and modify it to force the chatbot to be not just an empathetic listener, but a helpful listener in those situations,” Levenson said.

Asked whether his exit strategy might involve an acquisition by a company like Meta, Levenson acknowledged how well Moonbounce could fit into his former employer’s technology stack, while also noting his fiduciary duties as CEO.

“My investors would kill me for saying this, but I would hate to see someone buy us and then restrict the technology,” he said. “Like, ‘Okay, this is ours now, and nobody else can benefit from it.’”