From Deepfake Detection to AI Avatar Generation: Open-source Deepfake Detection Models from TrueMedia.org

Introduction

TrueMedia.org is a non-profit organization committed to detecting political deepfakes and supporting fact-checking efforts for the 2024 election. It was founded by Oren Etzioni, the former founding CEO of Allen Institute of AI, and its entire first year of operations was made possible by a generous donation from a single donor, Garrett Camp, the co-founder of Uber. In 2024, TrueMedia developed free-of-charge deepfake detection tools for news organizations, social media companies, investigative reporters, and anyone with internet access. Our tools empowered users to verify the authenticity of digital content. This technology is used not only by people in the United States but also by people all over the world.

Open-Sourcing Technology for the Public Benefit

The mission of TrueMedia is to protect truth and integrity online, prevent misinformation and disinformation, and therefore uphold democracy. The organization's primary focus is on developing advanced technologies to address these challenges. When Oren first shared the idea for this project, I was immediately excited— not just by the technical aspect of building ML applications from scratch, particularly leveraging generative techniques to identify generated content, but also because it felt like an opportunity to tithe professionally. After working for decades in profit-driven ventures, I wanted to give at least one year to a higher cause. I joined the project as its first ML engineer and later led the ML team. We started developing the app in December 2023 with remarkable velocity, driven by the pressing timeline of the 2024 election cycle. Through aligned effort, we built the initial product in just three months. A notable contributor that I want to mention is Michael Bayne, one of the most talented engineers I've ever worked with. He built the entire TrueMedia backend and infrastructure without compensation. His dedication set the tone for the project and inspired everyone involved. After the 2024 election, we decided to open-source the code. We hope that by open-sourcing our technology, we can empower others in the machine learning community to build on what we've created. I believe the tools and infrastructure we built offer significant value for anyone interested in building deepfake detection, media analysis, or more generally, multiple AI/ML model evaluation, or robust backend systems for apps requiring real-time data processing.

Open Source Components

Four (4) plug-and-play components that you can easily use in TrueMedia's open-source code:

1. ML models
https://github.com/truemediaorg/ml-models 2. MediaRes
https://github.com/truemediaorg/media-resolver 3. TwitterBot
https://github.com/truemediaorg/socialbot 4. TrueMedia Datasets
We're preparing the paper and dataset for release.

1. ML Models

We are open-sourcing all our models for deepfake detection across video, image, and audio modalities. The truemediaorg/ml-models repository has details on environment setup, dependencies, pre-training, fine-tuning, and deployment in production. Basically, we ran every available deepfake detection model from GitHub, selected a few promising candidates, fine-tuned these models, and tested them extensively in our production with user-uploaded data. The code we are releasing includes these enhancements, giving access to highly performant deepfake detection models. For the final prediction in the production, we deployed an ensemble of multiple models for each modality, using a voting-based approach based on the confidence levels observed in each model. An exciting opportunity of this code is that you can also create your own AI/ML evaluation app that uses ensemble methods to combine multiple API results. This isn't limited to deepfake detection; it's compatible with any AI/ML API.

Performance Metrics
Our models were benchmarked over time, achieving an accuracy exceeding 90%, with a precision of approximately 92% and a recall of around 75% on previously unseen user-uploaded video data. For benchmarking, we rely on user-submitted data, excluding the portion used for re-training our models. We are grateful and want to give credit to the original model creators, who released their models and allowed us to build upon their work. We hope our contributions will likewise benefit society. Here are the key features of the models we deployed.

Video Deepfake Detection Models

GenConViT: An updated implementation of the GenConViT model by Wodajo et al. It analyzes both visual artifacts and latent data distributions. Features: - Hybrid Architecture: Combines ConvNeXt and Swin Transformer for robust feature extraction, leveraging CNNs for local features and Transformers for global context. - Generative Learning: Uses Autoencoder and Variational Autoencoder to capture latent data distributions. - Frame-based video processing capabilities. - Streamlined training and testing scripts. - Docker deployment support. - FP16 precision support. FTCN: An updated implementation of the FTCN (Fully Temporal Convolution Network) model (Paper Link). The goal is to leverage long-term temporal coherence using a Temporal Transformer Network. - Uses temporal-only convolutions by reducing spatial kernel sizes to focus on temporal features, encouraging the model to learn temporal incoherence. - End-to-End Training Without Pre-training: The framework can be trained from scratch without any pre-trained models or external datasets. StyleFlow: By combining the StyleGRU module and Style Attention Module (SAM), it effectively captures temporal and visual inconsistencies (Paper Link). Features: - Style Latent Flow Analysis: Captures anomalies in the temporal changes of style latent vectors, highlighting suppressed variance in deepfake videos. - StyleGRU Module with Contrastive Learning: Encodes the dynamics of style latent flows using a GRU network trained with supervised contrastive learning to extract style-based temporal features.

Image Deepfake Detection Models

DistilDIRE: A lightweight version of DIRE, suitable for real-time applications. Instead of calculating DIRE images directly, DistilDIRE aims to reconstruct the features of the corresponding DIRE image forwarded by an ImageNet-pretrained classifier with one-step noise of DDIM inversion (Paper Link). - Utilizes a knowledge distillation approach from pre-trained diffusion models. - Achieves a 3.2x faster inference speed compared to the traditional DIRE framework. - Capable of detecting diffusion-generated images as well as those produced by GANs. UniversalFakeDetectV2: An updated implementation of the "UniversalFakeDetect" model by Ojha et al. for detecting AI-generated images. Key features: - Untrained Feature Space Utilization: Uses the feature space of large pre-trained vision-language models like CLIP-ViT, which are not specifically trained for distinguishing real from fake images. - Enhanced Generalization: Demonstrates superior performance in detecting fake images from a wide variety of unseen generative models, including diffusion and autoregressive models. - Simple and Efficient Classification: Uses nearest neighbor and linear probing methods in the pre-trained feature space, avoiding extensive model training and reducing computational overhead. - Streamlined codebase to allow training on large image datasets (100k+).

Audio Deepfake Detection Models

Transcript Based Detector: - Uses a speech-recognition model and an LLM to make predictions on audio veracity. - Predictions are based on language construction, narrative coherence, and factual alignment.

Learning from Real-World Deepfake Detection Deployment

Here's what we've learned about deepfake detection and what you can take away if you're considering building a similar system.

The Mother of Detection is Generation.

One of the biggest challenges we faced was the lack of high-quality, labeled training data. Real-world deepfakes were not only scarce but also lacked diversity in terms of manipulation techniques, demographics, resolutions, and other critical factors. Without a diverse and representative dataset, training reliable detection models was nearly impossible. To overcome this challenge, we turned to generative models to create synthetic training data. By leveraging a variety of generative models, we were able to simulate edge cases, such as: - Low-light conditions. - Diverse expressions matching utterances. - Diverse ethnicities. This significantly enriched our training data. Some generative models even allowed us to introduce controlled, incremental variations in manipulations—such as subtle morphing or blending artifacts—helping our detection models learn finer-grained features. This synthetic data proved to be a highly effective tool for improving our models' performance. By exposing the models to scenarios they might never encounter in real-world datasets, we enhanced their ability to generalize and detect nuanced manipulations. This experience highlighted a fundamental truth: "You cannot detect what you can't generate."

Balancing Speed and Accuracy in Video Deepfake Detection

One of our key achievements in video deepfake detection was the discovery and implementation of the GenConViT model, originally developed by Deressa Wodajo, a student from Africa. Despite its seemingly simple architecture, this model emerged as the top performer among our ML models when fine-tuned properly. Our talented ML engineer, Lin, optimized the model further to enhance its performance and reliability. However, GenConViT has limitations: it doesn't account for temporal dynamics. While it excels at analyzing individual frames, deepfakes often reveal themselves through inconsistencies in motion and transitions over time. To address this, we incorporated models like FTCN and StyleFlow, which analyze sequences of frames. These models leverage temporal inconsistencies to detect deepfakes more effectively. Video analysis poses unique challenges compared to image analysis. Processing videos means handling over 25 frames per second for clips that typically exceed 10 seconds, resulting in much higher computational demands and latency issues. This creates an inherent trade-off between real-time detection and accuracy. While one of our commercial partners opted to analyze every frame with significant latency, our approach with GenConViT involved strategically sampling frames to balance speed and accuracy. Looking ahead, we believe the following two areas are critical for advancing video deepfake detection: 1. Region-Agnostic Approaches
Many current deepfake detection models rely on face detection as a starting point. Future advancements could focus on region-agnostic methods that assess the authenticity of any part of a video, moving beyond the assumption that manipulations are limited to faces. 2. Multimodal Detection
We believe analyzing multiple modalities will offer promising opportunities. For example, in addition to visual analysis, audio analysis can be used to verify if the spoken content aligns with reality. We've begun experimenting with this by combining video spatial signals with audio transcriptions to assess feasibility.

Cost-Effective Image Deepfake Detection

The first deepfake image detection model we experimented with was DIRE. In early 2024, it was a relatively new model with a good number of citations, making it a promising candidate to try. The idea of DIRE was smart: it first reconstructs an input image using a pre-trained diffusion model and then compares the errors between the original and reconstructed images. The rationale is that generated (fake) images are easier to reconstruct than real images, making reconstruction errors a potential signal for detection. Initially, this method seemed promising. However, after deploying DIRE in production, we encountered a significant issue—it was extremely slow and required a lot of compute. To address this, our engineers CY and Yewon developed a distilled version called DistilDIRE. Despite this improvement, our experience ultimately led us to a different conclusion: the "reconstruct and compare" approach, while clever, proved unnecessarily computationally expensive. Other models achieved comparable performance with far less computational overhead. The reconstruction process not only consumed considerable time but also demanded substantial compute resources—raising the question: why rely on such an inefficient method? Our team's ML engineer, Hannah, found a more cost-effective model by fine-tuning the UniversalFakeDetect (UFD) model. UFD achieves superior deepfake detection performance with significantly less compute, eliminating the need for reconstruction entirely. Instead, UFD maps images into the feature space of a pre-trained CLIP-ViT model and applies simple classification techniques, such as nearest neighbor classification based on cosine similarity or a linear classifier on top of these features. This approach, though simple, has proven to be very effective. The success of UFD highlights the potential of utilizing pre-trained vision-language models for deepfake detection. For those entering this field, exploring ways to utilize large, pre-trained embeddings can offer substantial gains in both efficiency and effectiveness, making them a compelling alternative to more resource-intensive methods.

Balancing False Positives and False Negatives

Over the past year, we have identified several recurring patterns in deepfake detection. One of the challenges is the impact of media compression and noise on detection accuracy. Compression artifacts—such as pixelation and the loss of fine details—along with background noise, often lead to an increase in false positives. These are cases where genuine media is mistakenly flagged as manipulated, which undermines trust in our tool. High-resolution media significantly improves detection accuracy. With higher resolution, subtle artifacts—such as unnatural skin textures, inconsistencies in lighting reflections, or irregularities in fine details—become easier to identify. However, low-resolution media, which is common in real-world scenarios such as social media uploads or heavily compressed video streams, is not easy to detect. Adjusting the sensitivity of detection models has a critical trade-off. Lowering sensitivity to reduce false positives can cause models to overlook key indicators of manipulation, such as mismatched lighting reflections, unsynchronized mouth movements, or irregular blinking patterns. On the other hand, increasing sensitivity to catch these subtle signs of manipulation often results in legitimate media being falsely flagged as fake, perpetuating a cycle of trade-offs between false positives and false negatives. At TrueMedia, we addressed this challenge by implementing a voting-based ensemble approach based on the confidence of each model. Striking the right balance in model sensitivity remains a complex but essential challenge for the deployment of reliable detection systems.

The Inference

To serve our eight in-house detection models in production, I initially deployed all ML models directly on AWS EC2 instances using Docker and a Sanic server to handle asynchronous HTTP requests. Sanic was chosen for its capabilities in managing concurrent inference requests, with each model exposed through dedicated API endpoints (e.g., /predict/model1 , /predict/model2 ). This combination of Docker and Sanic was my first time trying it at TrueMedia, and it exceeded my expectations in terms of both stability and scalability. I highly recommend others try Sanic + Docker combo—it's good. Previously, I had been using Flask.

Octo AI: The Honest Inference Platform

While preparing to implement Kubernetes for autoscaling, Oren reconnected me with Luis, the CEO of Octo, who had been my professor in a computer architecture class seven years ago (for which I got an A!). There are countless inference platforms out there, but honestly, I don't get the appeal of most of them. For most use cases, engineers can handle it themselves using tools like Docker and Sanic. The only real work left is to implement autoscaling. If cost is a concern and you don't have a large user base, you might need a solution that scales pods down to zero when idle. However, in practice, most teams end up setting the minimum replicas to one anyway. Why? Because cold start times are rarely fast enough to avoid negatively affecting the user experience. So the primary value I see that most inference platforms offer boils down to two things: 1. Handling spikes during traffic surges. 2. Pricing advantages through bulk GPU reservations—something individuals or smaller teams don't get directly from AWS or GCP. What I appreciated most about Octo AI was its refreshingly honest and straightforward approach: no forced libraries installation or code refactoring. Many other services—like Runpod, Replicate, Baseten, and Coreweave—require you to install their libraries and modify your code to enable autoscaling. To me, these changes often feel more like vendor lock-in than a technical necessity. I also doubt these code modifications provide significant performance gains unless you use different hardware. To me, they are more about keeping you tied to their platform than delivering actual value. We had no interest in modifying our code for an inference service. After all, how would you switch to a different provider later? Octo kept it real and took care of autoscale entirely at a container level, requiring no changes to existing codebases. Ironically, this approach may have made Octo's customers less "sticky," but it gained our trust. We ultimately chose Octo and deployed all eight of our ML models into Octo's inference pods. The process was straightforward, and I would highly recommend Octo to any other ML team. But unfortunately, Octo AI was acquired by Nvidia last December, and the service is discontinued for now. I haven't yet found an alternative to container-level autoscaling. Octo was remarkable in the ML infra space with its transparent scaling, free from vendor lock-in or needless adjustments. I think there's an opportunity for any startup to step in and fill the niche that Octo has left. I worked with eleven amazing ML engineers on this project. Some of them were still in school just last semester but still gave it their all, working like full-time. I'm very grateful for their dedication over an extended period. A huge shoutout to Lin, Hannah, Alina, Arnab, Kevin, Ben, Iman, Max, Changyeon, Yewon, and Jongwook for their hard work and commitment. Several of these talented engineers are currently seeking new opportunities, and I wholeheartedly recommend them for hire. Their GitHub profiles can be found at truemediaorg/ml-models.

2. MediaRes (Res from Resolve): Unifying Media Extraction for Deepfake Detection

Mediares is a microservice designed to address a critical bottleneck in deepfake detection: extracting media URLs and saving them from a wide range of social media platforms, including TikTok, X (Twitter), Reddit, Google Drive, Instagram, Facebook, Truth Social, and more. It is one of the most important components of the TrueMedia app. Although it might sound simple, building a reliable and universal media extraction service is not easy. What makes MediaRes particularly impressive is its ability to handle the inconsistent ways different social platforms embed media. Only after media is resolved can ML models download and analyze them. Many services struggle with the diverse APIs and metadata of social networks. MediaRes overcomes these challenges by offering a unified and elegant interface for media extraction across platforms. In my opinion, very few services out there right now can match MediaRes in terms of scope and reliability. I've met a few startups in this field that couldn't navigate the complexities of media extraction—they often resorted to having users manually upload media, an approach that's impractical. For anyone interested in developing similar products, MediaRes will serve as a good example of success. Micahael Bayne wrote this entire system in Scala.

Challenges in Media Extraction and How MediaRes Solved Them

1. API Heterogeneity
Each platform (e.g., TikTok, Instagram, X) uses different API structures, authentication mechanisms, and response formats. - MediaRes unifies these disparate APIs into a single, consistent interface. - It also handles platform-specific rate limiting and manages API version changes across multiple platforms. 2. Media Embedding Variations
Social platforms embed media using a range of different techniques: direct URLs, iframes, JavaScript-based players, and even encrypted streams. - MediaRes implemented specialized extractors for each platform. - Integrated with yt-dlp for broader compatibility with lesser-known platforms. 3. Authentication Challenges
Handling platforms requiring logged-in states or OAuth-based authentication is challenging. - MediaRes managed complex OAuth flows, session persistence, and secure access to authenticated content. - AWS secrets management was used to store credentials. 4. Platform Restrictions
Many platforms implemented anti-scraping measures to protect their data. - MediaRes has fallback mechanisms when primary extraction methods failed. - YouTube was later excluded from the service because of its stringent terms of service. 5. Scalability Solutions
Managed concurrent requests towards multiple social media platforms.

3. Twitter Bot

The Twitter bot is one of the amazing features of TrueMedia. I haven't seen other startups implement something similar. This feature was developed because it's much easier to tag the Twitter handle @truemediabot than to copy the URL and then come to TrueMedia.org to paste the URL. The Twitter bot was built by a talented former Allen Institute of AI engineer, Michael Langan. It works amazingly well, and I've tagged many Twitter posts to see how our ML models perform. We were also building a deepfake evaluation dataset, and this bot proved to be the easiest tool for gathering data. With our paid access to the Twitter API, the system seamlessly integrates tagged tweets into the TrueMedia platform for analysis. The bot is written in Go. The core architecture has two main components: a Watcher and a Responder, both running as concurrent goroutines. - Watcher: Polls the Twitter API every 5 minutes to check for new mentions, working within rate limits. When it finds media-containing posts where the bot is mentioned, it submits them to TrueMedia's analysis API. - Responder: Runs on a 5-second loop to check for pending analyses, managing the bot's responses based on the analysis results. It implements different response templates based on confidence levels (low, uncertain, high). For local development, the system includes a test mode that simulates responses without actually posting to social media, making it easier to develop new features without requiring full API access. We open-sourced this bot as well. The system is actually designed to be platform-agnostic so it can be expanded to other social media platforms. For developers looking to build on this foundation, several expansion possibilities exist: - Platform Extensions: The model package is structured to support additional platforms. Implementing new platform handlers would primarily involve creating corresponding service packages and updating the Platform type. - Analysis Integration: The TrueMedia client could be abstracted to support multiple analysis services, allowing developers to plug in different media analysis backends. - Custom Response Templates: The response generation system can be extended to support different response formats or languages, potentially using template files instead of hardcoded messages. - Monitoring Extensions: The health check system could be expanded to include metrics about processing times, queue lengths, API rate limit usage, and so on.

4. TrueMedia Datasets

The heart of the matter—the dataset is the most important aspect of deepfake detection! We are in the process of submitting our dataset paper to academic venues, so we'll keep the details brief for now, as it hasn't been uploaded to arXiv yet. However, here are some key statistics: Video Data
- 44 hours of video content, consisting of 1,072 real videos and 964 fake videos. Audio Data
- 56.5 hours of audio, comprising 1,110 real audio clips and 710 fake audio clips. Image Data
- A total of 1,975 images: 1,208 real images and 767 fake images. - These images feature varying resolutions to simulate authentic media conditions. The data is contributed by TrueMedia's users and is exceptionally diverse. It spans 52 different languages and is sourced from 88 distinct websites, providing a rich representation of linguistic and cultural contexts. It incorporates state-of-the-art deepfake generation techniques and captures the complexities of "in-the-wild" data. This will make it an invaluable resource for evaluating deepfake detection systems in real-world scenarios. We'll provide a link to the paper once it's officially released!

Non-engineering Lessons

1. Market Realities

One of the most important lessons we learned was the disconnect between the perceived importance of deepfake detection and the market's willingness to invest in such solutions. For example, TrueMedia once conducted a pilot with one of the leading social media companies. However, the budget allocated for the project was surprisingly modest. This stood in stark contrast to my previous experience at Scale AI, where enterprises allocated massive budgets for training data in industries like self-driving cars or large language models. These customers had a clear and significant demand, with money readily available on the customer side. This highlighted a critical market reality: enterprises had not yet prioritized or budgeted for deepfake detection. For a company to achieve unicorn-level growth, I believe strong market pull is essential—major enterprise customers need to have dedicated budgets for the solution being offered. However, we learned the following challenges: - Social Media Companies were Ambivalent about Deepfake Detection: Deepfake content often increases user engagement, making it more entertaining for audiences. Additionally, as long as the content did not cause public harm, it is free speech. - Reliance on Community Fact-Checking: Platforms relied on community-driven fact-checking tools (e.g., comments or initiatives like Twitter Notes) to moderate content. These methods were free and often fairly reliable. - The real threat was personalized scams (like fake emergency calls requesting wire transfers) where community verification wasn't possible.

2. Source Authentication vs. Detection

The key insight we've learned is that detecting deepfakes after they've spread (downstream) is a lot more expensive (you need a lot of models!) than authenticating content at its source. Instead of focusing on downstream detection, it is much more economic to apply watermarking and authentication technologies when content is first created. By doing so, we can prevent the spread of deepfakes rather than attempting to identify them after the damage is done. One proposed solution for source authentication is the Content Authenticity Initiative (C2PA). However, the initiative has stagnated. It remains outdated and not really maintained, with founding members having disengaged from the project. Also, there are no meaningful incentives for companies to adopt or comply with the standard. No stakeholders are actively championing its advancement, leaving C2PA as a symbolic effort rather than an effective solution. This gap between concept and implementation highlights a challenge: while promising concepts exist, there is a lack of organizational incentives to ensure their effective implementation.

Oren

Over the past year, I genuinely looked forward to our weekly ML team meetings with Oren. I believe time is finite, making it crucial to be intentional about who you work with—whether it's your team, employees, or vendors. Oren was the reason I joined this project when nothing was built yet. What impressed me most about Oren was his sharp, clear thinking. No fluff, no nonsense—just pure clarity. He's also very honest. For example, he once told me directly, "Aerin, your tense is all over the place." Most people would avoid saying something like that out of politeness, but Oren doesn't shy away from telling it like it is. I really love that about him. Another thing I deeply respect about Oren is his obsession with the technical aspects of deepfake detection. Typically, CEOs tend to focus more on business matters, becoming less and less involved in the technical side. But not Oren. He benchmarked models daily, dove into the technical details, worked closely with us, and provided critical feedback. In fact, he was the one who came up with the vote-based ensemble method using the confidence of each model. He actively experimented with our models every single day and consistently pushed us to enhance model performance. This year, Oren turned 60. I've worked with CEOs in their 20s, and yet, Oren's energy, passion, and drive could easily match theirs. It's incredible to see someone stay so sharp, so committed, and so full of energy at 60. Oren shows how experience and passion can create an exceptional leader.