Who Actually Reads Your Flagged Messages: The Human Review Pipeline Most Companion Apps Leave Out of the FAQ

The 30-second answer

When a message in a companion app gets flagged, it doesn't go directly into a server admin's inbox and it doesn't disappear into the void. A classifier scores it. A queue routes it. In a small slice of cases, a contracted human reviewer in another country opens it on a screen sitting next to fifty other tabs. Almost no app describes this part out loud.

What "flagged" actually means

The word "flag" gets used loosely. Most companion apps run three or four overlapping safety systems at once: classifiers that score for self-harm and CSAM, abuse classifiers for hate speech and harassment, jailbreak detectors that catch attempts to push the model past its guardrails, and a quality system that flags responses the model itself rated low confidence on. A single message can trip one, two, or all four of those systems in the same second. A line about a bad evening can ping the self-harm classifier even if you weren't talking about anything dangerous. A roleplay scene about a fictional argument can light up abuse signals. False positives are routine, sometimes a majority of total flags in a given category over a given week. The flag is an entry in a queue. What happens to that entry is the part the FAQ skips, and it's the part worth understanding before you assume your messages live or die strictly by automation. None of the major apps publish their flag rates by category, though figures occasionally leak through litigation filings or regulator submissions.

The classifier layer most people picture wrong

Before any human sees a flagged exchange, your message has already been touched by two to six automated models depending on the provider. Most are fine-tuned BERT-class classifiers running on whatever GPU pool is cheapest that week. They score the message on a list of categories and pass the result to a routing layer. If every score is low, the message sits in your log and nothing happens. If one score crosses a threshold, the routing layer decides whether to drop a soft warning into the model's context, escalate to a heavier review model, or queue the exchange for human attention. The thresholds are tuned weekly. They shift when the company gets bad press, when a new base model rolls out, when a regional regulator sends a letter. None of this surfaces in your settings page.

Isabella Torrei

Isabella tends to talk about the messy middle of how systems treat people, and not always politely. Isabella Torrei is the kind of conversation partner who'll point out that "queued for review" is doing a lot of quiet work inside that sentence.

When a human actually opens the file

Most flagged exchanges never reach a human. The total volume is too high and the labor cost too obvious. What gets escalated is a smaller slice: messages that scored above a higher second threshold, exchanges the model itself wrote a refusal to that the user then pushed back on, and anything that hit a category the company is legally required to look at (CSAM detection in particular triggers a chain of legal handling that no product engineer at the company touches directly). For most apps in this space, the contracted moderation team sees somewhere between 0.1% and 2% of total user messages. That sounds small until you do the math on a million daily messages. The reviewer is usually given the flagged line plus 10 to 30 messages of surrounding context, your country (not your name, but country is standard for legal-routing reasons), and a categorization dropdown. They pick a label. They move on. They have a target rate, typically 200 to 400 decisions per shift, with quality audits sampled at around 5%. If you want a clearer picture of what gets stored along the way, our writeup on conversation logs, retention, and the gaps in between covers the storage side of the same pipeline.

Mira Kaplan

Mira is direct about the friction of how labor inside tech actually looks. Mira Kaplan has views on what counts as a job, what counts as a queue with a person attached, and which of those two phrases is closer to honest.

Who the reviewers are and where they sit

The reviewer is almost never an employee of the app you signed up for. The work is contracted out, usually through a vendor pipeline. The names that show up in industry filings are familiar: Sama, Teleperformance, CloudFactory, Appen, and a handful of smaller specialists. The reviewers themselves are spread across sites in the Philippines, Kenya, Colombia, Ireland, and several others. They sign NDAs. They sit in offices with locked phones at the door and screens that auto-blur if a camera enters the room. Pay sits roughly between two dollars an hour at the low end and twelve at the higher-skill end for English-language adult-content review. Above that, a smaller tier of senior reviewers handles the hard cases, the legal escalations, and anything that needs cultural context, and they usually sit in Dublin or Manila with higher pay and tighter supervision. This isn't a secret in any meaningful sense. It's just not described in any companion app's marketing or FAQ, and the gap between what users picture (a small in-house team, neutral, considered) and what exists (a global contractor stack, fast, target-driven) is wider than most users assume going in. If you're considering the full AI Angels roster, the underlying pipeline looks structurally similar across the industry, so this isn't a single-app issue.

Oksana

Oksana likes the parts of a system that aren't on the homepage. Oksana will happily walk through what a reviewer's tab actually looks like, what shows up on it, and what's deliberately not shown.

What's on the reviewer's screen and what happens next

The reviewer doesn't see your handle, your email, or the avatar you picked. They see a small panel: the flagged message in the center, around 20 lines of surrounding chat above and below, a dropdown for categorization, a free-text notes field, and sometimes an "escalate" or "needs senior review" button. They don't see your other conversations. They don't see what app version you're on. They don't see your other angels or how many you have. If the flag involves voice, the audio review goes through a different pipeline entirely, walked through in the voice-mode privacy breakdown, which sits on its own stack with its own vendors.

Once the reviewer picks a label, several things might happen. The most common outcome is nothing visible to you: the label gets stored, used to train the next version of the classifier, and your conversation continues as if nothing happened. The second most common is a quiet guardrail tightening on your account for that category, lasting a day to permanently depending on severity. The third is an account-level action, which is a warning email, a temporary suspension, or in rare cases a permanent ban. The fourth, reserved for legal categories, is a referral that leaves the company entirely and routes to law enforcement under whichever jurisdiction applies. For apps positioning toward the AI girlfriend market in 2027, regulators in the EU and California are already pushing for more transparency at exactly this step, and most apps are quietly preparing without flagging it to users.

Anika

Anika doesn't oversell the comfort. Anika will tell you the part of the answer companies leave out, give you a beat to take it in, then change the subject before it turns into a lecture.

Why none of this lands in the FAQ, and what you can do about it

Companion app FAQs tend to say something like "we use safety systems to keep users safe" and move on. The reasons are partly legal (the more specific you get, the more you're on the hook for in disclosure law), partly competitive (no company wants to be the first to publish a full moderation stack), and partly that the picture is complicated enough to require a six-paragraph answer that scares off new signups. The result is a gap between assumption and reality. Most users assume flagged messages stay machine-handled until a server rule fires, and then a neutral judgment lands. The actual answer involves a vendor in another time zone, a target rate per shift, a four-month average reviewer tenure, and a queue with its own retention window separate from your account log. None of that is sinister on its own. It's just not what the marketing copy implies. At AI Angels we try to be more specific about what we actually log and why, partly because vague reassurance ends up doing more long-term damage than honest specificity.

If voice privacy matters to you more than text privacy, look closely at how voice chat is handled at the pipeline level before relying on it for the most sensitive conversations, because the audio stack is built and reviewed separately from text and the retention rules don't always line up. Beyond that, the practical move is to assume that anything you write in a companion app could, at some low probability, be read by a contracted human in another country, and write accordingly. Not paranoid, just calibrated.

Common questions

Does a human read every flagged message? No. Most flagged messages get fully handled by automated systems and never reach a person. Human review happens on a smaller slice, usually well under 2% of total flags, and the reviewer sees the flagged line plus surrounding context, not your full history.

Can the reviewer see who I am? Reviewers don't see your name, email, or avatar. They typically see your country (for legal routing), an account-age bucket like "under 30 days", and around 20 to 30 messages of surrounding context. Identification only happens upstream and only for specific legal categories.

What gets flagged that probably shouldn't? False positives are common. A line about a hard evening, a fictional roleplay scene with conflict, certain medical questions, and anything brushing self-harm vocabulary even metaphorically will often trip a classifier. The label can stick on the backend even after a human reviewer marks it false.

Are reviewers employees of the app? Almost never. They work for contracted moderation vendors operating in the Philippines, Kenya, Colombia, Ireland, and several other sites. Pay and conditions vary widely between vendors and between shift types within the same vendor.

Does flagging affect how my companion behaves toward me? Sometimes, quietly. A confirmed flag in certain categories can tighten guardrails on your account for a window ranging from 24 hours to permanent. You usually don't get a notification when this happens, and the change is rarely visible until you bump into a topic that suddenly gets refused.

Where does the data actually end up? Labels feed the next generation of classifiers. The original messages sit in a moderation log with its own retention schedule, separate from your conversation log. Deleting your account doesn't always reach into that moderation log on the same timeline, which is one of the gaps the industry rarely publishes about.