Model filters versus app filters

Model providers may use several techniques:

• curate and filter training data;
• tune the model to follow safety policies;
• teach refusals and safer alternatives;
• run classifiers before or after generation;
• evaluate behavior against adversarial test sets;
• monitor broad abuse patterns and update mitigations.

These controls can reduce risk across many customers. They also have limits. The provider does not know every product rule, user age, regional requirement, retrieved document, tool permission, or downstream action. Provider behavior can change with model versions. Different languages and unusual contexts may perform unevenly. The model can misunderstand a benign request or comply with a harmful one.

Treat provider safeguards as a documented dependency, not an invisible guarantee.

An input filter examines material before the main model handles it. It can detect clear policy violations, strip unsupported file types, limit input size, identify likely private data, or route a request to a special flow. Input filters can reduce exposure, but they should not rely on a keyword list alone. Benign educational text often contains the same terms as disallowed requests.

An output filter checks generated material before display or action. It may classify content, redact personal identifiers, enforce a structured format, or prevent a tool call. Output checks matter because an allowed-looking request can still produce an unsafe answer, and retrieved data may introduce unexpected content.

Application controls can also include:

• age and account settings;
• tool allowlists and spending limits;
• retrieval permissions and source controls;
• rate limits and abuse detection;
• warnings, blurring, and click-through friction;
• reporting, appeals, and human-review queues;
• logging with privacy-aware access and retention;
• release gates, monitoring, and rollback.

Many of these are not “AI filters.” They are ordinary product and security engineering, which is precisely why they are dependable.

Consider a study assistant receiving a message.

1. Access layer: Confirm that the user may use the feature and apply the correct audience policy.
2. Input processing: Scan attachments, remove unsupported data, and classify obvious high-risk content.
3. Context assembly: Retrieve only sources the user may access. Mark retrieved text as untrusted data.
4. Model instruction: State the product’s role, allowed help, boundaries, and preferred safe alternatives.
5. Generation: Ask the model for a bounded response.
6. Output processing: Check for policy violations, private data, unsupported links, and required formatting.
7. Action gate: If the response would publish, message another person, or use a tool, apply permissions and review.
8. Feedback loop: Record an appropriate event, provide reporting or appeal, and use reviewed cases to improve tests.

Each layer catches different failures. Duplicating one weak classifier eight times is not true defense in depth.

Use deterministic code for facts the application already knows. If a child account must not receive direct messages from strangers, enforce that in permissions rather than asking a model whether a message “seems okay.” If a tool may only read records, do not grant write access and hope the prompt prevents edits. If output must exclude account numbers, apply structured data handling and redaction.

Use model-based classification where language and context require judgment, but plan for uncertainty. Store policy version, category, and decision reason when appropriate. Route severe or ambiguous cases to humans. Give reviewers enough context without exposing unnecessary personal data.

What happens when a filter times out or the moderation service is unavailable? A low-risk private drafting feature might continue with reduced functionality. A public-posting or real-world action flow might pause. The fallback should follow consequence, not convenience.

Also decide what happens when layers disagree. A provider may refuse content that your policy permits; the app cannot force an answer safely without understanding why. An app-level classifier may flag a response the model considered acceptable. Keep the stricter result temporarily for high-risk cases, but review patterns rather than accumulating unexplained blocks.

A visible warning or moderation badge is not evidence that the system works. Ask for measurable coverage:

• Which policy categories does each control address?
• What inputs and languages were tested?
• Where are the hard permission boundaries?
• How are disagreements and outages handled?
• Can users report mistakes and receive an appeal?
• Who can change thresholds or policies?