Case 04 / Prompt quality, local by default

Sapa

A prompt-quality workbench and model research project that separates instant browser checks from an experimental fine-tuned model for deeper scoring and rewriting.

Write-up only. Source is private; prototype captures and evaluation are included below.

Role: Builder; product, data, and model development
Status: PrototypeWorking prototype; private source; write-up only
Period: 2026

Next.js
TypeScript
Qwen3-4B
MLX
Data QA

Key decision: Keep deterministic linting in the browser and treat the fine-tuned model as a separately evaluated research track, not as an invisible dependency of the current interface.
Evidence: Prompt linting in the browser, with a separately evaluated fine-tuned model for deeper analysis.

Sapa prompt composer showing a local prompt score, a specific issue, and an enhanced version. — The implemented composer runs immediate rules in the browser; the fine-tuned model remains a separate research artifact.

01 / Problem

Catch prompt defects before the model call.

Prompt defects are often discovered after a model call: ambiguous labels, conflicting instructions, sensitive data, missing output constraints, or context that is simply too large.

A useful assistant has to catch obvious failures immediately without sending the prompt away, while reserving a model for the cases that require semantic judgment rather than pattern matching.

02 / Architecture

Instant browser checks, separate model research.

The shipped workbench analyzes prompts with deterministic TypeScript rules and stores settings and history locally. In parallel, an offline pipeline collected, audited, labeled, and split a training corpus for full supervised fine-tuning of Qwen3-4B. The two paths are intentionally not represented as one finished production system.

System map / Sapa

01PromptText stays in the browser for linting
02Local rule engineStructure, safety, and clarity checks
03ComposerIssues, score, and deterministic fixes
04Offline model trackCurated data plus Qwen3-4B SFT
05EvaluationParsing, score accuracy, failure review

The implemented browser path and the Qwen3-4B research artifact are deliberately not presented as one production system.

03 / Decisions

The trade-offs.

Make obvious checks local and immediate

Instead of: Send every keystroke or completed prompt to a hosted model for a single all-purpose quality score.
Choice: Run structural, privacy, injection, and context checks in a deterministic browser rule engine.
Why it holds: The fast path is inspectable, inexpensive, and does not need to transmit the prompt.
Cost: Rules can identify known patterns but cannot reliably judge every semantic defect.

Fix the dataset before scaling the model

Instead of: Continue training on a larger first-version corpus despite inconsistent labels and generic rewrites.
Choice: Rebuild the corpus with schema validation, calibration, diversity checks, consistency rules, and a held-out test split.
Why it holds: The first 89,000-example attempt failed for data reasons. More training would have amplified those errors.
Cost: The revised dataset was smaller and required a much longer collection and review process.

Separate parseable accuracy from end-to-end reliability

Instead of: Quote the strongest score metric and hide malformed outputs behind a sanitizer.
Choice: Report conditional accuracy, recovery rate, and end-to-end accuracy together.
Why it holds: A correct score inside unusable output is still a product failure.
Cost: The headline is less flattering, but it identifies constrained decoding as the real next step.

04 / Build & evidence

A working composer and a small held-out evaluation.

Built

A Next.js prompt composer with inline issues, deterministic fixes, templates, analytics, security checks, and local browser storage.
A labeling and audit pipeline with strict JSON schemas, score calibration, source filtering, deduplication, and consistency checks.
A full-SFT Qwen3-4B training run using 22,933 training, 1,482 validation, and 1,480 held-out test examples.
Inference sanitization and a documented evaluation harness that keeps malformed-output failures in the end-to-end result.

Evaluation

The model evaluation sampled 43 held-out examples with a fixed seed and classified every output by the recovery method needed to obtain valid JSON.
Among 31 parseable outputs, 90.3% of scores were within one point of the expected score. Across all 43 examples, that rate was 65.1%.
Ninety-one percent of outputs were recoverable after stripping or repair; 9% fell back to a default. The sample is small and the report says broader rubric-based evaluation is still needed.

Supported result

The browser prototype demonstrates the product loop without making a model call for each edit.
The model research produced a concrete artifact, a reproducible pipeline, and evidence that output conformance, not only score calibration, limits current reliability.

05 / Limits

What this does not solve.

Current limitations

The implemented web app uses deterministic rules; it is not yet wired to the fine-tuned Qwen model.
The previous preview is inactive and the source repository is private. This write-up includes prototype captures and evaluation, not a public demo.
The model evaluation has only 43 sampled test cases, and local latency estimates remain targets rather than validated product measurements.

Next improvements

Add constrained decoding so valid JSON is guaranteed instead of repaired after generation.
Run a larger, rubric-based held-out evaluation across prompt types and adversarial inputs.
Define the product boundary between deterministic findings and model-generated suggestions before integrating the two paths.