Reliable JSON from a Local LLM: Structured Output with the Prompt API

Free-text prompting is fine when a human reads the answer. The moment your code needs to parse the answer - a title here, a list of topics there - an LLM that occasionally wraps its JSON in prose or trails a stray comma becomes a reliability problem. Small on-device models are especially prone to it.

When OmniBrief extracts structured metadata for a brief - title, one-line summary, topics, named entities, keywords - it doesn't hope the model returns JSON. It constrains the model to a schema using the Prompt API's responseConstraint, so the output is machine-parseable by construction.

Part of the engineering behind OmniBrief.

Describe the Shape You Want

The schema is an ordinary JSON Schema object. It is the single source of truth for what a valid brief looks like:

const BRIEF_JSON_SCHEMA = {
  type: 'object',
  properties: {
    briefTitle:     { type: 'string' },
    oneLineSummary: { type: 'string' },
    topics:         { type: 'array', items: { type: 'string' } },
    namedEntities:  { type: 'array', items: { type: 'string' } },
    keywords:       { type: 'array', items: { type: 'string' } },
  },
  required: ['briefTitle', 'oneLineSummary'],
};

Constrain the Decoder

Chrome's on-device LanguageModel accepts the schema as a responseConstraint on the prompt() call. The model is then only allowed to emit tokens that keep the output valid against the schema - no prose preamble, no malformed arrays:

const session = await LanguageModel.create({
  initialPrompts: [
    { role: 'system', content: 'You extract research metadata. Respond with JSON only.' },
  ],
});

const raw = await session.prompt(buildBriefPrompt(text), {
  responseConstraint: BRIEF_JSON_SCHEMA,   // structured output
});
const brief = JSON.parse(raw);

Degrade Gracefully

Not every build supports responseConstraint. Rather than fail on older engines, OmniBrief retries without the constraint - the system prompt still asks for JSON - and only then parses defensively:

async function promptForMetadata(session, prompt) {
  try {
    return await session.prompt(prompt, { responseConstraint: BRIEF_JSON_SCHEMA });
  } catch (err) {
    if (/too large|quota|token/i.test(String(err?.message))) throw err;
    return session.prompt(prompt);   // older builds: no responseConstraint
  }
}

The caller wraps JSON.parse in a validator that checks the required fields exist before the brief is trusted - belt and suspenders for the fallback path.

Conclusion: Make the Model Obey the Type

Treating the schema as a hard constraint rather than a polite request is what makes on-device metadata extraction dependable. With responseConstraint, a graceful fallback, and defensive parsing, OmniBrief turns a small local model into a reliable structured-data source - no regex scraping of half-formed JSON required.