The Rise of On-Device AI: Running Gemini Nano Locally in Google Chrome

Integrating artificial intelligence into developer extensions has historically meant routing text strings to centralized servers. While cloud LLMs provide vast knowledge bases, they introduce three complications for everyday workflows: network latency, monthly API fees, and critical data privacy questions. Having your page content or corporate bookmarks sent to third-party endpoints is a security boundary many organizations block.

With OmniBrief, our on-device research manager, we made Chrome's built-in Gemini Nano the default provider - one adapter among several (the others route to cloud models or a local Ollama/LM Studio server), but the only one that keeps everything on the device. When it is active, summaries, structured metadata extraction, and the "Ask AI" chat all run on-device, with no network calls.

Part of the engineering behind OmniBrief.

Two Globals, No window.ai

Chrome's built-in AI shipped as two standalone global classes - Summarizer and LanguageModel - not the older experimental window.ai namespace that early previews used. Inside an extension they are available in every context (service worker, side panel, offscreen document) with no extra manifest permissions, so OmniBrief simply feature-detects the globals:

function nanoSupported() {
  return typeof globalThis.Summarizer !== 'undefined'
      || typeof globalThis.LanguageModel !== 'undefined';
}

Each API reports its readiness through a static availability() call that returns one of 'available', 'downloadable', 'downloading', or 'unavailable'. One important gotcha: an output language is now mandatory - omit outputLanguage and the call throws. OmniBrief clamps the user's language to the set Nano actually supports (en, es, fr, de, ja) before probing:

const status = await Summarizer.availability({
  type: 'key-points',
  format: 'markdown',
  outputLanguage: 'en',   // required - omitting it errors
});
// status: 'available' | 'downloadable' | 'downloading' | 'unavailable'

If the status is downloadable, calling create() with a monitor triggers the background model download and streams downloadprogress events, which the onboarding screen turns into a percentage bar.

Warm, Reusable Handles

The first time a device loads the on-device weights into memory, create() can take several seconds. Creating a fresh summarizer per request would pay that cost every time. Instead, OmniBrief keeps one warm, length-and-language-keyed Summarizer instance for the lifetime of the context and only rebuilds it when those parameters change:

let cachedSummarizer = null;

async function getSummarizer(length, language) {
  if (cachedSummarizer?.__length === length &&
      cachedSummarizer?.__language === language) {
    return cachedSummarizer;              // reuse the warm handle
  }
  cachedSummarizer?.destroy?.();
  cachedSummarizer = await Summarizer.create({
    type: 'key-points',
    format: 'markdown',
    length,
    outputLanguage: language,
  });
  cachedSummarizer.__length = length;
  cachedSummarizer.__language = language;
  return cachedSummarizer;
}

The service worker also warms the model ahead of time - a fire-and-forget create() right after a capture lands - so the first real summarize call runs against an already-resident model.

Summarizer for Bullets, Prompt API for Structure

A good brief is more than a paragraph of prose. OmniBrief splits the work across both APIs: the dedicated Summarizer produces the key-points list, while the LanguageModel Prompt API returns structured metadata - title, one-line summary, topics, named entities, and keywords - using a JSON responseConstraint so the output is machine-parseable rather than free text.

// 1) Summarizer API → bullet summary
const summarizer = await getSummarizer(opts.length || 'medium', language);
const md = await summarizer.summarize(text, { context: opts.instruction });
const bulletPoints = md
  .split(/\r?\n/)
  .map((l) => l.replace(/^[-*•]\s*/, '').trim())
  .filter(Boolean);

// 2) Prompt API (LanguageModel) → structured JSON metadata
const session = await LanguageModel.create({
  expectedInputs:  [{ type: 'text', languages: [language] }],
  expectedOutputs: [{ type: 'text', languages: [language] }],
  initialPrompts: [
    { role: 'system', content: 'You extract research metadata. Respond with JSON only.' },
  ],
});
const raw = await session.prompt(buildBriefPrompt(text), {
  responseConstraint: BRIEF_JSON_SCHEMA,   // structured output
});

If Summarizer is missing but LanguageModel is present (or vice-versa), OmniBrief degrades gracefully - synthesizing bullet points from the Prompt API, or minimal metadata from the summarizer alone.

Stall Watchdog and Overflow Retries

Local execution shares the machine with everything else the user is running. Under VRAM pressure the on-device model can hang without ever throwing, which would freeze the background job queue. Every provider call is therefore wrapped in a shared watchdog - 30 seconds for on-device work, 90 for cloud providers - that rejects with a stalled flag the UI turns into a "Local AI Engine Stalled" status:

export function withWatchdog(promise, ms, label = 'operation') {
  let timer;
  const timeout = new Promise((_, reject) => {
    timer = setTimeout(() => {
      const err = new Error(`${label} exceeded ${Math.round(ms / 1000)}s watchdog`);
      err.stalled = true;
      reject(err);
    }, ms);
  });
  return Promise.race([promise, timeout]).finally(() => clearTimeout(timer));
}

// On-device gets 30s; cloud adapters get 90s.
const brief = await withWatchdog(adapter.summarize(text, opts), 30_000, 'Local AI');

Nano's real context window is small, so a long article can be rejected as "too large." Rather than fail outright, OmniBrief retries with progressively trimmed input - shrinking to 60% each attempt down to a floor - so oversized pages still produce a brief instead of an error:

async function withShrink(run, text, { minChars = 600, attempts = 5 } = {}) {
  let lastErr;
  for (let i = 0; i < attempts; i++) {
    try {
      return await run(text);
    } catch (err) {
      const msg = String(err?.message || err);
      if (!/too large|quota|token|context/i.test(msg) || text.length <= minChars) throw err;
      lastErr = err;
      text = text.slice(0, Math.max(minChars, Math.floor(text.length * 0.6)));
    }
  }
  throw lastErr;
}

Conclusion: Local Privacy, Unlocked

By building on Chrome's native Summarizer and LanguageModel APIs - with warm handles, a stall watchdog, and overflow-aware retries around them - OmniBrief delivers a fast, zero-cost, fully private research workflow. Your page reads, documentation summaries, and bookmarks are analyzed entirely on your own device, showing that local-first system design is a viable reality for modern developer tools.