incr completion
**Incremental Completion (Streaming)** is the **UX pattern used by modern AI coding tools where code suggestions appear token-by-token as ghost text in real-time while the developer types** — requiring sub-100ms latency to feel instantaneous, implemented through streaming RPCs where the server pushes partial completions to the IDE as they're generated rather than waiting for the full suggestion to complete, creating the seamless autocomplete experience that makes tools like Copilot and Cursor feel responsive.
**What Is Incremental Completion?**
- **Definition**: The technique of displaying AI code suggestions progressively (token by token or chunk by chunk) as the model generates them — shown as translucent "ghost text" ahead of the cursor that the developer can accept with Tab or ignore by continuing to type.
- **Streaming Architecture**: Instead of request-response (send context → wait → receive full suggestion), streaming RPCs push tokens to the IDE immediately as they're generated — the first token appears in ~100ms while the model continues generating subsequent tokens in the background.
- **IDE Integration**: The IDE renders incoming tokens as light gray ghost text that updates in real-time — if the developer types a character that conflicts with the suggestion, it's immediately dismissed and a new completion request fires.
**Technical Requirements**
| Requirement | Target | Why It Matters |
|------------|--------|---------------|
| **First token latency** | <100ms | Anything slower feels laggy and disrupts flow |
| **Token throughput** | 30-100 tokens/sec | Must keep ahead of fast typers |
| **Cancellation** | <10ms | Dismiss stale suggestions instantly when user types |
| **Context update** | Real-time | New keystrokes must invalidate/update suggestions |
| **Memory** | <500MB | IDE plugin can't consume excessive resources |
**Implementation Challenges**
- **Debouncing**: Don't fire a completion request on every keystroke — wait 50-100ms after the last keypress to avoid overwhelming the server with requests that will be immediately cancelled.
- **Speculative Execution**: Some systems generate completions speculatively (before the user pauses) using fast, small models — then refine with larger models if the user stops typing.
- **Cache Management**: Recently generated completions are cached — if the user undoes a character and retypes, the cached suggestion can be restored instantly.
- **Context Invalidation**: Every typed character potentially invalidates the current suggestion — the IDE must check whether new input is consistent with the streaming suggestion or requires a new request.
- **Multi-Line Handling**: Single-line suggestions are straightforward, but multi-line completions (generating an entire function body) require careful rendering that doesn't disrupt the visible code layout.
**Streaming Protocols**
| Protocol | Used By | Characteristics |
|----------|---------|----------------|
| **Server-Sent Events (SSE)** | OpenAI API, most cloud models | Simple, HTTP-based, one-way streaming |
| **gRPC Streaming** | Internal tools, low-latency systems | Bidirectional, efficient binary protocol |
| **WebSocket** | IDE extensions, web-based editors | Full-duplex, persistent connection |
| **Language Server Protocol (LSP)** | VS Code extensions | Standardized IDE communication |
**Incremental Completion is the technical foundation that makes AI coding assistance feel magical** — transforming the raw output of language models into a seamless, responsive editing experience where code appears to write itself, requiring careful engineering of streaming protocols, latency optimization, and IDE integration to maintain the sub-100ms responsiveness that developers expect.