Technology
A chip for continuing
the conversation.
Our architecture starts with a specific job: generate the next tokens of a hybrid model from conversation memory prepared by the customer’s GPUs.
Generation shapes the design.
In a serving system, prompt processing builds the context for a request. Generation then repeatedly uses that context to produce the answer. Markov Silicon focuses on this second phase.
That focus guides the compute, memory and execution architecture. The proposed product uses high-bandwidth memory and supports recurrent, attention and expert layers as parts of one hybrid model.
- 01 / PREPARE
Your GPUs
Process the prompt
Read the input. Build the conversation memory. - One handoff
per request - 02 / GENERATE
Markov Silicon
Continue the conversation
Generate the next tokens across the hybrid model.
The handoff includes what the model remembers.
For a hybrid model, conversation memory can include both recurrent state and attention key/value history. A serving integration transfers that memory from the prompt processor once per request. Our card then continues generation from the imported state.
This is a design for disaggregated serving, with prompt processing and generation assigned to different hardware. The handoff and serving integration are essential parts of the system we are building.
Different layers. One execution architecture.
Recurrent layers
Update a running state as tokens arrive. The state carries information forward through the conversation.
Attention layers
Use key/value history to retrieve information from earlier tokens. Attention remains part of the hybrid execution path.
Expert layers
Execute the experts selected by the model’s routing, alongside its recurrent and attention layers.
Hardware and software, designed together.
The compiler and runtime coordinate model execution, data movement and the imported conversation state. Shared hardware provides the arithmetic and memory access those layers need.
We are developing the architecture as a complete generation system, including the path from a serving request to computed output and updated state.