Inference

Running a trained model on a new input to get an output.

concept

Training is when a model learns; inference is when it is used. Every time you send a prompt and get a response, the model is performing inference: computing an output from the patterns it already holds.

During ordinary inference the model’s parameters do not change, so using a model does not, by itself, teach it anything new.

Each run of inference takes computing power, which is one reason fast and efficient models matter.

Example

When an app turns a voice memo into text, it is running inference on a speech model.

Related