kernelfoundry.algorithm.inference_server

Inference classes for submitting LLM queries.

Classes

InferenceServer([server_type, model_name, ...])

Class for sending queries to an LLM API.

LLMEnsemble(servers[, weights, ...])

Ensemble of inference servers for LLM queries, allowing for weighted probabilistic selection of different servers / models.
class kernelfoundry.algorithm.inference_server.InferenceServer(server_type: str | None = None, model_name: str = 'default', greedy_sample: bool = False, verbose: bool = False, timeout: int = 400, max_retry: int = 5, **kwargs)[source]

Class for sending queries to an LLM API.

__init__(server_type: str | None = None, model_name: str = 'default', greedy_sample: bool = False, verbose: bool = False, timeout: int = 400, max_retry: int = 5, **kwargs)[source]

Initialize an inference server for LLM queries.

Parameters:

  • server_type (str) – Type of inference server (“openai” or “anthropic”).

  • model_name (str) – ID of the model to use (must be compatible with available models in API). Defaults to “default” which is the first model in a hard-coded list.

  • greedy_sample (bool) – Whether to use greedy sampling (temperature=0). Defaults to False.

  • verbose (bool) – Whether to print initialization information. Defaults to False.

  • timeout (int) – Timeout in seconds for inference requests. Defaults to 400.

  • max_retry (int) – Maximum number of retries for failed requests. Defaults to 5.

  • **kwargs – Additional arguments to pass to the server configuration.

class kernelfoundry.algorithm.inference_server.LLMEnsemble(servers: list[InferenceServer], weights: list | str = 'uniform', weights_warmstart: list | None = None, trials_warmstart: int = 0)[source]

Ensemble of inference servers for LLM queries, allowing for weighted probabilistic selection of different servers / models.

__init__(servers: list[InferenceServer], weights: list | str = 'uniform', weights_warmstart: list | None = None, trials_warmstart: int = 0)[source]

Initialize an ensemble of inference servers.

Parameters:

  • servers (list[InferenceServer]) – List of inference server instances to ensemble.

  • weights (list | str) – Weighting scheme for server selection. Can be “uniform” (default) or a list of weights.

  • weights_warmstart (list | None) – Alternative weights for warmstart trials. Defaults to None.

  • trials_warmstart (int) – Number of trials to use warmstart weights. Defaults to 0.