kernelfoundry.eval_pipeline.profiler_feedback

Profiler feedback generators for different profilers (Unitrace, NCU, VTune)

Functions

get_profiler_feedback_class(language, gpu_arch)

Get the appropriate profiler feedback class based on user-specified profiler type or language & GPU architecture.

get_reference_language_for_profiling(config)

Get the reference language for profiler feedback based on the current language.

Classes

NCUProfilerFeedback()

Profiler feedback for CUDA kernels using NVIDIA Nsight Compute (NCU).

OCLUnitraceProfilerFeedback()

Profiler feedback for OpenCL using unitrace metric-sampling.

ProfilerFeedback()

Abstract base class for profiler feedback generators.

UnitraceProfilerFeedback()

Profiler feedback for SYCL kernels using Intel unitrace (metric-query mode).

VTuneProfilerFeedback()

VTune profiler feedback generator for Intel GPUs.
class kernelfoundry.eval_pipeline.profiler_feedback.ProfilerFeedback[source]

Abstract base class for profiler feedback generators.

Subclasses implement profiler-specific logic to collate raw output data from multiple profiling passes and produce human-readable feedback strings that can be injected into LLM prompts.

__init__()[source]
abstract property name: str

Return the name of the profiler

abstract collate_data(outputs: dict[str, dict]) → dict[source]

Collate the output data from multiple profiling runs into a single dictionary for analysis.

Parameters:

outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}

Returns:

A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:

{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}

Note that the data keys in outputs and the collated dictionary are specific to the profiler.

abstract create_feedback(data: dict, worker_info: dict) → str[source]

Create feedback based on the collated data and worker information.

Parameters:

  • data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().

  • worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.

Returns:

A string containing the generated feedback based on the collated data and worker information.

collate_and_create_feedback(outputs: dict[str, dict], worker_info: dict) → tuple[dict, str][source]

Collate the output data from multiple profiling runs and create feedback.

Parameters:

  • outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘trace.metrics.pid’: ‘…’}, ‘unitrace.002’: {…}, …}

  • worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.

Returns:

A tuple containing the collated data dictionary and the generated feedback string. Example:

(
    {'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'},
    "The kernel is memory bound ..."
)

class kernelfoundry.eval_pipeline.profiler_feedback.UnitraceProfilerFeedback[source]

Profiler feedback for SYCL kernels using Intel unitrace (metric-query mode).

Supports multi-pass profiling with ComputeBasic, MemoryProfile, and VectorEngineProfile metric groups.

MODEL_RUN_LOOP_EVENT_PREFIX = 'ittapi::model run loop'
__init__()[source]
property name: str

Return the name of the profiler

analyze_kernel(compute_basic: Series, memory_profile: Series, vector_engine_profile: Series, roofs: HardwareRoofs) → tuple[str, dict][source]

Analyze the kernel performance based on profiling data. :param df: A pandas Series containing profiling metrics for a kernel. :type df: pd.Series

Returns:

A list of tuples containing importance scores and corresponding analysis messages.

Return type:

list

create_feedback(data: dict, worker_info: dict) → str[source]

Create feedback based on the collated data and worker information.

Parameters:

  • data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().

  • worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.

Returns:

A string containing the generated feedback based on the collated data and worker information.

collate_data(outputs: dict[str, dict]) → dict[source]

Collate the output data from multiple profiling runs into a single dictionary for analysis.

Parameters:

outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}

Returns:

A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:

{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}

Note that the data keys in outputs and the collated dictionary are specific to the profiler.

class kernelfoundry.eval_pipeline.profiler_feedback.OCLUnitraceProfilerFeedback[source]

Profiler feedback for OpenCL using unitrace metric-sampling.

Metric-sampling produces independent samples from separate program runs for each metric group (ComputeBasic, MemoryProfile, VectorEngineProfile). Unlike metric-query, the samples are not aligned by GlobalInstanceId or timestamp across groups, and GpuTime[ns] is just the fixed sampling interval duration. Segments cannot be reliably filtered from sampling data, so this class averages all samples per kernel and matches the three groups by kernel name only.

property name: str

Return the name of the profiler

collate_data(outputs: dict[str, dict]) → dict[source]

Collate the output data from multiple profiling runs into a single dictionary for analysis.

Parameters:

outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}

Returns:

A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:

{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}

Note that the data keys in outputs and the collated dictionary are specific to the profiler.

class kernelfoundry.eval_pipeline.profiler_feedback.NCUProfilerFeedback[source]

Profiler feedback for CUDA kernels using NVIDIA Nsight Compute (NCU).

Parses NCU CSV reports to extract throughput metrics, roofline analysis, and optimization hints for the slowest kernel in the profile.

__init__()[source]
property name: str

Return the name of the profiler

kernel_id_runtime(df: DataFrame) → list[tuple[int, float]][source]

Get a list of kernel IDs and their runtimes from the DataFrame.

kernel_feedback(df: DataFrame, kernel_id: int) → str[source]

Generate feedback for a specific kernel based on its metrics.

collate_data(outputs: dict[str, dict]) → dict[source]

Collate the output data from multiple profiling runs into a single dictionary for analysis.

Parameters:

outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}

Returns:

A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:

{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}

Note that the data keys in outputs and the collated dictionary are specific to the profiler.

create_feedback(data: dict, worker_info: dict) → str[source]

Create feedback based on the collated data and worker information.

Parameters:

  • data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().

  • worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.

Returns:

A string containing the generated feedback based on the collated data and worker information.

class kernelfoundry.eval_pipeline.profiler_feedback.VTuneProfilerFeedback[source]

VTune profiler feedback generator for Intel GPUs.

Works with the counters dict produced by VTune._extract_counters, serialised as JSON under the key ‘vtune_counters.json’.

property name: str

Return the name of the profiler

collate_data(outputs: dict[str, dict]) → dict[source]

Collate VTune output: deserialise the first vtune_counters.json found.

create_feedback(data: dict, worker_info: dict) → str[source]

Create feedback string from VTune profiling data.

kernelfoundry.eval_pipeline.profiler_feedback.get_reference_language_for_profiling(config) → str[source]

Get the reference language for profiler feedback based on the current language.

kernelfoundry.eval_pipeline.profiler_feedback.get_profiler_feedback_class(language: str, gpu_arch: str, profiler_type: str | None = None) → type[ProfilerFeedback][source]

Get the appropriate profiler feedback class based on user-specified profiler type or language & GPU architecture.