profiler_feedback#

Profiler feedback generators for different profilers (Unitrace, NCU, VTune)

Functions

compute_analytical_roofline_pct(entry, ...)

Compute a roofline summary dict from an agent-derived ideal FLOPs/bytes entry.

compute_roofline_summary(profiler_name, ...)

Summarize roofline performance per benchmark for one profiler run.

get_profiler_feedback_class(language, gpu_arch)

Get the appropriate profiler feedback class based on user-specified profiler type or language & GPU architecture.

get_reference_language_for_profiling(config)

Get the reference language for profiler feedback based on the current language.

Classes

NCUProfilerFeedback()

Profiler feedback for CUDA kernels using NVIDIA Nsight Compute (NCU).

OCLUnitraceProfilerFeedback()

Profiler feedback for OpenCL using unitrace metric-sampling.

ProfilerFeedback()

Abstract base class for profiler feedback generators.

UnitraceProfilerFeedback()

Profiler feedback for SYCL kernels using Intel unitrace (metric-query mode).

VTuneProfilerFeedback()

VTune profiler feedback generator for Intel GPUs.

kernelfoundry.eval_pipeline.profiler_feedback.compute_analytical_roofline_pct(entry: dict, runtime_ms: float | None, worker_info: dict) → dict | None[source]#

Compute a roofline summary dict from an agent-derived ideal FLOPs/bytes entry.

Unlike compute_roofline_summary(), which measures achieved instructions/bytes per second from profiler counters, entry describes the minimum work implied by the reference’s math and the benchmark’s shapes (see skills/analytical-roofline). The resulting percentage is therefore independent of how many instructions the generated kernel itself happens to execute. Returns None if hardware roofs, runtime or usable flops/bytes are unavailable.

class kernelfoundry.eval_pipeline.profiler_feedback.ProfilerFeedback[source]#

Abstract base class for profiler feedback generators.

Subclasses implement profiler-specific logic to collate raw output data from multiple profiling passes and produce human-readable feedback strings that can be injected into LLM prompts.

__init__()[source]#
abstract property name: str#

Return the name of the profiler

abstract collate_data(outputs: dict[str, dict]) → dict[source]#

Collate the output data from multiple profiling runs into a single dictionary for analysis.

Parameters:

outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}

Returns:

A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:

{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}

Note that the data keys in outputs and the collated dictionary are specific to the profiler.

abstract create_feedback(data: dict, worker_info: dict) → str[source]#

Create feedback based on the collated data and worker information.

Parameters:
  • data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().

  • worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.

Returns:

A string containing the generated feedback based on the collated data and worker information.

collate_and_create_feedback(outputs: dict[str, dict], worker_info: dict) → tuple[dict, str][source]#

Collate the output data from multiple profiling runs and create feedback.

Parameters:
  • outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘trace.metrics.pid’: ‘…’}, ‘unitrace.002’: {…}, …}

  • worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.

Returns:

A tuple containing the collated data dictionary and the generated feedback string. Example:

(
    {'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'},
    "The kernel is memory bound ..."
)

compute_roofline_summary(raw_data: dict, worker_info: dict) → dict[str, dict] | None[source]#

Summarize roofline performance per benchmark from this profiler’s collated data.

Returns a mapping from benchmark label to a {"bound", "compute_pct", "membw_pct", "roofline_pct"} dict (the worst-case kernel within that benchmark), or None if this backend doesn’t compute structured roofline numbers. Benchmark labels should match the keys used in EvalResult.runtime_stats where possible; backends that can’t attribute data to individual benchmarks should key their one result under ROOFLINE_ALL_BENCHMARKS_KEY instead.

class kernelfoundry.eval_pipeline.profiler_feedback.UnitraceProfilerFeedback[source]#

Profiler feedback for SYCL kernels using Intel unitrace (metric-query mode).

Supports multi-pass profiling with ComputeBasic, MemoryProfile, and VectorEngineProfile metric groups.

MODEL_RUN_LOOP_EVENT_PREFIX = 'ittapi::model run loop'#
__init__()[source]#
property name: str#

Return the name of the profiler

analyze_kernel(compute_basic: Series, memory_profile: Series, vector_engine_profile: Series, roofs: HardwareRoofs) → tuple[str, dict][source]#

Analyze the kernel performance based on profiling data. :param df: A pandas Series containing profiling metrics for a kernel. :type df: pd.Series

Returns:

A list of tuples containing importance scores and corresponding analysis messages.

Return type:

list

compute_roofline_summary(raw_data: dict, worker_info: dict) → dict[str, dict] | None[source]#

Summarize roofline performance per benchmark from Unitrace’s collated data.

Benchmark labels are the pytest node ids embedded in ITT “model run loop::<label>” markers by the autouse profiler_test_label fixture in conftest.py, which are the same node ids used as keys in EvalResult.runtime_stats - so segments map 1:1 onto benchmarks. Runs with no matching ITT markers fall back to ROOFLINE_ALL_BENCHMARKS_KEY.

create_feedback(data: dict, worker_info: dict) → str[source]#

Create feedback based on the collated data and worker information.

Parameters:
  • data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().

  • worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.

Returns:

A string containing the generated feedback based on the collated data and worker information.

collate_data(outputs: dict[str, dict]) → dict[source]#

Collate the output data from multiple profiling runs into a single dictionary for analysis.

Parameters:

outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}

Returns:

A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:

{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}

Note that the data keys in outputs and the collated dictionary are specific to the profiler.

class kernelfoundry.eval_pipeline.profiler_feedback.OCLUnitraceProfilerFeedback[source]#

Profiler feedback for OpenCL using unitrace metric-sampling.

Metric-sampling produces independent samples from separate program runs for each metric group (ComputeBasic, MemoryProfile, VectorEngineProfile). Unlike metric-query, the samples are not aligned by GlobalInstanceId or timestamp across groups, and GpuTime[ns] is just the fixed sampling interval duration. Segments cannot be reliably filtered from sampling data, so this class averages all samples per kernel and matches the three groups by kernel name only.

property name: str#

Return the name of the profiler

collate_data(outputs: dict[str, dict]) → dict[source]#

Collate the output data from multiple profiling runs into a single dictionary for analysis.

Parameters:

outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}

Returns:

A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:

{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}

Note that the data keys in outputs and the collated dictionary are specific to the profiler.

class kernelfoundry.eval_pipeline.profiler_feedback.NCUProfilerFeedback[source]#

Profiler feedback for CUDA kernels using NVIDIA Nsight Compute (NCU).

Parses NCU CSV reports to extract throughput metrics, roofline analysis, and optimization hints for the slowest kernel in the profile.

__init__()[source]#
property name: str#

Return the name of the profiler

kernel_id_runtime(df: DataFrame) → list[tuple[int, float]][source]#

Get a list of kernel IDs and their runtimes from the DataFrame.

kernel_feedback(df: DataFrame, kernel_id: int) → str[source]#

Generate feedback for a specific kernel based on its metrics.

collate_data(outputs: dict[str, dict]) → dict[source]#

Collate the output data from multiple profiling runs into a single dictionary for analysis.

Parameters:

outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}

Returns:

A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:

{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}

Note that the data keys in outputs and the collated dictionary are specific to the profiler.

create_feedback(data: dict, worker_info: dict) → str[source]#

Create feedback based on the collated data and worker information.

Parameters:
  • data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().

  • worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.

Returns:

A string containing the generated feedback based on the collated data and worker information.

class kernelfoundry.eval_pipeline.profiler_feedback.VTuneProfilerFeedback[source]#

VTune profiler feedback generator for Intel GPUs.

Works with the counters dict produced by VTune._extract_counters, serialised as JSON under the key ‘vtune_counters.json’.

property name: str#

Return the name of the profiler

collate_data(outputs: dict[str, dict]) → dict[source]#

Collate VTune output: deserialise the first vtune_counters.json found.

create_feedback(data: dict, worker_info: dict) → str[source]#

Create feedback string from VTune profiling data.

compute_roofline_summary(raw_data: dict, worker_info: dict) → dict[str, dict] | None[source]#

Summarize roofline performance from VTune’s collated counter data.

VTune’s counters aren’t segmented per benchmark today, so the one result computed here is returned under ROOFLINE_ALL_BENCHMARKS_KEY and applies to every benchmark.

kernelfoundry.eval_pipeline.profiler_feedback.compute_roofline_summary(profiler_name: str, raw_data: dict, worker_info: dict) → dict[str, dict] | None[source]#

Summarize roofline performance per benchmark for one profiler run.

Dispatches to the profiler-specific implementation (see ProfilerFeedback.compute_roofline_summary and its overrides on UnitraceProfilerFeedback/VTuneProfilerFeedback), keyed by profiler_name (i.e. ProfilerFeedback.name, the same key used in EvalResult.profiler_data). Returns None for backends without structured roofline support (e.g. NCU).

kernelfoundry.eval_pipeline.profiler_feedback.get_reference_language_for_profiling(config) → str[source]#

Get the reference language for profiler feedback based on the current language.

kernelfoundry.eval_pipeline.profiler_feedback.get_profiler_feedback_class(language: str, gpu_arch: str, profiler_type: str | None = None) → type[ProfilerFeedback] | None[source]#

Get the appropriate profiler feedback class based on user-specified profiler type or language & GPU architecture.

Returns None if no profiler is configured or available for this language (profiling is skipped).