profiler_feedback#
Profiler feedback generators for different profilers (Unitrace, NCU, VTune)
Functions
|
Compute a roofline summary dict from an agent-derived ideal FLOPs/bytes |
|
Summarize roofline performance per benchmark for one profiler run. |
|
Get the appropriate profiler feedback class based on user-specified profiler type or language & GPU architecture. |
Get the reference language for profiler feedback based on the current language. |
Classes
Profiler feedback for CUDA kernels using NVIDIA Nsight Compute (NCU). |
|
Profiler feedback for OpenCL using unitrace metric-sampling. |
|
Abstract base class for profiler feedback generators. |
|
Profiler feedback for SYCL kernels using Intel unitrace (metric-query mode). |
|
VTune profiler feedback generator for Intel GPUs. |
- kernelfoundry.eval_pipeline.profiler_feedback.compute_analytical_roofline_pct(entry: dict, runtime_ms: float | None, worker_info: dict) dict | None[source]#
Compute a roofline summary dict from an agent-derived ideal FLOPs/bytes
entry.Unlike
compute_roofline_summary(), which measures achieved instructions/bytes per second from profiler counters,entrydescribes the minimum work implied by the reference’s math and the benchmark’s shapes (seeskills/analytical-roofline). The resulting percentage is therefore independent of how many instructions the generated kernel itself happens to execute. ReturnsNoneif hardware roofs, runtime or usable flops/bytes are unavailable.
- class kernelfoundry.eval_pipeline.profiler_feedback.ProfilerFeedback[source]#
Abstract base class for profiler feedback generators.
Subclasses implement profiler-specific logic to collate raw output data from multiple profiling passes and produce human-readable feedback strings that can be injected into LLM prompts.
- abstract collate_data(outputs: dict[str, dict]) dict[source]#
Collate the output data from multiple profiling runs into a single dictionary for analysis.
- Parameters:
outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}
- Returns:
A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:
{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}Note that the data keys in outputs and the collated dictionary are specific to the profiler.
- abstract create_feedback(data: dict, worker_info: dict) str[source]#
Create feedback based on the collated data and worker information.
- Parameters:
data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().
worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.
- Returns:
A string containing the generated feedback based on the collated data and worker information.
- collate_and_create_feedback(outputs: dict[str, dict], worker_info: dict) tuple[dict, str][source]#
Collate the output data from multiple profiling runs and create feedback.
- Parameters:
outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘trace.metrics.pid’: ‘…’}, ‘unitrace.002’: {…}, …}
worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.
- Returns:
A tuple containing the collated data dictionary and the generated feedback string. Example:
( {'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}, "The kernel is memory bound ..." )
- compute_roofline_summary(raw_data: dict, worker_info: dict) dict[str, dict] | None[source]#
Summarize roofline performance per benchmark from this profiler’s collated data.
Returns a mapping from benchmark label to a
{"bound", "compute_pct", "membw_pct", "roofline_pct"}dict (the worst-case kernel within that benchmark), orNoneif this backend doesn’t compute structured roofline numbers. Benchmark labels should match the keys used inEvalResult.runtime_statswhere possible; backends that can’t attribute data to individual benchmarks should key their one result underROOFLINE_ALL_BENCHMARKS_KEYinstead.
- class kernelfoundry.eval_pipeline.profiler_feedback.UnitraceProfilerFeedback[source]#
Profiler feedback for SYCL kernels using Intel unitrace (metric-query mode).
Supports multi-pass profiling with ComputeBasic, MemoryProfile, and VectorEngineProfile metric groups.
- MODEL_RUN_LOOP_EVENT_PREFIX = 'ittapi::model run loop'#
- analyze_kernel(compute_basic: Series, memory_profile: Series, vector_engine_profile: Series, roofs: HardwareRoofs) tuple[str, dict][source]#
Analyze the kernel performance based on profiling data. :param df: A pandas Series containing profiling metrics for a kernel. :type df: pd.Series
- Returns:
A list of tuples containing importance scores and corresponding analysis messages.
- Return type:
- compute_roofline_summary(raw_data: dict, worker_info: dict) dict[str, dict] | None[source]#
Summarize roofline performance per benchmark from Unitrace’s collated data.
Benchmark labels are the pytest node ids embedded in ITT “model run loop::<label>” markers by the autouse
profiler_test_labelfixture in conftest.py, which are the same node ids used as keys inEvalResult.runtime_stats- so segments map 1:1 onto benchmarks. Runs with no matching ITT markers fall back toROOFLINE_ALL_BENCHMARKS_KEY.
- create_feedback(data: dict, worker_info: dict) str[source]#
Create feedback based on the collated data and worker information.
- Parameters:
data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().
worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.
- Returns:
A string containing the generated feedback based on the collated data and worker information.
- collate_data(outputs: dict[str, dict]) dict[source]#
Collate the output data from multiple profiling runs into a single dictionary for analysis.
- Parameters:
outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}
- Returns:
A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:
{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}Note that the data keys in outputs and the collated dictionary are specific to the profiler.
- class kernelfoundry.eval_pipeline.profiler_feedback.OCLUnitraceProfilerFeedback[source]#
Profiler feedback for OpenCL using unitrace metric-sampling.
Metric-sampling produces independent samples from separate program runs for each metric group (ComputeBasic, MemoryProfile, VectorEngineProfile). Unlike metric-query, the samples are not aligned by GlobalInstanceId or timestamp across groups, and GpuTime[ns] is just the fixed sampling interval duration. Segments cannot be reliably filtered from sampling data, so this class averages all samples per kernel and matches the three groups by kernel name only.
- collate_data(outputs: dict[str, dict]) dict[source]#
Collate the output data from multiple profiling runs into a single dictionary for analysis.
- Parameters:
outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}
- Returns:
A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:
{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}Note that the data keys in outputs and the collated dictionary are specific to the profiler.
- class kernelfoundry.eval_pipeline.profiler_feedback.NCUProfilerFeedback[source]#
Profiler feedback for CUDA kernels using NVIDIA Nsight Compute (NCU).
Parses NCU CSV reports to extract throughput metrics, roofline analysis, and optimization hints for the slowest kernel in the profile.
- kernel_id_runtime(df: DataFrame) list[tuple[int, float]][source]#
Get a list of kernel IDs and their runtimes from the DataFrame.
- kernel_feedback(df: DataFrame, kernel_id: int) str[source]#
Generate feedback for a specific kernel based on its metrics.
- collate_data(outputs: dict[str, dict]) dict[source]#
Collate the output data from multiple profiling runs into a single dictionary for analysis.
- Parameters:
outputs – A dictionary of dictionaries containing the output data from multiple profiling runs. Example: {‘unitrace.001’: {‘timeline’: ‘…’, ‘metrics’: ‘…’}, ‘unitrace.002’: {…}, …}
- Returns:
A single dictionary that collates the relevant data from the multiple profiling runs for analysis. Example:
{'timeline': '...', 'ComputeBasic.metrics.pid': '...', 'MemoryBasic.metrics.pid': '...'}Note that the data keys in outputs and the collated dictionary are specific to the profiler.
- create_feedback(data: dict, worker_info: dict) str[source]#
Create feedback based on the collated data and worker information.
- Parameters:
data – A dictionary containing the collated data from the profiling runs, as returned by collate_data().
worker_info – A dictionary containing information about the worker that executed the profiling runs, such as GPU architecture, device ID, etc.
- Returns:
A string containing the generated feedback based on the collated data and worker information.
- class kernelfoundry.eval_pipeline.profiler_feedback.VTuneProfilerFeedback[source]#
VTune profiler feedback generator for Intel GPUs.
Works with the counters dict produced by VTune._extract_counters, serialised as JSON under the key ‘vtune_counters.json’.
- collate_data(outputs: dict[str, dict]) dict[source]#
Collate VTune output: deserialise the first vtune_counters.json found.
- create_feedback(data: dict, worker_info: dict) str[source]#
Create feedback string from VTune profiling data.
- compute_roofline_summary(raw_data: dict, worker_info: dict) dict[str, dict] | None[source]#
Summarize roofline performance from VTune’s collated counter data.
VTune’s counters aren’t segmented per benchmark today, so the one result computed here is returned under
ROOFLINE_ALL_BENCHMARKS_KEYand applies to every benchmark.
- kernelfoundry.eval_pipeline.profiler_feedback.compute_roofline_summary(profiler_name: str, raw_data: dict, worker_info: dict) dict[str, dict] | None[source]#
Summarize roofline performance per benchmark for one profiler run.
Dispatches to the profiler-specific implementation (see
ProfilerFeedback.compute_roofline_summaryand its overrides onUnitraceProfilerFeedback/VTuneProfilerFeedback), keyed byprofiler_name(i.e.ProfilerFeedback.name, the same key used inEvalResult.profiler_data). Returns None for backends without structured roofline support (e.g. NCU).
- kernelfoundry.eval_pipeline.profiler_feedback.get_reference_language_for_profiling(config) str[source]#
Get the reference language for profiler feedback based on the current language.
- kernelfoundry.eval_pipeline.profiler_feedback.get_profiler_feedback_class(language: str, gpu_arch: str, profiler_type: str | None = None) type[ProfilerFeedback] | None[source]#
Get the appropriate profiler feedback class based on user-specified profiler type or language & GPU architecture.
Returns None if no profiler is configured or available for this language (profiling is skipped).