benchdnn_helper#

Helpers for invoking benchdnn correctness/performance runs and parsing output.

Functions

`check_correctness_benchdnn`(benchdnn_bin, driver)	Run benchdnn in correctness mode and report pass/fail.
`measure_runtime_benchdnn`(benchdnn_bin, driver)	Run benchdnn in performance mode and return measured runtime.
`parse_benchdnn_correctness_check`(stdout)	Detect correctness success from benchdnn stdout.
`parse_benchdnn_perf_stats`(stdout)	Extract `avg_time` from benchdnn performance stdout.

kernelfoundry.eval_pipeline.utils.benchdnn_helper.parse_benchdnn_correctness_check(stdout: str) → bool[source]#

Detect correctness success from benchdnn stdout.

This parser looks for benchdnn result lines in the REPRO format: case_num:status[ ... ] __REPRO: ....

Expected input includes exactly one REPRO line.

Parameters:: stdout – Raw stdout text produced by benchdnn in correctness mode.
Returns:: True when the single REPRO line has PASSED status, otherwise False.
Raises:: ValueError – If stdout contains zero or multiple REPRO lines, or if the REPRO line does not match the expected case_num:status prefix format.

kernelfoundry.eval_pipeline.utils.benchdnn_helper.parse_benchdnn_perf_stats(stdout: str) → dict[source]#

Extract avg_time from benchdnn performance stdout.

Expected input includes exactly two CSV-like perf, lines: one header row and one value row.

Parameters:: stdout – Raw stdout text produced by benchdnn in performance mode.
Returns:: A dictionary containing the parsed avg_time value as float.
Raises:: ValueError – If expected perf rows are missing, malformed, missing avg_time, or contain a non-numeric avg_time value.

kernelfoundry.eval_pipeline.utils.benchdnn_helper.check_correctness_benchdnn(benchdnn_bin: str, driver: str, test_shape: str | None = None, ld_preload: str | None = None, use_custom_kernel: bool = False, verbosity_level: int = 0, log_stdout: bool = False) → bool[source]#

Run benchdnn in correctness mode and report pass/fail.

Parameters:

benchdnn_bin – Path to the benchdnn executable.
driver – benchdnn driver name (for example, --matmul).
test_shape – Shape string for the benchdnn CLI call.
ld_preload – Value for LD_PRELOAD pointing to custom kernel .so.
use_custom_kernel – Whether to set DNNL_USE_CUSTOM_KERNEL=1.
verbosity_level – Value written to ONEDNN_VERBOSE.
log_stdout – Whether to emit benchdnn stdout through logging.info.

Returns:

True when benchdnn stdout indicates a pass, otherwise False.

Raises:

ValueError – If test_shape is empty.
RuntimeError – If benchdnn exits with a non-zero return code.

kernelfoundry.eval_pipeline.utils.benchdnn_helper.measure_runtime_benchdnn(benchdnn_bin: str, driver: str, test_shape: str | None = None, ld_preload: str | None = None, use_custom_kernel: bool = False, verbosity_level: int = 0, log_stdout: bool = False, output: list[float] | None = None) → list[float][source]#

Run benchdnn in performance mode and return measured runtime.

Parameters:

benchdnn_bin – Path to the benchdnn executable.
driver – benchdnn driver name (for example, --matmul).
test_shape – Shape string for the benchdnn CLI call.
ld_preload – Value for LD_PRELOAD pointing to custom kernel .so.
use_custom_kernel – Whether to set DNNL_USE_CUSTOM_KERNEL=1.
verbosity_level – Value written to ONEDNN_VERBOSE.
log_stdout – Whether to emit benchdnn stdout through logging.info.
output – Optional list that, when provided, is extended in-place with the returned runtime values.

Returns:

A single-item list containing avg_time parsed from benchdnn output.

Raises:

ValueError – If test_shape is empty or perf output parsing fails.
RuntimeError – If benchdnn exits with a non-zero return code.