custom_task_helper#
Functions for analyzing and manipulating custom tasks.
Functions
Returns a list of allowed text file extensions for custom tasks. |
|
|
Ordered list of labels identifying each block in a (file_path -> list[content]) blocks dict. |
|
Converts a blocks dict (file_path -> list[content]) to a single string. |
|
Converts a dict to a yaml string. |
|
Find the path to the file containing the block annotated with the given key. |
|
Returns the text block annotated with the given key. |
|
Reads the configuration yaml with the specified name in the task root directory. |
|
Returns the names of the correctness and profile tests defined in task.py. |
|
Updates the block annotated with the given key in the specified file. |
- kernelfoundry.eval_pipeline.utils.custom_task_helper.allowed_text_file_extensions() list[str][source]#
Returns a list of allowed text file extensions for custom tasks.
- kernelfoundry.eval_pipeline.utils.custom_task_helper.get_test_names(task_root: Path | str) dict[str, list[str]][source]#
Returns the names of the correctness and profile tests defined in task.py.
Test discovery is done by statically parsing task.py’s AST rather than importing it (e.g. via pytest collection), since importing it may require dependencies - like a GPU runtime - that are only available inside the container the task is later built and tested in.
- kernelfoundry.eval_pipeline.utils.custom_task_helper.get_block(task_root: Path | str | MemoryFileMap, key: str) dict[str, str][source]#
Returns the text block annotated with the given key.
- Parameters:
task_root (Path | str | MemoryFileMap) – Path to the root directory of the custom task or the in-memory file map.
key (str) – The key used to annotate the block (e.g., ‘REFERENCE’, ‘EVOLVE’,..)
- Returns:
- A dictionary with the annotated blocks as values and keys
as the relative path to the file where the block was found.
- Return type:
- kernelfoundry.eval_pipeline.utils.custom_task_helper.get_config(task_root: Path | str | MemoryFileMap, config_stem: str = 'config') dict[source]#
Reads the configuration yaml with the specified name in the task root directory.
- Parameters:
task_root (Path | str | MemoryFileMap) – Path to the root directory of the custom task or a MemoryFileMap.
config_name (str, optional) – Name of the config file without extension. Defaults to “config”.
- Returns:
The task configuration as a dictionary.
- Return type:
- kernelfoundry.eval_pipeline.utils.custom_task_helper.update_block(task: MemoryFileMap, key: str, path: str | Path, content: str, block_index: int = 0) None[source]#
Updates the block annotated with the given key in the specified file.
This updates the block in-place within the in-memory file map.
- Parameters:
task (MemoryFileMap) – The in-memory file map of the task.
key (str) – The key used to annotate the block (e.g., ‘REFERENCE’, ‘EVOLVE’,..)
path (str | Path) – The path to the file within the task.
content (str) – The new content to insert into the block.
- kernelfoundry.eval_pipeline.utils.custom_task_helper.blocks_to_str(blocks: dict[str, list[str]]) str[source]#
Converts a blocks dict (file_path -> list[content]) to a single string.
For a single block total, returns the content directly. For multiple blocks, combines them with ### Block: <label> ### separator lines, using the same labels (disambiguated with a “#index” suffix for files with more than one block) as
block_labels, so headers stay unique and match whatextract_multi_block_codeexpects. Split back with:re.split(r'^### Block: (.+?) ###$', text, flags=re.MULTILINE)- Parameters:
blocks – A dictionary mapping file paths to lists of block contents.
- Returns:
The combined string representation of the blocks.
- Return type:
- kernelfoundry.eval_pipeline.utils.custom_task_helper.block_labels(blocks: dict[str, list[str]]) list[str][source]#
Ordered list of labels identifying each block in a (file_path -> list[content]) blocks dict.
A file’s blocks are labeled with just its path when it has a single block; when a file has multiple blocks, each is disambiguated with a block-index suffix (
path#0,path#1, …). Both prompt construction and LLM-response parsing use this so multi-block responses round-trip to the right(path, block_index)regardless of iteration order.