kernelfoundry.eval_pipeline.utils.memory_file_map

Classes

MemoryFileMap()

In-memory representation of a file archive

class kernelfoundry.eval_pipeline.utils.memory_file_map.MemoryFileMap

In-memory representation of a file archive

__init__()

__contains__(file_path: str | Path) → bool

Checks if a file exists in the archive.

clear()

Clears all files from the archive.

open(file_path: str | Path, mode: str = 'r') → BytesIO | TextIOWrapper

Opens a file in the archive with the specified mode.

Parameters:

• file_path (str | Path) – Path of the file within the archive.

• mode (str) – Mode to open the file (‘r’, ‘w’, ‘rb’, ‘wb’). - ‘r’: Read text mode (returns TextIOWrapper) - ‘w’: Write text mode (returns TextIOWrapper) - ‘rb’: Read binary mode (returns BytesIO) - ‘wb’: Write binary mode (returns BytesIO)

Returns:

The file stream in binary or text mode.

Return type:

io.BytesIO | io.TextIOWrapper

Raises:

• ValueError – If an invalid mode is provided.

• FileNotFoundError – If the file doesn’t exist in read mode.

list_files() → list[str]

Lists all files in the archive.

Returns:

List of file paths in the archive.

Return type:

list[str]

remove_root_dir() → bool

Removes a common root directory or directories from all files in the archive.

Returns:

True if a common root directory was found and removed, False otherwise.

Return type:

bool

__getitem__(file_path: str | Path) → bytes | None

Gets the bytes of a file from the archive.

Parameters:

file_path (str | Path) – Path of the file within the archive.

Returns:

The file content as bytes, or None if the file doesn’t exist.

Return type:

bytes | None

__setitem__(file_path: str | Path, content: bytes)

Sets a file in the archive.

Parameters:

• file_path (str | Path) – Path of the file within the archive.

• content (bytes) – Content of the file.

to_tarball(mode='w:gz') → bytes

Creates a tarball from the in-memory files.

Returns:

The tarball as a byte string.

Return type:

bytes

from_tarball(*, tarball_bytes: bytes | None = None, tarball_path: Path | str | None = None)

Loads files from a tarball into the in-memory archive.

Parameters:

• tarball_bytes (bytes) – The tarball as a byte string.

• tarball_path (str | Path) – Path to the tarball file.

from_zip(*, zip_bytes: bytes | None = None, zip_path: Path | str | None = None)

Loads files from a zip archive into the in-memory archive.

Parameters:

• zip_bytes (bytes) – The zip archive as a byte string.

• zip_path (str | Path) – Path to the zip file.

from_archive(*, archive_bytes: bytes | None = None, archive_path: Path | str | None = None)

Loads files from an archive (zip or tarball) into the in-memory archive.

Parameters:

• archive_bytes (bytes) – The archive as a byte string.

• archive_path (str | Path) – Path to the archive file.

from_path(path: str | Path, include_extensions: list[str] | None = None)

Loads files from a given path to an archive or directory into the in-memory archive.

Parameters:

• path (str | Path) – Path to the archive file or directory.

• include_extensions (list[str] | None) – If provided, only files with these extensions will be included when loading from a directory.

to_disk(output_dir: str | Path)

Extracts the archive to disk.

Parameters:

output_dir (str | Path) – Directory where files will be extracted.

from_disk(input_dir: str | Path, include_extensions: list[str] | None = None)

Loads files from disk into the in-memory archive.

Parameters:

• input_dir (str | Path) – Directory to load files from.

• include_extensions (list[str] | None) – If provided, only files with these extensions will be included.

encode() → dict

Encodes the archive to a serializable dictionary.

Returns:

Dictionary mapping file paths to base64-encoded content.

Return type:

dict

decode(encoded_map: dict)

Decodes a serializable dictionary into the archive.

Parameters:

encoded_map (dict) – Dictionary mapping file paths to base64-encoded content.