memory_file_map#

Classes

MemoryFileMap()

In-memory representation of a file archive
class kernelfoundry.eval_pipeline.utils.memory_file_map.MemoryFileMap[source]#

In-memory representation of a file archive

__init__()[source]#
__contains__(file_path: str | Path) → bool[source]#

Checks if a file exists in the archive.

clear()[source]#

Clears all files from the archive.

open(file_path: str | Path, mode: str = 'r') → BytesIO | TextIOWrapper[source]#

Opens a file in the archive with the specified mode.

Parameters:

  • file_path (str | Path) – Path of the file within the archive.

  • mode (str) – Mode to open the file (‘r’, ‘w’, ‘rb’, ‘wb’). - ‘r’: Read text mode (returns TextIOWrapper) - ‘w’: Write text mode (returns TextIOWrapper) - ‘rb’: Read binary mode (returns BytesIO) - ‘wb’: Write binary mode (returns BytesIO)

Returns:

The file stream in binary or text mode.

Return type:

io.BytesIO | io.TextIOWrapper

Raises:
list_files() → list[str][source]#

Lists all files in the archive.

Returns:

List of file paths in the archive.

Return type:

list[str]

remove_root_dir() → bool[source]#

Removes a common root directory or directories from all files in the archive.

Returns:

True if a common root directory was found and removed, False otherwise.

Return type:

bool

__getitem__(file_path: str | Path) → bytes | None[source]#

Gets the bytes of a file from the archive.

Parameters:

file_path (str | Path) – Path of the file within the archive.

Returns:

The file content as bytes, or None if the file doesn’t exist.

Return type:

bytes | None

__setitem__(file_path: str | Path, content: bytes)[source]#

Sets a file in the archive.

Parameters:

  • file_path (str | Path) – Path of the file within the archive.

  • content (bytes) – Content of the file.

to_tarball(mode='w:gz') → bytes[source]#

Creates a tarball from the in-memory files.

Returns:

The tarball as a byte string.

Return type:

bytes

from_tarball(*, tarball_bytes: bytes | None = None, tarball_path: Path | str | None = None)[source]#

Loads files from a tarball into the in-memory archive.

Parameters:

  • tarball_bytes (bytes) – The tarball as a byte string.

  • tarball_path (str | Path) – Path to the tarball file.

from_zip(*, zip_bytes: bytes | None = None, zip_path: Path | str | None = None)[source]#

Loads files from a zip archive into the in-memory archive.

Parameters:

  • zip_bytes (bytes) – The zip archive as a byte string.

  • zip_path (str | Path) – Path to the zip file.

from_archive(*, archive_bytes: bytes | None = None, archive_path: Path | str | None = None)[source]#

Loads files from an archive (zip or tarball) into the in-memory archive.

Parameters:

  • archive_bytes (bytes) – The archive as a byte string.

  • archive_path (str | Path) – Path to the archive file.

from_path(path: str | Path, include_extensions: list[str] | None = None)[source]#

Loads files from a given path to an archive or directory into the in-memory archive.

Parameters:

  • path (str | Path) – Path to the archive file or directory.

  • include_extensions (list[str] | None) – If provided, only files with these extensions will be included when loading from a directory.

to_disk(output_dir: str | Path)[source]#

Extracts the archive to disk.

Parameters:

output_dir (str | Path) – Directory where files will be extracted.

from_disk(input_dir: str | Path, include_extensions: list[str] | None = None)[source]#

Loads files from disk into the in-memory archive.

Parameters:

  • input_dir (str | Path) – Directory to load files from.

  • include_extensions (list[str] | None) – If provided, only files with these extensions will be included.

encode() → dict[source]#

Encodes the archive to a serializable dictionary.

Returns:

Mapping of file path -> {“content”: base64 str, “mode”: int, “mtime”: float}.

mode/mtime are omitted for files with no recorded metadata.

Return type:

dict

decode(encoded_map: dict)[source]#

Decodes a serializable dictionary into the archive.

Accepts both the current format (per-file dict with content/mode/mtime) and the legacy format (path -> base64 string) for backward compatibility.

Parameters:

encoded_map (dict) – Mapping produced by encode (new or legacy format).