hf3fs¶

`Client`
`DirEntry`
`BinaryFile`
`iovec`	用于批量读取和批量写入的共享内存，支持 buffer protocol
`listdir`
`scandir`
`walk`

class hf3fs.Client¶

__init__(self: hf3fs_pyclient.Client, mount_name: str, token: str, *, as_super: bool = False) → None¶

创建 client 对象

Parameters

mount_name – 挂载名，支持 prod / cpu / stage
token – 访问 3FS 所需的认证信息，可联系运维获取
as_super – 是否创建有 super 权限的 client（需 token 有 root 权限）

chdir(self: hf3fs_pyclient.Client, arg0: str) → None¶

chmod(self: hf3fs_pyclient.Client, path: str, mode: int, *, dir_fd: Optional[int] = None, follow_symlinks: bool = True) → None¶

chown(self: hf3fs_pyclient.Client, path: str, uid: int, gid: int, *, dir_fd: Optional[int] = None, follow_symlinks: bool = True) → None¶

close(self: hf3fs_pyclient.Client, arg0: int) → None¶

creat(self: hf3fs_pyclient.Client, path: str, mode: int = 438, *, dir_fd: Optional[int] = None, excl: bool = False) → int¶

fstat(self: hf3fs_pyclient.Client, arg0: int) → hf3fs_pyclient.stat_result¶

ftruncate(self: hf3fs_pyclient.Client, arg0: int, arg1: int) → None¶

iovalloc(self: hf3fs_pyclient.Client, bytes: int, *, numa: int = -1, global: bool = False, block_size: int = 0) → hf3fs_pyclient.iovec¶

创建 iovec 对象在给定 numa 上

Parameters

bytes – 分配的 iovec 大小（以 byte 为单位）
numa – iovec 绑定的 numa 编号，默认为不绑定numa
global – 是否是所有进程都能访问到的iovec，是的话可以share给其它进程，默认为false

Examples:

client.iovalloc(1 << 30)
client.iovalloc(1 << 30, numa=1)

iovfree(self: hf3fs_pyclient.Client, arg0: hf3fs_pyclient.iovec) → None¶

link(self: hf3fs_pyclient.Client, src: str, dst: str, *, old_dir_fd: Optional[int] = None, new_dir_fd: Optional[int] = None, follow_symlinks: bool = True) → None¶

lseek(self: hf3fs_pyclient.Client, fd: int, pos: int, how: int, *, readahead: Optional[int] = None) → int¶

mkdir(self: hf3fs_pyclient.Client, path: str, mode: int = 511, *, dir_fd: Optional[int] = None, recursive: bool = False) → None¶

open(self: hf3fs_pyclient.Client, path: str, flags: int, mode: int = 438, *, dir_fd: Optional[int] = None) → int¶: 使用 client 打开一个 3fs 上的文件，并返回 fd（与操作系统的 fd 不互通）路径需要使用服务端路径（详见 hf3fs.fuse.serverPath）

openIovecHandle(self: hf3fs_pyclient.Client, iovh: str) → hf3fs_pyclient.iovec¶

openWithFileHandles(self: hf3fs_pyclient.Client, fhs: List[int]) → List[int]¶

opendir(self: hf3fs_pyclient.Client, name: str, *, dir_fd: Optional[int] = None) → hf3fs_pyclient._DIR¶

preadv(self: hf3fs_pyclient.Client, arg0: List[Tuple[hf3fs_pyclient.iovec, int, int]]) → List[int]¶

批量读取数据到 iovec 上

Parameters: piov – 一个列表，其中每一项是 (iovec, fd, offset) 的格式

import hf3fs as h3
iov = client.iovalloc(1 << 30)
client.preadv([(h3.iovec(iov, 3072, 2048), 3, 1024)])
# 从 3 号 fd 中的文件，从 1024 字节开始，读取 2048 字节到 iov 上从 3072 字节开始的内存

pwritev(self: hf3fs_pyclient.Client, arg0: List[Tuple[hf3fs_pyclient.iovec, int, int]]) → List[int]¶

批量写入 iovec 上的数据到文件系统

Parameters: piov – 一个列表，其中每一项是 (iovec, fd, offset) 的格式

import hf3fs as h3
iov = client.iovalloc(1 << 30)
memoryview(h3.iovec(iov, 3072, 2048))[:] = bytes([1] * 2048)
client.pwrite([(h3.iovec(iov, 3072, 2048), 3, 1024)])
# 将从 iov 中，3072 字节开始的连续 2048 个字节写入到 3 号 fd 中的文件 offset 为 1024 的位置

read(self: hf3fs_pyclient.Client, fd: int, buf: buffer, *, readahead: Optional[int] = None) → int¶

readdir(self: hf3fs_pyclient.Client, arg0: hf3fs_pyclient._DIR) → Optional[hf3fs_pyclient.dirent]¶

readlink(self: hf3fs_pyclient.Client, path: str, *, dir_fd: Optional[int] = None) → str¶

realpath(self: hf3fs_pyclient.Client, path: str, *, dir_fd: Optional[int] = None, absolute: bool = False) → str¶

remove(self: hf3fs_pyclient.Client, path: str, *, dir_fd: Optional[int] = None, recursive: bool = False) → None¶

rename(self: hf3fs_pyclient.Client, src: str, dst: str, *, src_dir_fd: Optional[int] = None, dst_dir_fd: Optional[int] = None) → None¶

rewinddir(self: hf3fs_pyclient.Client, arg0: hf3fs_pyclient._DIR) → None¶

rmdir(self: hf3fs_pyclient.Client, path: str, *, dir_fd: Optional[int] = None, recursive: bool = False) → None¶

sharedFileHandles(self: hf3fs_pyclient.Client, fds: List[int]) → List[int]¶

sharedIovecHandle(self: hf3fs_pyclient.Client, iov: hf3fs_pyclient.iovec) → str¶

stat(self: hf3fs_pyclient.Client, path: str, *, dir_fd: Optional[int] = None, follow_symlinks: bool = True) → hf3fs_pyclient.stat_result¶

symlink(self: hf3fs_pyclient.Client, src: str, dst: str, *, dir_fd: Optional[int] = None) → None¶

unlink(self: hf3fs_pyclient.Client, path: str, *, dir_fd: Optional[int] = None) → None¶

utime(self: hf3fs_pyclient.Client, path: str, *, times: Optional[Tuple[float, float]] = None, ns: Optional[Tuple[int, int]] = None, dir_fd: Optional[int] = None, follow_symlinks: bool = True) → None¶

write(self: hf3fs_pyclient.Client, fd: int, buf: buffer, *, flush: bool = False) → int¶

class hf3fs.DirEntry(parentPath, name, etype, parentFd, client=None)[source]¶

class hf3fs.BinaryFile(path, mode, dir_fd=None, client=None, ignore_cache=False)[source]¶

class hf3fs.iovec¶

用于批量读取和批量写入的共享内存，支持 buffer protocol

可以通过 iovalloc 方法分配

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: hf3fs_pyclient.iovec, other: hf3fs_pyclient.iovec, buf: Optional[buffer]) -> None

从给定的 Python buffer 地址创建子 iovec

Args:
other: 父 iovec buf: Python buffer 对象，需要保证 buffer 完全在在父 iovec 地址区间内，且指向的内存是连续的
__init__(self: hf3fs_pyclient.iovec, other: hf3fs_pyclient.iovec, off: Optional[int] = None, bytes: Optional[int] = None) -> None

从父 iovec 的一段连续内存创建子 iovec

Args:
other: 父 iovec off: 子 iovec 在父 iovec 中的起始位置（以 byte 为单位） bytes: 子 iovec 的长度（以 byte 为单位）

hf3fs.listdir(path='.', client=None)[source]¶

hf3fs.scandir(path='.', dir_fd=None, client=None)[source]¶

hf3fs.walk(top, topdown=True, onerror=None, followlinks=False, dir_fd=None, client=None)[source]¶