A very minimal distributed filesystem meant solely for write-once-read-many workloads using differently sized drives under my homelab. Taking inspiration from lizardfs, moosefs, and razorfs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
hak8or df59591f28 Moreeee 2 weeks ago
CMake Initial commit 4 weeks ago
docs Initial commit 4 weeks ago
sources Moreeee 2 weeks ago
.gitignore Initial commit 4 weeks ago
CMakeLists.txt Moreeee 2 weeks ago
readme.md Initial commit 4 weeks ago

readme.md

MyFS

A dead simple file system optimized for homelab use, where we care primarly for read-many-write-once use-cases while making use of many variable sized drives via configurable erasure encoding chunks.

Example Case

Lets say we have a homelab of various Linux ISO’s, where we have a total of 23,108, of which 2154 are directories (remaining 20955 are files), giving us 7,710,013,041 kB or ~7.7 TB.

# With 32 kB chunks
> filter(d, V1 %% 32 != 0) %>% mutate(Overhead_32KB = 32 - (V1 %% 32)) %$% sum(Overhead_32KB)
[1] 349007

| File Chunk | Files Overhead | Padding Overhead (%) | |------------|----------------|----------------------| | 0 kB | | %0.000 | | 32 KB | 349,007 KB | %0.005 | | 64 kB | 726,031 kB | %0.010 | | 256 kB | 3,264,399 kB | %0.042 |

Stuff to grab

Random links

Ideally would use Mojette transform but no libraries for it exist yet, and it looks really complicated (when doing transform-inverse, which cells to attempt to populate? Just corners?).

Instead, maybe can just do Reed-Solomon, and if user wants to change the parity bits, then can just re-create original data, re-encode using new encoding table with more/less parity, and then save again for every chunk.