S4 Core is an S3-compatible object storage server written in Rust. Append-only volumes keep a billion objects in about a thousand files, deduplication stores every byte once, and a leaderless cluster with erasure coding keeps it all online.
$ docker run -d -p 9000:9000 -v s4-data:/data \ -e S4_ACCESS_KEY_ID=admin \ -e S4_SECRET_ACCESS_KEY=change-me \ s4core/s4core:latest $ export AWS_ACCESS_KEY_ID=admin AWS_SECRET_ACCESS_KEY=change-me $ aws --endpoint-url http://localhost:9000 s3 mb s3://photos make_bucket: photos $ aws --endpoint-url http://localhost:9000 s3 cp ./cat.jpg s3://photos/ upload: ./cat.jpg to s3://photos/cat.jpg # That's it. Every S3 tool you already use now talks to S4.
Speaks S3 — tested with the tools you already run
~1,000
files on disk for a billion objects
1.375×
raw capacity with erasure coding, instead of 3× replication
8.2×
faster EC encoding with the ISA-L codec
0
lines of unsafe Rust in the Community Edition — the compiler forbids it
The inode problem
Storage that keeps one file per object hits a wall long before the disk fills up. S4 was built around the opposite idea: many objects, few files.
1,000,000,000 objects → 1,000,000,000 files & inodes
No space left on device with terabytes still freefsck and rsync crawl through billions of directory entries1,000,000,000 objects → ~1,000 volume files + one index
fsync before 200 OKEverything in one binary
No sidecars, no external database, no JVM. Every feature below ships in the free Community Edition.
Signature V4 and V2, multipart uploads, CopyObject, CORS and ListObjects V1/V2. Point your SDK at a new endpoint and keep your code.
Bitcask-style append-only volumes and a fjall LSM index. Crash-safe journal, fast recovery, and no small-file penalty.
Content-addressed by SHA-256: identical data is stored once, however many keys point to it. Typically 30–50% less disk.
S3 Select on Apache DataFusion for CSV, JSON and Parquet — plus multi-object queries: SELECT … FROM 'logs/*.csv' with JOINs, CTEs and window functions.
Quorum replication (N=3, W=2, R=2): any node serves any request and one node can fail. SWIM gossip, gRPC, HLC clocks.
Merkle-tree anti-entropy, read repair, hinted handoff, and a bit-rot scrubber that rewrites damaged bytes from a healthy replica.
S3 object versioning with delete markers, and lifecycle rules that expire objects and old versions on schedule.
Retention periods and legal holds that make backups immutable — the last line of defence against ransomware.
Users with Reader, Writer and SuperUser roles, per-user S3 keys and JWT-authenticated admin endpoints.
Data is on disk before the client hears 200 OK, and a write cut off mid-flight is discarded, never half-kept. CI kills nodes mid-write to prove it.
Prometheus metrics for latency, IOPS, dedup ratio and capacity, a JSON stats API, and MinIO-compatible health endpoints.
Manage buckets, objects, users and keys from the browser. TLS through rustls end to end — no OpenSSL anywhere.
Enterprise Federation erasure coding
Erasure-coded pools store each object as data and parity shards across an erasure set. Reed-Solomon RS(6,3) keeps your data through three lost shards at 1.5× raw capacity — replication needs 3× to survive two. Clients see no difference: body, ETag, size and metadata are preserved.
Acknowledged on RF=3
A write lands on three replicas first, so latency stays the same as a replicated pool.
Encoded in the background
A transcode worker splits the object into data and parity shards and commits a manifest through the metadata quorum.
Verified before anything is deleted
The object is read back from its shards and checked against its SHA-256. Only then, after a grace period, are the hot copies released.
How much raw disk your data needs.
1 TB10 PB
Steady state for objects of 1 MiB and larger; smaller objects stay replicated unless packed segments are on. New writes land on RF=3 first and are encoded in the background, so keep headroom for them. “Survives” is the loss count guaranteed in every pattern within one erasure set.
MiB/s on one core, RS(6,3), 6 MiB chunks
| Operation | Portable | ISA-L |
|---|---|---|
| Encode | 39 | 318 |
| Degraded read | 44 | 475 |
| Shard repair | 119 | 928 |
Both engines write byte-identical shards: mix them in one cluster and switch with a restart. The x86_64 Enterprise image ships with ISA-L. Measured on an AMD Ryzen 9 7940HS.
Architecture
One process from the HTTP socket to the disk. The cluster layer switches on when you need it.
Hash the body with SHA-256
Seen this content before? Reference it instead of writing it again
New content is appended to the active volume
Index, dedup and version records commit in one atomic batch
fsync, then 200 OK — never the other way round
Editions
Both editions are built from the same code and released together, with the same version number.
For teams that want solid S3 storage today.
Freeforever · Apache 2.0
For petabytes, many pools and strict budgets.
Licenseverified offline
On the roadmap: LDAP, SAML and OIDC sign-in, WebDAV access, audit log and a deep SHA-256 scrubber.
Talk to us about Enterprise| Capability | Community | Enterprise |
|---|---|---|
| S3 API, versioning, lifecycle, Object Lock | ||
| Deduplication, compaction, crash recovery | ||
| IAM, web console, S3 Select & multi-object SQL | ||
| Quorum replication, anti-entropy, bit-rot healing | ||
| Cluster size | 1 pool · 3 nodes | unlimited |
| Erasure-coded pools (RS, LRC, custom) | ||
| ISA-L accelerated codec | ||
| Auto-tiering RF=3 ⇄ EC, multi-set pools, packed segments | ||
| Topology-aware LRC placement, node replacement with shard rebuild | ||
| License | Apache 2.0 | Elastic 2.0 |
Live demo
A public sandbox with the web console and the S3 API. It resets every 10 minutes — don't upload anything private.
Web console
Openlogin root · password password12345
S3 API
$ export AWS_ACCESS_KEY_ID=my-secret-key_id $ export AWS_SECRET_ACCESS_KEY=my-secret-access-key $ aws --endpoint-url http://63.141.251.44:10117 s3 ls
Quick start
Pick how you want to run it. Every snippet is complete — copy, paste, done.
# Run the Community Edition $ docker run -d --name s4core -p 9000:9000 -v s4-data:/data \ -e S4_ACCESS_KEY_ID=admin \ -e S4_SECRET_ACCESS_KEY=change-me \ s4core/s4core:latest # Talk to it with any S3 client $ export AWS_ACCESS_KEY_ID=admin AWS_SECRET_ACCESS_KEY=change-me $ aws --endpoint-url http://localhost:9000 s3 mb s3://my-bucket $ aws --endpoint-url http://localhost:9000 s3 cp ./file.txt s3://my-bucket/
# Server + web console (log in as root / password12345) $ git clone https://github.com/s4core/s4core.git && cd s4core $ S4_ROOT_PASSWORD=password12345 \ docker compose -f docker-compose-single-node.yml up -d --build # S3 API → http://localhost:9000 # Console → http://localhost:3000
# Three replicated nodes behind HAProxy (N=3, W=2, R=2) $ git clone https://github.com/s4core/s4core.git && cd s4core $ docker compose -f docker-compose-cluster.yml up -d # S3 API through HAProxy → http://localhost:9000 # HAProxy stats → http://localhost:8404 # Stop any one node — reads and writes keep working.
# Enterprise image; the license is verified offline $ docker run -d --name s4core -p 9000:9000 -v s4-data:/data \ -e S4_ACCESS_KEY_ID=admin \ -e S4_SECRET_ACCESS_KEY=change-me \ -e S4_LICENSE_KEY="$(cat license.key)" \ s4core/s4core:ee # Without a license key the EE image runs as Community Edition.
# Needs a stable Rust toolchain and protoc (protobuf compiler) $ git clone https://github.com/s4core/s4core.git && cd s4core $ cargo build --release $ S4_ACCESS_KEY_ID=admin S4_SECRET_ACCESS_KEY=change-me \ ./target/release/s4-server # Listening on http://127.0.0.1:9000
FAQ
Yes. Since v1.1 S4 Core ships as a production release. Every write is on disk before the client gets
200 OK, and crash recovery, cluster durability and bit-rot healing are covered by automated suites in CI.
Erasure-coded pools also report a production_ready flag, set
only on builds that passed the full EC acceptance suite.
Yes. S4 speaks the S3 API, so rclone sync or
mc mirror copy buckets across without any special tooling. S4 also
answers MinIO-compatible health and Prometheus endpoints, so existing probes and dashboards keep working.
Only cluster size and erasure coding. Community runs one node or one pool of up to three nodes with full quorum replication, self-healing and every S3 feature. There is no time limit, no user limit and no data limit, and the code is Apache 2.0.
No. The license key is signed with Ed25519 and verified locally, so air-gapped clusters work. Without a key, the Enterprise image simply runs as Community Edition.
Anything that speaks S3, including the AWS SDKs. Our test suites drive S4 with the AWS CLI, rclone, restic,
MinIO mc and the warp benchmark; both Signature V4 and V2 are
supported.
A Linux machine on x86_64 or aarch64 and a disk. S4 is a single binary with no external database. For erasure coding, the ISA-L codec runs on x86_64; AVX2 is enough — AVX-512 adds under a percent.
It is invisible to clients. Each object's content is hashed with SHA-256; identical content is stored once and reference-counted. A blob that fails its hash on read is never reused for new writes, so damage cannot spread through deduplication.
One command gives you an S3 endpoint on your own hardware.
$ docker run -d -p 9000:9000 -v s4-data:/data -e S4_ACCESS_KEY_ID=admin -e S4_SECRET_ACCESS_KEY=change-me s4core/s4core:latest