Blob-Cloud
Cloud-native distributed object store with global block-level deduplication
Blob-Cloud is a production-grade, cloud-native file storage and collaboration platform engineered in Go and React. It offloads server memory load via direct-to-S3 presigned PUTs and S3 Multipart Uploads for files up to 10 TB, slices data into 4 MB SHA-256 blocks for cross-tenant global deduplication, and streams upload verification through Cloudflare Worker edge compute. Real-time updates fan out across horizontally scaled API pods via a Redis Pub/Sub backplane, backed by PostgreSQL atomic transactions and an asynchronous SQS worker fleet.
Application servers should never be file proxies: decoupling ingestion to the storage edge enables linear horizontal scalability and multi-terabyte uploads while preserving cryptographic consistency.
Block Dedup
4 MB
client-sliced SHA-256 blocks indexed globally across all users
Upload Ceiling
10 TB
unbounded object size via S3 multipart upload orchestration
Edge Validation
SHA-256
in-stream cryptographic integrity verified at Cloudflare Worker
The page keeps the strongest metrics and the strongest caveats in the same frame. Numbers are included because they clarify the system, not because they decorate it.
Server I/O
Zero Proxy
direct-to-cloud presigned PUT URLs bypass Go application RAM
Real-Time Fanout
Redis Backplane
pub/sub distributes upload and share events to all connected pods
Worker Fleet
SQS + GC
asynchronous thumbnail processing and dry-run capable orphan block GC
The project is explained across seven recurring dimensions so the reader can understand not only what was built, but why the implementation matters, where the proof is strongest, and where the limits still are.
Case-study dimension
Why This Exists
Traditional file storage web apps proxy large uploads directly through the web application backend, consuming immense memory buffers, blocking event loops, and creating severe throughput bottlenecks.
Blob-Cloud was engineered to eliminate application server I/O entirely by generating short-lived S3 presigned URLs, enabling clients to stream multi-gigabyte files directly to object storage with cryptographic edge validation.
Case-study dimension
Why It Matters Now
With datasets, creative media, and model checkpoints scaling into hundreds of gigabytes, object storage systems must support unbounded file sizes without driving cloud compute bills through the roof.
Global content-addressed deduplication saves massive physical storage and egress bandwidth by storing identical blocks only once across the entire infrastructure.
Case-study dimension
What It Improves
The Go backend acts as a pure control plane: files are sliced into 4 MB blocks client-side, fingerprinted with SHA-256, and verified against a global block index before generating presigned URLs.
Already-uploaded blocks are instantly acknowledged without re-uploading bytes, slashing ingestion latency and network consumption for duplicate files.
Case-study dimension
What Makes It Different
For direct uploads through Cloudflare Workers, a custom Web Crypto streaming validator computes and validates SHA-256 hashes in-flight, rejecting corrupted blocks before they reach persistent storage.
A Redis Pub/Sub backplane fans out real-time WebSocket notifications across arbitrary API instances, allowing client tabs to reflect file additions and shared collaborator updates instantly.
Case-study dimension
Why It Is Hard To Replicate
Orchestrating S3 Multipart Uploads with client-side block indexing requires atomic database state transitions across files, blocks, and permissions tables.
Cleaning up unreferenced blocks requires a dedicated garbage collection engine that safely accounts for inflight upload sessions before pruning physical storage.
Case-study dimension
How It Can Grow
Partitioning the PostgreSQL database by user_id or migrating to a globally distributed database like CockroachDB for multi-region scale.
Adding client-side zero-knowledge encryption before block slicing to provide end-to-end encrypted storage without forfeiting deduplication within private teams.
Case-study dimension
Why It Lasts
The fundamental architectural principle—decoupling control metadata from bulk binary transfer—is the proven design pattern powering hyperscale cloud infrastructure like AWS S3 and Google Drive.
Built with clean Go standard library conventions, chi routing, and standard SQL migrations, the codebase is free of ephemeral framework baggage.
Ingestion Pipeline
Direct-to-cloud presigned URLs eliminate server memory bottlenecks
The client requests upload URLs by providing block hashes and sizes. The Go backend returns presigned S3 PUT URLs only for missing blocks, while files over 5 GiB trigger S3 Multipart Upload part URLs.
Clients stream blocks directly to the storage bucket or Cloudflare edge, shielding application pods from network saturation.
Cryptographic Edge
Cloudflare Workers validate SHA-256 stream integrity before persistence
To guarantee data integrity without proxying through EC2, a Cloudflare Worker intercepts direct block uploads and calculates SHA-256 hashes on the incoming byte stream.
Blocks with mismatched checksums are immediately aborted with HTTP 400, ensuring corrupted or malicious payloads never pollute the deduplicated block store.
Next case
NullRing