Menu
Cloud Infrastructure / Distributed Storage
Case 082026Distributed Systems / Cloud Storage Infrastructure

Blob-Cloud

Cloud-native distributed object store with global block-level deduplication

Blob-Cloud is a production-grade, cloud-native file storage and collaboration platform engineered in Go and React. It offloads server memory load via direct-to-S3 presigned PUTs and S3 Multipart Uploads for files up to 10 TB, slices data into 4 MB SHA-256 blocks for cross-tenant global deduplication, and streams upload verification through Cloudflare Worker edge compute. Real-time updates fan out across horizontally scaled API pods via a Redis Pub/Sub backplane, backed by PostgreSQL atomic transactions and an asynchronous SQS worker fleet.

Application servers should never be file proxies: decoupling ingestion to the storage edge enables linear horizontal scalability and multi-terabyte uploads while preserving cryptographic consistency.

GoReactAWS S3PostgreSQLRedisCloudflare Workers

Block Dedup

4 MB

client-sliced SHA-256 blocks indexed globally across all users

Upload Ceiling

10 TB

unbounded object size via S3 multipart upload orchestration

Edge Validation

SHA-256

in-stream cryptographic integrity verified at Cloudflare Worker

Blob-Cloud architecture diagram showing direct-to-cloud presigned ingestion and edge validation
Proof strip

The page keeps the strongest metrics and the strongest caveats in the same frame. Numbers are included because they clarify the system, not because they decorate it.

Server I/O

Zero Proxy

direct-to-cloud presigned PUT URLs bypass Go application RAM

Real-Time Fanout

Redis Backplane

pub/sub distributes upload and share events to all connected pods

Worker Fleet

SQS + GC

asynchronous thumbnail processing and dry-run capable orphan block GC

Case-study frame

The project is explained across seven recurring dimensions so the reader can understand not only what was built, but why the implementation matters, where the proof is strongest, and where the limits still are.

Case-study dimension

Why This Exists

Traditional file storage web apps proxy large uploads directly through the web application backend, consuming immense memory buffers, blocking event loops, and creating severe throughput bottlenecks.

Blob-Cloud was engineered to eliminate application server I/O entirely by generating short-lived S3 presigned URLs, enabling clients to stream multi-gigabyte files directly to object storage with cryptographic edge validation.

Case-study dimension

Why It Matters Now

With datasets, creative media, and model checkpoints scaling into hundreds of gigabytes, object storage systems must support unbounded file sizes without driving cloud compute bills through the roof.

Global content-addressed deduplication saves massive physical storage and egress bandwidth by storing identical blocks only once across the entire infrastructure.

Case-study dimension

What It Improves

The Go backend acts as a pure control plane: files are sliced into 4 MB blocks client-side, fingerprinted with SHA-256, and verified against a global block index before generating presigned URLs.

Already-uploaded blocks are instantly acknowledged without re-uploading bytes, slashing ingestion latency and network consumption for duplicate files.

Case-study dimension

What Makes It Different

For direct uploads through Cloudflare Workers, a custom Web Crypto streaming validator computes and validates SHA-256 hashes in-flight, rejecting corrupted blocks before they reach persistent storage.

A Redis Pub/Sub backplane fans out real-time WebSocket notifications across arbitrary API instances, allowing client tabs to reflect file additions and shared collaborator updates instantly.

Case-study dimension

Why It Is Hard To Replicate

Orchestrating S3 Multipart Uploads with client-side block indexing requires atomic database state transitions across files, blocks, and permissions tables.

Cleaning up unreferenced blocks requires a dedicated garbage collection engine that safely accounts for inflight upload sessions before pruning physical storage.

Case-study dimension

How It Can Grow

Partitioning the PostgreSQL database by user_id or migrating to a globally distributed database like CockroachDB for multi-region scale.

Adding client-side zero-knowledge encryption before block slicing to provide end-to-end encrypted storage without forfeiting deduplication within private teams.

Case-study dimension

Why It Lasts

The fundamental architectural principle—decoupling control metadata from bulk binary transfer—is the proven design pattern powering hyperscale cloud infrastructure like AWS S3 and Google Drive.

Built with clean Go standard library conventions, chi routing, and standard SQL migrations, the codebase is free of ephemeral framework baggage.

Technical proof

Ingestion Pipeline

Direct-to-cloud presigned URLs eliminate server memory bottlenecks

The client requests upload URLs by providing block hashes and sizes. The Go backend returns presigned S3 PUT URLs only for missing blocks, while files over 5 GiB trigger S3 Multipart Upload part URLs.

Clients stream blocks directly to the storage bucket or Cloudflare edge, shielding application pods from network saturation.

zero-proxy server architecture preserves backend RAMS3 Multipart Upload removes file size limits up to 10 TBresumable sessions return upload URLs only for missing chunks
Blob-Cloud direct upload pipeline sequence diagram

Cryptographic Edge

Cloudflare Workers validate SHA-256 stream integrity before persistence

To guarantee data integrity without proxying through EC2, a Cloudflare Worker intercepts direct block uploads and calculates SHA-256 hashes on the incoming byte stream.

Blocks with mismatched checksums are immediately aborted with HTTP 400, ensuring corrupted or malicious payloads never pollute the deduplicated block store.

Web Crypto streaming hash validation at the network edgeimmutable audit trail in PostgreSQL records every mutationper-zone rate limiting defends authentication and upload endpoints
Blob-Cloud edge validation and system architecture
Supplementary visuals
Blob-Cloud system architecture overview
End-to-end cloud infrastructure: Go chi router, PostgreSQL, S3, Redis, and Cloudflare Workers.
Blob-Cloud presigned upload sequence
Direct-to-S3 presigned pipeline with in-flight cryptographic verification.