Why we price on incremental deduplicated bytes

Most data infrastructure is priced on logical bytes: the size of the files you hand it. That is easy to measure and easy to invoice. It is also a quiet conflict of interest. Every duplicated shard, every re-exported dataset version, every copy of an evaluation set that ended up in a training bucket, is revenue for the vendor and waste for you.

Needset exists to remove that waste, so we cannot bill on it. We meter incremental deduplicated bytes: the bytes that are new to your catalog after chunk-level deduplication.

What the meter counts

When an asset is registered, it is split with a content-defined chunker and each chunk is hashed. Chunks that already exist in the tenant's catalog are referenced, not stored. The meter increases only by the size of chunks that did not exist before.

• Upload a 40 GB shard for the first time: 40 GB metered, minus internal duplication.
• Upload the same shard again under a different path: 0 bytes metered.
• Re-export it with two records changed: roughly the size of the changed chunks, typically a few megabytes.

Why this is safe for you

The meter is not a black box. It is exported on a Prometheus endpoint and as CSV, in real time, from the same counter that produces the invoice. If your dashboard and our invoice disagree, one of us has a bug and you can prove which.

Why this is good for us

It forces us to be good at deduplication. If our chunker misses shared bytes, we bill more and lose the customer's trust. If it finds them, we bill less on that upload and keep the customer for years. The incentives point the same way.

Assessments and pilots are fixed price for the same reason: the first decision should be made on evidence, not on a meter running against an unmeasured workload.

All posts