What changed, and when.

All notable changes to the Needset platform. Dates are release dates.

2026-09-30

Edge hardening

• Unauthenticated signup, sign-in link and lead posts are throttled per address (10/min, burst 20, configurable); repeated bad credentials lock an address out (20 failures in 10 minutes, 15-minute lockout, logged as auth.lockout); public reads go through the per-address limiter; /readyz is cached for 3 seconds.
• The database thread has a bounded queue (NEEDSET_MAX_PENDING, default 64): beyond it requests get 503 busy with Retry-After instead of queueing behind one abusive client. Long-running operations (compile, quote, deliver, scan, optimize, verify, activate, retire, download) are capped per workspace (NEEDSET_HEAVY_IN_FLIGHT, default 2). Downloads stream on a separate I/O pool; byte-range reads of unoptimized sources are single ranged reads, never a spool of the whole object.
• Sign-in links: a duplicate signup answers exactly like a new one and sends the existing user a sign-in link; each address receives at most 5 sign-in mails per 15 minutes whoever asked; unknown addresses take as long as a send; NEEDSET_PUBLIC_URL is required in production. Multipart text fields capped (16 fields, 1 MB each).
• Tenant-supplied URLs (webhooks, storage endpoints) must resolve to public addresses; redirects are not followed; allow_insecure webhooks are refused in production (needset/netsafety.py).
• Gateway: body cap 16 MB on every route but upload; unexpected methods 405; HSTS preload, COOP/CORP, Permissions-Policy; the site's CSP has no inline scripts (script-src 'self'); the .html redirect is absolute on our host. Local 8443 listener bound to loopback. Build context excludes secrets, .env, deploy files and scripts. bin/harden-droplet.sh: firewall, key-only SSH, fail2ban, unattended security upgrades, off-box backup copy. GO_LIVE.md covers Cloudflare in front of both hostnames.
• Marketing site rebuilt: plain-language home page with a plan receipt and portal screenshots, outcome-first product page, engineer-facing technology page, public changelog, downloadable SDK. process-wide counters.

The joint plan (schema needset.plan.v4)

The compiler now decides selection and execution together by minimising one objective,

J = a*stored_bytes + b*moved_bytes + g*decode_ms + d*stall_ms + e*energy_j + l*quality_risk, at weights

the request sets (cost_model; defaults reproduce the previous selection exactly). Spec: JOINT_PLAN_V4.md.

• Joint execution placement. Every selected candidate carries an execution decision inside the signed body: decode_at (consumer or service, chosen by cost or pinned), route (transfer, partial_reuse, reuse), codec, tier, target_memory, and the moved bytes, decode, stall, energy and quality-risk terms it costs. Chunks the consumer already holds (delivery ledger) or the plan already moves cost nothing to move again. metrics.objective totals the terms. execution request section: consumer, decode_at, target_memory, stall_ms_per_mib, decode_stall_fraction, cold_first_byte_ms, service_decode_factor, energy rates.
• Compile context in the plan. context records the consumer, the held chunks and validations that are in the pool, each source's tier and the calibration used, so verify_plan recompiles from the plan body alone. NeedsetPolicy.from_body rebuilds a policy from a stored plan for offline verifiers.
• Fidelity contracts and dual identity. Candidates have representation (exact | semantic), and semantic ones name substitutes_for (the exact content hash) with a substitution_error. The request's fidelity (exact default, bounded_error with max_error, task_validated with optional task) decides eligibility; a plan never selects both identities of one content (substituted). New exclusion reasons: fidelity_contract, fidelity_error_bound, fidelity_unvalidated, substituted. POST /v1/validations (approver) records a task validation for a semantic candidate; :revoke withdraws it; GET /v1/validations.
• Request compiler: ranges and records. POST /v1/data:deliver takes requests of asset ids or {asset_id, ranges: [[start, end]], records: [ordinal]} and returns the fewest chunks that reconstruct exactly that (needset.delivery.v2, with bytes_requested, per-asset chunks_in_asset, requests spans). Records resolve through POST /v1/data/assets/{id}:index-records (non-empty newline records, 16 bytes per record, keyed by source digest). Range: bytes=a-b on :download reads only the covering chunks (206). At most 100,000 spans per delivery; inverted, empty or out-of-range spans are refused.
• Chunk transport honours the decision. GET /v1/data/chunks/{sha256}?encoding=stored returns stored bytes with X-Needset-Codec for consumers that decode themselves; the default decodes and verifies on the service.
• Dependency-aware schedule and prefetch. Candidates declare depends_on; the plan carries schedule (dependencies first, otherwise selection order; cycles are refused; metrics.unmet_dependencies names in-pool dependencies the budget excluded). Delivery by plan lists chunks in schedule order. The reader (needset.torch) follows the schedule, prefetches within a bounded window across asset boundaries, decodes where the plan says, and reports the run when it completes.
• Telemetry loop. `POST /v1/runs:report {consumer, run_id, plan_id, bytes_moved, decode_ms, stall_ms, energy_j, quality, samples, marginal_quality_gain, run_cost_usd, minimum_gain_per_usd}` stores one report (idempotent on run_id), returns the calibration the next plan for that consumer will use (median observed over declared decode time, median stall per MiB moved) and, with gain and cost, a continue | stop verdict. A report by the plan's own consumer under a task-validated plan whose quality is below the plan's minimum_quality revokes that plan's substitutions (validation.revoked webhook). GET /v1/runs.
• Closed-loop re-representation. optimize takes codec (auto | identity | zlib-6); GET /v1/data/rerepresentation names assets whose active codec should change from selection frequency and measured decode density. The new representation goes through verify, approve, activate; rollback reverses it.
• Asset kinds. kind on ingest, upload and register (dataset, checkpoint, index, embedding, other); the storage report groups by kind.
• Isolation hardening. Asset ids are validated ([A-Za-z0-9][A-Za-z0-9._/-]{0,255}, no .., no leading slash); hosted object keys must live under the tenant's prefix; hosted prefix scans see only the tenant's prefix and name assets relative to it.
• SDK: plans.validate, plans.revoke_validations, plans.validations, plans.report_run, plans.runs, data.index_records, data.rerepresentation_advice, data.optimize(codec), data.deliver(requests=), data.open(byte_range=), data.chunk(encoded=). Portal: objective, consumer, fidelity and per-asset execution in the explain view; "Moves" column on plans. Runtime accepts v3 and v4 plans.
• Not built (roadmap, stated as such on the site): token-span requests, storage-side and DPU decode.

Evaluation-set fencing

• Candidates and data assets carry a role (training | evaluation). Evaluation candidates, and any candidate whose source or lineage is an evaluation asset, are refused by the compiler with reason evaluation_fenced and listed under fenced_evaluation in the signed plan body. Verification recompiles, so a plan that drops the fence fails. Savings proofs restate the fenced list. The role is its own column (added by an additive migration), so no other write can drop it. Changing a role is an approver action and is audited with the approver. POST /v1/data/assets/{id}:role, Needset.data.set_role, portal toggle on Data assets.

Cross-run reuse

• Per-tenant delivery ledger: POST /v1/data:deliver {consumer, asset_ids | plan_id, dry_run} returns the chunks a consumer (job, cluster, region) has not already received and records the delivery. Consumers fetch exactly those chunks through GET /v1/data/chunks/{sha256} (served from a per-tenant chunk index) and read chunk order from GET /v1/data/assets/{id}:representation; repeated deliveries transfer nothing. Whole (unoptimized) objects are ledgered by source digest. Every delivery writes a delivery_events row with bytes transferred and bytes skipped; savings proofs sum those rows for their window. Consumer names are limited to [A-Za-z0-9._-]. GET /v1/data/reuse, DELETE /v1/data/reuse/{consumer}, needset_reuse_bytes_held in Prometheus (label values escaped). Portal: Reuse panel and Deliver on plans.

Demand-driven retention

• GET /v1/data/retention?window_days= names assets no plan selected, nothing delivered and nothing changed within the window, with the evidence for every asset kept. Evaluation assets are never candidates. Retirement stays behind the approver-gated retire flow. Portal: Retention panel.

Decode-aware placement

• GET /v1/plans/{id}:placement tiers a plan's assets hot/warm/cold from selection frequency and the representation's measured decode cost (verify now records decode_ms). POST /v1/data/placement:apply (approver) sets S3 storage classes by in-place copy, limited to STANDARD and STANDARD_IA (archive classes are not reversible in place); objects over 5 GiB are skipped with a note; copy errors returned inside a 200 body are detected. Unsupported stores report so. Portal: Placement.

Adoption

• needset.torch: plan_records() (framework-free) and NeedsetPlanDataset (torch IterableDataset, sharded across workers) assemble a plan's assets from chunks through a local, digest-verified chunk cache and record the delivery only when every shard of an epoch has finished.
• POST /v1/plans:quote: the same deterministic plan compile would store (shared policy path), plus a cost quote, stored nowhere, readable by viewers. Needset.plans.quote, Needset.data.open, .chunk, .representation.
• Test fixtures run the whole suite on SQLite or, with NEEDSET_TEST_DATABASE_URL, on a live PostgreSQL (testing.py); both are green.

2026-09-29

Self-serve (signup, billing, storage connections)

• Workspaces can be created from the portal (/signup): company, name, work email. Free plan is active instantly; Enterprise requests are created pending and approved by a platform admin.
• Passwordless sign-in with one-time email links (15 minutes, single use) and HttpOnly session cookies; cookie-authenticated writes require the portal header (CSRF). API keys and OIDC unchanged.
• Plans and quotas enforced in the service (needset/accounts.py): bytes under management after deduplication, seats, keys, schedules, webhooks. Over quota returns HTTP 402 quota_exceeded.
• Stripe billing without the SDK (needset/billing.py): one-time Checkout for Assessment and Pilot, metered subscription for Platform (daily meter events), Customer Portal, signed webhooks with idempotent processing. bin/stripe-setup.py creates the catalog.
• Customer-owned storage: a tenant connects its own S3-compatible bucket from the portal; credentials are Fernet-encrypted at rest; the lifecycle resolves the tenant store per asset (source.store == "tenant"). Hosted uploads keep using the deployment's store.
• Team management: invites by email with roles, removal revokes sessions.
• Platform admin surface (/v1/admin/*, portal Admin tab): tenants, approval, plan and quota, leads. Website contact form posts leads to /v1/leads (CORS-limited to the site origin).
• Transactional email over SMTP (needset/mail.py); local runs without SMTP return the sign-in link in the response for development.
• Nightly pg_dump backup service in compose.yaml; optional secrets as empty files.
• Portal: sign-in and signup screens, onboarding checklist, Storage, Billing, Access (team + keys) and Admin views.

Deployment

• Gateway configuration split into deploy/Caddyfile (local) and deploy/Caddyfile.hosted (public hostname, automatic certificates), selected with NEEDSET_CADDYFILE; the previous single file used placeholders Caddy could not parse for the tls directive.
• Optional marketing site served by the same gateway (deploy/site.caddy, NEEDSET_SITE_DOMAIN, NEEDSET_SITE_DIR) with clean URLs, www redirect, 404 page and a strict CSP.
• Gateway attached to a non-internal frontend network so ports 80/443 are reachable.
• bin/needset-up writes secrets readable by the unprivileged service user (directory 700, files 644); the previous umask made the container fail to read /run/secrets/*.
• .env.example documents NEEDSET_ALLOW_API_KEY_APPROVAL=1 for NDA/demo tenants.

Added

• demo_seed.py: populates a tenant with a coherent, idempotent demonstration pilot through the public API. DEMO.md: laptop walkthrough and hosted reviewer-portal setup with per-reviewer revocable keys.
• Gateway takes NEEDSET_DOMAIN for hosted deployments and obtains public certificates automatically.
• Upload progress card with stage and percentage; buttons show a busy state while an operation runs.
• Connections overview: browser upload, S3-compatible storage, API and SDK, scheduled jobs and SSO, each with live status. Only integrations the product implements are listed.

Fixed

• SDK uploads never sent the closing multipart boundary, so the server waited indefinitely.
• Re-creating an existing pilot returned an internal error; it is now idempotent for an identical definition and a 409 conflict otherwise.
• Content-Security-Policy header construction is compatible with Python 3.11 as well as 3.12.
• Compose project name no longer embeds a release version.
• The Windows start script copies the administrator key to the clipboard only when it creates one.
• Large-dataset guidance: trees beyond the browser file limit are uploaded as one archive or registered in place with data-scan.

2026-09-23.2

Fixed

• Declared the scikit-learn dependency used by the semantic representation graph and made its import lazy; the service starts without it.
• Startup refuses a bootstrap tenant that differs from the tenant the configured API key already belongs to, instead of proceeding under the other tenant.
• Added clear startup errors when a Docker secret path is missing, empty, or is a directory instead of a file.
• Aligned the Compose bootstrap defaults with .env.example.
• Published the application only on the Windows host loopback interface for straightforward local testing.

Added

• Windows PowerShell setup and diagnostic scripts with cryptographically secure secret generation, validation, readiness checks, and actionable failure output.
• Windows quick-start and extracted-folder upload instructions.

2026-09-23

Added

• Portal single sign-on: OpenID Connect authorization-code flow with PKCE, executed in the browser against the configured issuer. Enabled by NEEDSET_OIDC_CLIENT_ID.
• Evaluation import: lm-eval-harness results and per-sample files, and any JSONL with a label and a score, become failure signals (needset/evalimport.py, POST /v1/failures:batch, needsetctl failures-import, portal Demand tab).
• Source prefix scan: register every object under a bucket or directory prefix in place, idempotently, reporting conflicts rather than overwriting (POST /v1/data/assets:scan, needsetctl data-scan, S3 ListObjectsV2 support).
• Plan explanation: per-candidate reasons, the failures that drove each selection, and the change that would admit each exclusion (GET /v1/plans/{id}:explain, portal "Why").
• Metering: Prometheus exposition endpoint and savings CSV export.
• Python SDK (needset/client.py), standard library only, streaming uploads and downloads.
• Schedules for recurring compile and scan jobs; HMAC-signed webhooks for plan, pilot, scan and savings events.
• Multi-file and folder upload as a deterministic tar container, written in a single pass while parts arrive.
• Content-Security-Policy with per-load nonces on the portal; inline script forbidden on API responses.
• Signed audit anchors and attestation to detect chain truncation or rewriting.
• Streaming upload and download endpoints; browser and CLI uploads of any size within the configured limit.
• Operator portal served by the service itself; role-aware; covers the full lifecycle, demand, planning, pilots, economics, audit and key management.
• Scoped API keys with least-privilege roles; key issuance and revocation.
• Deployment: one-command bring-up (bin/needset-up), environment template, Kubernetes manifest with S3 and spool volumes, operator and pilot runbooks.

Changed

• Compiler: single deterministic greedy over an explicit selection state; coverage floors are hard constraints; exact lazy evaluation with a block-overlap index (verified identical to exhaustive greedy on random pools). Plan schema needset.plan.v3.
• Plan verification is deterministic recompilation plus optional HMAC signature.
• Chunking: one windowed gear-hash content-defined chunker used by every path.
• Codec selection is deterministic (nominal cost model); measured selection is available for benchmarking only.
• All catalog, lifecycle, pilot and identity-provider calls run on a dedicated database thread; the event loop never blocks.
• Object reads are memory-mapped; S3 objects are spooled by ranged reads. Peak memory is independent of object size.
• Audit log rows carry an HMAC; pre-2026-09-21 catalogs are refused at startup.
• Approvals record the authenticated principal; production requires an OIDC identity for approvals unless explicitly overridden.
• Scout (needset_cli.py) measures block-level deduplication with the production chunker and codec instead of file-level gzip.
• RAG workload benchmark rebuilt so routing never sees the query's ground-truth label; both arms use an inverted index.
• GPU benchmark compares matched codecs on a realistic payload; evidence.py rejects cross-codec results.

Removed

• Separate in-memory block store and its CLI benchmark; the lifecycle store is the only storage path.
• The separate Next.js portal and its third-party hosted sign-in dependency.
• A bounded brute-force "novelty" comparison that could not support a claim.