updateIdQuery() and upsertQueryFor() now use
{$or: [{identifier: UUID}, {_id: UUID}]}
instead of only {identifier: UUID}. This matches both:
- New documents (UUID in identifier field, ObjectId in _id)
- Legacy documents (UUID directly in _id, no identifier field)
Gated behind env.uuidHandling (UUID_HANDLING env var, default true).
All 30 treatment tests pass:
- 3 legacy UUID tests (issue-6923): DELETE, PUT, GET all work
- 12 gap-treat-012 tests: new data paths unaffected
- 15 uuid-handling tests: edge cases, UUID_HANDLING=false still works
Fixes#6923 (unable to edit/save/delete overrides for legacy data)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
_.isEmpty(new ObjectId()) returns true on driver 5.x because ObjectId
no longer has enumerable own properties (Object.keys returns []).
This caused filterForAge() to silently drop all entries with ObjectId
_id from the server cache.
check. This correctly accepts ObjectId instances and strings while
still rejecting null, undefined, and empty string.
16 tests confirm the fix:
- 3 root cause tests (_.isEmpty regression)
- 7 filterForAge logic tests (old vs fixed behavior)
- 6 integration tests (actual cache.js with data-update events)
Fixes AAPS backfill display bug where entries were in MongoDB but
invisible on the chart when using API V3 (mongoCachedCollection
emits data-update with raw ObjectId _id).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MongoDB driver 5.x ObjectId has no enumerable properties, so
_.isEmpty(new ObjectId()) returns true. This breaks cache.js
7 tests confirm:
- _.isEmpty(ObjectId) returns true (regression)
- filterForAge rejects ObjectId _id documents
- processRawDataForRuntime mitigates by converting to string
- Proposed fix (_id != null) works correctly
Currently mitigated in V1 paths by processRawDataForRuntime, but
unmitigated in API V3 mongoCachedCollection path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Inserts a treatment directly into MongoDB with UUID as _id (no identifier
field) — the shape of overrides created before normalizeTreatmentId(). All 3
tests fail as expected:
- DELETE: responds 200 but deletedCount=0 (silent no-op)
- PUT: creates duplicate document instead of updating in place
- GET: returns 0 results (query rewrite misses legacy doc)
These tests document the legacy data gap and will pass once updateIdQuery()
is updated to use a $or fallback: {identifier: UUID} || {_id: UUID}.
Relates to: #6923, #8450
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Change 'FAILED' to 'ACTIVATED' - this is protective, not a failure
- Explain that tests WILL DELETE data in the database
- Explain the purpose: preventing accidental production data loss
- List all override options clearly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The ctx.entries module isn't always available depending on boot context.
Access the entries collection directly via ctx.store.db.collection().
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds multi-layer protection against running destructive tests on production:
1. Pre-flight check (hooks.js): Verifies NODE_ENV=test before any DB connection
2. Database name check: Requires 'test' substring in database name
3. Entry count threshold: Refuses if database has >100 entries (configurable)
Environment Variables:
- TEST_SAFETY_MAX_ENTRIES: Max entries before refusing (default: 100)
- TEST_SAFETY_REQUIRE_TEST_DB: Require 'test' in DB name (default: true)
- TEST_SAFETY_SKIP: Emergency bypass for all checks (default: false)
Files:
- tests/lib/production-safety.js: Core safety check module
- tests/00_production-safety.test.js: Runs first to gate test suite
- tests/production-safety.test.js: Unit tests for safety module
- tests/hooks.js: Updated to use new module
This addresses concerns about users with 'test' in production DB names
by adding the entry count threshold as a secondary safety measure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wip/test improvements - extends same fixes for data shape across all remaining api surface areas and includes test coverage across the test matrix spectrum.
- Add test for single entry returns array with one item
- Add test for empty array returns empty result
- Validates response format consistency for entries API
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add test for single activity returns array with one item
- Add test for activity array returns array
- Add test for empty array returns empty array
- Rename test file to follow *.test.js convention
Validates array normalization behavior for activity API.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add tests for POST with array of foods
- Add tests for PUT with array of foods
- Add test for empty array returning empty array
- Fix PUT endpoint to normalize array input like POST
- Fix food.save() storage to handle arrays with bulkWrite
- Rename test file to follow *.test.js convention
Validates fix from ef7bff3d for complete array handling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace sequential replaceOne calls with bulkWrite for batch upserts.
This improves performance when inserting multiple documents at once.
- activity.js: Use bulkWrite with replaceOne ops instead of forEach loop
- food.js: Same optimization for consistency
Both maintain exact same upsert behavior (query by _id+created_at or doc).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add array normalization to food API POST endpoint:
- API layer: normalize single object to array (like activity/profile)
- Storage layer: use replaceOne loop with upsert (same as activity pattern)
- Storage layer: accept both single object and array for backward compat
Previously POST /api/food/ with array input would crash:
insertOne([{...}]) → MongoDB error
Now supports both single object and array input consistently.
Response format is now array (matching treatments pattern).
Fixes#8447
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in activity and food APIs:
- activity: POST, PUT, DELETE now validate _id format
- food: POST, PUT, DELETE now validate _id format
Accepts: undefined, null, or 24-character hex string
Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request
Previously:
- activity: 500 crash on invalid _id in save/remove
- food: silently replaced invalid _id with new ObjectId (data loss)
Tests added covering all validation cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in devicestatus API:
- POST: validates each document's _id before storage
- DELETE: validates _id parameter (allows wildcard '*')
Accepts: undefined, null, or 24-character hex string
Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request
Previously, invalid _id values were silently stored as strings instead
of ObjectIds, causing inconsistent data and query issues.
Tests added for all validation cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in profile API:
- POST: validates each document's _id before storage
- PUT: validates _id format before update
- DELETE: validates _id parameter before removal
Accepts: undefined, null, or 24-character hex string
Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request
This prevents 500 errors from BSONError when clients send
UUID-style _ids (e.g., NightscoutKit).
Tests added for all validation cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The API layer was calling purifyObject() on the raw req.body without
handling arrays. When NightscoutKit sends [status], only the outer
array would be purified (no-op), not the individual status objects.
Now normalizes to array and purifies each devicestatus object,
matching the treatments pattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NightscoutKit (Loop) sends profiles wrapped in arrays: [profile].
The MongoDB driver migration changed insert() to insertOne(), breaking
array support.
Changes:
- API layer: normalize input to array, purify each item
- Storage layer: use insertMany() instead of insertOne()
- Tests: verify single, array, and empty array handling
This matches the proven pattern from treatments API.
Fixes array handling regression introduced in d46c5b41.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UUID_HANDLING should ONLY affect UUID values in the _id field.
Previous commit incorrectly added server-side dedup for syncIdentifier
and uuid fields, which was never part of the original behavior.
Changes:
- upsertQueryFor(): Remove syncIdentifier/uuid as dedup keys
- Batch POST: Only fetch existing IDs by identifier, not by
syncIdentifier/uuid
- tests: Update TEST-CACHE-003/004 to document actual behavior
(duplicates occur without ObjectIdCache - this is by design)
- docs: Correct treatments-schema.md (syncIdentifier/uuid preserved,
not copied to identifier)
- docs: Remove external link from entries-schema.md
Loop carbs/doses rely on ObjectIdCache for dedup, not server-side logic.
This matches the original (pre-change) server behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syncIdentifier and uuid fields are used for dedup, not copied to
identifier. Only UUID values in _id field are extracted to identifier.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
REQ-SYNC-072 scope correction: normalizeTreatmentId() should ONLY
handle UUID values in the _id field, not copy syncIdentifier or uuid
fields to identifier.
Changes:
- normalizeTreatmentId(): Only extract UUID from _id to identifier
- normalizeEntryId(): Same fix for entries collection
- upsertQueryFor(): Add syncIdentifier and uuid as dedup fallbacks
(fields are preserved, not copied to identifier)
- Batch POST: Fetch _id for docs deduped by syncIdentifier/uuid
Test updates:
- TEST-ID-003, TEST-V1-ID-004: Updated to expect identifier NOT copied
from syncIdentifier (scope fix)
Affected clients:
- Loop overrides (UUID _id → identifier): Still works
- Loop carbs/doses (syncIdentifier): Dedup works, no identifier copy
- xDrip+ (uuid): Dedup works, no identifier copy
- AAPS (identifier): Unchanged
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UUID_HANDLING default changed to true in 15.0.7. Test now explicitly
sets UUID_HANDLING=false rather than deleting the env var.
742 passing, 1 pending.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Write normalization (syncIdentifier/uuid → identifier) is ALWAYS on
- UUID_HANDLING flag only controls READ path (GET/DELETE by UUID)
- Fix default: UUID_HANDLING=true (not false)
- Remove incorrect xDrip+ mention from entries (doesn't use UUID _id)
- Clarify that flag only affects API calls with UUID as _id parameter
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The feature only handles the specific case where a UUID is sent as
the _id field itself. It does NOT affect:
- AAPS (uses 'identifier' field)
- xDrip+ (uses 'uuid' field)
- Loop carbs/doses (uses 'syncIdentifier' field)
Only affects:
- Loop overrides (_id: syncIdentifier.uuidString)
- Trio CGM entries (_id: UUID)
See docs/10-domain/client-id-handling-deep-dive.md for full analysis.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid implying all AID clients use the same pattern or that any
specific implementation is incorrect. Different clients have
divergent sync patterns - the feature accommodates this variety.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update default from false to true to enable AID client compatibility
out of the box:
- lib/server/env.js: readENVTruthy('UUID_HANDLING', true)
- README.md: Document UUID_HANDLING in Features section
- docs/example-template.env: Update comments, show true as default
Rationale:
- Loop, Trio, AAPS, xDrip+ use UUID sync patterns by default
- Before MongoDB 5.x, UUID _id didn't crash (just didn't CRUD properly)
- ObjectID users completely unaffected (quirk only triggers on UUID)
- Can set UUID_HANDLING=false for strict mode if needed
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
API returns _id as plain string "507f1f77bcf86cd799439011", not
MongoDB Extended JSON format {"$oid": "..."}.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document MongoDB pool settings and AUTH_FAIL_DELAY for test tuning.
Useful for CI or resource-constrained environments.
Refs: DOC-ENV-002
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 6 tests verifying UUID_HANDLING env var behavior:
- UUID-OFF-001: GET by UUID returns empty (no crash)
- UUID-OFF-002: DELETE by UUID deletes nothing (no crash)
- UUID-ON-001: GET by UUID finds treatment via identifier
- UUID-ON-002: DELETE by UUID removes treatment via identifier
- UUID-ON-003: ObjectId still works normally
- UUID-ON-004: Non-matching UUID returns empty
Tests use clearModuleCache() to reload env.js with different flag values.
Refs: uuid-test-flag-off, uuid-test-flag-on, REQ-SYNC-072
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace Date.now() with the pre-captured 'now' variable in the
'set a pill to BWP with infos' test. This prevents timing drift
between when test data timestamps are set and when the sandbox
is initialized, eliminating flaky failures in CI environments.
Refs: BWP-TIME-001, GAP-TEST-001
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- env.js: Add UUID_HANDLING env var (default: false)
- query.js: Add UUID detection in normalizeIdValue()
- When UUID_HANDLING=true and _id is UUID, search by identifier field
- Returns searchByIdentifier flag to redirect query
- treatments.js: Move queryOpts inside query_for() for env access
- entries.js: Same pattern for entries collection
When UUID_HANDLING=true:
- GET /treatments/{uuid} searches by identifier field
- DELETE /treatments/{uuid} deletes by identifier field
- Same behavior for entries collection
When UUID_HANDLING=false (default):
- UUID _id values return empty results (safe, no crash)
- Maintains backwards compatibility
Refs: uuid-feature-flag, uuid-query-impl from uuid-identifier-lookup.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
copilot-swe-agent[bot]andbewest