This reverts commit 3b786ab3.
On review the V3 (app, defaultProfile) collapse was too aggressive and
broke parity with how the rest of the ecosystem treats the profile
collection:
- Loop (NightscoutKit) and Trio (NightscoutAPI.swift:411) both POST
/api/v1/profile without _id on every settings edit, accumulating one
doc per upload via lib/server/profile.js:create(). They have done so
for years.
- The Nightscout profile collection is historical/append-only by
design; the NS UI profile editor lets users navigate prior
snapshots, and lib/server/profile.js:last() picks the most recent
for display.
- Collapsing AAPS V3 edits onto a single (app, defaultProfile) row
diverged from Loop/Trio/AAPS-V1 behavior and erased the upload
history that NS UI exposes.
The original 'AAPS edits not appearing' user complaint is sufficiently
addressed by:
- the V1 websocket retry dedup (commit 85f7e6ac), which kills the
60s ack-window race; and
- the {startDate: -1, _id: -1} secondary sort in profile.last()
(also 85f7e6ac), which deterministically picks the newest row
when startDate ties.
Both of those help every uploader (Loop, Trio, AAPS V1, AAPS V3)
without changing the ecosystem-wide profile-as-history semantic. The
characterization tests added in ddabdc6c are restored by this revert
and continue to document V3's request-level (date-based) dedup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Profile-store documents are singleton-per-(app, defaultProfile) by design:
each source (e.g. AAPS) has one current profile snapshot at a time. The
prior identifier scheme (uuidv5 of "undefined_<doc.date>") created a new
identifier on every edit because AAPS sends a new `date`
(LocalProfileLastChange) per save, accumulating duplicate profile docs in
MongoDB and causing 'AAPS profile edits not appearing' user reports.
Changes:
- operationTools.calculateIdentifier: special-case profile-store shape
(has `defaultProfile` + `store`, no `eventType`) -> identifier =
uuidv5("profilestore_<app>_<defaultProfile>"), so re-sends and edits
collapse onto the same row.
- update/validate: relax immutability of `date`, `created_at`,
`startDate` during deduplication when the storage doc is a
profile-store, since those fields are expected to advance per edit.
- api3.aaps-patterns tests updated to assert post-fix behavior:
edits return 200 + same identifier + single doc; distinct
defaultProfile names still produce distinct docs.
This complements the V1 (websocket) profile dedup fix in 85f7e6ac so
both AAPS sync paths now converge on a single profile document per
source.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
V3 POST /api/v3/profile uses uuid.v5("undefined_<doc.date>") as the
identifier (no device, no eventType in profile docs), so:
- Identical resends (retry) dedup in place (200) — request-level dedup works
- AAPS edits with a new LocalProfileLastChange produce a new identifier and
insert a new doc (201) — edit-level dedup does not exist
This characterizes the V3 behavior alongside the V1 websocket fix in the
prior commit. Both V1 (post-fix) and V3 rely on profile.last() with
{startDate: -1, _id: -1} for deterministic ordering.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AAPS V1 NSClient sync only ever calls nsAdd("profile", ...) — there is no
nsUpdate path for profiles (DataSyncSelectorV1.processChangedProfileStore).
Every profile edit reaches the server's websocket dbAdd handler.
Previously the dbAdd handler had no dedup branch for the 'profile' collection
and fell through to the generic else, which called insertOne() unconditionally
and then silently swallowed any insertion error via console.log + return [].
Result: each AAPS edit either created a duplicate profile document or failed
silently if the source JSONObject still carried an _id (E11000 dup key), so
users perceived their profile updates as not taking effect.
Changes:
- websocket.js: add a profile dedup branch — match on NSCLIENT_ID if present,
otherwise on startDate, and replaceOne in place rather than insertOne.
Returns the existing _id so the AAPS ack worker sees a stable identifier.
- websocket.js: upgrade the silent 'insertion error' console.log to
console.warn for both the profile branch and the generic fallback so
MongoDB write failures are visible in server logs.
- profile.js: add _id as a secondary sort key in last() so duplicate
startDate values resolve deterministically (newest insert wins) for any
legacy duplicates already present.
- tests/websocket.shape-handling.test.js: regression coverage for the
AAPS-shaped profile flow — first insert, repeated dbAdd with same
startDate (expect replace, not duplicate), and distinct startDate
(expect insert + last() returns newest).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The DB name check now only blocks when entries EXCEED the threshold.
If the database has fewer entries than the threshold (default 100),
it's treated as safe to test regardless of DB name — the name check
becomes a warning suggesting you rename for best practice.
Logic: entry count is the primary safety signal. DB name is secondary.
Both must fail to block. Entry count alone blocks. Name alone warns.
Changes:
- Entry count checked first to determine safety baseline
- DB name check downgrades to warning when entries within threshold
- Contextual override hints (only show relevant suggestions)
- Clarify CUSTOMCONNSTR_mongo vs CUSTOMCONNSTR_mongo_collection
Tests (5 new):
- 0 entries + non-test name → passes with warning
- 50 entries (below threshold) + non-test name → passes with warning
- Entries above threshold + non-test name → blocks (both errors)
- Entries above threshold alone → blocks with 'real data' message
- Non-test name hint mentions correct env var
Fixesnightscout/cgm-remote-monitor#8464
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
updateIdQuery() and upsertQueryFor() now use
{$or: [{identifier: UUID}, {_id: UUID}]}
instead of only {identifier: UUID}. This matches both:
- New documents (UUID in identifier field, ObjectId in _id)
- Legacy documents (UUID directly in _id, no identifier field)
Gated behind env.uuidHandling (UUID_HANDLING env var, default true).
All 30 treatment tests pass:
- 3 legacy UUID tests (issue-6923): DELETE, PUT, GET all work
- 12 gap-treat-012 tests: new data paths unaffected
- 15 uuid-handling tests: edge cases, UUID_HANDLING=false still works
Fixes#6923 (unable to edit/save/delete overrides for legacy data)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
_.isEmpty(new ObjectId()) returns true on driver 5.x because ObjectId
no longer has enumerable own properties (Object.keys returns []).
This caused filterForAge() to silently drop all entries with ObjectId
_id from the server cache.
check. This correctly accepts ObjectId instances and strings while
still rejecting null, undefined, and empty string.
16 tests confirm the fix:
- 3 root cause tests (_.isEmpty regression)
- 7 filterForAge logic tests (old vs fixed behavior)
- 6 integration tests (actual cache.js with data-update events)
Fixes AAPS backfill display bug where entries were in MongoDB but
invisible on the chart when using API V3 (mongoCachedCollection
emits data-update with raw ObjectId _id).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MongoDB driver 5.x ObjectId has no enumerable properties, so
_.isEmpty(new ObjectId()) returns true. This breaks cache.js
7 tests confirm:
- _.isEmpty(ObjectId) returns true (regression)
- filterForAge rejects ObjectId _id documents
- processRawDataForRuntime mitigates by converting to string
- Proposed fix (_id != null) works correctly
Currently mitigated in V1 paths by processRawDataForRuntime, but
unmitigated in API V3 mongoCachedCollection path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Inserts a treatment directly into MongoDB with UUID as _id (no identifier
field) — the shape of overrides created before normalizeTreatmentId(). All 3
tests fail as expected:
- DELETE: responds 200 but deletedCount=0 (silent no-op)
- PUT: creates duplicate document instead of updating in place
- GET: returns 0 results (query rewrite misses legacy doc)
These tests document the legacy data gap and will pass once updateIdQuery()
is updated to use a $or fallback: {identifier: UUID} || {_id: UUID}.
Relates to: #6923, #8450
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Change 'FAILED' to 'ACTIVATED' - this is protective, not a failure
- Explain that tests WILL DELETE data in the database
- Explain the purpose: preventing accidental production data loss
- List all override options clearly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The ctx.entries module isn't always available depending on boot context.
Access the entries collection directly via ctx.store.db.collection().
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds multi-layer protection against running destructive tests on production:
1. Pre-flight check (hooks.js): Verifies NODE_ENV=test before any DB connection
2. Database name check: Requires 'test' substring in database name
3. Entry count threshold: Refuses if database has >100 entries (configurable)
Environment Variables:
- TEST_SAFETY_MAX_ENTRIES: Max entries before refusing (default: 100)
- TEST_SAFETY_REQUIRE_TEST_DB: Require 'test' in DB name (default: true)
- TEST_SAFETY_SKIP: Emergency bypass for all checks (default: false)
Files:
- tests/lib/production-safety.js: Core safety check module
- tests/00_production-safety.test.js: Runs first to gate test suite
- tests/production-safety.test.js: Unit tests for safety module
- tests/hooks.js: Updated to use new module
This addresses concerns about users with 'test' in production DB names
by adding the entry count threshold as a secondary safety measure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add test for single entry returns array with one item
- Add test for empty array returns empty result
- Validates response format consistency for entries API
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add test for single activity returns array with one item
- Add test for activity array returns array
- Add test for empty array returns empty array
- Rename test file to follow *.test.js convention
Validates array normalization behavior for activity API.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add tests for POST with array of foods
- Add tests for PUT with array of foods
- Add test for empty array returning empty array
- Fix PUT endpoint to normalize array input like POST
- Fix food.save() storage to handle arrays with bulkWrite
- Rename test file to follow *.test.js convention
Validates fix from ef7bff3d for complete array handling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add array normalization to food API POST endpoint:
- API layer: normalize single object to array (like activity/profile)
- Storage layer: use replaceOne loop with upsert (same as activity pattern)
- Storage layer: accept both single object and array for backward compat
Previously POST /api/food/ with array input would crash:
insertOne([{...}]) → MongoDB error
Now supports both single object and array input consistently.
Response format is now array (matching treatments pattern).
Fixes#8447
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in activity and food APIs:
- activity: POST, PUT, DELETE now validate _id format
- food: POST, PUT, DELETE now validate _id format
Accepts: undefined, null, or 24-character hex string
Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request
Previously:
- activity: 500 crash on invalid _id in save/remove
- food: silently replaced invalid _id with new ObjectId (data loss)
Tests added covering all validation cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in devicestatus API:
- POST: validates each document's _id before storage
- DELETE: validates _id parameter (allows wildcard '*')
Accepts: undefined, null, or 24-character hex string
Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request
Previously, invalid _id values were silently stored as strings instead
of ObjectIds, causing inconsistent data and query issues.
Tests added for all validation cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in profile API:
- POST: validates each document's _id before storage
- PUT: validates _id format before update
- DELETE: validates _id parameter before removal
Accepts: undefined, null, or 24-character hex string
Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request
This prevents 500 errors from BSONError when clients send
UUID-style _ids (e.g., NightscoutKit).
Tests added for all validation cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NightscoutKit (Loop) sends profiles wrapped in arrays: [profile].
The MongoDB driver migration changed insert() to insertOne(), breaking
array support.
Changes:
- API layer: normalize input to array, purify each item
- Storage layer: use insertMany() instead of insertOne()
- Tests: verify single, array, and empty array handling
This matches the proven pattern from treatments API.
Fixes array handling regression introduced in d46c5b41.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UUID_HANDLING should ONLY affect UUID values in the _id field.
Previous commit incorrectly added server-side dedup for syncIdentifier
and uuid fields, which was never part of the original behavior.
Changes:
- upsertQueryFor(): Remove syncIdentifier/uuid as dedup keys
- Batch POST: Only fetch existing IDs by identifier, not by
syncIdentifier/uuid
- tests: Update TEST-CACHE-003/004 to document actual behavior
(duplicates occur without ObjectIdCache - this is by design)
- docs: Correct treatments-schema.md (syncIdentifier/uuid preserved,
not copied to identifier)
- docs: Remove external link from entries-schema.md
Loop carbs/doses rely on ObjectIdCache for dedup, not server-side logic.
This matches the original (pre-change) server behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
REQ-SYNC-072 scope correction: normalizeTreatmentId() should ONLY
handle UUID values in the _id field, not copy syncIdentifier or uuid
fields to identifier.
Changes:
- normalizeTreatmentId(): Only extract UUID from _id to identifier
- normalizeEntryId(): Same fix for entries collection
- upsertQueryFor(): Add syncIdentifier and uuid as dedup fallbacks
(fields are preserved, not copied to identifier)
- Batch POST: Fetch _id for docs deduped by syncIdentifier/uuid
Test updates:
- TEST-ID-003, TEST-V1-ID-004: Updated to expect identifier NOT copied
from syncIdentifier (scope fix)
Affected clients:
- Loop overrides (UUID _id → identifier): Still works
- Loop carbs/doses (syncIdentifier): Dedup works, no identifier copy
- xDrip+ (uuid): Dedup works, no identifier copy
- AAPS (identifier): Unchanged
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UUID_HANDLING default changed to true in 15.0.7. Test now explicitly
sets UUID_HANDLING=false rather than deleting the env var.
742 passing, 1 pending.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 6 tests verifying UUID_HANDLING env var behavior:
- UUID-OFF-001: GET by UUID returns empty (no crash)
- UUID-OFF-002: DELETE by UUID deletes nothing (no crash)
- UUID-ON-001: GET by UUID finds treatment via identifier
- UUID-ON-002: DELETE by UUID removes treatment via identifier
- UUID-ON-003: ObjectId still works normally
- UUID-ON-004: Non-matching UUID returns empty
Tests use clearModuleCache() to reload env.js with different flag values.
Refs: uuid-test-flag-off, uuid-test-flag-on, REQ-SYNC-072
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace Date.now() with the pre-captured 'now' variable in the
'set a pill to BWP with infos' test. This prevents timing drift
between when test data timestamps are set and when the sandbox
is initialized, eliminating flaky failures in CI environments.
Refs: BWP-TIME-001, GAP-TEST-001
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This test passes locally in ~50ms but occasionally times out at 30s
in constrained GitHub runners. Adding retries(2) allows it to recover
from transient CI resource contention.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The 'WebSocket dbAdd Array Handling Investigation' block was R&D to
understand insertOne behavior with arrays. The investigation concluded:
- MongoDB's insertOne([a,b]) creates single doc (not multiple)
- Fix: sequential processing via processNextItem() in websocket.js
Production tests now cover this behavior:
- 'dbAdd with array input for treatments - current behavior test'
- 'dbAdd with array input for devicestatus - current behavior test'
- 'dbAdd with array input for entries - current behavior test'
Removes 2 flaky investigative tests, keeps 729 production tests passing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Change from warning to process.exit(1) to prevent any possibility of
running destructive test operations against a production database.
Tests now fail immediately if NODE_ENV !== 'test', with clear instructions
on how to fix.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SAFETY-001: Fix tests/ci.test.env to use NODE_ENV=test instead of production
SAFETY-002: Add NODE_ENV check to tests/hooks.js with warning
SAFETY-003: Create tests/fixtures/test-guard.js with guarded deleteMany/drop helpers
This prevents deleteMany({}) from accidentally running against production
databases if test environment is misconfigured.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Trio/Loop upload CGM entries with UUID strings as _id field.
This caused MongoDB errors when re-uploading with different UUID
at same timestamp: "immutable field '_id'" error.
Fix:
- Add normalizeEntryId() to extract UUID from _id to identifier field
- Add upsertQueryFor() to strip non-ObjectId _id before $set
- Maintain sysTime+type as primary dedup key for CGM data integrity
- Add identifier to indexed fields
Tests:
- 3 baseline tests document current sysTime+type dedup behavior
- 6 UUID handling tests including the previously-failing scenario
Refs: GAP-SYNC-045, REQ-SYNC-072
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 3 new tests for explicit identifier field handling:
- supports identifier field for AAPS-style treatments
- deduplicates by identifier on re-upload
- supports batch upload with identifiers
These complement existing UUID _id tests (Loop pattern) to cover
both AID client sync patterns.
Refs: REQ-SYNC-072, GAP-TREAT-012
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TEST-CACHE-001: POST carb → cache syncIdentifier → PUT with id
TEST-CACHE-002: POST dose → cache syncIdentifier → DELETE with id
TEST-CACHE-003: Cache miss (24hr expiry) → POST same syncIdentifier
TEST-CACHE-004: App restart (cache empty) → POST existing syncIdentifier
TEST-CACHE-005: Batch POST → verify response order → cache mapping
7 new tests validating Loop's ObjectIdCache behavior:
- syncIdentifier → ObjectId mapping
- Response order for batch operations
- Deduplication by syncIdentifier
- Hex string syncIdentifier handling
All 7 tests passing.
Refs: Loop ObjectIdCache.swift analysis
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TEST-GAP-001: Loop override POST with UUID _id
TEST-GAP-002: Loop override DELETE by UUID
TEST-GAP-003: Loop override UPDATE by UUID
TEST-GAP-004: Loop override re-POST (upsert)
12 new tests validating REQ-SYNC-072 behavior:
- UUID _id promoted to identifier field
- Server generates valid ObjectId for _id
- Updates/deletes work via identifier lookup
- Duplicate detection via identifier
- Batch and edge case handling
New fixtures:
- loop-override.js: Real Loop override payload patterns
All 12 tests passing.
Refs: GAP-TREAT-012, REQ-SYNC-072
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes#8450 - Loop Temporary Override sync breaks due to UUID _id handling
Option G Implementation:
- Extract client sync identity (identifier) from any source:
- Loop overrides: UUID in _id field → moved to identifier
- Loop carbs/doses: syncIdentifier → copied to identifier
- AAPS: identifier already present
- xDrip+: uuid → copied to identifier
- Server generates proper ObjectId for _id field
- Deduplication uses identifier (not _id) as primary key
- No database migration needed - gradual adoption
Changes:
- normalizeTreatmentId(): extracts client identity to identifier field
- upsertQueryFor(): identifier-first lookup, strips UUID _id for upsert
- create()/upsert()/save(): fetch _id from DB after update by identifier
- Added 'identifier' to indexedFields for efficient querying
- Updated UUID treatment test with full workflow coverage
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ben WestandCopilot
Andy Low
copilot-swe-agent[bot]andbewest