Update documentation and tooling for identifying and running flaky tests

Updates flaky test documentation with observed flakiness rates and corrects npm script syntax for isolating tests. Introduces a generic isolate runner and clarifies harness behavior regarding database state, removing the claim of automatic resets and noting reliance on Mocha hooks for cleanup.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 3be0ef95-aa69-441e-a33e-a8bd64e908de
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: b7dbc77f-8b96-4ac3-8453-8976ddb7ee24
Replit-Helium-Checkpoint-Created: true

.gitignore

@@ -21,6 +21,9 @@ static/bower_components/
 # istanbul output
 coverage/
 
+# flaky test results
+flaky-test-results/
+
 npm-debug.log
 *.heapsnapshot
 

docs/test-specs/flaky-tests.md

@@ -30,7 +30,7 @@ Flaky tests are tests that pass sometimes and fail other times without any code
 - Database state pollution from previous test runs
 - Missing proper test isolation and setup fixtures
 
-**Flakiness Rate:** ~20% (1 in 5 runs on clean database state)
+**Observed Flakiness:** Failed 4/19 tests on initial run, passed all 19 on subsequent runs (observed during manual testing session - actual flakiness rate may vary based on database state)
 
 **Harness:** `npm run test:flaky:entries`
 
@@ -53,7 +53,7 @@ Flaky tests are tests that pass sometimes and fail other times without any code
 - Event emission timing is non-deterministic
 - Server may not be fully ready when socket connects
 
-**Flakiness Rate:** ~10-15% (sporadic failures)
+**Observed Flakiness:** 2/8 tests failed in one run out of five consecutive runs (observed during manual testing - sporadic failures)
 
 **Harness:** `npm run test:flaky:socket`
 
@@ -76,7 +76,7 @@ Flaky tests are tests that pass sometimes and fail other times without any code
 - Database connection pooling affects timing
 - Server response time variability
 
-**Flakiness Rate:** ~10% (sporadic timeouts)
+**Observed Flakiness:** 1/11 tests failed in one observed run (sporadic timeouts)
 
 **Harness:** `npm run test:flaky:partial-failures`
 
@@ -125,7 +125,7 @@ The following npm scripts are available to run flaky tests in isolation:
 | `npm run test:flaky:entries` | Run entries tests in isolation |
 | `npm run test:flaky:socket` | Run socket tests in isolation |
 | `npm run test:flaky:partial-failures` | Run partial-failures tests in isolation |
-| `npm run test:flaky:isolate TEST=testname` | Run specific test file in isolation |
+| `TEST=testname npm run test:flaky:isolate` | Run any test file in isolation |
 
 ### Using the Flaky Test Runner
 
@@ -157,15 +157,25 @@ For debugging specific flaky tests, use the isolation harnesses:
 # Run entries tests 10 times in isolation
 npm run test:flaky:entries
 
+# Run socket tests 10 times
+npm run test:flaky:socket
+
+# Run any test file in isolation
+TEST=api.entries npm run test:flaky:isolate
+TEST=api3.socket npm run test:flaky:isolate
+
 # Run with custom iterations
 FLAKY_ITERATIONS=5 npm run test:flaky:entries
+FLAKY_ITERATIONS=5 TEST=api.entries npm run test:flaky:isolate
 ```
 
 These harnesses:
-1. Run tests in complete isolation from other test files
-2. Reset database state before each iteration
+1. Run the specified test file in isolation from other test files
+2. Execute multiple iterations sequentially and track pass/fail rates
 3. Capture detailed timing and error information
-4. Generate focused reports for the specific test file
+4. Generate JSON reports for the specific test file
+
+**Note:** The harnesses rely on the existing Mocha test hooks (`tests/hooks.js`) for any test cleanup. They do not perform additional database resets between iterations. Database state from one iteration may affect subsequent iterations, which can help identify state-dependent flakiness.
 
 ---
 
@@ -175,11 +185,12 @@ These harnesses:
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `FLAKY_TEST_ITERATIONS` | 10 | Number of test iterations |
+| `FLAKY_TEST_ITERATIONS` | 10 | Number of test iterations (main runner) |
 | `FLAKY_TEST_TIMEOUT` | 300000 | Timeout per iteration (ms) |
 | `FLAKY_OUTPUT_DIR` | ./flaky-test-results | Output directory |
 | `FLAKY_TEST_ENV_FILE` | ./my.test.env | Test environment file |
 | `FLAKY_ITERATIONS` | 10 | Iterations for isolation harnesses |
+| `TEST` | (required for isolate) | Test file name for generic isolate runner |
 
 ---
 
@@ -198,15 +209,12 @@ done
 
 ### Method 2: Fresh Database State
 
-Flaky tests often fail on clean database state. To reproduce:
+Flaky tests often fail on clean database state. To reproduce state-dependent failures:
 
-1. Clear the test database
+1. Clear the test database manually
 2. Run tests immediately after
 
-```bash
-# Reset database state and run tests
-npm run test:flaky:entries
-```
+This exposes tests that incorrectly assume pre-existing data.
 
 ### Method 3: Stress Testing
 
@@ -223,9 +231,9 @@ npm test & npm test
 
 ### Priority Order
 
-1. **High Impact**: Tests that fail frequently (>20% failure rate)
-2. **Medium Impact**: Tests that occasionally fail (5-20%)
-3. **Low Impact**: Rare failures (<5%)
+1. **High Impact**: Tests that fail frequently (>20% failure rate in observed runs)
+2. **Medium Impact**: Tests that occasionally fail (5-20% in observed runs)
+3. **Low Impact**: Rare failures (<5% in observed runs)
 
 ### General Fixes
 
@@ -266,4 +274,5 @@ Monitor flaky test trends over time by:
 - [Mocha Documentation](https://mochajs.org/)
 - [Testing Best Practices](https://github.com/goldbergyoni/javascript-testing-best-practices)
 - Main test runner: `scripts/flaky-test-runner.js`
+- Isolation harnesses: `scripts/flaky-harnesses/`
 - Existing test specs: `docs/test-specs/`

flaky-test-results/entries-isolation-results.json

@@ -1,27 +0,0 @@
-{
-  "testFile": "tests/api.entries.test.js",
-  "iterations": 2,
-  "passed": 2,
-  "failed": 0,
-  "passRate": "100.0",
-  "results": [
-    {
-      "iteration": 1,
-      "success": true,
-      "duration": 15313,
-      "exitCode": 0,
-      "passing": 19,
-      "failing": 0,
-      "error": null
-    },
-    {
-      "iteration": 2,
-      "success": true,
-      "duration": 14027,
-      "exitCode": 0,
-      "passing": 19,
-      "failing": 0,
-      "error": null
-    }
-  ]
-}

package.json

@@ -46,7 +46,8 @@
     "test:flaky:thorough": "FLAKY_TEST_ITERATIONS=20 node scripts/flaky-test-runner.js",
     "test:flaky:entries": "node scripts/flaky-harnesses/run-entries-isolation.js",
     "test:flaky:socket": "node scripts/flaky-harnesses/run-socket-isolation.js",
-    "test:flaky:partial-failures": "node scripts/flaky-harnesses/run-partial-failures-isolation.js"
+    "test:flaky:partial-failures": "node scripts/flaky-harnesses/run-partial-failures-isolation.js",
+    "test:flaky:isolate": "node scripts/flaky-harnesses/run-isolate.js"
   },
   "main": "lib/server/server.js",
   "nodemonConfig": {

scripts/flaky-harnesses/run-isolate.js

@@ -0,0 +1,173 @@
+#!/usr/bin/env node
+
+'use strict';
+
+const { spawn } = require('child_process');
+const fs = require('fs');
+const path = require('path');
+
+const TEST_FILE = process.env.TEST;
+if (!TEST_FILE) {
+  console.error('Error: TEST environment variable is required');
+  console.error('Usage: TEST=api.entries npm run test:flaky:isolate');
+  console.error('       TEST=api3.socket npm run test:flaky:isolate');
+  process.exit(1);
+}
+
+const testPath = TEST_FILE.includes('.test.js') 
+  ? `tests/${TEST_FILE}` 
+  : `tests/${TEST_FILE}.test.js`;
+
+if (!fs.existsSync(testPath)) {
+  console.error(`Error: Test file not found: ${testPath}`);
+  process.exit(1);
+}
+
+const CONFIG = {
+  testFile: testPath,
+  iterations: parseInt(process.env.FLAKY_ITERATIONS, 10) || 10,
+  timeout: parseInt(process.env.FLAKY_TEST_TIMEOUT, 10) || 60000,
+  outputDir: process.env.FLAKY_OUTPUT_DIR || './flaky-test-results',
+  testEnvFile: process.env.FLAKY_TEST_ENV_FILE || './my.test.env'
+};
+
+const results = [];
+
+function loadEnv() {
+  const env = { ...process.env };
+  if (fs.existsSync(CONFIG.testEnvFile)) {
+    const content = fs.readFileSync(CONFIG.testEnvFile, 'utf8');
+    content.split('\n').forEach(line => {
+      const trimmed = line.trim();
+      if (trimmed && !trimmed.startsWith('#')) {
+        const eqIndex = trimmed.indexOf('=');
+        if (eqIndex > 0) {
+          env[trimmed.substring(0, eqIndex)] = trimmed.substring(eqIndex + 1);
+        }
+      }
+    });
+  }
+  return env;
+}
+
+function runTest(iteration) {
+  return new Promise((resolve) => {
+    console.log(`\n--- Iteration ${iteration + 1}/${CONFIG.iterations} ---`);
+    const startTime = Date.now();
+    
+    const mochaPath = path.join(process.cwd(), 'node_modules', '.bin', 'mocha');
+    const args = [
+      '--timeout', '30000',
+      '--require', './tests/hooks.js',
+      '--exit',
+      CONFIG.testFile
+    ];
+    
+    let stdout = '';
+    let stderr = '';
+    
+    const testProcess = spawn(mochaPath, args, {
+      cwd: process.cwd(),
+      env: loadEnv(),
+      shell: false
+    });
+    
+    testProcess.stdout.on('data', data => {
+      stdout += data.toString();
+      process.stdout.write(data);
+    });
+    
+    testProcess.stderr.on('data', data => {
+      stderr += data.toString();
+      process.stderr.write(data);
+    });
+    
+    const timeoutId = setTimeout(() => {
+      testProcess.kill('SIGTERM');
+      resolve({
+        iteration: iteration + 1,
+        success: false,
+        duration: Date.now() - startTime,
+        error: 'Timeout'
+      });
+    }, CONFIG.timeout);
+    
+    testProcess.on('close', (code) => {
+      clearTimeout(timeoutId);
+      const duration = Date.now() - startTime;
+      
+      const passingMatch = stdout.match(/(\d+) passing/);
+      const failingMatch = stdout.match(/(\d+) failing/);
+      
+      resolve({
+        iteration: iteration + 1,
+        success: code === 0,
+        duration,
+        exitCode: code,
+        passing: passingMatch ? parseInt(passingMatch[1], 10) : 0,
+        failing: failingMatch ? parseInt(failingMatch[1], 10) : 0,
+        error: code !== 0 ? stderr.substring(0, 500) : null
+      });
+    });
+  });
+}
+
+async function main() {
+  console.log(`Flaky Test Isolation Harness: ${CONFIG.testFile}`);
+  console.log('='.repeat(50));
+  console.log(`Iterations: ${CONFIG.iterations}`);
+  console.log(`Timeout: ${CONFIG.timeout}ms`);
+  console.log('='.repeat(50));
+  
+  for (let i = 0; i < CONFIG.iterations; i++) {
+    const result = await runTest(i);
+    results.push(result);
+    console.log(`Result: ${result.success ? 'PASS' : 'FAIL'} (${result.duration}ms)`);
+  }
+  
+  const passed = results.filter(r => r.success).length;
+  const failed = results.filter(r => !r.success).length;
+  const passRate = ((passed / CONFIG.iterations) * 100).toFixed(1);
+  
+  console.log('\n' + '='.repeat(50));
+  console.log('SUMMARY');
+  console.log('='.repeat(50));
+  console.log(`Test file: ${CONFIG.testFile}`);
+  console.log(`Total iterations: ${CONFIG.iterations}`);
+  console.log(`Passed: ${passed}`);
+  console.log(`Failed: ${failed}`);
+  console.log(`Pass rate: ${passRate}%`);
+  
+  if (failed > 0) {
+    console.log('\nFailed iterations:');
+    results.filter(r => !r.success).forEach(r => {
+      console.log(`  - Iteration ${r.iteration}: ${r.error || 'Unknown error'}`);
+    });
+    console.log('\nThis test is FLAKY');
+  } else {
+    console.log('\nNo failures detected - test appears stable');
+  }
+  
+  if (!fs.existsSync(CONFIG.outputDir)) {
+    fs.mkdirSync(CONFIG.outputDir, { recursive: true });
+  }
+  
+  const baseName = path.basename(CONFIG.testFile, '.test.js');
+  const reportPath = path.join(CONFIG.outputDir, `${baseName}-isolation-results.json`);
+  fs.writeFileSync(reportPath, JSON.stringify({
+    testFile: CONFIG.testFile,
+    iterations: CONFIG.iterations,
+    passed,
+    failed,
+    passRate,
+    results
+  }, null, 2));
+  
+  console.log(`\nResults saved to: ${reportPath}`);
+  process.exit(failed > 0 ? 1 : 0);
+}
+
+main().catch(err => {
+  console.error('Fatal error:', err);
+  process.exit(1);
+});