v0.8.4

%s was missing from the format string. This enhances the message to
report the unexpected thearpy type along with the dump of the event that
trigger the exception.

.github/ISSUE_TEMPLATE/setup-help--tconnectsync-heroku.md

@@ -0,0 +1,53 @@
+---
+name: 'Setup help: tconnectsync-heroku'
+about: When experiencing an issue setting up tconnectsync-heroku
+title: ''
+labels: heroku, setup help
+assignees: ''
+
+---
+
+**Describe the problem**
+A clear and concise description of the issue you're experiencing.
+
+
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. ...
+2. ...
+
+
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+
+
+**Have you followed the [Troubleshooting steps in the README?](https://github.com/jwoglom/tconnectsync-heroku#troubleshooting)** (Yes or No)
+
+
+
+**Setup details**
+
+* **On what platform are you using the t:connect mobile app?** (Android or iOS)
+* **What version are you using of the t:connect mobile app?**
+
+
+
+**Heroku log output**
+If applicable, add the full output from the heroku logs in More > View Logs. ([See the instructions in the README.](https://github.com/jwoglom/tconnectsync-heroku#testing))
+
+
+
+**Check Login output**
+If applicable, [please follow the instructions to visit the `check_login` page](https://github.com/jwoglom/tconnectsync-heroku#testing), and copy and paste the output here.
+
+This file may contain sensitive details like your Nightscout URL and pump serial number. Copy the output into a text editor like Textedit or Notepad first and find-and-replace sensitive strings, like your Nightscout URL and pump serial number, if they appear.
+
+The output will also contain pump and CGM related information such as delivery events and blood sugars. If you would prefer to keep the contents private, send an email to tconnectsync<at>wogloms.net with the log output so @jwoglom can investigate.
+
+
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/setup-help--tconnectsync.md

@@ -0,0 +1,49 @@
+---
+name: 'Setup help: tconnectsync'
+about: When experiencing an issue setting up tconnectsync on Windows, Linux, or MacOS.
+title: ''
+labels: setup help
+assignees: ''
+
+---
+
+**Describe the problem**
+A clear and concise description of the issue you're experiencing.
+
+
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. ...
+2. ...
+
+
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+
+
+**Setup details**
+
+* **Operating system:** 
+* **tconnectsync version** (from `--version`): 
+* **On what platform are you using the t:connect mobile app?** (Android or iOS)
+* **What version are you using of the t:connect mobile app?**
+
+
+
+**Terminal output**
+If applicable, add the full output from your terminal from running tconnectsync. Copy the output into a text editor like Textedit or Notepad first and find-and-replace sensitive strings, like your Nightscout URL and pump serial number, if they appear.
+
+
+
+**Check Login output**
+If applicable, please run `--check-login` and upload the tconnectsync-check-output.log file as an attachment on the issue. This file should have sensitive details like your Nightscout URL and pump serial number automatically removed, but double-check before uploading.
+
+This log will contain pump and CGM related information such as delivery events and blood sugars. If you would prefer to keep the contents private, send an email to tconnectsync<at>wogloms.net with the log output so @jwoglom can investigate.
+
+
+
+**Additional context**
+Add any other context about the problem here.

README.md

@@ -116,7 +116,10 @@ First, ensure that you have **Python 3** with **Pip** installed:
 
 * **On MacOS:** Open Terminal. Install [Homebrew](https://brew.sh/), and then run `brew install python3`
 * **On Linux:** Follow your distribution's specific instructions.
-  For Debian/Ubuntu based distros, `sudo apt install python3 python3-pip`
+  - For Debian/Ubuntu based distros, `sudo apt install python3 python3-pip`
+  - For CentOS/Rocky Linux 8: 
+    - `sudo dnf install python39-pip`
+    - `sudo alternatives --set python /usr/bin/python3.9` 
 * **On Windows:** Install Ubuntu under the [Windows Subsystem for Linux](https://ubuntu.com/wsl).
   Open the Ubuntu Terminal, then run `sudo apt install python3 python3-pip`.
   Perform the remainder of the steps under the Ubuntu environment.
@@ -126,6 +129,12 @@ Now install the `tconnectsync` package with pip:
 ```
 $ pip3 install tconnectsync
 ```
+To install into a user environment instead of system-wide for a more contained install:
+````
+$ pip3 install --user tconnectsync
+````
+  - This will place the tconnectsync binary file at ``/home/<username>/.local/bin/tconnectsync``
+
 
 If the pip3 command is not found, run `python3 -m pip install tconnectsync` instead.
 
@@ -377,22 +386,27 @@ invoking tconnectsync with no arguments via cron.
 
 If using Pipenv or a virtualenv, make sure that you either prefix the call to main.py with `pipenv run` or source the `bin/activate` file within the virtualenv, so that the proper dependencies are loaded. If not using any kind of virtualenv, you can instead just install the necessary dependencies as specified inside Pipfile globally.
 
-An example configuration in `/etc/crontab` which runs every 15 minutes:
+An system-wide example configuration in `/etc/crontab` which runs every 15 minutes, on the 15 minute mark:
 
 ```bash
 # m         h  dom mon dow user   command
 0,15,30,45  *  *   *   *   root   /path/to/tconnectsync/run.sh
 ```
 
+An example of a user crontab `crontab -e` if not running system-wide, which runs every 15 minutes:
+```
+*/15 * * * * /path/to/tconnectsync/run.sh
+```
+
 You can use one of the same `run.sh` files referenced above, but remove the `--auto-update` flag since you are handling the functionality for running the script periodically yourself.
 
 ## Tandem APIs
 
 This application utilizes three separate Tandem APIs for obtaining t:connect data, referenced here by the identifying part of their URLs:
 
-* [**controliq**](https://github.com/jwoglom/tconnectsync/blob/master/tconnectsync/api/controliq.py) - Contains Control:IQ related data, namely a timeline of all Basal events uploaded by the pump, separated by type (temp basals, algorithmically-updated basals, or profile-updated basals).
+* [**controliq**](https://github.com/jwoglom/tconnectsync/blob/master/tconnectsync/api/controliq.py) - Contains Control:IQ related data, namely a timeline of all Basal events uploaded by the pump, separated by type (temp basals, algorithmically-updated basals, or profile-updated basals). Additionally includes CGM and Bolus data.
 * [**android**](https://github.com/jwoglom/tconnectsync/blob/master/tconnectsync/api/android.py) - Used internally by the t:connect Android app, these API endpoints were discovered by reverse-engineering the Android app. Most of the API endpoints are used for uploading pump data, and tconnectsync uses one endpoint which returns the most recent event ID uploaded by the pump, so we know when more data has been uploaded.
-* [**tconnectws2**](https://github.com/jwoglom/tconnectsync/blob/master/tconnectsync/api/ws2.py) - More legacy than the others, this seems to power the bulk of the main t:connect website. It is used to retrieve a CSV export of non-ControlIQ basal data, as well as bolus and IOB data. (I haven't found any mentions of bolus or IOB data in the Control:IQ-specific API.)
+* [**tconnectws2**](https://github.com/jwoglom/tconnectsync/blob/master/tconnectsync/api/ws2.py) - More legacy than the others, this seems to power the bulk of the main t:connect website. It is used as a last resort due to severe performance issues with this API (see https://github.com/jwoglom/tconnectsync/issues/43). We can use it to retrieve a CSV export of non-ControlIQ basal data, as well as bolus and IOB data. It is only used for bolus data as a fallback, and for pump-reported IOB data if requested. Full tracking of pump events also uses a limited version of this API.
 
 I have only tested tconnectsync with a Tandem pump set in the US Eastern timezone. Tandem's (to us, undocumented) APIs are [a bit loose with timezones](https://github.com/jwoglom/tconnectsync/blob/master/tconnectsync/parser.py#L15), so please let me know if you notice any timezone-related bugs.
 ## Backfilling t:connect Data
@@ -406,3 +420,15 @@ python3 main.py --start-date 2020-01-01 --end-date 2020-03-01
 In order to bulk-import a lot of data, you may need to use shorter intervals, and invoke tconnectsync multiple times. Tandem's API endpoints occasionally return invalid data if you request too large of a data window which causes tconnectsync to error out mid-way through.
 
 One oddity when backfilling data is that the Control:IQ specific API endpoints return errors if they are queried before you updated your pump to utilize Control:IQ. This is [partially worked around in tconnectsync's code](https://github.com/jwoglom/tconnectsync/blob/d841c3811aeff3671d941a7d3ff4b80cce6a219e/main.py#L238), but you might need to update the logic if you did not switch to a Control:IQ enabled pump immediately after launch.
+
+## t:connect API Testing
+
+To test t:connect API endpoints in a Python shell, you can do something like the following:
+
+```python
+import tconnectsync
+tconnectsync.util.cli.enable_logging()
+api = tconnectsync.util.cli.get_api()
+# Make API calls, e.g.
+therapy_timeline = api.controliq.therapy_timeline('2022-08-01', '2022-08-10')
+```

codecov.yml

@@ -0,0 +1,6 @@
+coverage:
+  status:
+    project:
+      default:
+        target: 50%
+        threshold: null

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = tconnectsync
-version = 0.8.0
+version = 0.8.4
 author = James Woglom 
 author_email = j@wogloms.net
 description = Syncs Tandem t:connect pump data to Nightscout for the t:slim X2

tconnectsync/api/controliq.py

@@ -14,9 +14,12 @@ class ControlIQApi:
     BASE_URL = 'https://tdcservices.tandemdiabetes.com/'
     LOGIN_URL = 'https://tconnect.tandemdiabetes.com/login.aspx?ReturnUrl=%2f'
 
+    LAST_CONFIRMED_SOFTWARE_VERSION = 't:connect 7.14.0.1'
+
     userGuid = None
     accessToken = None
     accessTokenExpiresAt = None
+    tconnect_software_ver = None
 
     def __init__(self, email, password):
         self.login(email, password)
@@ -47,6 +50,16 @@ class ControlIQApi:
             return True
 
     def _build_login_data(self, email, password, soup):
+        try:
+            version = soup.select_one("#footer_version").text.strip()
+            self.tconnect_software_ver = version
+            logger.info("Reported tconnect software version: %s" % version)
+            if version != self.LAST_CONFIRMED_SOFTWARE_VERSION:
+                logger.warn("Newer API version than last confirmed working. Saw %s and expected %s" % (version, self.LAST_CONFIRMED_SOFTWARE_VERSION))
+                logger.warn("If you experience any issues, please report them to https://github.com/jwoglom/tconnectsync")
+        except Exception:
+            logger.warn("Unable to find tconnect software version")
+            pass
         return {
             "__LASTFOCUS": "",
             "__EVENTTARGET": "ctl00$ContentBody$LoginControl$linkLogin",
@@ -126,10 +139,7 @@ class ControlIQApi:
         startDate = parse_date(start)
         endDate = parse_date(end)
 
-        return self.get('tconnect/controliq/api/summary/users/%s' % (self.userGuid), {
-            "startDate": startDate,
-            "endDate": endDate
-        })
+        return self.get('tconnect/controliq/api/summary/users/%s?startDate=%s&endDate=%s' % (self.userGuid, startDate, endDate), {})
     
     """
     Returns active account features, including the date when ControlIQ was enabled.

tconnectsync/api/webui.py

@@ -1,3 +1,4 @@
+from typing import List
 import requests
 import urllib
 import datetime
@@ -7,6 +8,10 @@ import logging
 
 from bs4 import BeautifulSoup
 
+from tconnectsync.domain.device_settings import Device, Profile, ProfileSegment
+from tconnectsync.util import removesuffix, removeprefix
+from tconnectsync.util.constants import MMOLL_TO_MGDL
+
 from .common import base_headers, ApiException
 
 logger = logging.getLogger(__name__)
@@ -92,12 +97,11 @@ class WebUIScraper:
                 settings_guid = settings_a.attrs['href'].split('?guid=')[1]
             
             if serial_number:
-                devices[serial_number] = {
-                    'name': device_name,
-                    'model_number': model_number,
-                    'status': status,
-                    'guid': settings_guid
-                }
+                devices[serial_number] = Device(
+                    name=device_name,
+                    model_number=model_number,
+                    status=status,
+                    guid=settings_guid)
         
         return devices
     
@@ -106,7 +110,7 @@ class WebUIScraper:
     Note that pump_guid is NOT the serial number of the pump, and
     should be obtained from my_devices()[str(serial_number)]['guid']
     """
-    def device_settings_from_guid(self, pump_guid):
+    def device_settings_from_guid(self, pump_guid: str) -> List[Profile]:
         profiles = []
         settings = {}
         r = self.get('myaccount/DeviceSettings.aspx?guid=%s' % pump_guid)
@@ -124,12 +128,33 @@ class WebUIScraper:
         
         return profiles, settings
     
-    def _parse_profile_tbl(self, tbl):
+    def _parse_profile_tbl(self, tbl) -> Profile:
         profile = {}
         profile["title"] = self.strip(tbl.select_one('.setting_title').text)
         profile["active"] = bool(tbl.find(text='Active at the time of upload'))
         profile["segments"] = []
 
+        def parse_basal_rate(rate) -> float:
+            return float(removesuffix(rate, ' u/hr'))
+        
+        def parse_factor(ratio) -> int:
+            return parse_bg_mgdl(removeprefix(ratio, '1u:'))
+
+        def parse_ratio(ratio) -> float:
+            return float(removesuffix(removeprefix(ratio, '1u:'), ' g'))
+        
+        def parse_bg_mgdl(bg) -> int:
+            if bg.endswith(' mg/dL'):
+                return float(removesuffix(bg, ' mg/dL'))
+            elif bg.endswith(' mmol/L'):
+                return float(removesuffix(bg, ' mmol/L')) * MMOLL_TO_MGDL
+            raise ValueError(bg)
+        
+        def hours_to_mins(text) -> int:
+            hrmin = removesuffix(text, " hours")
+            hr, min = hrmin.split(":", 1)
+            return int(min) + int(hr)*60
+
         for tr in tbl.select('tr'):
             # Skip header rows
             if tr.select_one('.setting_bg'):
@@ -149,19 +174,20 @@ class WebUIScraper:
                     t = "12:00 AM"
                 elif display_time == "Noon":
                     t = "12:00 PM"
+
                 segment = {
                     "display_time": display_time,
                     "time": t,
-                    "basal_rate": self.strip(tds[1].text),
-                    "correction_factor": self.strip(tds[2].text),
-                    "carb_ratio": self.strip(tds[3].text),
-                    "target_bg": self.strip(tds[4].text)
+                    "basal_rate": parse_basal_rate(self.strip(tds[1].text)),
+                    "correction_factor": parse_factor(self.strip(tds[2].text)),
+                    "carb_ratio": parse_ratio(self.strip(tds[3].text)),
+                    "target_bg_mgdl": parse_bg_mgdl(self.strip(tds[4].text))
                 }
-                profile["segments"].append(segment)
+                profile["segments"].append(ProfileSegment(**segment))
                 continue
             
             if tr.find(text='Calculated Total Daily Basal'):
-                profile["calculated_total_daily_basal"] = self.strip(tds[1].text)
+                profile["calculated_total_daily_basal"] = float(removesuffix(self.strip(tds[1].text), " units"))
                 continue
             
             # Last row
@@ -175,12 +201,12 @@ class WebUIScraper:
                     key = self.strip(key)
                     val = self.strip(val)
                     if key == 'Duration of Insulin':
-                        profile["insulin_duration"] = val
+                        profile["insulin_duration_min"] = hours_to_mins(val)
                     elif key == 'Carbohydrates':
-                        profile["carbohydrates"] = val
+                        profile["carbs_enabled"] = self.strip(val.lower()) == "on"
 
 
-        return profile
+        return Profile(**profile)
 
     def _parse_settings_tbl(self, tbl):
         outer_tr = tbl.select('tr')[2]

tconnectsync/api/ws2.py

@@ -21,14 +21,14 @@ class WS2Api:
         self.userGuid = userGuid
         self.session = base_session()
 
-    def get(self, endpoint):
-        r = self.session.get(self.BASE_URL + endpoint, headers=base_headers())
+    def get(self, endpoint, **kwargs):
+        r = self.session.get(self.BASE_URL + endpoint, headers=base_headers(), **kwargs)
         if r.status_code != 200:
             raise ApiException(r.status_code, "WS2 API HTTP %s response: %s" % (str(r.status_code), r.text))
         return r.text
 
-    def get_jsonp(self, endpoint):
-        r = self.session.get(self.BASE_URL + endpoint + '?callback=cb', headers=base_headers())
+    def get_jsonp(self, endpoint, **kwargs):
+        r = self.session.get(self.BASE_URL + endpoint + '?callback=cb', headers=base_headers(), **kwargs)
         if r.status_code != 200:
             raise ApiException(r.status_code, "WS2 API HTTP %s response: %s" % (str(r.status_code), r.text))
 
@@ -80,7 +80,7 @@ class WS2Api:
         endDate = parse_date(end)
 
         try:
-            req_text = self.get('therapytimeline2csv/%s/%s/%s?format=csv' % (self.userGuid, startDate, endDate))
+            req_text = self.get('therapytimeline2csv/%s/%s/%s?format=csv' % (self.userGuid, startDate, endDate), timeout=10)
         except ApiException as e:
             # This seems to occur as some kind of soft rate-limit.
             logger.warning("Received ApiException in therapy_timeline_csv: (retry count %d) %s" % (tries, e))
@@ -137,7 +137,7 @@ class WS2Api:
         endDate = parse_date(end)
         arg = "filterbasal/1" if filterbasal else ""
 
-        return self.get_jsonp('basalsuspension/%s/%s/%s/%s' % (self.userGuid, startDate, endDate, arg))
+        return self.get_jsonp('basalsuspension/%s/%s/%s/%s' % (self.userGuid, startDate, endDate, arg), timeout=10)
 
     """
     Returns info on BasalIQ in JSONP format.
@@ -146,4 +146,4 @@ class WS2Api:
         startDate = parse_date(start)
         endDate = parse_date(end)
 
-        return self.get_jsonp('basaliqtech/%s/%s/%s' % (self.userGuid, startDate, endDate))
+        return self.get_jsonp('basaliqtech/%s/%s/%s' % (self.userGuid, startDate, endDate), timeout=10)

tconnectsync/autoupdate.py

@@ -1,5 +1,6 @@
 import time
 import logging
+import datetime
 import sys
 
 from .process import process_time_range
@@ -67,7 +68,7 @@ class Autoupdate:
                             # (we can see the indexes increasing, so we know something's happening!)
                             if (now - last_action_or_start) >= 60 * self.secret.AUTOUPDATE_FAILURE_MINUTES:
                                 logger.error(AutoupdateFailureError(
-                                        "An event index change was recorded, but no new data was found via the API. " +
+                                        ("%s: An event index change was recorded, but no new data was found via the API. " % datetime.datetime.now()) +
                                         "The %s was %d minutes ago. This is a problem with tconnectsync." % 
                                         ("last processed event" if self.last_successful_process_time_range else "start of autoupdate", (now - last_action_or_start)//60)))
 
@@ -75,7 +76,7 @@ class Autoupdate:
                                     logger.error("Exiting with error code due to AUTOUPDATE_RESTART_ON_FAILURE")
                                     return 1
                             else:
-                                logger.warn(AutoupdateFailureWarning("An event index change was recorded, but no new data was found via the API. " +
+                                logger.warn(AutoupdateFailureWarning(("%s: An event index change was recorded, but no new data was found via the API. " % datetime.datetime.now()) +
                                             "The %s was %d minutes ago. Resetting TConnectApi to attempt to solve this problem." % 
                                             ("last processed event" if self.last_successful_process_time_range else "start of autoupdate", (now - last_action_or_start)//60)))
                                 
@@ -107,7 +108,7 @@ class Autoupdate:
                 # The most likely case here is that the pump isn't uploading right now.
                 if self.last_event_time and (now - self.last_event_time) >= 60 * self.secret.AUTOUPDATE_NO_DATA_FAILURE_MINUTES:
                     logger.error(AutoupdateNoEventIndexesDetectedError(
-                        "No new data event indexes have been detected for %d minutes. " % ((now - self.last_event_time)//60) +
+                        "%s: No new data event indexes have been detected for %d minutes. " % (datetime.datetime.now(), (now - self.last_event_time)//60) +
                         "The t:connect app might no longer be functioning."))
 
                     # TODO: restarting doesn't really help anything here.
@@ -124,11 +125,11 @@ class Autoupdate:
                 # above no indexes warning.
                 elif self.last_successful_process_time_range and (now - self.last_successful_process_time_range) >= 60 * self.secret.AUTOUPDATE_FAILURE_MINUTES:
                     logger.error(AutoupdateNoNewDataDetectedError(
-                        "No new data has been detected via the API for %d minutes. " % (now - self.last_successful_process_time_range)//60 +
+                        "%s: No new data has been detected via the API for %d minutes. " % (datetime.datetime.now(), now - self.last_successful_process_time_range)//60 +
                         "tconnectsync might not be functioning properly."))
 
                     if self.secret.AUTOUPDATE_RESTART_ON_FAILURE:
-                        logger.error("Exiting with error code due to AUTOUPDATE_RESTART_ON_FAILURE")
+                        logger.error("%s: Exiting with error code due to AUTOUPDATE_RESTART_ON_FAILURE" % datetime.datetime.now())
                         return 1
 
                 # Track how long we've been retrying

tconnectsync/check.py

@@ -21,7 +21,7 @@ Attempts to authenticate with each t:connect API,
 and returns the output of a sample API call from each.
 Also attempts to connect to the Nightscout API.
 """
-def check_login(tconnect, time_start, time_end, verbose=False, sanitize=False):
+def check_login(tconnect, time_start, time_end, verbose=False, sanitize=True):
     errors = 0
 
     loglines = []
@@ -76,6 +76,7 @@ def check_login(tconnect, time_start, time_end, verbose=False, sanitize=False):
     try:
         summary = tconnect.controliq.dashboard_summary(time_start, time_end)
         debug("ControlIQ dashboard summary: \n%s" % pformat(summary))
+        log("tconnect_software_ver: %s" % tconnect.controliq.tconnect_software_ver)
     except Exception as e:
         log("Error occurred querying ControlIQ API for dashboard_summary:")
         log(e)
@@ -110,27 +111,32 @@ def check_login(tconnect, time_start, time_end, verbose=False, sanitize=False):
     
     log("-----")
 
-    log("Logging in to t:connect WS2 API...")
+    log("Initializing t:connect WS2 API...")
+    ws2_loggedin = False
     try:
         summary = tconnect.ws2.basaliqtech(time_start, time_end)
         debug("WS2 basaliq status: \n%s" % pformat(summary))
+        ws2_loggedin = True
     except Exception as e:
-        log("Error occurred querying WS2 API:")
+        log("Error occurred querying WS2 API. This is okay so long as you are not using the PUMP_EVENTS or IOB sync features.")
         log(e)
         errors += 1
     
-    log("Querying WS2 therapy_timeline_csv...")
     lastReadingTime = None
-    try:
-        ttcsv = tconnect.ws2.therapy_timeline_csv(time_start, time_end)
-        debug("therapy_timeline_csv: \n%s" % pformat(ttcsv))
-        if ttcsv and "readingData" in ttcsv and len(ttcsv["readingData"]) > 0:
-            log("Last therapy_timeline_csv reading: \n%s" % pformat(ttcsv["readingData"][-1]))
-            lastReadingTime = TConnectEntry._datetime_parse(ttcsv["readingData"][-1]['EventDateTime'])
-    except Exception as e:
-        log("Error occurred querying WS2 therapy_timeline_csv:")
-        log(e)
-        errors += 1
+    if ws2_loggedin:
+        log("Querying WS2 therapy_timeline_csv...")
+        try:
+            ttcsv = tconnect.ws2.therapy_timeline_csv(time_start, time_end)
+            debug("therapy_timeline_csv: \n%s" % pformat(ttcsv))
+            if ttcsv and "readingData" in ttcsv and len(ttcsv["readingData"]) > 0:
+                log("Last therapy_timeline_csv reading: \n%s" % pformat(ttcsv["readingData"][-1]))
+                lastReadingTime = TConnectEntry._datetime_parse(ttcsv["readingData"][-1]['EventDateTime'])
+        except Exception as e:
+            log("Error occurred querying WS2 therapy_timeline_csv. This is okay so long as you are not using the PUMP_EVENTS or IOB sync features.")
+            log(e)
+            errors += 1
+    else:
+        log("Not able to log in to WS2 API, so skipping therapy_timeline_csv")
     
     log("-----")
 

tconnectsync/domain/device_settings.py

@@ -0,0 +1,27 @@
+from dataclasses import dataclass
+from typing import List, Optional
+
+@dataclass
+class Device:
+    name: str
+    model_number: str
+    status: str
+    guid: Optional[str]
+
+@dataclass
+class ProfileSegment:
+    display_time: str # Identical to time except written out as Midnight or Noon
+    time: str
+    basal_rate: float # _ u/hr
+    correction_factor: int # 1u: _ mg/dL
+    carb_ratio: float # 1u: _ g
+    target_bg_mgdl: int
+
+@dataclass
+class Profile:
+    title: str
+    active: bool
+    segments: List[ProfileSegment]
+    calculated_total_daily_basal: float # in units
+    insulin_duration_min: int
+    carbs_enabled: bool

tconnectsync/domain/therapy_event.py

@@ -51,12 +51,52 @@ class CGMTherapyEvent(TherapyEvent):
     },
     """
     @classmethod
-    def parse(self, json):
+    def parse(_, json):
+        self = CGMTherapyEvent()
         TherapyEvent.parse(self, json)
         self.eventID = json['eventID']
         self.egv = json['egv']['estimatedGlucoseValue']
         return self
 
+class BGTherapyEvent(TherapyEvent):
+    eventID = None
+    egv = None
+    """
+    {   
+        'bg': 160, # note in EGV
+        'cgmCalibration': 1, # not in EGV
+        'description': 'BG',
+        'deviceType': 't:slim X2 Insulin Pump',
+        'eventDateTime': '2022-08-20T07:25:24',
+        'eventTypeId': 16,
+        'indexId': 844955,
+        'interactive': 0,
+        'iob': 0.75,
+        'note': {   'active': False,
+                'eventId': 0, # different location than EGV
+                'eventTypeId': 16,
+                'id': 0,
+                'indexId': '',
+                'sourceRecordId': 0},
+        'requestDateTime': '0001-01-01T00:00:00',
+        'serialNumber': 'xxx',
+        'sourceRecId': 793549667,
+        'tempRateActivated': 0,
+        'tempRateCompleted': 0,
+        'tempRateId': 0,
+        'type': 'BG',
+        'uploadId': 748700213}
+    """    
+    @classmethod
+    def parse(_, json):
+        self = BGTherapyEvent()
+        TherapyEvent.parse(self, json)
+        self.eventID = json['note']['eventId']
+        # This is probably not how we want to provide CGM calibrations to Nightscout,
+        # but will just include it as egv data for now to keep the thing from crashing :)
+        self.egv = json['bg']
+        return self
+
 class BolusTherapyEvent(TherapyEvent):
     bolusRequestOptions = None
     REQUEST_AUTOMATIC = "Automatic Bolus/Correction"
@@ -364,4 +404,28 @@ class BolusTherapyEvent(TherapyEvent):
         "tempRateCompleted": 0,
         "tempRateActivated": 0
     }
-    """
+    CGM Calibration (Therapy Event Type BG):
+    {   'bg': 160,
+        'cgmCalibration': 1,
+        'description': 'BG',
+        'deviceType': 't:slim X2 Insulin Pump',
+        'eventDateTime': '2022-08-20T07:25:24',
+        'eventTypeId': 16,
+        'indexId': 844955,
+        'interactive': 0,
+        'iob': 0.75,
+        'note': {   'active': False,
+                    'eventId': 0,
+                    'eventTypeId': 16,
+                    'id': 0,
+                    'indexId': '',
+                    'sourceRecordId': 0},
+        'requestDateTime': '0001-01-01T00:00:00',
+        'serialNumber': 'xxx',
+        'sourceRecId': 793549667,
+        'tempRateActivated': 0,
+        'tempRateCompleted': 0,
+        'tempRateId': 0,
+        'type': 'BG',
+        'uploadId': 0}
+    """

tconnectsync/domain/utility.py

@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+
+class Time:
+    def __init__(self, hour: int, min: int):
+        self.hour = hour
+        self.min = min
+    
+    @classmethod
+    def parse(cls, input):
+        if ' ' not in input:
+            raise ValueError('unable to parse time: %s' % input)
+
+        hrmin, ampm = input.split(' ')
+        
+        hr, min = hrmin.split(':')
+        hr = int(hr)
+        min = int(min)
+
+        if ampm.lower() == 'pm':
+            hr += 12
+        elif ampm.lower() != 'am':
+            raise ValueError('unable to parse time: %s' % input)
+        
+        return cls(hr, min)

tconnectsync/nightscout.py

@@ -11,7 +11,7 @@ from .api.common import ApiException
 from .parser.nightscout import ENTERED_BY
 
 def format_datetime(date):
-	return arrow.get(date).strftime('%Y-%m-%d %H:%M:%S')
+        return arrow.get(date).isoformat()
 
 def time_range(field_name, start_time, end_time):
 	arg = ''

tconnectsync/parser/ciq_therapy_events.py

@@ -1,4 +1,4 @@
-from tconnectsync.domain.therapy_event import BolusTherapyEvent, CGMTherapyEvent
+from tconnectsync.domain.therapy_event import BolusTherapyEvent, CGMTherapyEvent, BGTherapyEvent
 from tconnectsync.parser.tconnect import TConnectEntry
 
 import logging
@@ -8,12 +8,16 @@ logger = logging.getLogger(__name__)
 def split_therapy_events(ciqTherapyEvents):
     bolusEvents = []
     cgmEvents = []
+    bgEvents = []
     for e in ciqTherapyEvents['event']:
         event = TConnectEntry.parse_therapy_event(e)
-        if type(event) == BolusTherapyEvent:
+        if isinstance(event, BolusTherapyEvent):
             bolusEvents.append(event)
-        elif type(event) == CGMTherapyEvent:
+        elif isinstance(event, CGMTherapyEvent): 
             cgmEvents.append(event)
+        elif isinstance(event, BGTherapyEvent):
+            bgEvents.append(event)
         
-    logger.debug("split_therapy_events: %d bolus, %d CGM" % (len(bolusEvents), len(cgmEvents)))
+    logger.debug("split_therapy_events: %d bolus, %d CGM, %d BG" % (len(bolusEvents), len(cgmEvents), len(bgEvents)))
+    # TODO: BG events (CGM Calibration) values are not currently returned from ciq_therapy_events.py
     return bolusEvents, cgmEvents

tconnectsync/parser/tconnect.py

@@ -3,7 +3,7 @@ import sys
 import arrow
 from tconnectsync.domain.bolus import Bolus
 
-from tconnectsync.domain.therapy_event import BolusTherapyEvent, CGMTherapyEvent
+from tconnectsync.domain.therapy_event import BolusTherapyEvent, CGMTherapyEvent, BGTherapyEvent
 
 try:
     from ..secret import TIMEZONE_NAME
@@ -200,6 +200,8 @@ class TConnectEntry:
             return BolusTherapyEvent.parse(data)
         elif data["type"] == "CGM":
             return CGMTherapyEvent.parse(data)
+        elif data["type"] == "BG":
+            return BGTherapyEvent.parse(data)
         
         raise UnknownTherapyEventException(data)
 
@@ -213,4 +215,5 @@ class UnknownBasalSuspensionEventException(Exception):
 
 class UnknownTherapyEventException(Exception):
     def __init__(self, data):
-        super().__init__("Unknown therapy event type: " % data)
+        typ = data["type"]
+        super().__init__(f"Unknown therapy event type: {typ} in {data}")

tconnectsync/process.py

@@ -87,17 +87,18 @@ def process_time_range(tconnect, nightscout, time_start, time_end, pretend, feat
        BOLUS_BG in features or \
        IOB in features:
         logger.warn("Downloading t:connect CSV data")
-        logger.warn("<!!> This data source is unreliable and may prevent timely synchronization")
         if bolusFallingBack:
-            logger.warn("Falling back on WS2 data source because BOLUS is an enabled feature and CIQ bolus data was empty!!")
+            logger.warn("Falling back on WS2 CSV data source because BOLUS is an enabled feature and CIQ bolus data was empty!!")
         if ciqFallingBack:
-            logger.warn("Falling back on WS2 data source because CGM is an enabled feature and CIQ cgm data was empty!!")
+            logger.warn("Falling back on WS2 CSV data source because CGM is an enabled feature and CIQ cgm data was empty!!")
         if BOLUS_BG in features:
-            logger.warn("<!!> Falling back on WS2 data source because BOLUS_BG is an enabled feature. " +
+            logger.warn("Falling back on WS2 CSV data source because BOLUS_BG is an enabled feature. " +
                         "Please consider disabling this feature to improve synchronization reliability.")
         if IOB in features:
-            logger.warn("<!!> Falling back on WS2 data source because IOB is an enabled feature. " +
+            logger.warn("Falling back on WS2 CSV data source because IOB is an enabled feature. " +
                         "Please consider disabling this feature to improve synchronization reliability.")
+        
+        logger.warn("<!!> The WS2 data source is unreliable and may prevent timely synchronization")
         csvdata = tconnect.ws2.therapy_timeline_csv(time_start, time_end)
 
         csvReadingData = csvdata["readingData"]
@@ -125,6 +126,7 @@ def process_time_range(tconnect, nightscout, time_start, time_end, pretend, feat
         if CGM in features:
             logger.debug("Writing CGM events")
             added += ns_write_cgm_events(nightscout, cgmData, pretend, time_start=time_start, time_end=time_end)
+            logger.debug("Finished writing CGM events")
 
     if BASAL in features:
         basalEvents = process_ciq_basal_events(ciqTherapyTimelineData)
@@ -137,12 +139,16 @@ def process_time_range(tconnect, nightscout, time_start, time_end, pretend, feat
         if basalEvents and len(basalEvents) > 0:
             logger.info("Last basal event from CIQ: %s" % basalEvents[-1])
 
+        logger.debug("Writing basal events")
         added += ns_write_basal_events(nightscout, basalEvents, pretend=pretend, time_start=time_start, time_end=time_end)
+        logger.debug("Finished writing basal events")
     
     if PUMP_EVENTS in features:
         pumpEvents = process_ciq_activity_events(ciqTherapyTimelineData)
         logger.debug("CIQ activity events: %s" % pumpEvents)
 
+        logger.warn("Using WS2 data source for basalsuspension because PUMP_EVENTS is an enabled feature")
+        logger.warn("<!!> The WS2 data source is unreliable and may prevent timely synchronization")
         ws2BasalSuspension = tconnect.ws2.basalsuspension(time_start, time_end)
 
         bsPumpEvents = process_basalsuspension_events(ws2BasalSuspension)
@@ -150,7 +156,9 @@ def process_time_range(tconnect, nightscout, time_start, time_end, pretend, feat
 
         pumpEvents += bsPumpEvents
 
+        logger.debug("Writing pump events")
         added += ns_write_pump_events(nightscout, pumpEvents, pretend=pretend, time_start=time_start, time_end=time_end)
+        logger.debug("Finished writing basal events")
 
     if BOLUS in features:
         bolusEvents = []
@@ -164,12 +172,16 @@ def process_time_range(tconnect, nightscout, time_start, time_end, pretend, feat
             logger.debug("ciq bolusEvents: %s" % bolusEvents)
         
         logger.info("finalized bolusEvents: %s" % bolusEvents)
+        logger.debug("Writing bolus events")
         added += ns_write_bolus_events(nightscout, bolusEvents, pretend=pretend, include_bg=(BOLUS_BG in features), time_start=time_start, time_end=time_end)
+        logger.debug("Finished writing bolus events")
 
     if csvIobData:
         if IOB in features:
             iobEvents = process_iob_events(csvIobData)
+            logger.debug("Writing iob events")
             added += ns_write_iob_events(nightscout, iobEvents, pretend=pretend)
+            logger.debug("Finished writing iob events")
 
     logger.info("Wrote %d events to Nightscout this process cycle" % added)
     return added

tconnectsync/secret.py

@@ -31,7 +31,7 @@ def get_bool(name, default):
 TCONNECT_EMAIL = get('TCONNECT_EMAIL', 'email@email.com')
 TCONNECT_PASSWORD = get('TCONNECT_PASSWORD', 'password')
 
-PUMP_SERIAL_NUMBER = get_number('PUMP_SERIAL_NUMBER', '11111111')
+PUMP_SERIAL_NUMBER = int(get_number('PUMP_SERIAL_NUMBER', '11111111'))
 
 NS_URL = get('NS_URL', 'https://yournightscouturl/')
 NS_SECRET = get('NS_SECRET', 'apisecret')

tconnectsync/util/__init__.py

@@ -1,5 +1,8 @@
 import arrow
 
+from . import cli
+from . import constants
+
 def timeago(timestamp):
     seconds = (arrow.get() - arrow.get(timestamp)).total_seconds()
     fmt = '%s ago' if seconds >= 0 else 'in %s'
@@ -15,3 +18,14 @@ def timeago(timestamp):
     ret += '%d minutes' % (seconds//60)
 
     return fmt % ret
+
+# String methods only available in python 3.9+
+def removesuffix(input_string, suffix):
+    if suffix and input_string.endswith(suffix):
+        return input_string[:-len(suffix)]
+    return input_string
+
+def removeprefix(input_string, prefix):
+    if prefix and input_string.startswith(prefix):
+        return input_string[len(prefix):]
+    return input_string

tconnectsync/util/cli.py

@@ -0,0 +1,20 @@
+import logging
+
+"""
+Enables logging at the specified level for all loggers
+inside the tconnectsync package, and sets up a basicConfig
+to print those log messages to stderr.
+"""
+def enable_logging(level=logging.DEBUG):
+    logging.basicConfig()
+    for logger in logging.root.manager.loggerDict:
+        if logger.startswith('tconnectsync'):
+            logging.getLogger(logger).setLevel(level)
+
+"""
+Returns a TConnectApi object with default secret parameters.
+"""
+def get_api():
+    from ..api import TConnectApi
+    from ..secret import TCONNECT_EMAIL, TCONNECT_PASSWORD
+    return TConnectApi(TCONNECT_EMAIL, TCONNECT_PASSWORD)

tconnectsync/util/constants.py

@@ -0,0 +1,4 @@
+
+# http://www.soc-bdr.org/rds/authors/unit_tables_conversions_and_genetic_dictionaries/conversion_glucose_mg_dl_to_mmol_l/index_en.html
+MMOLL_TO_MGDL = 18.0182
+MGDL_TO_MMOLL = 0.0555

tests/api/test_controliq.py

@@ -221,12 +221,10 @@ class TestControlIQApi(unittest.TestCase):
         ciq = ControlIQApi()
         ciq.userGuid = 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee'
 
-        def fake_get(endpoint, query):
+        def fake_get(raw_endpoint, ignored_query):
+            endpoint, query = raw_endpoint.split("?")
             self.assertTrue(endpoint.endswith("summary/users/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"))
-            self.assertEqual(query, {
-                "startDate": "04-01-2021",
-                "endDate": "04-02-2021"
-            })
+            self.assertEqual(query, "startDate=04-01-2021&endDate=04-02-2021")
 
             return {"faked_json": True}
 

tests/api/test_webui.py

@@ -11,6 +11,7 @@ import requests_mock
 from bs4 import BeautifulSoup
 
 from tconnectsync.api.webui import WebUIScraper
+from tconnectsync.domain.device_settings import Device, Profile, ProfileSegment
 
 from .fake import ControlIQApi
 
@@ -293,30 +294,30 @@ class TestWebUIScraper(unittest.TestCase):
 
             devices = webui.my_devices()
             self.assertDictEqual(devices, {
-                '100001': {
+                '100001': Device(**{
                     'name': 't:slim X2™ Insulin Pump', 
                     'model_number': '001002717', 
                     'status': 'Activated —  Dec 30 2021', 
                     'guid': '00000000-0000-0000-0000-000000000001'
-                },
-                '10000002': {
+                }),
+                '10000002': Device(**{
                     'name': 't:slim X2™ Insulin Pump', 
                     'model_number': '001000354', 
                     'status': 'Activated —  Oct 26 2021', 
                     'guid': '00000000-0000-0000-0000-000000000002'
-                }, 
-                '100003': {
+                }), 
+                '100003': Device(**{
                     'name': 't:slim X2™ Insulin Pump', 
                     'model_number': '001000096', 
                     'status': 'Activated —  Nov 20 2017', 
                     'guid': '00000000-0000-0000-0000-000000000003'
-                }, 
-                'ABCDEFGH': {
+                }), 
+                'ABCDEFGH': Device(**{
                     'name': 'OneTouch Verio IQ', 
                     'model_number': 'VERIO IQ', 
                     'status': 'Activated —  Jan 17 2018', 
                     'guid': None
-                }})
+                })})
     
     PUMP_SETTINGS_HTML = """
 <!DOCTYPE html
@@ -956,42 +957,42 @@ class TestWebUIScraper(unittest.TestCase):
                 text=self.PUMP_SETTINGS_HTML)
 
             profiles, settings = webui.device_settings_from_guid('00000000-0000-0000-0000-000000000001')
-            self.assertListEqual(profiles, [{
+            self.assertListEqual(profiles, [Profile(**{
                 'title': 'A', 
                 'active': True, 
-                'segments': [{
+                'segments': [ProfileSegment(**{
                     'display_time': 'Midnight', 
                     'time': '12:00 AM', 
-                    'basal_rate': '0.800 u/hr', 
-                    'correction_factor': '1u:30 mg/dL', 
-                    'carb_ratio': '1u:6.0 g', 
-                    'target_bg': '110 mg/dL'
-                }, {
+                    'basal_rate': 0.800,
+                    'correction_factor': 30.0,
+                    'carb_ratio': 6.0,
+                    'target_bg_mgdl': 110
+                }), ProfileSegment(**{
                     'display_time': '6:00 AM', 
                     'time': '6:00 AM', 
-                    'basal_rate': '1.250 u/hr', 
-                    'correction_factor': '1u:30 mg/dL', 
-                    'carb_ratio': '1u:6.0 g', 
-                    'target_bg': '110 mg/dL'
-                }, {
+                    'basal_rate': 1.250,
+                    'correction_factor': 30,
+                    'carb_ratio': 6.0,
+                    'target_bg_mgdl': 110
+                }), ProfileSegment(**{
                     'display_time': '11:00 AM', 
                     'time': '11:00 AM', 
-                    'basal_rate': '1.000 u/hr', 
-                    'correction_factor': '1u:30 mg/dL', 
-                    'carb_ratio': '1u:6.0 g', 
-                    'target_bg': '110 mg/dL'
-                }, {
+                    'basal_rate': 1.000,
+                    'correction_factor': 30,
+                    'carb_ratio': 6.0,
+                    'target_bg_mgdl': 110
+                }), ProfileSegment(**{
                     'display_time': 'Noon', 
                     'time': '12:00 PM', 
-                    'basal_rate': '0.800 u/hr', 
-                    'correction_factor': '1u:30 mg/dL', 
-                    'carb_ratio': '1u:6.0 g', 
-                    'target_bg': '110 mg/dL'
-                }], 
-                'calculated_total_daily_basal': '21.65 units', 
-                'insulin_duration': '5:00 hours', 
-                'carbohydrates': 'On'
-            }])
+                    'basal_rate': 0.800,
+                    'correction_factor': 30,
+                    'carb_ratio': 6.0,
+                    'target_bg_mgdl': 110
+                })], 
+                'calculated_total_daily_basal': 21.65, 
+                'insulin_duration_min': 5*60,
+                'carbs_enabled': True
+            })])
 
             self.assertDictEqual(settings, {
                 'Alerts': {

tests/api/test_ws2.py

@@ -10,7 +10,7 @@ from tconnectsync.api.common import ApiException
 class TestWS2Api(unittest.TestCase):
     def fake_get_with_http_500(self, num_times):
         tries = 0
-        def fake_get(endpoint):
+        def fake_get(endpoint, **kwargs):
             nonlocal tries, num_times
             if "therapytimeline2csv" in endpoint:
                 if tries < num_times:
@@ -94,7 +94,7 @@ Report Generated On, 4/24/2021 7:50:04 PM
 
         rawData = self.RAW_DATA_FULL
 
-        def fake_get(endpoint):
+        def fake_get(endpoint, **kwargs):
             nonlocal rawData
             if endpoint == 'therapytimeline2csv/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/2021-04-01/2021-04-02?format=csv':
                 return rawData
@@ -111,7 +111,7 @@ Report Generated On, 4/24/2021 7:50:04 PM
 
         rawData = ""
 
-        def fake_get(endpoint):
+        def fake_get(endpoint, **kwargs):
             nonlocal rawData
             if endpoint == 'therapytimeline2csv/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/2021-04-01/2021-04-02?format=csv':
                 return rawData