Merge pull request #161 from Michael-C-Strobel/master

Swap to endpoint for fetching crumb

README.md

@@ -9,22 +9,43 @@ spreadsheets - currently supporting Yahoo's (FX, crypto, equities, indices, futu
 ## Latest version vs Yahoo HTTPS fingerprinting
 
 Latest version 3.8.0 was created to bypass Yahoo's recently adding crazy HTTPS fingerprinting 
-to their website. If you don't use Yahoo, no further changes are required.
+to their website. In a step back to before or rather a return to times long gone some Python 
+modules need to be installed such that LibreOffice can find them - otherwise Yahoo will not work. 
 
-If you do use Yahoo as a source, here is what I had to do to get Yahoo working again on my Ubuntu system:
-- install system-wide Python module curl_cffi - on my system as root: `pip3 install curl_cffi --upgrade`
-- download latest binary of [curl-impersonate](https://github.com/lwthiker/curl-impersonate/releases) e.g. 
-libcurl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz and untar it somewhere 
+Update for version: 3.8.2 - this bundles the Python module 'requests' and dependencies so users only 
+using 'FT' as source should not require anything else.
 
-Now some of these bits need to be loaded/initialised before running LibreOffice: I used the below (adjust your location 
-to libcurl-impersonate-chrome.so) to run LibreOffice Calc directly from command line:
+Everyone else using 'Yahoo' as source needs to install module 'curl_cffi'.
+
+### Ubuntu / Linux Mint / etc.
+
+Install Python curl_cffi module as root (such that LibroOffice can find it)
+
+- Optionally, if you don't have pip3 installed: ```sudo apt install python3-pip```
+
+- Then ```sudo pip3 install curl_cffi --upgrade```
+
+Note: For a normal Python script just installing curl_cffi is enough to bypass Yahoo's HTTPS fingerprinting. 
+Because LibreOffice on Linux is loading the stock curl library long before executing the extension 
+code directly, a second step are required. 
+
+The second bit requires a download of [curl-impersonate](https://github.com/lwthiker/curl-impersonate/releases) e.g.
+(currently) libcurl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz - unpack it somewhere 
+
+Then I used the below (adjust your location of libcurl-impersonate-chrome.so) to run LibreOffice Calc 
+directly from command line - alternatively you could define/export LD_PRELOAD and CURL_IMPERSONATE 
+e.g. in /etc/environment or ~/.bashrc - but make sure the variables are really set when you run LibreOffice. 
+Note: the setting chrome101 is just that - a setting on what browser to "impersonate". You don't 
+need to use or install Chrome for this.
 
 ```
 LD_PRELOAD=/tmp/curl-impersonate/libcurl-impersonate-chrome.so CURL_IMPERSONATE=chrome101 /usr/lib/libreoffice/program/soffice.bin --calc
 ```
 
-With this I can see the below in the output from `=GETREALTIME("SUPPORT")` and the examples.ods file from this repo 
-can load data for Yahoo again.
+In LibreOffice Calc this I can see something like the below in the output 
+from `=GETREALTIME("SUPPORT")` and the examples.ods file from this repo can load data 
+for Yahoo again.
+
 ```
 ...
 requests=curl_cffi_0.10.0
@@ -33,20 +54,22 @@ CURL_IMPERSONATE=chrome101
 curl_version="libcurl/8.1.1 BoringSSL zlib/1.2.11 brotli/1.0.9 nghttp2/1.56.0"
 ```
 
-Similar things should be possible on Windows - let me know if [this](https://stackoverflow.com/questions/1178257/ld-preload-equivalent-for-windows-to-preload-shared-libraries)
-is helpful and share your experience.
+### Windows
+ 
+- Download the script https://bootstrap.pypa.io/get-pip.py to your computer
 
-Background: for a normal Python script just installing curl_cffi is enough to bypass Yahoo's HTTPS fingerprinting. 
-Because LibreOffice is loading the stock curl library before executing the extension code directly, the above hack 
-is required. Unless someone tells me otherwise...
+- Start a Command Prompt (CMD) as Administrator on the command prompt run (change path as required)
 
-### Feedback requested:
+    ```"c:\Program Files\LibreOffice\program\python.exe" c:\temp\get-pip.py``` and then
 
-Please provide feedback about using the extension [here](https://github.com/cmallwitz/Financials-Extension/issues/10)
+    ```"c:\Program Files\LibreOffice\program\python.exe" -m pip install curl_cffi --upgrade```
 
-### Usage:
+- Add a new user environment variable CURL_IMPERSONATE, setting it to value chrome101, the Linux 
+LD_PRELOAD is not needed for the LibreOffice 7.1 I tested this with.  
 
-Under 'Releases' on GitHub [there](https://github.com/cmallwitz/Financials-Extension/releases) is a downloadable **Financials-Extension.oxt** file - load it into Calc 
+## Usage of extension:
+
+Under 'Releases' on GitHub is a [downloadable](https://github.com/cmallwitz/Financials-Extension/releases)  **Financials-Extension.oxt** file - load it into Calc 
 under menu item: Tools, Extension Manager...
 
 Please make sure, not to rename the OXT file when downloading and before installing: LO will mess up the installation otherwise and the extension won't work.
@@ -61,7 +84,7 @@ Getting data should be as simple as having this in a cell:
 Codes 21 and 90 stand for "last price" and "close" (see below), respectively. 
 Only Yahoo has historic data available.
 
-There is a file **examples.ods** in the Release area too with usage examples 
+There is a file **examples.ods** in the same Release area with usage examples 
 and possible arguments to functions.
 
 You have to check the respective websites to work out what symbol is the right one for you. Make sure today or the date 
@@ -152,13 +175,14 @@ refresh things.
 
 ### Build:
 
-You will need the LibreOffice SDK installed. 
+I only ever tried building on a Linux box. 
 
-On my system (Ubuntu) I installed packages: libreoffice-dev libreoffice-java-common libreoffice-script-provider-python
+You need to install LibreOffice SDK packages: libreoffice-dev libreoffice-java-common libreoffice-script-provider-python
+Since the Yahoo HTTPS fingerprinting issue, additionally curl_cffi needs to be installed (see beginning of README) 
 
 \# depending on your location...
 
-cd ~/tech/IdeaProjects/Financials-Extension/
+cd ~/tech/Financials-Extension/
 
 \# Assuming curl-cffi is installed, LD_PRELOAD is not required here 
 

compile.sh

@@ -59,27 +59,52 @@ cp -f "${PWD}"/src/financials_ft.py "${PWD}"/build/
 cp -f "${PWD}"/src/financials_yahoo.py "${PWD}"/build/
 cp -f "${PWD}"/src/financials_coinbase.py "${PWD}"/build/
 
-# this copies python modules dateutil, pytz, pyparsing to extension so it doesn't have to be installed by user
+# this copies python some modules to extension so they doesn't have to be installed by user
 
 TMPFILE=`mktemp`
 
+# https://pypi.org/project/python-dateutil/
 wget "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl" -O $TMPFILE
 unzip $TMPFILE dateutil/\* -d "${PWD}"/build/
 rm $TMPFILE
 
+# https://pypi.org/project/pytz/
 wget "https://files.pythonhosted.org/packages/81/c4/34e93fe5f5429d7570ec1fa436f1986fb1f00c3e0f43a589fe2bbcd22c3f/pytz-2025.2-py2.py3-none-any.whl" -O $TMPFILE
 unzip $TMPFILE pytz/\* -d "${PWD}"/build/
 rm $TMPFILE
 
+# https://pypi.org/project/pyparsing/
+# lastest version of "single-file" pyparsing 2.x - used by Ubuntu 22.04 as python3-pyparsing
 wget "https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl" -O $TMPFILE
 unzip $TMPFILE pyparsing.py -d "${PWD}"/build/
 rm $TMPFILE
 
-# Windows LibreOffice 7.1 Python is missing this...
+# https://pypi.org/project/six/
 wget "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl" -O $TMPFILE
 unzip $TMPFILE six.py -d "${PWD}"/build/
 rm $TMPFILE
 
+# https://pypi.org/project/requests/
+wget "https://files.pythonhosted.org/packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl" -O $TMPFILE
+unzip $TMPFILE requests/\* -d "${PWD}"/build/
+rm $TMPFILE
+
+# https://pypi.org/project/urllib3/
+# urllib3-2.2.3 is last version supporting Python 3.8 used by LibreOffice 7.1
+wget "https://files.pythonhosted.org/packages/ce/d9/5f4c13cecde62396b0d3fe530a50ccea91e7dfc1ccf0e09c228841bb5ba8/urllib3-2.2.3-py3-none-any.whl" -O $TMPFILE
+unzip $TMPFILE urllib3/\* -d "${PWD}"/build/
+rm $TMPFILE
+
+# https://pypi.org/project/certifi/
+wget "https://files.pythonhosted.org/packages/4a/7e/3db2bd1b1f9e95f7cddca6d6e75e2f2bd9f51b1246e546d88addca0106bd/certifi-2025.4.26-py3-none-any.whl" -O $TMPFILE
+unzip $TMPFILE certifi/\* -d "${PWD}"/build/
+rm $TMPFILE
+
+# https://pypi.org/project/idna/
+wget "https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl" -O $TMPFILE
+unzip $TMPFILE idna/\* -d "${PWD}"/build/
+rm $TMPFILE
+
 echo "Package into oxt file..."
 pushd "${PWD}"/build/
 zip -r "${PWD}"/Financials-Extension.zip ./*

src/financials_yahoo.py

@@ -168,6 +168,21 @@ class Yahoo(BaseClient):
 
         return text
 
+    def fetch_crumb(self):
+        """Refreshes the crumb using the current session."""
+        try:
+            # Grab cookies
+            self.session.get("https://finance.yahoo.com/quote/SPY", timeout=10)
+            
+            # Request crumb endpoint
+            response = self.session.get("https://query1.finance.yahoo.com/v1/test/getcrumb", timeout=10)
+            
+            if response.status_code == 200:
+                return response.text.strip()
+        except Exception as e:
+            logger.error(f"Failed to fetch crumb: {e}")
+        return None
+
     def getRealtime(self, ticker, datacode):
 
         """
@@ -195,33 +210,13 @@ class Yahoo(BaseClient):
         tick = self.realtime[ticker]
 
         if not self.crumb:
-
-            url = f'https://finance.yahoo.com/quote/{ticker}'
-            text = self.handleCookiesAndConsent(url, ticker, datacode, f'yahoo-{ticker}.html')
-
-            if text is None:
+            new_crumb = self.fetch_crumb()
+            if new_crumb:
+                self.crumb = new_crumb
+                logger.debug(f"Crumb successfully set: {self.crumb}")
+            else:
                 del self.realtime[ticker]
-                return 'Yahoo.getRealtime({}, {}) - handleCookiesAndConsent'.format(ticker, datacode)
-
-            # crumbs like 'TKkC\u002FZBwoUA' may contain unicode _text_ (not encoded code points)
-            try:
-                r = r'\bcrumb=([^"]{11,})"'
-                pattern = re.compile(r)
-                match = pattern.search(text)
-                if match:
-                    self.crumb = urllib.parse.unquote(match.group(1).encode('unicode-escape').decode('ascii'))
-                    logger.debug(f"crumb='{match.group(1)}' self.crumb='{self.crumb}'")
-                else:
-                    r = r'"crumb"\s*:\s*"([^"]{11,})"'
-                    pattern = re.compile(r)
-                    match = pattern.search(text)
-                    if match:
-                        self.crumb = urllib.parse.unquote(match.group(1).encode('unicode-escape').decode('ascii'))
-                        logger.debug(f"crumb='{match.group(1)}' self.crumb='{self.crumb}'")
-            except BaseException as e:
-                logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-                del self.realtime[ticker]
-                return 'Yahoo.getRealtime({}, {}) - crumb: {}'.format(ticker, datacode, e)
+                return f'Yahoo.getRealtime({ticker}, {datacode}) - failed to fetch crumb'
 
         if not self.crumb:
             return 'Yahoo.getRealtime({}, {}) - crumb missing'.format(ticker, datacode)

src/generate_metainfo.py

@@ -14,7 +14,7 @@ import os
 cur_dir = os.getcwd()
 
 addin_id = "com.financials.getinfo"
-addin_version = "3.8.0"
+addin_version = "3.8.2"
 addin_displayname = "Financial Market Extension"
 addin_publisher_link = "https://github.com/cmallwitz/Financials-Extension"
 addin_publisher_name = "The Publisher"