Fix for FT redirect to unencoded URLs

Improve Yahoo crumb handling
Fix Yahoo crumb handling
2026-08-24 10:04:10 -05:00 · 2024-09-02 20:37:37 +01:00 · 2024-09-01 17:35:26 +01:00 · 2024-05-17 15:56:43 +01:00 · 2024-05-01 18:35:41 +01:00 · 2024-03-13 12:11:42 +00:00
22 changed files with 1440 additions and 1162 deletions
@@ -1,4 +1,5 @@
 .idea
 src/__pycache__
 build/*
-Financials-Extension.oxt
+Financials-Extension.oxt
+_trial_temp
@@ -1,83 +1,108 @@
 # Financials-Extension

-This is a Python based extension for LibreOffice Calc to make stock market, index and FX data available in Calc 
-spreadsheets - currently supporting Yahoo's, Financial Times' and Google's (without FX data) finance websites 
-using old-fashioned web scraping.
+Version 3.3.0 includes improved cookie handling and somewhat improved logic to deal with network issues.   

-### System requirements:
+## Overview

-This extension requires the following Python 3 packages (on top of standard libs): dateutil, pytz, pyparsing. These are 
-from within LibreOffice, so if your LibreOffice comes with it's own Python runtime (e.g. Windows) they need to be
-installed there instead somewhere else in the system.  
+This is a Python based extension for LibreOffice Calc to make market data available in Calc 
+spreadsheets - currently supporting Yahoo's (FX, crypto, equities, indices, futures, options) and Financial Times' 
+(FX, equities, indices, futures) websites using old-fashioned web scraping. 

-On Ubuntu 18 and 20, dateutil and pytz may already be installed, but you can get all you need by running
- sudo apt-get install python3-dateutil python3-tz python3-pyparsing
+Starting with version 3.1.0, we received a contribution to get crypto data directly from Coinbase

-On Debian 10, the following steps have worked for me (as root)
- apt install python3-pip
- pip3 install python-dateutil
- pip3 install pytz
- pip3 install pyparsing
+### Feedback requested:
+
+Please provide feedback about using the extension [here](https://github.com/cmallwitz/Financials-Extension/issues/10)

 ### Usage:

-Have a look at the 'releases' tab on GitHub and download a prebuilt **Financials-Extension.oxt** file and load it into Calc 
+Under 'Releases' on GitHub [there](https://github.com/cmallwitz/Financials-Extension/releases) is a downloadable **Financials-Extension.oxt** file - load it into Calc 
 under menu item: Tools, Extension Manager...

-Getting data should be a simple as having this in a cell: 
+Please make sure, not to rename the OXT file when downloading and before installing: LO will mess up the installation otherwise and the extension won't work.
+
+Getting data should be as simple as having this in a cell: 
 - `=GETREALTIME("IBM",21,"YAHOO")`
- `=GETREALTIME("NYSE:IBM",21,"GOOGLE")`
 - `=GETREALTIME("IBM:NYQ",21,"FT")`
+- `=GETREALTIME("EURUSD","LAST_PRICE","FT")`
 - `=GETHISTORIC("IBM",90,"2020-12-01","YAHOO")`
+- `=GETREALTIME("ETH-USD","LAST_PRICE","COINBASE")`

 Codes 21 and 90 stand for "last price" and "close" (see below), respectively. 
-Only Yahoo ahs historic data available.
+Only Yahoo has historic data available.

 There is a file **examples.ods** there too with usage examples and possible arguments to functions.

-You have to check the respective websites to work out what symbol is the right one for you. If a website doesn't have 
+You have to check the respective websites to work out what symbol is the right one for you. Make sure today or the date 
+requested is a trading day (exchange is not closed). If a website doesn't have 
 the symbol/asset you want, this extension can't help you either. Having said that, I mostly look at US and West European 
 equities, ETFs and mutual funds and major FX rates - if you have issues with the data available for other assets or 
 assets in other regions, drop me a line (best to include full URLs and possibly the same asset listed on more than one 
 site for comparison). While data for last price is most likely consistent across sites, they may differ for other data 
 points.     

+List of example URLs for checking symbols. You can start from these and search for other symbols. If you think some data
+is available on the website but not from the extension, a good place to start when raising an issue is to include a similar 
+URL to compare results.
+
+| Website  | Symbol  | Example URL for Vodafone Group Plc UK                            |
+|:---------|:--------|:-----------------------------------------------------------------|
+| YAHOO    | VOD.L   | https://finance.yahoo.com/quote/VOD.L                            |
+| FT       | VOD:LSE | https://markets.ft.com/data/equities/tearsheet/summary?s=VOD:LSE |
+| COINBASE | ETH-EUR | https://api.exchange.coinbase.com/products/ETH-EUR/stats         |
+
+### LibreOffice: using , (comma) vs ; (semicolon) to separate arguments in formula
+
+There is a setting in "Tools" / "Options..." / "LibreOffice Calc" / "Formula" called "Functions". Here the user can specify the character used to separate arguments in formula.
+
+Mine is set to , (comma) - when I enter ; (semicolon) in a formula (Ubuntu / UK English), no error is reported but the semicolon is converted to , (comma)
+
+Depending on your system's language and default LibreOffice settings, you maybe better off using ; instead of , in your formulas.
+
 ### List of supported data points

-You can either specify numbers or names (lower or upper case) - not all bits are available from all sources and they are not necessarily consistent across sources either.
+You can either specify numbers or names (lower or upper case) - not all bits are available from all sources, and they are not necessarily consistent across sources either.

-|Name|Code|YAHOO|FT|GOOGLE|YAHOO (historic)|
-| :--- | :--- | :---: | :---: | :---: | :---: |
-|PREV_CLOSE|5|Yes|Yes|Yes||
-|OPEN|6|Yes|Yes|Yes|Yes|
-|CHANGE|7|Yes|Yes|Yes||
-|LAST_PRICE_DATE|8|Yes|Yes|Yes||
-|LAST_PRICE_TIME|10|Yes|Yes|Yes||
-|CHANGE_IN_PERCENT|11|Yes|Yes|Yes||
-|LOW|14|Yes|Yes|Yes|Yes|
-|HIGH|16|Yes|Yes|Yes|Yes|
-|LAST_PRICE|21|Yes|Yes|Yes||
-|HIGH_52_WEEK|24|Yes|Yes|Yes||
-|LOW_52_WEEK|26|Yes|Yes|Yes||
-|MARKET_CAP|27|Yes|Yes|Yes||
-|VOLUME|35|Yes|Yes|No|Yes|
-|AVG_DAILY_VOL_3MONTH|39|Yes|Yes|No||
-|BETA|67|Yes|Yes|No||
-|EPS|68|Yes|Yes|No||
-|PE_RATIO|69|Yes|Yes|No||
-|DIV|70|Yes|Yes|No||
-|DIV_YIELD|71|Yes|Yes|No||
-|EX_DIV_DATE|72|Yes|Yes|No||
-|PAYOUT_RATIO|73|Yes|No|No||
-|CLOSE|90|No|No|No|Yes|
-|ADJ_CLOSE|91|No|No|No|Yes|
-|SECTOR|98|Yes|Yes|No||
-|INDUSTRY|99|Yes|Yes|No||
-|TICKER|101|Yes|Yes|Yes||
-|EXCHANGE|102|Yes|No|Yes||
-|CURRENCY|103|Yes|Yes|Yes||
-|NAME|104|Yes|Yes|Yes||
-|TIMEZONE|105|Yes|Yes|Yes||
+| Name                 | Code |      YAHOO       | FT  | YAHOO (historic) | COINBASE |   Notes    |
+|:---------------------|:-----|:----------------:|:---:|:----------------:|----------|:----------:|
+| PREV_CLOSE           | 5    |       Yes        | Yes |||
+| OPEN                 | 6    |       Yes        | Yes |       Yes        | Yes      ||
+| CHANGE               | 7    |       Yes        | Yes |||
+| LAST_PRICE_DATE      | 8    |                  | Yes |                  |          | yyyy-mm-dd |
+| LAST_PRICE_TIME      | 10   |                  | Yes |||
+| CHANGE_IN_PERCENT    | 11   |       Yes        | Yes |||
+| LOW                  | 14   |       Yes        | Yes |       Yes        | Yes      ||
+| HIGH                 | 16   |       Yes        | Yes |       Yes        | Yes      ||
+| LAST_PRICE           | 21   |       Yes        | Yes |                  | Yes      |
+| BID                  | 22   |       Yes        | Yes |||
+| ASK                  | 25   |       Yes        | Yes |||
+| HIGH_52_WEEK         | 24   |       Yes        | Yes |||
+| LOW_52_WEEK          | 26   |       Yes        | Yes |||
+| MARKET_CAP           | 27   |       Yes        | Yes |||
+| BIDSIZE              | 30   |       Yes        | Yes |||
+| ASKSIZE              | 31   |       Yes        | Yes |||
+| VOLUME               | 35   |       Yes        | Yes |       Yes        | Yes      ||
+| AVG_DAILY_VOL_3MONTH | 39   |       Yes        | Yes |||
+| BETA                 | 67   |       Yes        | Yes |||
+| EPS                  | 68   |       Yes        | Yes |||
+| PE_RATIO             | 69   |       Yes        | Yes |||
+| DIV                  | 70   |       Yes        | Yes |||
+| DIV_YIELD            | 71   |       Yes        | Yes |||
+| EX_DIV_DATE          | 72   |       Yes        | Yes |                  |          | yyyy-mm-dd |
+| PAYOUT_RATIO         | 73   |       Yes        | No  |||
+| EXPIRY_DATE          | 74   | Yes (on options) | No  |                  |          | yyyy-mm-dd |
+| SHARES_OUT           | 75   |       Yes        | Yes |                  |          ||
+| FREE_FLOAT           | 76   |       Yes        | Yes |                  |          ||
+| SETTLEMENT_DATE      | 77   | Yes (on futures) |     |                  |          | yyyy-mm-dd |
+| CLOSE                | 90   |        No        | No  |       Yes        |          ||
+| ADJ_CLOSE            | 91   |        No        | No  |       Yes        |          ||
+| SECTOR               | 98   |       Yes        | Yes |||
+| INDUSTRY             | 99   |       Yes        | Yes |||
+| TICKER               | 101  |       Yes        | Yes |                  | Yes      |
+| EXCHANGE             | 102  |       Yes        | No  |||
+| CURRENCY             | 103  |       Yes        | Yes |                  | Yes      |
+| NAME                 | 104  |       Yes        | Yes |||
+| TIMEZONE             | 105  |       Yes        | Yes |||

 ### Dealing with missing data:

@@ -90,6 +115,10 @@ your favorite web browser (or other tools) to check if the page actually contain
 If it does, the file trace.log has a record of all calls to the extension with the value returned to LibreOffice. 
 Otherwise, the file extension.log in the same location might have more details about errors or exceptions.  

+In general, web scraping can't be compared to using a stable API - the websites might have issues - from a technical or
+data perspective. I have found especially on the weekend it can sometimes be "flaky" and closing/reopening LibreCalc can 
+refresh things. 
+
 ### Build:

 You will need the LibreOffice SDK installed. 
@@ -100,20 +129,18 @@ On my system (Ubuntu) I installed packages: libreoffice-dev libreoffice-java-com

 cd ~/tech/IdeaProjects/Financials-Extension/

-python3 src/test_yahoo.py
-
-python3 src/test_google.py
+python3 -m unittest discover src

 \# This builds file **Financials-Extension.oxt**

 ./compile.sh

 ### Tested with:
+- Windows 10 / LibreOffice Calc 7.1.2.2 / Python 3.8.8
+- Ubuntu 22.04.1 / LibreOffice Calc 7.3.7.2 / Python 3.10.6
+- MacOS 10.15.7 / LibreOffice Calc 7.2.0.4 / Python 3.8.10
+
+(Previous versions)
 - Debian 10.3  / LibreOffice Calc 6.1.5.2 / Python 3.7.3
- Ubuntu 20.10 / LibreOffice Calc 7.0.3.1 / Python 3.8.6
- Ubuntu 20.04 / LibreOffice Calc 6.4.3.2 / Python 3.8.2
- Ubuntu 19.10 / LibreOffice Calc 6.3.5.2 / Python 3.7.6
- Ubuntu 19.04 / LibreOffice Calc 6.2 / Python 3.7.3
+- Ubuntu 20.04.5 / LibreOffice Calc 6.4.7.2 / Python 3.8.10
 - Ubuntu 18.04.5 / LibreOffice Calc 6 / Python 3.6.9
- Ubuntu 18.04 / LibreOffice Calc 6 / Python 3.6.7
- Ubuntu 16.04 / LibreOffice Calc 5 (previous versions)
@@ -1,7 +1,24 @@
-#!/bin/bash
+#!/usr/bin/env bash

-export PATH=$PATH:/usr/lib/libreoffice/sdk/bin
-export PATH=$PATH:/usr/lib/libreoffice/program
+set -o errexit
+set -o nounset
+set -o pipefail
+set -o noclobber
+
+if [[ $OSTYPE == "darwin"* ]]; then
+  echo MacOS
+  # Assuming both are installed in the applications folder
+  # Required some steps to make it work for MacOS M1, mind the `find` call which could return more than one (shouldn't)
+  # install_name_tool -change @__VIA_LIBRARY_PATH__/libreglo.dylib $(find /Applications -name "libreglo.dylib") /Applications/LibreOffice7.4_SDK/bin/idlc
+  # install_name_tool -change @__VIA_LIBRARY_PATH__/libuno_sal.dylib.3 $(find /Applications -name "libuno_sal.dylib.3") /Applications/LibreOffice7.4_SDK/bin/idlc
+  # install_name_tool -change @__VIA_LIBRARY_PATH__/libuno_salhelpergcc3.dylib.3 $(find /Applications -name "libuno_salhelpergcc3.dylib.3") /Applications/LibreOffice7.4_SDK/bin/idlc
+  # codesign --force -s - $(find /Applications -name "idlc")
+  export PATH=$PATH:/Applications/LibreOffice7.4_SDK/bin
+  export PATH=$PATH:/Applications/LibreOffice.app/Contents/MacOS
+else
+  export PATH=$PATH:/usr/lib/libreoffice/sdk/bin
+  export PATH=$PATH:/usr/lib/libreoffice/program
+fi

 # Setup build directories

@@ -12,13 +29,23 @@ mkdir "${PWD}"/build/META-INF/

 # Compile the binaries

-echo "Calling idlc..."
-idlc -w -verbose "${PWD}"/idl/XFinancials.idl
+if command -v idlc &> /dev/null
+then
+  echo "Calling idlc..."
+  idlc -w -verbose "${PWD}"/idl/XFinancials.idl

-echo "Calling regmerge..."
-regmerge -v "${PWD}"/build/XFinancials.rdb UCR "${PWD}"/idl/XFinancials.urd
+  echo "Calling regmerge..."
+  regmerge -v "${PWD}"/build/XFinancials.rdb UCR "${PWD}"/idl/XFinancials.urd

-rm "${PWD}"/idl/XFinancials.urd
+  rm "${PWD}"/idl/XFinancials.urd
+else
+  # LibreOffice 7.4+
+  export unoTypes=/usr/lib/libreoffice/program/types.rdb
+  export offTypes=/usr/lib/libreoffice/program/types/offapi.rdb
+
+  echo "Calling unoidl-write..."
+  unoidl-write $unoTypes $offTypes "${PWD}"/idl/XFinancials.idl "${PWD}"/build/XFinancials.rdb
+fi

 echo "Generating meta files..."
 python3 "${PWD}"/src/generate_metainfo.py
@@ -29,9 +56,30 @@ cp -f "${PWD}"/src/baseclient.py "${PWD}"/build/
 cp -f "${PWD}"/src/jsonParser.py "${PWD}"/build/
 cp -f "${PWD}"/src/naivehtmlparser.py "${PWD}"/build/
 cp -f "${PWD}"/src/tz.py "${PWD}"/build/
-cp -f "${PWD}"/src/ft.py "${PWD}"/build/
-cp -f "${PWD}"/src/google.py "${PWD}"/build/
-cp -f "${PWD}"/src/yahoo.py "${PWD}"/build/
+cp -f "${PWD}"/src/financials_ft.py "${PWD}"/build/
+cp -f "${PWD}"/src/financials_yahoo.py "${PWD}"/build/
+cp -f "${PWD}"/src/financials_coinbase.py "${PWD}"/build/
+
+# this copies python modules dateutil, pytz, pyparsing to extension so it doesn't have to be installed by user
+
+TMPFILE=`mktemp`
+
+wget "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl" -O $TMPFILE
+unzip $TMPFILE dateutil/\* -d "${PWD}"/build/
+rm $TMPFILE
+
+wget "https://files.pythonhosted.org/packages/9c/3d/a121f284241f08268b21359bd425f7d4825cffc5ac5cd0e1b3d82ffd2b10/pytz-2024.1-py2.py3-none-any.whl" -O $TMPFILE
+unzip $TMPFILE pytz/\* -d "${PWD}"/build/
+rm $TMPFILE
+
+wget "https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl" -O $TMPFILE
+unzip $TMPFILE pyparsing.py -d "${PWD}"/build/
+rm $TMPFILE
+
+# Windows LibreOffice 7.1 Python is missing this...
+wget "https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl" -O $TMPFILE
+unzip $TMPFILE six.py -d "${PWD}"/build/
+rm $TMPFILE

 echo "Package into oxt file..."
 pushd "${PWD}"/build/
@@ -16,6 +16,7 @@ import pathlib
 import random
 import select
 import urllib.request
+import urllib.parse
 from http import cookiejar
 from http.client import HTTPConnection, HTTPSConnection, HTTPException

@@ -33,9 +34,20 @@ class RedirectException(HTTPException):


 class HttpException(HTTPException):
-    def __init__(self, url, status):
+    def __init__(self, url, response):
        self.url = url
-        self.status = status
+        self.response = response
+
+    def __str__(self):
+        if self.response is None:
+            return f"url='{self.url}'"
+        if type(self.response) is str:
+            return f"url='{self.url}' status='{self.response}'"
+        if self.response.headers:
+            h = '\n'.join(sorted(self.response.headers.__str__().splitlines(), key=lambda l: l.lower()))
+            return f"url='{self.url}' status={self.response.status} reason='{self.response.reason}'{h}\n"
+        else:
+            return f"url='{self.url}' status={self.response.status} reason='{self.response.reason}'"


 class BaseClient:
@@ -43,31 +55,22 @@ class BaseClient:
        self.connections = {}
        self.cookies = cookiejar.CookieJar()
        self.last_url = None
+        self.redirect_count = 0  # will be set later

        self.basedir = os.path.join(str(pathlib.Path.home()), '.financials-extension')
        os.makedirs(self.basedir, exist_ok=True)

        user_agents = [
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0',
-            'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0',
-            'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0',
-            'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0',
-
-            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36',
-            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
-            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36',
-            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36',
-            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4104.0 Safari/537.36',
-            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4149.0 Safari/537.36',
-            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0',
        ]

        self.default_headers = {
@@ -81,16 +84,15 @@ class BaseClient:

        self.response = None

-    def request(self, method: str, url: str, data=None, headers={}, cookies=[], **kwargs):
+    def request(self, method: str, url: str, data=None, headers={}, **kwargs):

        _headers = self.default_headers.copy()
        if headers:
            for key, value in headers.items():
                _headers[key] = value

-        if cookies:
-            for c in cookies:
-                self.cookies.set_cookie(c)
+        if method == 'POST' and 'Content-Type' not in _headers:
+            _headers['Content-Type'] = 'application/x-www-form-urlencoded'

        connection = None

@@ -108,7 +110,7 @@ class BaseClient:
            connection = HTTPConnection(host, **kwargs) if scheme == 'http:' else HTTPSConnection(host, **kwargs)

        logger.debug('Creating request -----------------------------------------------------')
-        logger.info("url='%s'", url)
+        logger.debug("%s %s", method, url)

        self.last_url = url

@@ -128,7 +130,7 @@ class BaseClient:

        logger.debug('Processing response --------------------------------------------------')

-        # logger.debug('response.status={}'.format(response.status))
+        logger.debug('response.status=%s', response.status)
        for key, value in response.getheaders():
            logger.debug('Header: %s=%s', key, value)

@@ -139,13 +141,30 @@ class BaseClient:

    def urlopen(self, url, redirect=True, data=None, headers={}, cookies=[], **kwargs):

-        self.response = self.request('POST' if data else 'GET', url, data, headers, cookies, **kwargs)
+        if cookies:
+            for c in cookies:
+                self.cookies.set_cookie(c)
+
+        self.last_url = None
+
+        self.response = self.request('POST' if data else 'GET', url, data, headers, **kwargs)
        text = self.response.read()

-        # Allow two redirects: used by Yahoo for some cookie based consent
+        # Allow redirects - used by Yahoo for some cookie based consent
+        self.redirect_count = 5

-        if 300 <= self.response.status < 400:
-            location = self.response.getheader('Location')
+        # (for Yahoo) AWS CloudFront occasionally returns an incorrect, cached error responses
+        # try mitigating by re-requesting straight away
+        if 400 <= self.response.status < 500:
+            if self.response.getheader('X-Cache') == 'Error from cloudfront':
+                self.response = self.request('POST' if data else 'GET', url, data, headers, **kwargs)
+                text = self.response.read()
+
+        while 300 <= self.response.status < 400 and self.redirect_count >= 0:
+
+            self.redirect_count -= 1
+            location = str(self.response.getheader('Location'))
+            location = location.replace(' ', '%20')  # FT bug workaround - this should not be necessary

            if location and redirect:

@@ -153,29 +172,17 @@ class BaseClient:
                    scheme, _, host, path = url.split('/', 3)
                    location = '{}//{}{}'.format(scheme, host, location)

-                self.response = self.request('POST' if data else 'GET', location, data, headers, cookies, **kwargs)
+                self.response = self.request('GET', location, None, headers, **kwargs)
                text = self.response.read()

-                if 300 <= self.response.status < 400:
-                    location = self.response.getheader('Location')
-
-                    if location and redirect:
-
-                        if location.startswith('/'):
-                            scheme, _, host, path = url.split('/', 3)
-                            location = '{}//{}{}'.format(scheme, host, location)
-
-                        self.response = self.request('POST' if data else 'GET', location, data, headers, cookies,
-                                                     **kwargs)
-                        text = self.response.read()
-                    else:
-                        raise RedirectException(location)
-
            else:
                raise RedirectException(location)

        if self.response.status >= 400:
-            raise HttpException(url, self.response.status)
+            logger.warning("last_url='%s' status=%s reason='%s' headers=%s", self.last_url, self.response.status,
+                           self.response.reason,
+                           '\n'.join(sorted(self.response.headers.__str__().splitlines(), key=lambda l: l.lower())))
+            raise HttpException(url, self.response)

        if self.response.getheader('Content-Encoding') == 'gzip':
            text = gzip.decompress(text)
@@ -187,6 +194,52 @@ class BaseClient:

        return text

+    def get_ticker(self):
+
+        tick = {}
+
+        tick[Datacode.ADJ_CLOSE] = None
+        tick[Datacode.ASKSIZE] = None
+        tick[Datacode.ASK] = None
+        tick[Datacode.AVG_DAILY_VOL_3MONTH] = None
+        tick[Datacode.BETA] = None
+        tick[Datacode.BIDSIZE] = None
+        tick[Datacode.BID] = None
+        tick[Datacode.CHANGE] = None
+        tick[Datacode.CHANGE_IN_PERCENT] = None
+        tick[Datacode.CURRENCY] = None
+        tick[Datacode.DIV] = None
+        tick[Datacode.DIV_YIELD] = None
+        tick[Datacode.EPS] = None
+        tick[Datacode.EXCHANGE] = None
+        tick[Datacode.EXPIRY_DATE] = None
+        tick[Datacode.EX_DIV_DATE] = None
+        tick[Datacode.FREE_FLOAT] = None
+        tick[Datacode.SETTLEMENT_DATE] = None
+        tick[Datacode.HIGH] = None
+        tick[Datacode.HIGH_52_WEEK] = None
+        tick[Datacode.INDUSTRY] = None
+        tick[Datacode.LAST_PRICE] = None
+        tick[Datacode.LAST_PRICE_DATE] = None
+        tick[Datacode.LAST_PRICE_TIME] = None
+        tick[Datacode.LOW] = None
+        tick[Datacode.LOW_52_WEEK] = None
+        tick[Datacode.MARKET_CAP] = None
+        tick[Datacode.NAME] = None
+        tick[Datacode.OPEN] = None
+        tick[Datacode.PAYOUT_RATIO] = None
+        tick[Datacode.PE_RATIO] = None
+        tick[Datacode.PREV_CLOSE] = None
+        tick[Datacode.SECTOR] = None
+        tick[Datacode.SHARES_OUT] = None
+        tick[Datacode.TICKER] = None
+        tick[Datacode.TIMEZONE] = None
+        tick[Datacode.VOLUME] = None
+
+        tick[Datacode.TIMESTAMP] = None
+
+        return tick
+
    def _return_value(self, data: dict, datacode: int):

        """
@@ -234,6 +287,18 @@ class BaseClient:
            elif datacode == Datacode.LAST_PRICE.value and Datacode.LAST_PRICE in data:
                return data[Datacode.LAST_PRICE]

+            elif datacode == Datacode.BID.value and Datacode.BID in data:
+                return data[Datacode.BID]
+
+            elif datacode == Datacode.ASK.value and Datacode.ASK in data:
+                return data[Datacode.ASK]
+
+            elif datacode == Datacode.BIDSIZE.value and Datacode.BIDSIZE in data:
+                return data[Datacode.BIDSIZE]
+
+            elif datacode == Datacode.ASKSIZE.value and Datacode.ASKSIZE in data:
+                return data[Datacode.ASKSIZE]
+
            elif datacode == Datacode.LOW_52_WEEK.value and Datacode.LOW_52_WEEK in data:
                return data[Datacode.LOW_52_WEEK]

@@ -273,6 +338,24 @@ class BaseClient:
            elif datacode == Datacode.PAYOUT_RATIO.value and Datacode.PAYOUT_RATIO in data:
                return data[Datacode.PAYOUT_RATIO]

+            elif datacode == Datacode.EXPIRY_DATE.value and Datacode.EXPIRY_DATE in data:
+                if data[Datacode.EXPIRY_DATE]:
+                    return data[Datacode.EXPIRY_DATE].isoformat()
+                else:
+                    return data[Datacode.EXPIRY_DATE]
+
+            elif datacode == Datacode.FREE_FLOAT.value and Datacode.FREE_FLOAT in data:
+                return data[Datacode.FREE_FLOAT]
+
+            elif datacode == Datacode.SETTLEMENT_DATE.value and Datacode.SETTLEMENT_DATE in data:
+                if data[Datacode.SETTLEMENT_DATE]:
+                    return data[Datacode.SETTLEMENT_DATE].isoformat()
+                else:
+                    return data[Datacode.SETTLEMENT_DATE]
+
+            elif datacode == Datacode.SHARES_OUT.value and Datacode.SHARES_OUT in data:
+                return data[Datacode.SHARES_OUT]
+
            elif datacode == Datacode.CLOSE.value and Datacode.CLOSE in data:
                return data[Datacode.CLOSE]

@@ -317,3 +400,11 @@ class BaseClient:
            pass

        return None
+
+    def close(self):
+        for connection in self.connections.values():
+            try:
+                connection.close()
+            except BaseException:
+                pass
+        self.connections = {}
@@ -24,6 +24,11 @@ class Datacode(Enum):

    LAST_PRICE = 21

+    BID = 22
+    ASK = 25
+    BIDSIZE = 30
+    ASKSIZE = 31
+
    HIGH_52_WEEK = 24
    LOW_52_WEEK = 26
    MARKET_CAP = 27
@@ -38,6 +43,10 @@ class Datacode(Enum):
    DIV_YIELD = 71
    EX_DIV_DATE = 72
    PAYOUT_RATIO = 73
+    EXPIRY_DATE = 74
+    SHARES_OUT = 75
+    FREE_FLOAT = 76
+    SETTLEMENT_DATE = 77

    CLOSE = 90
    ADJ_CLOSE = 91
@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<description xmlns="http://openoffice.org/extensions/description/2006" 
+xmlns:d="http://openoffice.org/extensions/description/2006" 
+xmlns:l="http://libreoffice.org/extensions/description/2011" 
+xmlns:xlink="http://www.w3.org/1999/xlink"> 
+
+<!-- only used for testing -->
+
+<dependencies> 
+    <l:LibreOffice-minimal-version value="5.0" d:name="LibreOffice 5.0" /> 
+</dependencies> 
+
+<identifier value="com.financials.getinfo" /> 
+<version value="3.0.0" />
+<display-name><name lang="en">Financial Market Extension</name></display-name>
+<publisher><name xlink:href="https://github.com/cmallwitz/Financials-Extension" lang="en">The Publisher</name></publisher>
+<extension-description><src xlink:href="description-en-US.txt" lang="en" /></extension-description>
+ 
+</description> 
@@ -14,12 +14,13 @@ import logging
 import os
 import pathlib
 import platform
+import ssl
 import sys
 import time
 from functools import wraps
 from importlib import util
+import xml.etree.ElementTree as ET

-import dateutil.parser
 import unohelper
 from com.financials.getinfo import Financials

@@ -29,7 +30,7 @@ os.makedirs(basedir, exist_ok=True)
 logging.basicConfig(
    handlers=[logging.FileHandler(filename=os.path.join(basedir, 'extension.log'), encoding='utf-8', mode='a+')],
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
-    level=logging.WARN)
+    level=logging.INFO)

 # Add current directory to import path
 current_dir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
@@ -47,15 +48,29 @@ if dateutil_missing or pyparsing_missing or pytz_missing:
    msg += ' pytz' if pytz_missing else ''
    raise Exception("THIS EXTENSION NEEDS THE FOLLOWING PYTHON 3 LIBRARIES INSTALLED:" + msg)

+import dateutil.parser
+import pytz
+import pyparsing
+import six
+
 from datacode import Datacode
-import google
-import yahoo
-import ft
-from version import version
+import financials_yahoo as yahoo
+import financials_coinbase as coinbase
+import financials_ft as ft

 implementation_name = "com.financials.getinfo.python.FinancialsImpl"  # as defined in Financials.xcu
 implementation_services = ("com.sun.star.sheet.AddIn",)

+# Disabling SSL certificate validation as Python setup on MacOS seems to be broken
+# Only reading public data so this should be safe
+
+try:
+    _create_unverified_https_context = ssl._create_unverified_context
+except AttributeError:
+    pass
+else:
+    ssl._create_default_https_context = _create_unverified_https_context
+

 def profile(fn):
    @wraps(fn)
@@ -79,14 +94,14 @@ class FinancialsImpl(unohelper.Base, Financials):

    def __init__(self, ctx):
        self.ctx = ctx
-        self.google = google.createInstance(ctx)
        self.yahoo = yahoo.createInstance(ctx)
+        self.coinbase = coinbase.createInstance(ctx)
        self.ft = ft.createInstance(ctx)

    @profile
    def getRealtime(self, ticker, datacode=None, source=None):

-        if ticker == 'SUPPORT':
+        if ticker == 'SUPPORT' or ticker == 'support':
            return self.support(datacode)

        try:
@@ -119,12 +134,12 @@ class FinancialsImpl(unohelper.Base, Financials):
            ticker = str(ticker).strip()
            source = str(source).upper()

-            if source == 'GOOGLE':
-                s = self.google.getRealtime(ticker, datacode)
-            elif source == 'YAHOO':
+            if source == 'YAHOO':
                s = self.yahoo.getRealtime(ticker, datacode)
            elif source == 'FT':
                s = self.ft.getRealtime(ticker, datacode)
+            elif source == 'COINBASE':
+                s = self.coinbase.getRealtime(ticker, datacode)
            else:
                s = 'Source \'{}\' not supported'.format(source)

@@ -181,7 +196,7 @@ class FinancialsImpl(unohelper.Base, Financials):

                try:
                    offset = int(date)  # offset for 1899-12-30
-                    d = dateutil.parser.parse('1899-12-30') + datetime.timedelta(days=offset)
+                    d = dateutil.parser.parse('1899-12-30', yearfirst=True, dayfirst=False) + datetime.timedelta(days=offset)
                    d = d.date().isoformat()
                except:
                    return 'Date format not supported: {}'.format(date)
@@ -190,7 +205,7 @@ class FinancialsImpl(unohelper.Base, Financials):
            elif type(date) == str:

                try:
-                    int(dateutil.parser.parse(date).strftime('%s'))
+                    int(dateutil.parser.parse(date, yearfirst=True, dayfirst=False).timestamp())
                except:
                    return 'Date format not supported: \'{}\''.format(date)

@@ -218,7 +233,14 @@ class FinancialsImpl(unohelper.Base, Financials):
    @profile
    def support(self, datacode):

-        s = 'ctx={}\nid(self)={}\nversion={}\nfile={}\ncwd={}\nhome={}\nuname={}\npid={}\nsys.executable={}\nsys.version={}\nlocale={}'.format(
+        version = '0.0.0'
+
+        description_file = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'description.xml')
+        for e in ET.parse(description_file).getroot():
+            if e.tag.endswith('version'):
+                version = e.attrib['value']
+
+        s = 'ctx={}\nid(self)={}\nversion={}\nfile={}\ncwd={}\nhome={}\nuname={}\npid={}\nsys.executable={}\nsys.version={}\nsys.path={}\nlocale={}\ndefaultlocale={}\ndateutil={}\npytz={}\npyparsing={}\nsix={}'.format(
            self.ctx,
            id(self),
            version,
@@ -229,7 +251,14 @@ class FinancialsImpl(unohelper.Base, Financials):
            os.getpid(),
            sys.executable,
            sys.version.replace("\n", " "),
-            locale.getlocale())
+            sys.path,
+            locale.getlocale(),
+            locale.getdefaultlocale(),
+            dateutil.__version__,
+            pytz.__version__,
+            pyparsing.__version__,
+            six.__version__,
+        )

        if datacode:
            s = '{}\ntype(datacode)={}\nstr(datacode)={}'.format(
@@ -239,11 +268,16 @@ class FinancialsImpl(unohelper.Base, Financials):

        return s

+    def close(self):
+        if self.yahoo: self.yahoo.close()
+        if self.coinbase: self.coinbase.close()
+        if self.ft: self.ft.close()
+

 def createInstance(ctx):
    return FinancialsImpl(ctx)


-# pythonloader looks for a static g_ImplementationHelper variable
+# python loader looks for a static g_ImplementationHelper variable
 g_ImplementationHelper = unohelper.ImplementationHelper()
 g_ImplementationHelper.addImplementation(createInstance, implementation_name, implementation_services, )
@@ -0,0 +1,112 @@
+#  financials_coinbase.py
+#
+#  license: GNU LGPL
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 3 of the License, or (at your option) any later version.
+
+
+import csv
+import datetime
+import logging
+import os
+import pprint
+import re
+import time
+import json
+
+import dateutil.parser
+import pytz
+
+import jsonParser
+from baseclient import BaseClient, HttpException
+from datacode import Datacode
+
+logger = logging.getLogger(__name__)
+
+
+# logger.setLevel(logging.DEBUG)
+
+class Coinbase(BaseClient):
+    def __init__(self, ctx):
+        super().__init__()
+
+        self.crumb = None
+        self.realtime = {}
+        self.js = jsonParser.jsonObject
+
+    def getRealtime(self, ticker, datacode):
+
+        """
+        Retrieve realtime data for ticker from Coinbase and cache it for further lookups
+
+        :param ticker: the ticker symbol e.g. ETH-EUR
+        :param datacode: the requested datacode, not all are supported
+        :return:
+        """
+
+        # remove white space
+        ticker = "".join(ticker.split())
+
+        # use cached value for up to 60 seconds
+        if ticker in self.realtime:
+            tick = self.realtime[ticker]
+            if Datacode.TIMESTAMP in tick and type(tick[Datacode.TIMESTAMP]) == float and time.time() - 60 < tick[Datacode.TIMESTAMP]:
+                return self._return_value(tick, datacode)
+            else:
+                del self.realtime[ticker]
+
+        url = 'https://api.exchange.coinbase.com/products/{}/stats'.format(ticker)
+
+        try:
+            text = self.urlopen(url, redirect=True, data=None, headers=None)
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s last_url=%s redirect_count=%s", ticker, datacode, self.last_url, self.redirect_count)
+            del self.realtime[ticker]
+            return 'Coinbase.getRealtime({}, {}) - urlopen: {}'.format(ticker, datacode, e)
+
+        try:
+            with open(os.path.join(self.basedir, 'coinbase-{}.json'.format(ticker)), "w", encoding="utf-8") as text_file:
+                print(f"<!-- '{self.last_url}' -->\r\n\r\n{text}", file=text_file)
+        except BaseException:
+            logger.exception("BaseException open/write ticker=%s datacode=%s", ticker, datacode)
+
+        try:
+            results = json.loads(text)
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            del self.realtime[ticker]
+            return 'Coinbase.getRealtime({}, {}) - exception: {}'.format(ticker, datacode, e)
+
+        try:
+            price = results['last']
+
+            if not price:
+                return 'Could not find price for \'{}\''.format(ticker)
+
+            if ticker not in self.realtime:
+                self.realtime[ticker] =  self.get_ticker()
+
+            tick = self.realtime[ticker]
+
+            tick[Datacode.TIMESTAMP] = time.time()
+            tick[Datacode.LAST_PRICE] = float(price)
+            tick[Datacode.OPEN] = float(results['open'])
+            tick[Datacode.HIGH] = float(results['high'])
+            tick[Datacode.LOW] = float(results['low'])
+            tick[Datacode.VOLUME] =  float(results['volume'])
+            tick[Datacode.TICKER] = ticker.split('-', 1)[0]
+            tick[Datacode.CURRENCY] = ticker.split('-', 1)[1]
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            del self.realtime[ticker]
+            return 'Coinbase.getRealtime({}, {}) - process: {}'.format(ticker, datacode, e)
+
+        return self._return_value(self.realtime[ticker], datacode)
+
+def createInstance(ctx):
+    return Coinbase(ctx)
@@ -1,4 +1,4 @@
-#  ft.py
+#  financials_ft.py
 #
 #  license: GNU LGPL
 #
@@ -12,6 +12,7 @@ import logging
 import os
 import re
 import time
+import urllib.parse

 import dateutil.parser

@@ -52,76 +53,46 @@ class FT(BaseClient):
    def getRealtime(self, ticker: str, datacode: int):

        """
-        Retrieve data for ticker from Alpha Vantage and cache it for further lookups
+        Retrieve data for ticker from Financial Times and cache it for further lookups

-        :param ticker: the ticker symbol e.g. VOD.LON
+        :param ticker: the ticker symbol e.g. VOD:LSE
        :param datacode: the requested datacode
        :return:
        """

-        # remove white space
-        ticker = "".join(ticker.split())
-
        # use cached value for up to 60 seconds
        if ticker in self.realtime:
            tick = self.realtime[ticker]
-            if time.time() - 60 < tick[Datacode.TIMESTAMP]:
+            if Datacode.TIMESTAMP in tick and type(tick[Datacode.TIMESTAMP]) == float and time.time() - 60 < tick[Datacode.TIMESTAMP]:
                return self._return_value(tick, datacode)
            else:
                del self.realtime[ticker]

        if ticker not in self.realtime:
-            self.realtime[ticker] = {}
+            self.realtime[ticker] = self.get_ticker()

        tick = self.realtime[ticker]

        asset_class = self.guess_asset_class(ticker)

-        url = f'https://markets.ft.com/data/{asset_class}/tearsheet/summary?s={ticker}'
+        url = f'https://markets.ft.com/data/{asset_class}/tearsheet/summary?s={urllib.parse.quote_plus(ticker)}'

        try:
            text = self.urlopen(url, redirect=True, data=None, headers=None)
        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            logger.exception("BaseException ticker=%s datacode=%s last_url=%s redirect_count=%s", ticker, datacode, self.last_url, self.redirect_count)
+            del self.realtime[ticker]
            return f'FT.getRealtime({ticker}, {datacode}) - urlopen endpoint: {str(e)}'

        try:
-            with open(os.path.join(self.basedir, f'ft-{ticker}.html'), "w", encoding="utf-8") as text_file:
+            temp = ticker.replace(':', '_') # Windows can't have ':' in file names
+            with open(os.path.join(self.basedir, f'ft-{temp}.html'), "w", encoding="utf-8") as text_file:
                print(f"<!-- '{self.last_url}' -->\r\n\r\n{text}", file=text_file)
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+        except BaseException:
+            logger.exception("BaseException open/write ticker=%s datacode=%s %s", ticker, datacode)

        tick[Datacode.TIMESTAMP] = time.time()

-        tick[Datacode.NAME] = None
-        tick[Datacode.TICKER] = None
-        tick[Datacode.CURRENCY] = None
-        tick[Datacode.LAST_PRICE] = None
-        tick[Datacode.CHANGE] = None
-        tick[Datacode.CHANGE_IN_PERCENT] = None
-        tick[Datacode.VOLUME] = None
-        tick[Datacode.LOW_52_WEEK] = None
-        tick[Datacode.HIGH_52_WEEK] = None
-        tick[Datacode.LAST_PRICE_DATE] = None
-        tick[Datacode.LAST_PRICE_TIME] = None
-        tick[Datacode.TIMEZONE] = None
-
-        tick[Datacode.OPEN] = None
-        tick[Datacode.HIGH] = None
-        tick[Datacode.LOW] = None
-        tick[Datacode.PREV_CLOSE] = None
-        tick[Datacode.MARKET_CAP] = None
-
-        tick[Datacode.EXCHANGE] = None
-        tick[Datacode.AVG_DAILY_VOL_3MONTH] = None
-        tick[Datacode.BETA] = None
-        tick[Datacode.EPS] = None
-        tick[Datacode.PE_RATIO] = None
-        tick[Datacode.DIV] = None
-        tick[Datacode.DIV_YIELD] = None
-        tick[Datacode.EX_DIV_DATE] = None
-        tick[Datacode.PAYOUT_RATIO] = None
-
        try:
            r = '<h1 class="mod-tearsheet-overview__header__name mod-tearsheet-overview__header__name--large">(.*?)</h1>'
            match = re.compile(r, flags=re.DOTALL).search(text)
@@ -148,7 +119,7 @@ class FT(BaseClient):
                tick[Datacode.INDUSTRY] = self.save_wrapper(
                    lambda: html.unescape(match.group(2)).strip())

-            r = r'<span [^>]*>Price \(([A-Z]+|--)\)</span><span [^>]*>([0-9,\.]+)</span>'
+            r = r'<span [^>]*>Price \(([A-Za-z]{3}|--)\)</span><span [^>]*>([0-9,\.]+)</span>'
            match = re.compile(r, flags=re.DOTALL).search(text, start)
            if match:
                start = match.span(0)[1]
@@ -197,7 +168,7 @@ class FT(BaseClient):

                try:
                    value = html.unescape(match.group(1)).strip()
-                    dt = dateutil.parser.parse(value, tzinfos=whois_timezone_info)
+                    dt = dateutil.parser.parse(value, yearfirst=True, dayfirst=False, tzinfos=whois_timezone_info)
                    tick[Datacode.LAST_PRICE_DATE] = dt.date()
                    tick[Datacode.LAST_PRICE_TIME] = dt.time()

@@ -205,7 +176,7 @@ class FT(BaseClient):
                    if len(time_bits) >= 4:
                        tick[Datacode.TIMEZONE] = time_bits[-1]

-                except BaseException as e:
+                except BaseException:
                    pass

            # second attempt at 52 week range
@@ -247,18 +218,42 @@ class FT(BaseClient):
                tick[Datacode.LOW] = self.save_wrapper(
                    lambda: float(html.unescape(match.group(1)).replace(',', '').strip()))

+            r = r'<th>\s*Bid\s*</th><td>([0-9,\.]+)</td>'
+            match = re.compile(r, flags=re.DOTALL).search(text, start)
+            if match:
+                tick[Datacode.BID] = self.save_wrapper(
+                    lambda: float(html.unescape(match.group(1)).replace(',', '').strip()))
+
+            r = r'<th>\s*Offer\s*</th><td>([0-9,\.]+)</td>'
+            match = re.compile(r, flags=re.DOTALL).search(text, start)
+            if match:
+                tick[Datacode.ASK] = self.save_wrapper(
+                    lambda: float(html.unescape(match.group(1)).replace(',', '').strip()))
+
            r = r'<th>\s*Previous close\s*</th><td>\s*([0-9,\.]+)\s*</td>'
            match = re.compile(r, flags=re.DOTALL).search(text, start)
            if match:
                tick[Datacode.PREV_CLOSE] = self.save_wrapper(
                    lambda: float(html.unescape(match.group(1)).replace(',', '').strip()))

-            r = r'<th>\s*Average volume\s*</th><td>\s*([0-9,\.btnm]+)\s*</td>'
+            r = r'<th>\s*Average volume\s*</th><td>\s*([0-9,\.btnmk]+)\s*</td>'
            match = re.compile(r, flags=re.DOTALL).search(text, start)
            if match:
                tick[Datacode.AVG_DAILY_VOL_3MONTH] = self.save_wrapper(
                    lambda: handle_abbreviations(html.unescape(match.group(1))))

+            r = r'<th>\s*Shares outstanding\s*</th><td>\s*([0-9,\.btnmk]+)\s*</td>'
+            match = re.compile(r, flags=re.DOTALL).search(text, start)
+            if match:
+                tick[Datacode.SHARES_OUT] = self.save_wrapper(
+                    lambda: handle_abbreviations(html.unescape(match.group(1))))
+
+            r = r'<th>\s*Free float\s*</th><td>\s*([0-9,\.btnmk]+)\s*</td>'
+            match = re.compile(r, flags=re.DOTALL).search(text, start)
+            if match:
+                tick[Datacode.FREE_FLOAT] = self.save_wrapper(
+                    lambda: handle_abbreviations(html.unescape(match.group(1))))
+
            r = r'<th>\s*P/E.*?</th><td>\s*([0-9,\.\-]+)\s*<'
            match = re.compile(r, flags=re.DOTALL).search(text, start)
            if match:
@@ -269,7 +264,7 @@ class FT(BaseClient):
                    tick[Datacode.PE_RATIO] = self.save_wrapper(
                        lambda: float(value))

-            r = r'<th>\s*Market cap\s*</th><td>\s*([0-9,\.btnm]+)\s*<'
+            r = r'<th>\s*Market cap\s*</th><td>\s*([0-9,\.btnmk]+)\s*<'
            match = re.compile(r, flags=re.DOTALL).search(text, start)
            if match:
                tick[Datacode.MARKET_CAP] = self.save_wrapper(
@@ -298,16 +293,17 @@ class FT(BaseClient):
            if match:
                try:
                    value = html.unescape(match.group(1)).strip()
-                    dt = dateutil.parser.parse(value, tzinfos=whois_timezone_info)
+                    dt = dateutil.parser.parse(value, yearfirst=True, dayfirst=False, tzinfos=whois_timezone_info)
                    tick[Datacode.EX_DIV_DATE] = dt.date()
-                except BaseException as e:
+                except BaseException:
                    pass

        except BaseException as e:
            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            del self.realtime[ticker]
            return f'FT.getRealtime({ticker}, {datacode}) - process: {str(e)}'

-        logger.info(tick)
+        logger.debug(tick)

        return self._return_value(self.realtime[ticker], datacode)

@@ -0,0 +1,450 @@
+#  financials_yahoo.py
+#
+#  license: GNU LGPL
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 3 of the License, or (at your option) any later version.
+
+
+import csv
+import datetime
+import json
+import logging
+import os
+import pytz
+import re
+import time
+import urllib.parse
+
+import dateutil.parser
+
+import jsonParser
+from baseclient import BaseClient, HttpException
+from datacode import Datacode
+from naivehtmlparser import NaiveHTMLParser
+
+logger = logging.getLogger(__name__)
+
+
+# logger.setLevel(logging.DEBUG)
+
+
+def default(obj, prop, fallback=''):
+    try:
+        if obj is None or property is None:
+            return fallback
+
+        v = None
+        if hasattr(obj, prop):
+            v = getattr(obj, prop)
+        elif prop in obj:
+            v = obj[prop]
+        return v if v is not None else fallback
+    except:
+        pass
+    return fallback
+
+
+def handle_abbreviations(s):
+    s = str(s).strip()
+    if s.endswith('M'):
+        return float(s[:-1]) * 1000000
+    elif s.endswith('B'):
+        return float(s[:-1]) * 1000000000
+    elif s.endswith('T'):
+        return float(s[:-1]) * 1000000000000
+    return float(s)
+
+
+class Yahoo(BaseClient):
+    def __init__(self, ctx):
+        super().__init__()
+
+        self.crumb = None
+        self.realtime = {}
+        self.historicdata = {}
+        self.js = jsonParser.jsonObject
+
+    def _read_ticker_csv_file(self, ticker):
+
+        fn = os.path.join(self.basedir, 'yahoo-{}.csv'.format(ticker))
+
+        if not os.path.isfile(fn):
+            return
+
+        with open(fn, newline='', encoding="utf-8") as csvfile:
+            reader = csv.DictReader(csvfile)
+
+            ticks = {}
+
+            for row in reader:
+                tick = self.get_ticker()
+                try:
+                    tick[Datacode.OPEN] = float(row['Open'])
+                    tick[Datacode.LOW] = float(row['Low'])
+                    tick[Datacode.HIGH] = float(row['High'])
+                    tick[Datacode.VOLUME] = float(row['Volume'])
+                    tick[Datacode.CLOSE] = float(row['Close'])
+                    tick[Datacode.ADJ_CLOSE] = float(row['Adj Close'])
+                except:
+                    pass
+
+                if len(tick) > 0:
+                    ticks[row['Date']] = tick
+
+            self.historicdata[ticker] = ticks
+
+
+    def handleCookiesAndConsent(self, url, ticker, datacode, html_file):
+
+        try:
+            text = self.urlopen(url, redirect=True)
+        except BaseException as e:
+            logger.exception("BaseException (1) ticker=%s datacode=%s last_url=%s redirect_count=%s %s",
+                             ticker, datacode, self.last_url, self.redirect_count, e)
+            return None
+
+        try:
+            with open(os.path.join(self.basedir, html_file), "w", encoding="utf-8") as text_file:
+                print(f"<!-- '{self.last_url}' -->\r\n\r\n{text}", file=text_file)
+        except BaseException as e:
+            logger.exception("BaseException (2) ticker=%s datacode=%s %s", ticker, datacode, e)
+
+        if not text:
+            return None
+
+        try:
+            parser = NaiveHTMLParser()
+            root = parser.feed(text)
+            parser.close()
+        except BaseException as e:
+            logger.exception("BaseException (3) ticker=%s datacode=%s - HTML parsing - %s", ticker, datacode, e)
+            return None
+
+        form = root.find(f".//form[@class='consent-form']")
+
+        if form:
+            inputs = form.findall(f".//input")
+
+            if inputs:
+
+                data = {'reject': 'reject'}
+                for d in inputs:
+                    data[d.attrib['name']] = d.attrib['value']
+
+                try:
+                    text = self.urlopen(self.last_url, redirect=True, data=urllib.parse.urlencode(data))
+                except BaseException as e:
+                    logger.exception("BaseException (4) ticker=%s datacode=%s last_url=%s redirect_count=%s %s",
+                                     ticker, datacode, self.last_url, self.redirect_count, e)
+
+                try:
+                    with open(os.path.join(self.basedir, html_file), "w", encoding="utf-8") as text_file:
+                        print(f"<!-- '{self.last_url}' (after consent handling) -->\r\n\r\n{text}", file=text_file)
+                except BaseException as e:
+                    logger.exception("BaseException (5) ticker=%s datacode=%s %s", ticker, datacode, e)
+
+        return text
+
+    def getRealtime(self, ticker, datacode):
+
+        """
+        Retrieve realtime data for ticker from Yahoo Finance and cache it for further lookups
+
+        :param ticker: the ticker symbol e.g. VOD.L
+        :param datacode: the requested datacode
+        :return:
+        """
+
+        # remove white space
+        ticker = "".join(ticker.split())
+
+        # use cached value for up to 60 seconds
+        if ticker in self.realtime:
+            tick = self.realtime[ticker]
+            if Datacode.TIMESTAMP in tick and type(tick[Datacode.TIMESTAMP]) == float and time.time() - 60 < tick[Datacode.TIMESTAMP]:
+                return self._return_value(tick, datacode)
+            else:
+                del self.realtime[ticker]
+
+        if ticker not in self.realtime:
+            self.realtime[ticker] = self.get_ticker()
+
+        tick = self.realtime[ticker]
+
+        if not self.crumb:
+
+            url = 'https://finance.yahoo.com/quote/{}?p={}'.format(ticker, ticker)
+            text = self.handleCookiesAndConsent(url, ticker, datacode, f'yahoo-{ticker}.html')
+
+            if text is None:
+                del self.realtime[ticker]
+                return 'Yahoo.getRealtime({}, {}) - handleCookiesAndConsent'.format(ticker, datacode)
+
+            # crumbs like 'TKkC\u002FZBwoUA' may contain unicode _text_ (not encoded code points)
+            try:
+                r = r'\bcrumb=([^"]{11,})"'
+                pattern = re.compile(r)
+                match = pattern.search(text)
+                if match:
+                    self.crumb = urllib.parse.unquote(match.group(1).encode('unicode-escape').decode('ascii'))
+                    logger.debug(f"crumb='{match.group(1)}' self.crumb='{self.crumb}'")
+                else:
+                    r = r'"crumb"\s*:\s*"([^"]{11,})"'
+                    pattern = re.compile(r)
+                    match = pattern.search(text)
+                    if match:
+                        self.crumb = urllib.parse.unquote(match.group(1).encode('unicode-escape').decode('ascii'))
+                        logger.debug(f"crumb='{match.group(1)}' self.crumb='{self.crumb}'")
+            except BaseException as e:
+                logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+                del self.realtime[ticker]
+                return 'Yahoo.getRealtime({}, {}) - crumb: {}'.format(ticker, datacode, e)
+
+        if not self.crumb:
+            return 'Yahoo.getRealtime({}, {}) - crumb missing'.format(ticker, datacode)
+
+        try:
+
+            url = 'https://query1.finance.yahoo.com/v10/finance/quoteSummary/{}?formatted=true&' \
+                  'modules=summaryProfile,financialData,quoteType,recommendationTrend,earnings,equityPerformance,summaryDetail,defaultKeyStatistics,calendarEvents,esgScores,price,pageViews,financialsTemplate&' \
+                  'lang=en-US&region=US&crumb={}' \
+                .format(ticker, urllib.parse.quote_plus(self.crumb))
+
+            js = self.urlopen(url)
+
+        except HttpException as e:
+            logger.exception("HttpException querying ticker=%s datacode=%s", ticker, datacode)
+            del self.realtime[ticker]
+            return None
+
+        try:
+
+            with open(os.path.join(self.basedir, 'yahoo-{}.json'.format(ticker)), "w", encoding="utf-8") as json_file:
+                print(f"<!-- '{self.last_url}' -->\r\n\r\n{js}", file=json_file)
+
+            parsed = json.loads(js)
+            parsed = parsed['quoteSummary']['result'][0]
+
+            summaryDetail = dict(sorted(parsed['summaryDetail'].items()))
+            price = dict(sorted(parsed['price'].items()))
+
+            if 'defaultKeyStatistics' in parsed:
+                defaultKeyStatistics = dict(sorted(parsed['defaultKeyStatistics'].items()))
+            else:
+                defaultKeyStatistics = {}
+
+            if 'summaryProfile' in parsed:
+                summaryProfile = dict(sorted(parsed['summaryProfile'].items()))
+            else:
+                summaryProfile = {}
+
+            quoteType = dict(sorted(parsed['quoteType'].items()))
+
+        except BaseException as e:
+            logger.exception("BaseException parsing ticker=%s datacode=%s", ticker, datacode)
+            del self.realtime[ticker]
+            return 'Yahoo.getRealtimeSummary({}, {}) - exception: {}'.format(ticker, datacode, e)
+
+        try:
+
+            tick[Datacode.TICKER] = ticker
+            tick[Datacode.TIMESTAMP] = time.time()
+
+            if 'regularMarketPrice' not in price:
+                return None
+
+            tick[Datacode.PREV_CLOSE] = self.save_wrapper(lambda: float(price['regularMarketPreviousClose']['raw']))
+            tick[Datacode.OPEN] = self.save_wrapper(lambda: float(price['regularMarketOpen']['raw']))
+            tick[Datacode.CHANGE] = self.save_wrapper(lambda: float(price['regularMarketChange']['raw']))
+            tick[Datacode.CHANGE_IN_PERCENT] = self.save_wrapper(lambda: float(price['regularMarketChangePercent']['raw']))
+
+            tick[Datacode.LOW] = self.save_wrapper(lambda: float(price['regularMarketDayLow']['raw']))
+            tick[Datacode.HIGH] = self.save_wrapper(lambda: float(price['regularMarketDayHigh']['raw']))
+
+            tick[Datacode.LAST_PRICE] = self.save_wrapper(lambda: float(price['regularMarketPrice']['raw']))
+            tick[Datacode.VOLUME] = self.save_wrapper(lambda: float(price['regularMarketVolume']['raw']))
+            tick[Datacode.AVG_DAILY_VOL_3MONTH] = self.save_wrapper(lambda: float(price['averageDailyVolume3Month']['raw']))
+            tick[Datacode.BETA] = self.save_wrapper(lambda: float(defaultKeyStatistics['beta']['raw']))
+            tick[Datacode.EPS] = self.save_wrapper(lambda: float(defaultKeyStatistics['trailingEps']['raw']))
+            tick[Datacode.PE_RATIO] = self.save_wrapper(lambda: float(summaryDetail['trailingPE']['raw']))
+
+            tick[Datacode.EX_DIV_DATE] = self.save_wrapper(
+                lambda: dateutil.parser.parse(summaryDetail['exDividendDate']['fmt'], yearfirst=True, dayfirst=False).date())
+
+            tick[Datacode.LOW_52_WEEK] = self.save_wrapper(lambda: float(summaryDetail['fiftyTwoWeekLow']['raw']))
+            tick[Datacode.HIGH_52_WEEK] = self.save_wrapper(lambda: float(summaryDetail['fiftyTwoWeekHigh']['raw']))
+
+            tick[Datacode.MARKET_CAP] = self.save_wrapper(lambda: float(price['marketCap']['raw']))
+
+            tick[Datacode.BID] = self.save_wrapper(lambda: float(summaryDetail['bid']['raw']))
+            tick[Datacode.BIDSIZE] = self.save_wrapper(lambda: float(summaryDetail['bidSize']['raw']))
+
+            tick[Datacode.ASK] = self.save_wrapper(lambda: float(summaryDetail['ask']['raw']))
+            tick[Datacode.ASKSIZE] = self.save_wrapper(lambda: float(summaryDetail['askSize']['raw']))
+
+            if quoteType:
+                t = int(price['regularMarketTime'])
+                tz = pytz.timezone(quoteType['timeZoneFullName'])
+
+                tick[Datacode.TIMEZONE] = tz
+                dt = datetime.datetime.fromtimestamp(t, tz)
+
+                tick[Datacode.LAST_PRICE_DATE] = dt.date()
+                tick[Datacode.LAST_PRICE_TIME] = dt.time()
+
+            tick[Datacode.EXCHANGE] = self.save_wrapper(lambda: price['exchangeName'])
+            tick[Datacode.CURRENCY] = self.save_wrapper(lambda: price['currency'])
+
+            tick[Datacode.DIV] = self.save_wrapper(lambda: float(summaryDetail['dividendRate']['raw']))
+            tick[Datacode.DIV_YIELD] = self.save_wrapper(lambda: float(summaryDetail['dividendYield']['raw']))
+
+            # fallback to last dividend on mutual funds and ETFs
+            if not tick[Datacode.DIV]:
+                tick[Datacode.DIV] = self.save_wrapper(lambda: float(defaultKeyStatistics['lastDividendValue']['raw']))
+
+            if default(price, 'quoteType') == 'FUTURE':
+                tick[Datacode.TICKER] = self.save_wrapper(lambda: price['underlyingSymbol'])
+                tick[Datacode.NAME] = self.save_wrapper(lambda: price['shortName'])
+                tick[Datacode.SETTLEMENT_DATE] = self.save_wrapper(
+                    lambda: dateutil.parser.parse(summaryDetail['expireDate']['fmt'], yearfirst=True, dayfirst=False).date())
+            else:
+                tick[Datacode.NAME] = self.save_wrapper(lambda: price['longName'])
+                tick[Datacode.EXPIRY_DATE] = self.save_wrapper(
+                    lambda: dateutil.parser.parse(summaryDetail['expireDate']['fmt'], yearfirst=True, dayfirst=False).date())
+                tick[Datacode.SETTLEMENT_DATE] = None
+
+            if not tick[Datacode.NAME]:
+                tick[Datacode.NAME] = tick[Datacode.TICKER]
+
+            tick[Datacode.SECTOR] = self.save_wrapper(lambda: summaryProfile['sector'])
+            tick[Datacode.INDUSTRY] = self.save_wrapper(lambda: summaryProfile['industry'])
+
+            tick[Datacode.SHARES_OUT] = self.save_wrapper(lambda: float(defaultKeyStatistics['sharesOutstanding']['raw']))
+            tick[Datacode.FREE_FLOAT] = self.save_wrapper(lambda: float(defaultKeyStatistics['floatShares']['raw']))
+            tick[Datacode.PAYOUT_RATIO] = self.save_wrapper(lambda: float(summaryDetail['payoutRatio']['raw']))
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            del self.realtime[ticker]
+            return 'Yahoo.getRealtime({}, {}) - process: {}'.format(ticker, datacode, e)
+
+        return self._return_value(self.realtime[ticker], datacode)
+
+    def getHistoric(self, ticker: str, datacode: int, date):
+
+        """
+        Retrieve historic data for ticker from Yahoo Finance and cache it for further lookups
+
+        :param ticker: the ticker symbol e.g. VOD.L
+        :param datacode: the requested datacode
+        :param date: the requested date
+        :return:
+        """
+
+        # remove white space
+        ticker = "".join(ticker.split())
+        min_tick_date = None
+
+        # dividend and splits will change past adjusted prices
+        # the moment we are asked for ADJ_CLOSE we ignore the ticker cache to refresh
+
+        if Datacode.ADJ_CLOSE != datacode and ticker not in self.historicdata:
+            self._read_ticker_csv_file(ticker)
+
+        try:
+            date_as_dt = dateutil.parser.parse(date, yearfirst=True, dayfirst=False)
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s last_url=%s redirect_count=%s", ticker, datacode, self.last_url, self.redirect_count)
+            return 'Yahoo.getHistoric({}, {}, {}) - date_as_dt: {}'.format(ticker, datacode, date, e)
+
+        if ticker in self.historicdata:
+            ticks = self.historicdata[ticker]
+
+            if date in ticks:
+                return self._return_value(ticks[date], datacode)
+
+            # weekend, trading holiday or as yet un-fetched
+            if min(ticks) <= date <= max(ticks):
+                return 'Not a trading day \'{}\''.format(date)
+
+            # (potentially) future date
+            if date > max(ticks):
+                t1 = int(date_as_dt.timestamp())
+                t2 = int(time.time())
+                if t1 > t2:
+                    return 'Future date \'{}\''.format(date)
+
+                min_tick_date = int(dateutil.parser.parse(min(ticks), yearfirst=True, dayfirst=False).timestamp())  # remember current earliest date
+
+        if not self.crumb:
+            self.getRealtime(ticker, Datacode.LAST_PRICE)
+
+        if not self.crumb:
+            return 'Yahoo.getHistoric({}, {}, {}) - crumb missing'.format(ticker, datacode, date)
+
+        try:
+            t1 = int(date_as_dt.timestamp())
+            t2 = int(time.time())
+
+            if min_tick_date:
+                t1 = min_tick_date
+
+            if t1 >= t2:
+                return 'Future date \'{}\''.format(date)
+
+            if t1 < int(dateutil.parser.parse('2000-01-01', yearfirst=True, dayfirst=False).timestamp()):
+                return 'Date before 2000 \'{}\''.format(date)
+
+            t1 = t1 - 2682000  # pad with extra month
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getHistoric({}, {}, {}) - date: {}'.format(ticker, datacode, date, e)
+
+        try:
+
+            url = 'https://query1.finance.yahoo.com/v7/finance/download/{}' \
+                  '?period1={}&period2={}&interval=1d&events=history&crumb={}' \
+                .format(ticker, t1, t2, urllib.parse.quote_plus(self.crumb))
+
+            text = self.urlopen(url)
+
+            with open(os.path.join(self.basedir, 'yahoo-{}.csv'.format(ticker)), "w", encoding="utf-8") as csv_file:
+                print(text, file=csv_file)
+
+            self._read_ticker_csv_file(ticker)
+
+        except HttpException:
+            logger.exception("HttpException ticker=%s datacode=%s date=%s", ticker, datacode, date)
+            return None
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s date=%s", ticker, datacode, date)
+            return 'Yahoo.getHistoric({}, {}, {}) - urlopen: {}'.format(ticker, datacode, date, e)
+
+        try:
+            if ticker in self.historicdata:
+                ticks = self.historicdata[ticker]
+
+                if date in ticks:
+                    return self._return_value(ticks[date], datacode)
+
+                # future date
+                if date > max(ticks):
+                    return 'Future date \'{}\''.format(date)
+
+                # weekend or trading holiday
+                return 'Not a trading day \'{}\''.format(date)
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getHistoric({}, {}, {}) - process: {}'.format(ticker, datacode, date, e)
+
+        return None
+
+
+def createInstance(ctx):
+    return Yahoo(ctx)
@@ -14,19 +14,13 @@ import os
 cur_dir = os.getcwd()

 addin_id = "com.financials.getinfo"
-addin_version = "2.3.0"
+addin_version = "3.6.3"
 addin_displayname = "Financial Market Extension"
 addin_publisher_link = "https://github.com/cmallwitz/Financials-Extension"
 addin_publisher_name = "The Publisher"

 print("Generating extension files for version", addin_version)

-################################################################################
-# version.py
-
-version_file = open(cur_dir + '/build/version.py', 'w')
-version_file.write(f"version = '{addin_version}'")
-version_file.close()

 ################################################################################
 # description.xml
@@ -1,269 +0,0 @@
-#  google.py
-#
-#  license: GNU LGPL
-#
-#  This library is free software; you can redistribute it and/or
-#  modify it under the terms of the GNU Lesser General Public
-#  License as published by the Free Software Foundation; either
-#  version 3 of the License, or (at your option) any later version.
-
-
-import html
-import logging
-import os
-import re
-import time
-import xml.etree.ElementTree as ET
-
-import dateutil
-
-from baseclient import BaseClient, RedirectException
-from datacode import Datacode
-from naivehtmlparser import NaiveHTMLParser
-from tz import whois_timezone_info
-
-logger = logging.getLogger(__name__)
-
-
-# logger.setLevel(logging.DEBUG)
-
-
-def handle_abbreviations(s):
-    s = str(s).strip()
-    if s.endswith('M'):
-        return float(s[:-1]) * 1000000
-    elif s.endswith('B'):
-        return float(s[:-1]) * 1000000000
-    elif s.endswith('T'):
-        return float(s[:-1]) * 1000000000000
-    return float(s)
-
-
-def un_span(s):
-    return re.sub(r'<span [^>]*>', '', s).replace('</span>', '')
-
-
-class Google(BaseClient):
-    def __init__(self, ctx):
-        super().__init__()
-
-        self.realtime = {}
-        self.location = None
-
-    def getRealtime(self, ticker: str, datacode: int):
-
-        """
-        Retrieve realtime data for ticker from Google Finance and cache it for further lookups
-
-        :param ticker: the ticker symbol e.g. LON:VOD
-        :param datacode: the requested datacode
-        :return:
-        """
-
-        # remove white space
-        ticker = "".join(ticker.split())
-
-        # use cached value for up to 60 seconds
-        if ticker in self.realtime:
-            tick = self.realtime[ticker]
-            if time.time() - 60 < tick[Datacode.TIMESTAMP]:
-                return self._return_value(tick, datacode)
-            else:
-                del self.realtime[ticker]
-
-        q_param = 'q=' + ticker
-
-        if not self.location:
-            url = 'https://www.google.com/search?hl=en&tbm=fin&' + q_param
-
-            try:
-                self.urlopen(url, redirect=False)
-            except RedirectException as e:
-                self.location = e.location.replace('&' + q_param, '')
-            except BaseException as e:
-                logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-                return 'Google.getRealtime(\'{}\', {}) - location: {}'.format(ticker, datacode, e)
-
-        if not self.location:
-            url = 'https://www.google.com/search?tbm=fin&' + q_param
-        else:
-            url = self.location + '&' + q_param
-
-        try:
-            text = self.urlopen(url)
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Google.getRealtime(\'{}\', {}) - urlopen: {} {}'.format(ticker, datacode, e, url)
-
-        try:
-            with open(os.path.join(self.basedir, 'google-{}.html'.format(ticker)), "w", encoding="utf-8") as text_file:
-                print(f"<!-- '{url}' -->\r\n\r\n{text}", file=text_file)
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-
-        if ticker not in self.realtime:
-            self.realtime[ticker] = {}
-
-        tick = self.realtime[ticker]
-
-        tick[Datacode.TIMESTAMP] = time.time()
-
-        tick[Datacode.NAME] = None
-        tick[Datacode.TICKER] = None
-        tick[Datacode.CURRENCY] = None
-        tick[Datacode.LAST_PRICE] = None
-        tick[Datacode.CHANGE] = None
-        tick[Datacode.CHANGE_IN_PERCENT] = None
-        tick[Datacode.VOLUME] = None
-        tick[Datacode.LOW_52_WEEK] = None
-        tick[Datacode.HIGH_52_WEEK] = None
-        tick[Datacode.LAST_PRICE_DATE] = None
-        tick[Datacode.LAST_PRICE_TIME] = None
-        tick[Datacode.TIMEZONE] = None
-
-        tick[Datacode.OPEN] = None
-        tick[Datacode.HIGH] = None
-        tick[Datacode.LOW] = None
-        tick[Datacode.PREV_CLOSE] = None
-        tick[Datacode.MARKET_CAP] = None
-
-        tick[Datacode.EXCHANGE] = None
-        tick[Datacode.AVG_DAILY_VOL_3MONTH] = None
-
-        try:
-            r = '<span[^>]+role="heading"[^>]+>(.*?)</span>'
-            pattern = re.compile(r)
-
-            match = pattern.search(text)
-            if not match:
-                return None
-            start = match.span(0)[1]
-
-            tick[Datacode.NAME] = self.save_wrapper(
-                lambda: html.unescape(un_span(match.group(1)).strip()))
-
-            # next div is TICKER
-            r = '<div [^>]*><div [^>]*>(.*?)</div></div>'
-            pattern = re.compile(r)
-
-            match = pattern.search(text, start)
-            if not match:
-                return 'Google.getRealtime({}, {}) - no match'.format(ticker, datacode)
-
-            ticker = self.save_wrapper(
-                lambda: html.unescape(match.group(1)).replace(' ', ''))
-
-            tick[Datacode.EXCHANGE] = self.save_wrapper(lambda: ticker.split(':')[0])
-            tick[Datacode.TICKER] = self.save_wrapper(lambda: ticker.split(':')[1])
-
-        except BaseException as e:
-            return 'Google.getRealtime({}, {}) - process: {}'.format(ticker, datacode, e)
-
-        try:
-            r = '<sticky-header [^>]*>(.*?)</sticky-header>'
-            pattern = re.compile(r, flags=re.DOTALL)
-            match = re.search(pattern, text)
-
-            if match:
-                text = match.group(1)
-            else:
-                return 'Data for \'{}\' not found'.format(ticker)
-
-            parser = NaiveHTMLParser()
-            root = parser.feed(text)
-            parser.close()
-
-            cards = root.findall('.//g-card-section')
-
-            if len(cards) < 4:
-                return 'Data for \'{}\' not found'.format(ticker)
-
-            header = cards[1]
-
-            tick[Datacode.LAST_PRICE] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(header.find('./span[1]/span[1]/span[1]').text).replace(',', '').strip()))
-
-            tick[Datacode.CURRENCY] = self.save_wrapper(
-                lambda: html.unescape(header.find('./span[1]/span[1]/span[2]').text).strip())
-
-            tick[Datacode.CHANGE] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(header.find('./span[2]/span[1]').text).replace('−', '-').replace(',', '').strip()))
-
-            # percentage is always wrapped in (...) and always positive even if there is a price drop
-            tick[Datacode.CHANGE_IN_PERCENT] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(header.find('./span[2]/span[2]/span[1]').text).strip()
-                        .replace('(', '').replace(')', '').replace('%', '')))
-
-            try:
-                value = html.unescape(header.find('./div[1]/span[1]/span[2]').text).replace('·', '').strip()
-                logger.debug(value)
-                dt = dateutil.parser.parse(value, tzinfos=whois_timezone_info)
-                tick[Datacode.LAST_PRICE_DATE] = dt.date()
-                tick[Datacode.LAST_PRICE_TIME] = dt.time()
-
-                time_bits = value.split(' ')
-                if len(time_bits) >= 4:
-                    tick[Datacode.TIMEZONE] = time_bits[-1]
-
-            except BaseException as e:
-                pass
-
-            footer = cards[3]
-            logger.debug(ET.tostring(footer))
-
-            # parse 'footer' for remaining fields
-            table = footer.find('./div[1]/div[1]/div[1]/table[1]')
-
-            tick[Datacode.OPEN] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(table.find('./tr[1]/td[2]').text).replace(',', '').strip()))
-
-            tick[Datacode.HIGH] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(table.find('./tr[2]/td[2]').text).replace(',', '').strip()))
-
-            tick[Datacode.LOW] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(table.find('./tr[3]/td[2]').text).replace(',', '').strip()))
-
-            tick[Datacode.MARKET_CAP] = self.save_wrapper(
-                lambda: handle_abbreviations(
-                    html.unescape(table.find('./tr[4]/td[2]').text).replace(',', '').replace('-', '').strip()))
-
-            table = footer.find('./div[1]/div[1]/div[2]/table[1]')
-
-            # for indices: first item on right side is LOW
-            if html.unescape(table.find('./tr[1]/td[1]').text).strip() == 'Low':
-                tick[Datacode.LOW] = self.save_wrapper(
-                    lambda: float(
-                        html.unescape(table.find('./tr[1]/td[2]').text).replace(',', '').strip()))
-
-            tick[Datacode.PREV_CLOSE] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(table.find('./tr[2]/td[2]').text).replace(',', '').strip()))
-
-            tick[Datacode.HIGH_52_WEEK] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(table.find('./tr[3]/td[2]').text).replace(',', '').strip()))
-
-            tick[Datacode.LOW_52_WEEK] = self.save_wrapper(
-                lambda: float(
-                    html.unescape(table.find('./tr[4]/td[2]').text).replace(',', '').strip()))
-
-            logger.info(tick)
-
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Google.getRealtime({}, {}) - process: {}'.format(ticker, datacode, e)
-
-        return self._return_value(self.realtime[ticker], datacode)
-
-    def getHistoric(self, ticker, datacode, date):
-        return 'Google.getHistoric: Historic Data not implemented.'
-
-
-def createInstance(ctx):
-    return Google(ctx)
@@ -0,0 +1,42 @@
+#  test_yahoo.py
+#
+#  license: GNU LGPL
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 3 of the License, or (at your option) any later version.
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+import unittest
+
+logging.basicConfig(level=logging.ERROR, format="%(asctime)s %(name)s %(levelname)s %(message)s")
+
+import financials
+from datacode import Datacode
+import testutils
+
+financials = financials.createInstance(None)
+
+
+class Test(unittest.TestCase):
+
+    @classmethod
+    def tearDownClass(cls):
+        # this avoids "ResourceWarning: unclosed..." on cached socket connections
+        financials.close()
+
+    def test_currency(self):
+        s = financials.getRealtime('ETH-EUR', Datacode.LAST_PRICE.value, 'COINBASE')
+        self.assertEqual(float, type(s), 'test_currency LAST_PRICE')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('unittest_args', nargs='*')
+    args = parser.parse_args()
+    unit_argv = [sys.argv[0]] + args.unittest_args
+    unittest.main(argv=unit_argv)
@@ -15,6 +15,7 @@ import unittest
 logging.basicConfig(level=logging.ERROR, format="%(asctime)s %(name)s %(levelname)s %(message)s")

 import financials
+from datacode import Datacode
 import testutils

 financials = financials.createInstance(None)
@@ -22,6 +23,11 @@ financials = financials.createInstance(None)

 class Test(unittest.TestCase):

+    @classmethod
+    def tearDownClass(cls):
+        # this avoids "ResourceWarning: unclosed..." on cached socket connections
+        financials.close()
+
    def test_currency(self):
        s = financials.getRealtime('EURGBP', 'LAST_PRICE', 'FT')
        self.assertEqual(float, type(s), 'test_currency LAST_PRICE')
@@ -70,6 +76,12 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('IBM:NYQ', 'EPS', 'FT')
        self.assertEqual(float, type(s), 'test_US_equity EPS {}'.format(s))

+        s = financials.getRealtime('IBM:NYQ', 'SHARES_OUT', 'FT')
+        self.assertEqual(float, type(s), 'test_US_equity SHARES_OUT {}'.format(s))
+
+        s = financials.getRealtime('IBM:NYQ', 'FREE_FLOAT', 'FT')
+        self.assertEqual(float, type(s), 'test_US_equity FREE_FLOAT {}'.format(s))
+
        s = financials.getRealtime('IBM:NYQ', 'PE_RATIO', 'FT')
        self.assertEqual(float, type(s), 'test_US_equity PE_RATIO {}'.format(s))

@@ -96,7 +108,7 @@ class Test(unittest.TestCase):

        s = financials.getRealtime('IBM:NYQ', 'INDUSTRY', 'FT')
        self.assertEqual(str, type(s), 'test_US_equity INDUSTRY {}'.format(s))
-        self.assertEqual(s, 'Software & Computer Services', 'test_US_equity INDUSTRY {}'.format(s))
+        self.assertEqual(s, 'Technology', 'test_US_equity INDUSTRY {}'.format(s))

        # may fail (s is None) on weekends when date/time displayed doesn't have time component with TZ
        s = financials.getRealtime('IBM:NYQ', 'TIMEZONE', 'FT')
@@ -128,20 +140,63 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('VFIAX', 'LAST_PRICE_TIME', 'FT')
        self.assertEqual(str, type(s), 'test_US_mutuals LAST_PRICE_TIME {}'.format(s))

+    def test_US_futures(self):
+
+        # https://markets.ft.com/data/commodities/tearsheet/summary?s=775326843 ESH25:IOM
+
+        s = financials.getRealtime('775326843', Datacode.NAME.value, 'FT')
+        self.assertEqual(str, type(s), 'test_realtime_US_futures NAME {}'.format(s))
+        self.assertEqual('EMINI S&P MAR25', s, 'test_US_futures NAME {}'.format(s))
+
+        s = financials.getRealtime('775326843', Datacode.LAST_PRICE.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures LAST_PRICE {}'.format(s))
+
+        # s = financials.getRealtime('775326843', Datacode.OPEN.value, 'FT')
+        # self.assertEqual(float, type(s), 'test_US_futures OPEN {}'.format(s))
+
+        s = financials.getRealtime('775326843', Datacode.VOLUME.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures VOLUME {}'.format(s))
+
+        s = financials.getRealtime('775326843', Datacode.LOW_52_WEEK.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures LOW_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('775326843', Datacode.HIGH_52_WEEK.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures HIGH_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('775326843', Datacode.CHANGE.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures CHANGE {}'.format(s))
+
+        s = financials.getRealtime('775326843', Datacode.CHANGE_IN_PERCENT.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures CHANGE_IN_PERCENT {}'.format(s))
+
    def test_UK_ETF(self):
        s = financials.getRealtime('CSP1:LSE:GBX', 'NAME', 'FT')
        self.assertEqual(str, type(s), 'test_UK_ETF NAME {}'.format(s))
        self.assertEqual('iShares Core S&P 500 UCITS ETF USD (Acc)', s, 'test_UK_ETF NAME {}'.format(s))

-        s = financials.getRealtime('C060:GER:EUR', 'NAME', 'FT')
-        self.assertEqual(str, type(s), 't_UK_ETF NAME {}'.format(s))
-
        s = financials.getRealtime('VERX:LSE:GBP', 'LAST_PRICE', 'FT')
        self.assertEqual(float, type(s), 'test_UK_ETF LAST_PRICE {}'.format(s))

        s = financials.getRealtime('VERX:LSE:GBP', 'TIMEZONE', 'FT')
        self.assertEqual(str, type(s), 'test_UK_ETF TIMEZONE {}'.format(s))

+    def test_UK_equity(self):
+        s = financials.getRealtime('VOD:LSE', 'NAME', 'FT')
+        self.assertEqual(str, type(s), 'test_UK_equity NAME {}'.format(s))
+        self.assertEqual('Vodafone Group PLC', s, 'test_UK_ETF NAME {}'.format(s))
+
+        s = financials.getRealtime('VOD:LSE', 'BID', 'FT')
+        self.assertEqual(float, type(s), 'test_UK_equity BID {}'.format(s))
+
+        s = financials.getRealtime('VOD:LSE', 'ASK', 'FT')
+        self.assertEqual(float, type(s), 'test_UK_equity ASK {}'.format(s))
+
+        s = financials.getRealtime('VOD:LSE', 'LAST_PRICE', 'FT')
+        self.assertEqual(float, type(s), 'test_UK_equity LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('VOD:LSE', 'TIMEZONE', 'FT')
+        self.assertEqual(str, type(s), 'test_UK_equity TIMEZONE {}'.format(s))
+
    def test_DE_equity(self):
        s = financials.getRealtime('SAPX:GER', 'NAME', 'FT')
        self.assertEqual('SAP SE', s, 'test_DE_equity NAME {}'.format(s))
@@ -185,7 +240,7 @@ class Test(unittest.TestCase):

        s = financials.getRealtime('SAPX:GER', 'INDUSTRY', 'FT')
        self.assertEqual(str, type(s), 'test_DE_equity INDUSTRY {}'.format(s))
-        self.assertEqual('Software & Computer Services', s, 'test_DE_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Technology', s, 'test_DE_equity INDUSTRY {}'.format(s))

        s = financials.getRealtime('SAPX:GER', 'LAST_PRICE_DATE', 'FT')
        self.assertEqual(str, type(s), 'test_DE_equity LAST_PRICE_DATE {}'.format(s))
@@ -200,6 +255,12 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('SAPX:GER', 'EPS', 'FT')
        self.assertEqual(float, type(s), 'test_DE_equity EPS {}'.format(s))

+        s = financials.getRealtime('SAPX:GER', 'SHARES_OUT', 'FT')
+        self.assertEqual(float, type(s), 'test_DE_equity SHARES_OUT {}'.format(s))
+
+        s = financials.getRealtime('SAPX:GER', 'FREE_FLOAT', 'FT')
+        self.assertEqual(float, type(s), 'test_DE_equity FREE_FLOAT {}'.format(s))
+
        s = financials.getRealtime('SAPX:GER', 'PE_RATIO', 'FT')
        self.assertEqual(float, type(s), 'test_DE_equity PE_RATIO {}'.format(s))

@@ -237,6 +298,24 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('ISHAX:GER', 'EX_DIV_DATE', 'FT')
        self.assertIsNone(s, 'test_DE_equity EX_DIV_DATE {}'.format(s))

+    def test_DK_equity(self):
+        s = financials.getRealtime('NOVO B:CPH', 'name', 'FT')
+        self.assertEqual('Novo Nordisk A/S', s, 'test_DK_equity NAME {}'.format(s))
+
+        s = financials.getRealtime('NOVO B:CPH', 'currency', 'FT')
+        self.assertEqual('DKK', s, 'test_DK_equity CURRENCY {}'.format(s))
+
+        s = financials.getRealtime('NOVO B:CPH', 'industry', 'FT')
+        self.assertEqual(str, type(s), 'test_DK_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Pharmaceuticals and Biotechnology', s, 'test_DK_equity INDUSTRY {}'.format(s))
+
+    def test_SE_equity(self):
+        s = financials.getRealtime('ACRI A:STO', 'name', 'FT')
+        self.assertEqual('Acrinova AB (publ)', s, 'test_SE_equity NAME {}'.format(s))
+
+        s = financials.getRealtime('SE0015660014', 'name', 'FT')
+        self.assertEqual('Acrinova AB (publ)', s, 'test_SE_equity NAME {}'.format(s))
+
    def test_TY_equity(self):
        s = financials.getRealtime('6503:TYO', 'OPEN', 'FT')
        self.assertEqual(float, type(s), 'test_TY_equity OPEN {}'.format(s))
@@ -270,6 +349,42 @@ class Test(unittest.TestCase):
        self.assertEqual(str, type(s), 'test_TY_equity INDUSTRY {}'.format(s))
        self.assertEqual('General Industrials', s, 'test_TY_equity INDUSTRY {}'.format(s))

+    def test_TLV_equity(self):
+        s = financials.getRealtime('LUMI:TLV', 'LAST_PRICE', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'OPEN', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity OPEN {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'LOW', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity LOW {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'HIGH', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity HIGH {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'LOW_52_WEEK', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity LOW_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'HIGH_52_WEEK', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity HIGH_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'MARKET_CAP', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity MARKET_CAP {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'VOLUME', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity VOLUME {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'CURRENCY', 'FT')
+        self.assertEqual('ILa', s, 'test_TLV_equity CURRENCY')
+
+        s = financials.getRealtime('LUMI:TLV', 'SECTOR', 'FT')
+        self.assertEqual(str, type(s), 'test_TLV_equity SECTOR {}'.format(s))
+        self.assertEqual('Financials', s, 'test_TY_equity SECTOR {}'.format(s))
+
+        s = financials.getRealtime('LUMI:TLV', 'INDUSTRY', 'FT')
+        self.assertEqual(str, type(s), 'test_TLV_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Banks', s, 'test_TY_equity INDUSTRY {}'.format(s))
+
    def test_index(self):
        s = financials.getRealtime('INX:IOM', 'ticker', 'FT')
        self.assertEqual('INX:IOM', s, 'test_index TICKER {}'.format(s))
@@ -1,284 +0,0 @@
-#  test_google.py
-#
-#  license: GNU LGPL
-#
-#  This library is free software; you can redistribute it and/or
-#  modify it under the terms of the GNU Lesser General Public
-#  License as published by the Free Software Foundation; either
-#  version 3 of the License, or (at your option) any later version.
-
-import argparse
-import logging
-import sys
-import unittest
-
-logging.basicConfig(level=logging.ERROR, format="%(asctime)s %(name)s %(levelname)s %(message)s")
-
-import financials
-from datacode import Datacode
-
-financials = financials.createInstance(None)
-
-
-class Test(unittest.TestCase):
-
-    def test_currency(self):
-        s = financials.getRealtime('EURGBP', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertIsNone(s, 'test_currency LAST_PRICE')
-
-    def test_UK_equity(self):
-        s = financials.getRealtime('LON:VOD', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_UK_equity LAST_PRICE {}'.format(s))
-
-        s = financials.getRealtime('LON:VOD', Datacode.TICKER.value, 'GOOGLE')
-        self.assertEqual(s, 'VOD', 'test_UK_equity TICKER')
-
-        s = financials.getRealtime('LON:VOD', Datacode.NAME.value, 'GOOGLE')
-        self.assertEqual(type(s), str, 'test_UK_equity NAME')
-
-        s = financials.getRealtime('LON:VOD', Datacode.EXCHANGE.value, 'GOOGLE')
-        self.assertEqual(s, 'LON', 'test_UK_equity EXCHANGE')
-
-        s = financials.getRealtime('LON:VOD', Datacode.PREV_CLOSE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_UK_equity PREV_CLOSE {}'.format(s))
-
-        # MARKET_CAP missing for UK stock but available for German stock - weekend issue (FX) ?
-        s = financials.getRealtime('LON:VOD', Datacode.MARKET_CAP.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_UK_equity MARKET_CAP {}'.format(s))
-
-    def test_UK_ETF(self):
-        s = financials.getRealtime('LON:CSP1', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_UK_ETF LAST_PRICE {}'.format(s))
-
-        s = financials.getRealtime('LON:CSP1', Datacode.CURRENCY.value, 'GOOGLE')
-        self.assertEqual(s, 'GBX', 'test_UK_ETF CURRENCY')
-
-        s = financials.getRealtime('LON:FTAL', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_UK_ETF LAST_PRICE {}'.format(s))
-
-        s = financials.getRealtime('LON:FTAL', Datacode.CURRENCY.value, 'GOOGLE')
-        self.assertEqual(s, 'GBP', 'test_UK_ETF CURRENCY')
-
-        s = financials.getRealtime('LON:FTAL', Datacode.NAME.value, 'GOOGLE')
-        self.assertEqual(type(s), str, 'test_UK_ETF NAME')
-
-    def test_DE_equity(self):
-        s = financials.getRealtime('FRA:SAP', 7, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_DE_equity 7')
-
-        s = financials.getRealtime('FRA:SAP', '7', 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_DE_equity \'7\'')
-
-        s = financials.getRealtime('FRA:SAP', 8, 'GOOGLE')
-        self.assertEqual(type(s), str, 'test_DE_equity 8')
-
-        s = financials.getRealtime('FRA:SAP', 8.1, 'GOOGLE')
-        self.assertEqual(type(s), str, 'test_DE_equity 8.1')
-
-        s = financials.getRealtime('FRA:SAP', '8.1', 'GOOGLE')
-        self.assertEqual(type(s), str, 'test_DE_equity \'8.1\'')
-
-        s = financials.getRealtime('FRA:SAP', 10, 'GOOGLE')
-        self.assertEqual(type(s), str, 'test_DE_equity 10')
-
-        s = financials.getRealtime('FRA:SAP', '11', 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_DE_equity \'11\'')
-
-        s = financials.getRealtime('FRA:SAP', '21', 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_DE_equity \'21\'')
-
-        s = financials.getRealtime('FRA:SAP', Datacode.TIMEZONE.value, 'GOOGLE')
-        # self.assertEqual(s, 'Europe/Berlin', 'test_DE_equity TIMEZONE')
-        self.assertTrue(s == 'CET' or s == 'CEST', 'test_DE_equity TIMEZONE: {}'.format(s))
-
-    def test_DE_ETF(self):
-        s = financials.getRealtime('FRA:C060', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_DE_ETF LAST_PRICE {}'.format(s))
-
-        s = financials.getRealtime('FRA:C060', Datacode.CURRENCY.value, 'GOOGLE')
-        self.assertEqual('EUR', s, 'test_DE_ETF CURRENCY')
-
-        s = financials.getRealtime('FRA:C060', Datacode.TICKER.value, 'GOOGLE')
-        self.assertEqual('C060', s, 'test_DE_ETF TICKER')
-
-        s = financials.getRealtime('FRA:C060', Datacode.EXCHANGE.value, 'GOOGLE')
-        self.assertEqual('FRA', s, 'test_DE_ETF EXCHANGE')
-
-        s = financials.getRealtime('FRA:C060', Datacode.CURRENCY.value, 'GOOGLE')
-        self.assertEqual('EUR', s, 'test_DE_ETF CURRENCY')
-
-        s = financials.getRealtime('FRA:C060', Datacode.MARKET_CAP.value, 'GOOGLE')
-        self.assertIsNone(s, 'test_DE_ETF MARKET_CAP {}'.format(s))
-
-    def test_TY_equity(self):
-        s = financials.getRealtime('TYO:6503', Datacode.OPEN.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_TY_equity OPEN {}'.format(s))
-
-        s = financials.getRealtime('TYO:6503', Datacode.LOW.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_TY_equity LOW {}'.format(s))
-
-        s = financials.getRealtime('TYO:6503', Datacode.HIGH.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_TY_equity HIGH {}'.format(s))
-
-        s = financials.getRealtime('TYO:6503', Datacode.LOW_52_WEEK.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_TY_equity LOW_52_WEEK {}'.format(s))
-
-        s = financials.getRealtime('TYO:6503', Datacode.HIGH_52_WEEK.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_TY_equity HIGH_52_WEEK {}'.format(s))
-
-        s = financials.getRealtime('TYO:6503', Datacode.MARKET_CAP.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_TY_equity MARKET_CAP {}'.format(s))
-
-        # s = financials.getRealtime('TYO:6503', Datacode.VOLUME.value, 'GOOGLE')
-        # self.assertEqual(float, type(s), 'test_TY_equity VOLUME {}'.format(s))
-
-        s = financials.getRealtime('TYO:6503', Datacode.CURRENCY.value, 'GOOGLE')
-        self.assertEqual(s, 'JPY', 'test_TY_equity CURRENCY')
-
-    def test_US_equity(self):
-        s = financials.getRealtime(' NASDAQ : AAPL ', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_US_equity LAST_PRICE {}'.format(s))
-
-        s = financials.getRealtime(' NASDAQ : AAPL ', Datacode.TICKER.value, 'GOOGLE')
-        self.assertEqual(s, 'AAPL', 'test_US_equity TICKER')
-
-        s = financials.getRealtime(' NASDAQ : AAPL ', Datacode.EXCHANGE.value, 'GOOGLE')
-        self.assertEqual(s, 'NASDAQ', 'test_US_equity EXCHANGE')
-
-        s = financials.getRealtime(' NASDAQ : AAPL ', Datacode.CURRENCY.value, 'GOOGLE')
-        self.assertEqual(s, 'USD', 'test_US_equity CURRENCY')
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_US_equity LAST_PRICE {}'.format(s))
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.TICKER.value, 'GOOGLE')
-        self.assertEqual(s, 'IBM', 'test_US_equity TICKER')
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.EXCHANGE.value, 'GOOGLE')
-        self.assertEqual(s, 'NYSE', 'test_US_equity EXCHANGE')
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.CURRENCY.value, 'GOOGLE')
-        self.assertEqual(s, 'USD', 'test_US_equity CURRENCY')
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.NAME.value, 'GOOGLE')
-        self.assertEqual(type(s), str, 'test_US_equity NAME')
-        self.assertEqual(s, 'IBM Common Stock', 'test_US_equity NAME')
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.LOW.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_US_equity LOW {}'.format(s))
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.HIGH.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_US_equity HIGH {}'.format(s))
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.LOW_52_WEEK.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_US_equity LOW_52_WEEK {}'.format(s))
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.HIGH_52_WEEK.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_US_equity HIGH_52_WEEK {}'.format(s))
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.MARKET_CAP.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_US_equity MARKET_CAP {}'.format(s))
-
-        # s = financials.getRealtime('NYSE:IBM', Datacode.VOLUME.value, 'GOOGLE')
-        # self.assertEqual(float, type(s), 'test_US_equity VOLUME {}'.format(s))
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.TIMESTAMP.value, 'GOOGLE')
-        self.assertEqual(s, 'Data doesn\'t exist - 999', 'test_US_equity TIMESTAMP')
-
-        s = financials.getRealtime('NYSE:IBM', Datacode.TIMEZONE.value, 'GOOGLE')
-        # self.assertEqual('America/New_York', s, 'test_US_equity TIMEZONE')
-        # self.assertEqual('GMT-4', s, 'test_US_equity TIMEZONE')
-        self.assertEqual('GMT-5', s, 'test_US_equity TIMEZONE')
-
-    def test_US_mutuals(self):
-        s = financials.getRealtime('MUTF:VFIAX', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_US_mutuals LAST_PRICE - {}'.format(s))
-
-        s = financials.getRealtime('MUTF:VFIAX', Datacode.CURRENCY.value, 'GOOGLE')
-        self.assertEqual(s, 'USD', 'test_US_mutuals CURRENCY')
-
-        s = financials.getRealtime('MUTF:VFIAX', Datacode.TIMEZONE.value, 'GOOGLE')
-        self.assertIsNone(s, 'test_US_mutuals TIMEZONE')
-
-    def test_index(self):
-        s = financials.getRealtime('INDEXDB:DAX', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_index LAST_PRICE {}'.format(s))
-
-        s = financials.getRealtime('INDEXDB:DAX', Datacode.CHANGE_IN_PERCENT.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_index CHANGE_IN_PERCENT')
-
-        s = financials.getRealtime('INDEXDB:DAX', Datacode.CHANGE.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_index CHANGE')
-
-        s = financials.getRealtime('INDEXDB:DAX', Datacode.OPEN.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_index OPEN')
-
-        s = financials.getRealtime('INDEXDB:DAX', Datacode.LOW.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_index LOW')
-
-        s = financials.getRealtime('INDEXDB:DAX', Datacode.HIGH.value, 'GOOGLE')
-        self.assertEqual(float, type(s), 'test_index HIGH')
-
-    def test_errors(self):
-        s = financials.getRealtime(None, Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(s, 'Ticker is empty', 'test_errors')
-
-        s = financials.getRealtime('NYS:IBM', None, 'GOOGLE')
-        self.assertEqual(s, 'Datacode is empty', 'test_errors')
-
-        s = financials.getRealtime('DOES_NOT_EXISTS', Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertIsNone(s, 'test_errors')
-
-        s = financials.getRealtime('NYS:IBM', 'Foo', 'GOOGLE')
-        self.assertEqual(s, 'Datacode is invalid', 'test_errors')
-
-        # Historic data not supported on GOOGLE
-
-        s = financials.getHistoric('NYS:IBM', Datacode.LAST_PRICE.value, '2017-01-01', 'GOOGLE')
-        self.assertEqual(s, 'Source \'GOOGLE\' not supported', 'test_errors')
-
-    def test_errors_cell_range_passed(self):
-        cell_range = ((1, 2), ('3', '4'), (5.0, 6.0))
-
-        s = financials.getRealtime(cell_range, Datacode.LAST_PRICE.value, 'GOOGLE')
-        self.assertEqual(s, 'Cell range not allowed for ticker', 'test_errors')
-
-        s = financials.getRealtime('NYS:IBM', cell_range, 'GOOGLE')
-        self.assertEqual(s, 'Cell range not allowed for datacode', 'test_errors')
-
-        s = financials.getRealtime('NYS:IBM', Datacode.LAST_PRICE.value, cell_range)
-        self.assertEqual(s, 'Cell range not allowed for source', 'test_errors')
-
-    def test_support(self):
-        cell_range = ((1, 2), ('3', '4'), (5.0, 6.0))
-
-        s = financials.getRealtime('SUPPORT')
-        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
-
-        s = financials.getRealtime('SUPPORT', 1)
-        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
-        self.assertTrue("type(datacode)=<class 'int'>" in s, 'test_errors SUPPORT {}'.format(s))
-        self.assertTrue("str(datacode)=1" in s, 'test_errors SUPPORT {}'.format(s))
-
-        s = financials.getRealtime('SUPPORT', 1.0)
-        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
-        self.assertTrue("type(datacode)=<class 'float'>" in s, 'test_errors SUPPORT {}'.format(s))
-        self.assertTrue("str(datacode)=1.0" in s, 'test_errors SUPPORT {}'.format(s))
-
-        s = financials.getRealtime('SUPPORT', '1')
-        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
-        self.assertTrue("type(datacode)=<class 'str'>" in s, 'test_errors SUPPORT {}'.format(s))
-        self.assertTrue("str(datacode)=1" in s, 'test_errors SUPPORT {}'.format(s))
-
-        s = financials.getRealtime('SUPPORT', cell_range)
-        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
-        self.assertTrue("type(datacode)=<class 'tuple'>" in s, 'test_errors SUPPORT {}'.format(s))
-        self.assertTrue("str(datacode)=((1, 2), ('3', '4'), (5.0, 6.0))" in s, 'test_errors SUPPORT {}'.format(s))
-
-
-if __name__ == '__main__':
-    parser = argparse.ArgumentParser()
-    parser.add_argument('unittest_args', nargs='*')
-    args = parser.parse_args()
-    unit_argv = [sys.argv[0]] + args.unittest_args
-    unittest.main(argv=unit_argv)
@@ -0,0 +1,57 @@
+#  test_support.py
+#
+#  license: GNU LGPL
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 3 of the License, or (at your option) any later version.
+
+import argparse
+import logging
+import sys
+import unittest
+
+logging.basicConfig(level=logging.ERROR, format="%(asctime)s %(name)s %(levelname)s %(message)s")
+
+import financials
+
+financials = financials.createInstance(None)
+
+
+class Test(unittest.TestCase):
+
+    def test_support(self):
+        cell_range = ((1, 2), ('3', '4'), (5.0, 6.0))
+
+        s = financials.getRealtime('SUPPORT')
+        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("version=3.0.0" in s, 'test_errors SUPPORT {}'.format(s))
+
+        s = financials.getRealtime('SUPPORT', 1)
+        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("type(datacode)=<class 'int'>" in s, 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("str(datacode)=1" in s, 'test_errors SUPPORT {}'.format(s))
+
+        s = financials.getRealtime('SUPPORT', 1.0)
+        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("type(datacode)=<class 'float'>" in s, 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("str(datacode)=1.0" in s, 'test_errors SUPPORT {}'.format(s))
+
+        s = financials.getRealtime('SUPPORT', '1')
+        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("type(datacode)=<class 'str'>" in s, 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("str(datacode)=1" in s, 'test_errors SUPPORT {}'.format(s))
+
+        s = financials.getRealtime('SUPPORT', cell_range)
+        self.assertTrue(s.startswith("ctx="), 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("type(datacode)=<class 'tuple'>" in s, 'test_errors SUPPORT {}'.format(s))
+        self.assertTrue("str(datacode)=((1, 2), ('3', '4'), (5.0, 6.0))" in s, 'test_errors SUPPORT {}'.format(s))
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('unittest_args', nargs='*')
+    args = parser.parse_args()
+    unit_argv = [sys.argv[0]] + args.unittest_args
+    unittest.main(argv=unit_argv)
@@ -16,6 +16,7 @@ import unittest

 logging.basicConfig(level=logging.ERROR, format="%(asctime)s %(name)s %(levelname)s %(message)s")

+import baseclient
 import financials
 from datacode import Datacode
 import testutils
@@ -23,8 +24,31 @@ import testutils
 financials = financials.createInstance(None)


+def urlopen_fail(self, url, redirect=True, data=None, headers={}, cookies=[], **kwargs):
+    raise baseclient.HttpException(url, 'ERROR: simulated urlopen() failed')
+
+
 class Test(unittest.TestCase):

+    @classmethod
+    def tearDownClass(cls):
+        # this avoids "ResourceWarning: unclosed..." on cached socket connections
+        financials.close()
+
+    def test_recovery_from_urlopen_error_issue(self):
+
+        financials.yahoo.last_url = 'blank'
+
+        urlopen_saved = baseclient.BaseClient.urlopen
+        baseclient.BaseClient.urlopen = urlopen_fail
+
+        s = financials.getRealtime('U1IH.F', Datacode.LAST_PRICE.value, 'YAHOO')
+
+        baseclient.BaseClient.urlopen = urlopen_saved
+
+        s = financials.getRealtime('U1IH.F', Datacode.PREV_CLOSE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_recovery_from_urlopen_error_issue PREV_CLOSE {}'.format(s))
+
    def test_currency(self):
        s = financials.getRealtime('EURGBP=X', Datacode.CURRENCY.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_currency CURRENCY')
@@ -32,6 +56,17 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('EURGBP=X', Datacode.LAST_PRICE.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_currency LAST_PRICE')

+    def test_realtime_US_ZVZZT(self):
+
+        s = financials.getRealtime('ZVZZT', Datacode.PAYOUT_RATIO.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_ZVZZT PAYOUT_RATIO {}'.format(s))
+
+        s = financials.getRealtime('ZVZZT', Datacode.SECTOR.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_ZVZZT SECTOR {}'.format(s))
+
+        s = financials.getRealtime('ZVZZT', Datacode.LAST_PRICE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_ZVZZT LAST_PRICE {}'.format(s))
+
    def test_realtime_US_equity(self):

        s = financials.getRealtime('^GSPC', Datacode.NAME.value, 'YAHOO')
@@ -73,16 +108,16 @@ class Test(unittest.TestCase):
        self.assertEqual(s, 'International Business Machines Corporation',
                         'test_realtime_US_equity NAME {}'.format(s))

-        s = financials.getRealtime('IBM', Datacode.SECTOR.value, 'YAHOO')
+        s = financials.getRealtime('IBM', 'SECTOR', 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_US_equity SECTOR {}'.format(s))
        self.assertEqual(s, 'Technology', 'test_realtime_US_equity SECTOR {}'.format(s))

-        s = financials.getRealtime('IBM', Datacode.INDUSTRY.value, 'YAHOO')
+        s = financials.getRealtime('IBM', 'INDUSTRY', 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_US_equity INDUSTRY {}'.format(s))
        self.assertEqual(s, 'Information Technology Services', 'test_realtime_US_equity INDUSTRY {}'.format(s))

        s = financials.getRealtime('IBM', Datacode.TIMEZONE.value, 'YAHOO')
-        self.assertEqual(s, 'America/New_York', 'test_realtime_US_equity TIMEZONE {}'.format(s))
+        self.assertTrue(s == 'America/New_York', 'test_realtime_US_equity TIMEZONE: {}'.format(s))

        s = financials.getRealtime('IBM', Datacode.BETA.value, 'YAHOO')
        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_equity BETA {}'.format(s))
@@ -103,11 +138,20 @@ class Test(unittest.TestCase):
        self.assertEqual(str, type(s), 'test_realtime_US_equity EX_DIV_DATE {}'.format(s))
        self.assertTrue(testutils.is_date(s), 'test_realtime_US_equity EX_DIV_DATE {}'.format(s))

-        s = financials.getRealtime('IBM', Datacode.PAYOUT_RATIO.value, 'YAHOO')
+        s = financials.getRealtime('IBM', 'PAYOUT_RATIO', 'YAHOO')
        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_equity PAYOUT_RATIO {}'.format(s))
+        # self.assertEqual(None, s, 'test_realtime_US_equity PAYOUT_RATIO {}'.format(s))
+
+        s = financials.getRealtime('IBM', 'SHARES_OUT', 'YAHOO')
+        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_equity SHARES_OUT {}'.format(s))
+        # self.assertEqual(None, s, 'test_realtime_US_equity SHARES_OUT {}'.format(s))
+
+        s = financials.getRealtime('IBM', 'FREE_FLOAT', 'YAHOO')
+        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_equity FREE_FLOAT {}'.format(s))
+        # self.assertEqual(None, s, 'test_realtime_US_equity FREE_FLOAT {}'.format(s))

        s = financials.getRealtime('IBM', Datacode.EXCHANGE.value, 'YAHOO')
-        self.assertEqual(s, 'NYQ', 'test_realtime_US_equity EXCHANGE')
+        self.assertEqual(s, 'NYSE', 'test_realtime_US_equity EXCHANGE')

    def test_realtime_US_mutuals(self):

@@ -117,11 +161,102 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('VFIAX', Datacode.LAST_PRICE.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_realtime_US_mutuals LAST_PRICE {}'.format(s))

-        s = financials.getRealtime('VFIAX', Datacode.LAST_PRICE_DATE.value, 'YAHOO')
-        self.assertEqual(str, type(s), 'test_realtime_US_mutuals LAST_PRICE_DATE {}'.format(s))
+        s = financials.getRealtime('VFIAX', Datacode.DIV.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_mutuals DIV {}'.format(s))

-        s = financials.getRealtime('VFIAX', Datacode.LAST_PRICE_TIME.value, 'YAHOO')
-        self.assertEqual(str, type(s), 'test_realtime_US_mutuals LAST_PRICE_TIME {}'.format(s))
+        s = financials.getRealtime('VFIAX', Datacode.DIV_YIELD.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_mutuals DIV_YIELD {}'.format(s)) # no yield
+
+        s = financials.getRealtime('SHRAX', Datacode.DIV.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_mutuals DIV {}'.format(s))
+
+        s = financials.getRealtime('SHRAX', Datacode.DIV_YIELD.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_mutuals DIV_YIELD {}'.format(s)) # no yield
+
+        s = financials.getRealtime('VERX.L', Datacode.DIV.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_mutuals DIV {}'.format(s)) # no dividend
+
+        s = financials.getRealtime('VERX.L', Datacode.DIV_YIELD.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_mutuals DIV_YIELD {}'.format(s)) # no yield
+
+    def test_realtime_US_options(self):
+
+        # symbol from https://finance.yahoo.com/quote/IBM/options?p=IBM
+        
+        s = financials.getRealtime('IBM250117C00165000', Datacode.PREV_CLOSE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options PREV_CLOSE {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.NAME.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_options NAME {}'.format(s))
+        self.assertEqual('IBM Jan 2025 165.000 call', s, 'test_realtime_US_options NAME {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.EXPIRY_DATE.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
+        self.assertTrue(testutils.is_date(s), 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
+        self.assertEqual("2025-01-17", s, 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.LAST_PRICE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.OPEN.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options OPEN {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.VOLUME.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options VOLUME {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.BID.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options BID {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.ASK.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options ASK {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.PAYOUT_RATIO.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_options PAYOUT_RATIO {}'.format(s))
+
+        s = financials.getRealtime('IBM250117C00165000', Datacode.SECTOR.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_options SECTOR {}'.format(s))
+
+    def test_realtime_US_futures(self):
+
+        s = financials.getRealtime('ES=F', Datacode.NAME.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_futures NAME {}'.format(s))
+        self.assertEqual('E-Mini S&P 500 Sep 24', s, 'test_realtime_US_futures NAME {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.TICKER.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_futures TICKER {}'.format(s))
+        self.assertEqual('ESU24.CME', s, 'test_realtime_US_futures TICKER {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.SETTLEMENT_DATE.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_futures SETTLEMENT_DATE {}'.format(s))
+        self.assertTrue(testutils.is_date(s), 'test_realtime_US_futures SETTLEMENT_DATE {}'.format(s))
+        self.assertEqual("2024-09-20", s, 'test_realtime_US_futures SETTLEMENT_DATE {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.LAST_PRICE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.OPEN.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures OPEN {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.VOLUME.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures VOLUME {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.BID.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures BID {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.ASK.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures ASK {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.CHANGE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures CHANGE {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.CHANGE_IN_PERCENT.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures CHANGE_IN_PERCENT {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.LOW.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures LOW {}'.format(s))
+
+        s = financials.getRealtime('ES=F', Datacode.HIGH.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures HIGH {}'.format(s))

    def test_realtime_UK_ETF(self):

@@ -129,12 +264,23 @@ class Test(unittest.TestCase):
        self.assertEqual(float, type(s), 'test_realtime_UK_ETF LAST_PRICE {}'.format(s))

        s = financials.getRealtime('VERX.L', Datacode.TIMEZONE.value, 'YAHOO')
-        self.assertEqual(s, 'Europe/London', 'test_realtime_UK_ETF TIMEZONE {}'.format(s))
+        self.assertTrue(s == 'Europe/London', 'test_realtime_UK_ETF TIMEZONE: {}'.format(s))

        s = financials.getRealtime('CSP1.L', Datacode.NAME.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_UK_ETF NAME {}'.format(s))
-        self.assertEqual(s, 'iShares VII Public Limited Company - iShares Core S&P 500 UCITS ETF',
-                         'test_realtime_UK_ETF NAME {}'.format(s))
+        self.assertEqual('iShares Core S&P 500 UCITS ETF USD (Acc)', s, 'test_realtime_UK_ETF NAME {}'.format(s))
+
+        s = financials.getRealtime('VERX.L', 'SECTOR', 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_UK_ETF SECTOR {}'.format(s))
+
+        s = financials.getRealtime('VERX.L', 'INDUSTRY', 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_UK_ETF INDUSTRY {}'.format(s))
+
+        s = financials.getRealtime('VERX.L', 'PAYOUT_RATIO', 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_UK_ETF PAYOUT_RATIO {}'.format(s))
+
+        s = financials.getRealtime('VERX.L', 'SHARES_OUT', 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_UK_ETF SHARES_OUT {}'.format(s))

    def test_realtime_DE_equity(self):

@@ -142,23 +288,85 @@ class Test(unittest.TestCase):
        self.assertEqual(float, type(s), 'test_realtime_DE_equity LAST_PRICE {}'.format(s))

        s = financials.getRealtime('SAP.DE', Datacode.TIMEZONE.value, 'YAHOO')
-        self.assertEqual(s, 'Europe/Berlin', 'test_realtime_DE_equity TIMEZONE {}'.format(s))
+        self.assertTrue(s == 'Europe/Berlin', 'test_realtime_DE_equity TIMEZONE: {}'.format(s))

        s = financials.getRealtime('SAP.DE', Datacode.SECTOR.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_DE_equity SECTOR {}'.format(s))
-        self.assertEqual(s, 'Technology', 'test_realtime_DE_equity SECTOR {}'.format(s))
+        self.assertEqual('Technology', s, 'test_realtime_DE_equity SECTOR {}'.format(s))

        s = financials.getRealtime('SAP.DE', Datacode.INDUSTRY.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_DE_equity INDUSTRY {}'.format(s))
-        self.assertEqual(s, 'Software—Application', 'test_realtime_DE_equity INDUSTRY {}'.format(s))
+        self.assertEqual(s, 'Software - Application', 'test_realtime_DE_equity INDUSTRY {}'.format(s))

-        s = financials.getRealtime('C060.DE', Datacode.NAME.value, 'YAHOO')
+        s = financials.getRealtime('LYY8.DE', Datacode.NAME.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_DE_equity NAME {}'.format(s))

-        s = financials.getRealtime('C060.DE', Datacode.EXCHANGE.value, 'YAHOO')
-        self.assertEqual(s, 'GER', 'test_realtime_DE_equity EXCHANGE')
+        s = financials.getRealtime('LYY8.DE', Datacode.EXCHANGE.value, 'YAHOO')
+        self.assertEqual(s, 'XETRA', 'test_realtime_DE_equity EXCHANGE')
+
+    def test_TA_equity(self):
+
+        s = financials.getRealtime('LUMI.TA', 'LAST_PRICE', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'OPEN', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity OPEN {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'LOW', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity LOW {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'HIGH', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity HIGH {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'LOW_52_WEEK', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity LOW_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'HIGH_52_WEEK', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity HIGH_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'MARKET_CAP', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity MARKET_CAP {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'VOLUME', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity VOLUME {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'CURRENCY', 'YAHOO')
+        self.assertEqual('ILA', s, 'test_TA_equity CURRENCY')
+
+        s = financials.getRealtime('LUMI.TA', 'SECTOR', 'YAHOO')
+        self.assertEqual(str, type(s), 'test_TLV_equity SECTOR {}'.format(s))
+        self.assertEqual('Financial Services', s, 'test_TA_equity SECTOR {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'INDUSTRY', 'YAHOO')
+        self.assertEqual(str, type(s), 'test_TLV_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Banks - Regional', s, 'test_TA_equity INDUSTRY {}'.format(s))
+
+    def test_DK_equity(self):
+        s = financials.getRealtime('NOVO-B.CO', 'last_price', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_DK_equity LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('NOVO-B.CO', 'name', 'YAHOO')
+        self.assertEqual('Novo Nordisk A/S', s, 'test_DK_equity NAME {}'.format(s))
+
+        s = financials.getRealtime('NOVO-B.CO', 'currency', 'YAHOO')
+        self.assertEqual('DKK', s, 'test_DK_equity CURRENCY {}'.format(s))
+
+        s = financials.getRealtime('NOVO-B.CO', 'industry', 'YAHOO')
+        self.assertEqual(str, type(s), 'test_DK_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Biotechnology', s, 'test_DK_equity INDUSTRY {}'.format(s))
+
+        s = financials.getRealtime('MAERSK-B.CO', 'currency', 'YAHOO')
+        self.assertEqual('DKK', s, 'test_DK_equity CURRENCY {}'.format(s))

    def test_realtime_TY_equity(self):
+        s = financials.getRealtime('6503.T', Datacode.SECTOR.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_TY_equity SECTOR {}'.format(s))
+        self.assertEqual(s, 'Industrials', 'test_TY_equity SECTOR {}'.format(s))
+
+        s = financials.getRealtime('6503.T', Datacode.INDUSTRY.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_TY_equity INDUSTRY {}'.format(s))
+        self.assertEqual(s, 'Electrical Equipment & Parts', 'test_TY_equity INDUSTRY {}'.format(s))
+
        s = financials.getRealtime('6503.T', Datacode.OPEN.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_TY_equity OPEN {}'.format(s))

@@ -183,13 +391,8 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('6503.T', Datacode.CURRENCY.value, 'YAHOO')
        self.assertEqual(s, 'JPY', 'test_TY_equity CURRENCY')

-        s = financials.getRealtime('6503.T', Datacode.SECTOR.value, 'YAHOO')
-        self.assertEqual(str, type(s), 'test_TY_equity SECTOR {}'.format(s))
-        self.assertEqual(s, 'Industrials', 'test_TY_equity SECTOR {}'.format(s))
-
-        s = financials.getRealtime('6503.T', Datacode.INDUSTRY.value, 'YAHOO')
-        self.assertEqual(str, type(s), 'test_TY_equity INDUSTRY {}'.format(s))
-        self.assertEqual(s, 'Electrical Equipment & Parts', 'test_TY_equity INDUSTRY {}'.format(s))
+        s = financials.getRealtime('6503.T', Datacode.TIMEZONE.value, 'YAHOO')
+        self.assertEqual(s, 'Asia/Tokyo', 'test_TY_equity TIMEZONE')

    def test_historic_US_equity(self):

@@ -200,15 +403,15 @@ class Test(unittest.TestCase):
        self.assertEqual('Not a trading day \'2017-01-01\'', s, 'test_historic_US_equity CLOSE {}'.format(s))

        s = financials.getHistoric('IBM', Datacode.LAST_PRICE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual('Data doesn\'t exist - 21', s, 'test_historic_US_equity LAST_PRICE {}'.format(s))
+        self.assertIsNone(s, 'test_historic_US_equity LAST_PRICE {}'.format(s))

        s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(167.190002, s, 'test_historic_US_equity CLOSE {}'.format(s))
+        self.assertEqual(159.837479, s, 'test_historic_US_equity CLOSE {}'.format(s))

        financials.yahoo.historicdata = {}

        s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(167.190002, s, 'test_historic_US_equity CLOSE {}'.format(s))
+        self.assertEqual(159.837479, s, 'test_historic_US_equity CLOSE {}'.format(s))

        directory = os.path.join(str(pathlib.Path.home()), '.financials-extension')
        ibm = os.path.join(directory, 'yahoo-IBM.csv')
@@ -220,11 +423,10 @@ class Test(unittest.TestCase):
        financials.yahoo.historicdata = {}

        s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(167.190002, s, 'test_historic_US_equity CLOSE {}'.format(s))
+        self.assertEqual(159.837479, s, 'test_historic_US_equity CLOSE {}'.format(s))

-        # Note: quarterly dividend and splits will change past adjusted prices - will fail after the next dividend
        s = financials.getHistoric('IBM', Datacode.ADJ_CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(139.61322, s, 'test_historic_US_equity ADJ_CLOSE {}'.format(s))
+        self.assertEqual(float, type(s), 'test_historic_US_equity ADJ_CLOSE {}'.format(s))

    def test_historic_UK_ETF(self):

@@ -237,14 +439,6 @@ class Test(unittest.TestCase):

        financials.yahoo.historicdata = {}

-        #  Inception Date 2014-09-30
-        s = financials.getHistoric('VERX.L', Datacode.CLOSE.value, '2018-04-02', 'YAHOO')  # Easter Monday
-        self.assertEqual(s, 'Not a trading day \'2018-04-02\'', 'test_historic_UK_ETF CLOSE {}'.format(s))
-
-        #  Inception Date 2014-09-30
-        s = financials.getHistoric('VERX.L', Datacode.CLOSE.value, '2015-01-01', 'YAHOO')
-        self.assertEqual(s, 'Not a trading day \'2015-01-01\'', 'test_historic_UK_ETF CLOSE {}'.format(s))
-
        s = financials.getHistoric('VERX.L', Datacode.LAST_PRICE.value, '2017-01-01', 'YAHOO')
        self.assertEqual(s, 'Not a trading day \'2017-01-01\'', 'test_historic_UK_ETF LAST_PRICE {}'.format(s))

@@ -279,14 +473,23 @@ class Test(unittest.TestCase):
        s = financials.getHistoric('SAP.DE', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
        self.assertEqual(s, 82.889999, 'test_historic_DE_equity CLOSE {}'.format(s))

-        s = financials.getHistoric('C060.DE', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(s, 68.209999, 'test_historic_DE_equity CLOSE {}'.format(s))
+        s = financials.getHistoric('LYY8.DE', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
+        self.assertEqual(s, 96.010002, 'test_historic_DE_equity CLOSE {}'.format(s))

    def test_realtime_errors(self):

        s = financials.getRealtime('NO_NAME', Datacode.LAST_PRICE.value, 'YAHOO')
        self.assertIsNone(s, 'test_realtime_errors LAST_PRICE {}'.format(s))

+        s = financials.getRealtime('NO_NAME', Datacode.PAYOUT_RATIO.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_errors PAYOUT_RATIO {}'.format(s))
+
+        s = financials.getRealtime('NO_NAME', Datacode.SECTOR.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_errors SECTOR {}'.format(s))
+
+        s = financials.getRealtime('LYY8.DE', -1, 'YAHOO')
+        self.assertEqual('Datacode -1 not supported', s, 'test_realtime_errors -1 {}'.format(s))
+
    def test_historic_errors(self):

        s = financials.getHistoric('NO_NAME', Datacode.LAST_PRICE.value, '2018-01-08', 'YAHOO')
@@ -299,7 +502,7 @@ class Test(unittest.TestCase):
        self.assertEqual(s, 'Datacode 9999 not supported', 'test_historic_errors 9999')

        s = financials.getRealtime('IBM', Datacode.ADJ_CLOSE.value, 'YAHOO')
-        self.assertEqual(s, 'Data doesn\'t exist - 91', 'test_historic_errors ADJ_CLOSE {}'.format(s))
+        self.assertIsNone(s, 'test_historic_errors ADJ_CLOSE {}'.format(s))

        s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2030-01-01', 'YAHOO')
        self.assertEqual(s, 'Future date \'2030-01-01\'', 'test_historic_errors CLOSE {}'.format(s))
@@ -1,15 +0,0 @@
-#  version.py
-#
-#  license: GNU LGPL
-#
-#  This library is free software; you can redistribute it and/or
-#  modify it under the terms of the GNU Lesser General Public
-#  License as published by the Free Software Foundation; either
-#  version 3 of the License, or (at your option) any later version.
-
-# This file will not be actually used as it is regenerated with the correct build version
-# by generate_metainfo.py in the "compile" step
-
-# This is a bit of a hack but I can't find out how to access the version in description.xml once deployed
-
-version = '0.0.0'
@@ -1,352 +0,0 @@
-#  yahoo.py
-#
-#  license: GNU LGPL
-#
-#  This library is free software; you can redistribute it and/or
-#  modify it under the terms of the GNU Lesser General Public
-#  License as published by the Free Software Foundation; either
-#  version 3 of the License, or (at your option) any later version.
-
-
-import csv
-import datetime
-import html
-import logging
-import os
-import pprint
-import re
-import time
-import urllib.parse
-from http import cookiejar
-
-import dateutil.parser
-import pytz
-
-import jsonParser
-from baseclient import BaseClient, HttpException
-from datacode import Datacode
-
-logger = logging.getLogger(__name__)
-
-
-# logger.setLevel(logging.DEBUG)
-
-
-def raw(m, key, default=0.0):
-    try:
-        return m[key]['raw']
-    except:
-        pass
-
-    return default
-
-
-def fmt(m, key, default=0.0):
-    try:
-        return m[key]['fmt']
-    except:
-        pass
-
-    return default
-
-
-class Yahoo(BaseClient):
-    def __init__(self, ctx):
-        super().__init__()
-
-        self.crumb = None
-        self.realtime = {}
-        self.historicdata = {}
-        self.js = jsonParser.jsonObject
-
-    def _read_ticker_csv_file(self, ticker):
-
-        fn = os.path.join(self.basedir, 'yahoo-{}.csv'.format(ticker))
-
-        if not os.path.isfile(fn):
-            return
-
-        with open(fn, newline='', encoding="utf-8") as csvfile:
-            reader = csv.DictReader(csvfile)
-
-            ticks = {}
-
-            for row in reader:
-                tick = {}
-                try:
-                    tick[Datacode.OPEN] = float(row['Open'])
-                    tick[Datacode.LOW] = float(row['Low'])
-                    tick[Datacode.HIGH] = float(row['High'])
-                    tick[Datacode.VOLUME] = float(row['Volume'])
-                    tick[Datacode.CLOSE] = float(row['Close'])
-                    tick[Datacode.ADJ_CLOSE] = float(row['Adj Close'])
-                except:
-                    pass
-
-                if len(tick) > 0:
-                    ticks[row['Date']] = tick
-
-            self.historicdata[ticker] = ticks
-
-    def getRealtime(self, ticker, datacode):
-
-        """
-        Retrieve realtime data for ticker from Yahoo Finance and cache it for further lookups
-
-        :param ticker: the ticker symbol e.g. VOD.L
-        :param datacode: the requested datacode
-        :return:
-        """
-
-        # remove white space
-        ticker = "".join(ticker.split())
-
-        # use cached value for up to 60 seconds
-        if ticker in self.realtime:
-            tick = self.realtime[ticker]
-            if time.time() - 60 < tick[Datacode.TIMESTAMP]:
-                return self._return_value(tick, datacode)
-            else:
-                del self.realtime[ticker]
-
-        url = 'https://finance.yahoo.com/quote/{}?p={}'.format(ticker, ticker)
-
-        cookies = [cookiejar.Cookie(version=0,
-                                    name="B",
-                                    value="7pbfivtfkl00m&b=3&s=if",
-                                    port=None, port_specified=False,
-                                    domain=".yahoo.com", domain_specified=True, domain_initial_dot=True,
-                                    path="/", path_specified=True,
-                                    secure=True,
-                                    expires=None,
-                                    discard=False,
-                                    comment=None,
-                                    comment_url=None,
-                                    rest=dict())
-                   ]
-
-        try:
-            text = self.urlopen(url, redirect=True, data=None, headers=None, cookies=cookies)
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getRealtime({}, {}) - urlopen: {}'.format(ticker, datacode, e)
-
-        try:
-            with open(os.path.join(self.basedir, 'yahoo-{}.html'.format(ticker)), "w", encoding="utf-8") as text_file:
-                print(f"<!-- '{url}' -->\r\n\r\n{text}", file=text_file)
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-
-        try:
-            text = urllib.parse.unquote(text)
-            text = text.replace('\\u002F', '/')
-
-            r = '"CrumbStore":{"crumb":"([^"]{11})"'
-            pattern = re.compile(r)
-            match = pattern.search(text)
-
-            if match:
-                self.crumb = match.group(1)
-
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getRealtime({}, {}) - crumb: {}'.format(ticker, datacode, e)
-
-        try:
-            start = text.find('"QuoteSummaryStore":{')
-
-            if start < 0:
-                return None
-
-            start = start + len('"QuoteSummaryStore":')
-            results = self.js.parseString(text[start:])
-
-            if not results:
-                return None
-
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getRealtime({}, {}) - parsing: {}'.format(ticker, datacode, e)
-
-        with open(os.path.join(self.basedir, 'yahoo-{}.js'.format(ticker)), "w", encoding="utf-8") as text_file:
-            print(f"// '{url}' QuoteSummaryStore:\n", file=text_file)
-            pprint.pprint(results.asList(), stream=text_file)
-
-        try:
-            price = results['price']
-            quoteType = results['quoteType']
-            summaryDetail = results['summaryDetail']
-
-            if not price:
-                return 'Could not find price for \'{}\''.format(ticker)
-
-            if ticker not in self.realtime:
-                self.realtime[ticker] = {}
-
-            tick = self.realtime[ticker]
-
-            tick[Datacode.TIMESTAMP] = time.time()
-
-            tick[Datacode.PREV_CLOSE] = float(raw(price, 'regularMarketPreviousClose'))
-            tick[Datacode.OPEN] = float(raw(price, 'regularMarketOpen'))
-            tick[Datacode.CHANGE] = float(raw(price, 'regularMarketChange'))
-            tick[Datacode.CHANGE_IN_PERCENT] = 100 * float(raw(price, 'regularMarketChangePercent'))
-            tick[Datacode.LOW] = float(raw(price, 'regularMarketDayLow'))
-            tick[Datacode.HIGH] = float(raw(price, 'regularMarketDayHigh'))
-            tick[Datacode.LAST_PRICE] = float(raw(price, 'regularMarketPrice'))
-            tick[Datacode.VOLUME] = float(raw(price, 'regularMarketVolume'))
-            tick[Datacode.AVG_DAILY_VOL_3MONTH] = float(raw(price, 'averageDailyVolume3Month'))
-            tick[Datacode.BETA] = float(raw(summaryDetail, 'beta'))
-            tick[Datacode.EPS] = self.save_wrapper(lambda: float(raw(results['defaultKeyStatistics'], 'trailingEps')))
-            tick[Datacode.PE_RATIO] = float(raw(summaryDetail, 'trailingPE'))
-            tick[Datacode.DIV] = float(raw(summaryDetail, 'dividendRate'))
-            tick[Datacode.DIV_YIELD] = float(raw(summaryDetail, 'dividendYield'))
-            tick[Datacode.EX_DIV_DATE] = self.save_wrapper(
-                lambda: dateutil.parser.parse(str(fmt(summaryDetail, 'exDividendDate'))).date())
-            tick[Datacode.PAYOUT_RATIO] = float(raw(summaryDetail, 'payoutRatio'))
-            tick[Datacode.LOW_52_WEEK] = float(raw(summaryDetail, 'fiftyTwoWeekLow'))
-            tick[Datacode.HIGH_52_WEEK] = float(raw(summaryDetail, 'fiftyTwoWeekHigh'))
-            tick[Datacode.MARKET_CAP] = float(raw(summaryDetail, 'marketCap'))
-
-            tick[Datacode.TIMEZONE] = None
-            tick[Datacode.LAST_PRICE_DATE] = None
-            tick[Datacode.LAST_PRICE_TIME] = None
-
-            if quoteType:
-                t = int(price['regularMarketTime'])
-                tz = pytz.timezone(quoteType['exchangeTimezoneName'])
-
-                tick[Datacode.TIMEZONE] = tz
-                dt = datetime.datetime.fromtimestamp(t, tz)
-
-                tick[Datacode.LAST_PRICE_DATE] = dt.date()
-                tick[Datacode.LAST_PRICE_TIME] = dt.time()
-
-            tick[Datacode.TICKER] = str(price['symbol'])
-            tick[Datacode.EXCHANGE] = str(price['exchange'])
-            tick[Datacode.CURRENCY] = str(price['currency'])
-
-            name = price['longName'] or price['shortName']
-            if name:
-                tick[Datacode.NAME] = html.unescape(str(name))
-            else:
-                tick[Datacode.NAME] = tick[Datacode.TICKER]
-
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getRealtime({}, {}) - process: {}'.format(ticker, datacode, e)
-
-        tick[Datacode.SECTOR] = self.save_wrapper(lambda: str(results['summaryProfile']['sector']))
-        tick[Datacode.INDUSTRY] = self.save_wrapper(lambda: str(results['summaryProfile']['industry']))
-
-        return self._return_value(self.realtime[ticker], datacode)
-
-    def getHistoric(self, ticker: str, datacode: int, date):
-
-        """
-        Retrieve historic data for ticker from Yahoo Finance and cache it for further lookups
-
-        :param ticker: the ticker symbol e.g. VOD.L
-        :param datacode: the requested datacode
-        :param date: the requested date
-        :return:
-        """
-
-        # remove white space
-        ticker = "".join(ticker.split())
-        min_tick_date = None
-
-        # dividend and splits will change past adjusted prices
-        # the moment we are asked for ADJ_CLOSE we ignore the ticker cache to refresh
-
-        if Datacode.ADJ_CLOSE != datacode and ticker not in self.historicdata:
-            self._read_ticker_csv_file(ticker)
-
-        if ticker in self.historicdata:
-            ticks = self.historicdata[ticker]
-
-            if date in ticks:
-                return self._return_value(ticks[date], datacode)
-
-            # weekend, trading holiday or as yet un-fetched
-            if min(ticks) <= date <= max(ticks):
-                return 'Not a trading day \'{}\''.format(date)
-
-            # (potentially) future date
-            if date > max(ticks):
-                t1 = int(dateutil.parser.parse(date).strftime('%s'))
-                t2 = int(time.time())
-                if t1 > t2:
-                    return 'Future date \'{}\''.format(date)
-
-                min_tick_date = int(dateutil.parser.parse(min(ticks)).strftime('%s'))  # remember current earliest date
-
-        if not self.crumb:
-            self.getRealtime(ticker, datacode)
-
-        if not self.crumb:
-            return 'Yahoo.getHistoric({}, {}, {}) - crumb'.format(ticker, datacode, date)
-
-        try:
-            t1 = int(dateutil.parser.parse(date).strftime('%s'))
-            t2 = int(time.time())
-
-            if min_tick_date:
-                t1 = min_tick_date
-
-            if t1 >= t2:
-                return 'Future date \'{}\''.format(date)
-
-            if t1 < int(dateutil.parser.parse('2000-01-01').strftime('%s')):
-                return 'Date before 2000 \'{}\''.format(date)
-
-            t1 = t1 - 2682000  # pad with extra month
-
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getHistoric({}, {}, {}) - date: {}'.format(ticker, datacode, date, e)
-
-        try:
-
-            url = 'https://query1.finance.yahoo.com/v7/finance/download/{}' \
-                  '?period1={}&period2={}&interval=1d&events=history&crumb={}' \
-                .format(ticker, t1, t2, urllib.parse.quote_plus(self.crumb))
-
-            text = self.urlopen(url)
-
-            with open(os.path.join(self.basedir, 'yahoo-{}.csv'.format(ticker)), "w", encoding="utf-8") as csv_file:
-                print(text, file=csv_file)
-
-            self._read_ticker_csv_file(ticker)
-
-        except HttpException:
-            logger.exception("HttpException ticker=%s datacode=%s date=%s", ticker, datacode, date)
-            return None
-
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s date=%s", ticker, datacode, date)
-            return 'Yahoo.getHistoric({}, {}, {}) - urlopen: {}'.format(ticker, datacode, date, e)
-
-        try:
-            if ticker in self.historicdata:
-                ticks = self.historicdata[ticker]
-
-                if date in ticks:
-                    return self._return_value(ticks[date], datacode)
-
-                # future date
-                if date > max(ticks):
-                    return 'Future date \'{}\''.format(date)
-
-                # weekend or trading holiday
-                return 'Not a trading day \'{}\''.format(date)
-
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getHistoric({}, {}, {}) - process: {}'.format(ticker, datacode, date, e)
-
-        return None
-
-
-def createInstance(ctx):
-    return Yahoo(ctx)
Author	SHA1	Message	Date
cmallwitz	28ae9efa74	Fix for FT redirect to unencoded URLs	2024-09-02 20:37:37 +01:00
cmallwitz	090e604d1d	Improve Yahoo crumb handling	2024-09-01 17:35:26 +01:00
cmallwitz	a4432a85e6	Fix Yahoo crumb handling	2024-05-17 15:56:43 +01:00
cmallwitz	a593336b08	Handle Yahoo site redesign	2024-05-01 18:35:41 +01:00
cmallwitz	06d9b91793	Update version and some cleanups	2024-03-13 12:11:42 +00:00
cmallwitzandGitHub	60d5d7b764	Merge pull request #99 from beckjake/handle-data-value Handle the case where there is a data-value but no value	2024-03-12 17:18:17 +00:00
Jacob Beck	345f57420c	Handle the case where there is a data-value but no value	2024-03-08 21:51:21 -07:00
cmallwitz	88b039d471	Yahoo: Don't re-fetch non-existing statistic/profile data	2023-12-29 22:22:28 +00:00
cmallwitz	7e751928de	Rework Yahoo cookies (always fetch from scratch)	2023-12-29 18:52:19 +00:00
cmallwitz	0c07c4c3ba	Update build for LibeOffice 7.4+	2023-12-24 19:49:56 +00:00
cmallwitz	3f6249a6b6	Update dependencies for Fedora 39	2023-11-19 21:37:46 +00:00
cmallwitz	d05ca9bb55	Reworked Yahoo cookie	2023-07-04 22:10:09 +01:00
cmallwitz	023fa71a81	Reworked Yahoo cookie and general network error handling	2023-01-20 16:04:44 +00:00
cmallwitz	a89edcc5fc	Rework Yahoo after website change (done)	2022-12-20 15:17:54 +00:00
cmallwitz	c71825143e	Rework Yahoo after website change (phase 1)	2022-12-18 18:58:59 +00:00
cmallwitz	587b391393	Update to README.md	2022-12-18 17:32:58 +00:00
cmallwitz	605e9ef185	Updated examples.ods for COINBASE	2022-12-13 13:52:29 +00:00
cmallwitz	e0491318c3	Fixing Coinbase merge	2022-12-13 13:27:52 +00:00
cmallwitz	24b1406772	Merge branch 'TheSerapher-coinbase-and-macos'	2022-12-13 13:25:08 +00:00
cmallwitz	1d9c4e824f	Merge branch 'coinbase-and-macos' of github.com:TheSerapher/Financials-Extension into TheSerapher-coinbase-and-macos	2022-12-13 13:24:24 +00:00
Sebastian Grewe	c3a6d8d69a	[WIP] Examples.ods	2022-12-13 10:29:00 +01:00
Sebastian Grewe	d0fc0ae2bf	Revert "[VERSION] 3.1.0" This reverts commit `e4d395240d`.	2022-12-09 15:44:48 +01:00
Sebastian Grewe	e4d395240d	[VERSION] 3.1.0	2022-12-09 15:44:15 +01:00
Sebastian Grewe	c3dc5c35fc	[ADDED] Super simple test	2022-12-09 15:43:47 +01:00
cmallwitzandSebastian Grewe	b356a1bca4	Updated Yahoo EU consent cookies	2022-12-09 15:43:47 +01:00
Sebastian Grewe	befd0956f0	[FIX] Compile for MacOS	2022-12-09 15:14:14 +01:00
cmallwitz	9e842b30de	Updated Yahoo EU consent cookies	2022-12-09 13:51:19 +00:00
Sebastian Grewe	6d0f56c87d	[ADDED] Coinbase Exchange for Crypto symbols	2022-12-09 14:43:35 +01:00
Sebastian Grewe	cd41959559	[UPDATE] added support for MacOS to compile.sh	2022-12-09 14:18:48 +01:00
cmallwitz	52e7119081	Updated Yahoo EU consent cookies	2022-11-04 13:25:40 +00:00
cmallwitz	520d2441cb	Refresh Yahoo cookies	2022-06-02 21:19:49 +01:00
cmallwitz	8b370dfc73	Updated version	2022-01-07 21:42:07 +00:00
cmallwitz	af82f16740	Updated Yahoo EU consent	2022-01-07 21:33:55 +00:00
cmallwitz	4e626c272e	Add shares outstanding and free float information	2021-09-28 20:28:00 +01:00
cmallwitz	e22dc7448f	Fix SSL issue on MacOS	2021-09-11 23:46:42 +01:00
cmallwitz	ab34a246e3	Update Readme	2021-08-19 19:50:41 +01:00
cmallwitz	aa20845ae4	Fix yield for US mutuals and EFTs	2021-06-08 17:29:47 +01:00
cmallwitz	7fccc7c0ad	Fix issue with FT avg vol	2021-05-19 21:44:06 +01:00
cmallwitz	7ddfbd084f	Fix for FT symbols containing space	2021-05-15 18:31:10 +01:00
cmallwitz	e22d4f46b8	Package all Python dependencies and disable Google as source	2021-04-03 11:23:45 +01:00
cmallwitz	6a3ab22801	Added BID/ASK/BIDSIZE/ASKSIZE/EXPIRY_DATE fields for FT/YAHOO and minor ccy fix	2021-02-22 21:09:03 +00:00
cmallwitz	46e6a68f6c	Fix currency issue with Yahoo / Moscow symbols	2021-02-01 20:33:13 +00:00
cmallwitz	ddaa4dc0aa	Merge branch 'master' of github.com:cmallwitz/Financials-Extension	2021-02-01 20:29:14 +00:00
cmallwitz	9c8a80b1e3	Fix currency issue with Yahoo / Moscow symbols	2021-02-01 20:28:11 +00:00
cmallwitz	e99a1109df	Fix currency issue with Yahho / Moscow symbols	2021-02-01 20:27:04 +00:00
cmallwitz	372c878bad	Fix typo	2021-01-03 18:37:06 +00:00
cmallwitz	c3308ee9ec	Historic data not working due to missing locale setting on Windows	2021-01-02 17:13:18 +00:00
cmallwitz	85bc40f6bd	Fixed typo	2020-12-29 15:04:52 +00:00
cmallwitz	601b91c8c3	Version 2.4.0 and fixed build	2020-12-28 15:55:04 +00:00
cmallwitz	7b806ce527	Renamed files to avoid conflicts with system libraries	2020-12-26 21:29:06 +00:00