Rework Yahoo after website change (done)

Rework Yahoo after website change (phase 1)
Update to README.md
2026-08-25 10:04:10 -05:00 · 2022-12-20 15:17:54 +00:00 · 2022-12-18 18:58:59 +00:00 · 2022-12-18 17:32:58 +00:00 · 2022-12-13 13:52:29 +00:00 · 2022-12-13 13:27:52 +00:00
14 changed files with 788 additions and 254 deletions
@@ -1,4 +1,5 @@
 .idea
 src/__pycache__
 build/*
-Financials-Extension.oxt
+Financials-Extension.oxt
+_trial_temp
@@ -1,14 +1,15 @@
 # Financials-Extension

+## Update (20 Dec 2022) Yahoo deployed a major website change 
+
+The latest version 3.2.1 restores almost all extension functionality. The only fields missing is LAST_PRICE_DATE and LAST_PRICE_TIME - somehow this is handled now using JavaScript making it a pain to extract.  
+
+## Overview
+
 This is a Python based extension for LibreOffice Calc to make stock market, index and FX data available in Calc 
 spreadsheets - currently supporting Yahoo's and Financial Times' finance websites using old-fashioned web scraping. 
-Starting with version 3 of the extension, Google was dropped as source as its finance website was changed dramatically
-requiring a complete re-work of the logic to pull data of it. No one seems to have complained, so I guess not a lot of 
-people rely on it. If there is popular demand, I can take a look at adding it back. Alternatively I'm open for 
-suggestions to add another provider with global exchange and asset class coverage.

-Additionally, starting with version 3 of the extension, it contains all non-standard Python packages it uses 
-(dateutil, pytz, pyparsing). Consequently, it is no longer necessary to install anything to your system.
+Starting with version 3.1.0, we received a contribution to get crypto data directly from Coinbase

 ### Feedback requested:

@@ -21,11 +22,12 @@ under menu item: Tools, Extension Manager...

 Please make sure, not to rename the OXT file when downloading and before installing: LO will mess up the installation otherwise and the extension won't work.

-Getting data should be a simple as having this in a cell: 
+Getting data should be as simple as having this in a cell: 
 - `=GETREALTIME("IBM",21,"YAHOO")`
 - `=GETREALTIME("IBM:NYQ",21,"FT")`
 - `=GETREALTIME("EURUSD","LAST_PRICE","FT")`
 - `=GETHISTORIC("IBM",90,"2020-12-01","YAHOO")`
+- `=GETREALTIME("ETH-USD","LAST_PRICE","COINBASE")`

 Codes 21 and 90 stand for "last price" and "close" (see below), respectively. 
 Only Yahoo has historic data available.
@@ -43,10 +45,11 @@ List of example URLs for checking symbols. You can start from these and search f
 is available on the website but not from the extension, a good place to start when raising an issue is to include a similar 
 URL to compare results.

-|Website|Symbol|Example URL for Vodafone Group Plc UK |
-| :--- | :--- | :--- |
-|YAHOO|VOD.L|https://finance.yahoo.com/quote/VOD.L|
-|FT|VOD:LSE|https://markets.ft.com/data/equities/tearsheet/summary?s=VOD:LSE|
+| Website  | Symbol  | Example URL for Vodafone Group Plc UK                            |
+|:---------|:--------|:-----------------------------------------------------------------|
+| YAHOO    | VOD.L   | https://finance.yahoo.com/quote/VOD.L                            |
+| FT       | VOD:LSE | https://markets.ft.com/data/equities/tearsheet/summary?s=VOD:LSE |
+| COINBASE | ETH-EUR | https://api.exchange.coinbase.com/products/ETH-EUR/stats         |

 ### LibreOffice: using , (comma) vs ; (semicolon) to separate arguments in formula

@@ -58,45 +61,48 @@ Depending on your system's language and default LibreOffice settings, you maybe

 ### List of supported data points

-You can either specify numbers or names (lower or upper case) - not all bits are available from all sources and they are not necessarily consistent across sources either.
+You can either specify numbers or names (lower or upper case) - not all bits are available from all sources, and they are not necessarily consistent across sources either.

-|Name|Code|YAHOO|FT|YAHOO (historic)|Notes|
-| :--- | :--- | :---: | :---: | :---: | :---: |
-|PREV_CLOSE|5|Yes|Yes|||
-|OPEN|6|Yes|Yes|Yes||
-|CHANGE|7|Yes|Yes|||
-|LAST_PRICE_DATE|8|Yes|Yes| |yyyy-mm-dd|
-|LAST_PRICE_TIME|10|Yes|Yes|||
-|CHANGE_IN_PERCENT|11|Yes|Yes|||
-|LOW|14|Yes|Yes|Yes||
-|HIGH|16|Yes|Yes|Yes||
-|LAST_PRICE|21|Yes|Yes|||
-|BID|22|Yes|Yes|||
-|ASK|25|Yes|Yes|||
-|HIGH_52_WEEK|24|Yes|Yes|||
-|LOW_52_WEEK|26|Yes|Yes|||
-|MARKET_CAP|27|Yes|Yes|||
-|BIDSIZE|30|Yes|Yes|||
-|ASKSIZE|31|Yes|Yes|||
-|VOLUME|35|Yes|Yes|Yes||
-|AVG_DAILY_VOL_3MONTH|39|Yes|Yes|||
-|BETA|67|Yes|Yes|||
-|EPS|68|Yes|Yes|||
-|PE_RATIO|69|Yes|Yes|||
-|DIV|70|Yes|Yes|||
-|DIV_YIELD|71|Yes|Yes|||
-|EX_DIV_DATE|72|Yes|Yes| |yyyy-mm-dd|
-|PAYOUT_RATIO|73|Yes|No|||
-|EXPIRY_DATE|74|Yes (on options)|No| |yyyy-mm-dd|
-|CLOSE|90|No|No|Yes||
-|ADJ_CLOSE|91|No|No|Yes||
-|SECTOR|98|Yes|Yes|||
-|INDUSTRY|99|Yes|Yes|||
-|TICKER|101|Yes|Yes|||
-|EXCHANGE|102|Yes|No|||
-|CURRENCY|103|Yes|Yes|||
-|NAME|104|Yes|Yes|||
-|TIMEZONE|105|Yes|Yes|||
+| Name                 | Code |      YAHOO       | FT  | YAHOO (historic) | COINBASE |   Notes    |
+|:---------------------|:-----|:----------------:|:---:|:----------------:|----------|:----------:|
+| PREV_CLOSE           | 5    |       Yes        | Yes |||
+| OPEN                 | 6    |       Yes        | Yes |       Yes        | Yes      ||
+| CHANGE               | 7    |       Yes        | Yes |||
+| LAST_PRICE_DATE      | 8    |                  | Yes |                  |          | yyyy-mm-dd |
+| LAST_PRICE_TIME      | 10   |                  | Yes |||
+| CHANGE_IN_PERCENT    | 11   |       Yes        | Yes |||
+| LOW                  | 14   |       Yes        | Yes |       Yes        | Yes      ||
+| HIGH                 | 16   |       Yes        | Yes |       Yes        | Yes      ||
+| LAST_PRICE           | 21   |       Yes        | Yes |                  | Yes      |
+| BID                  | 22   |       Yes        | Yes |||
+| ASK                  | 25   |       Yes        | Yes |||
+| HIGH_52_WEEK         | 24   |       Yes        | Yes |||
+| LOW_52_WEEK          | 26   |       Yes        | Yes |||
+| MARKET_CAP           | 27   |       Yes        | Yes |||
+| BIDSIZE              | 30   |       Yes        | Yes |||
+| ASKSIZE              | 31   |       Yes        | Yes |||
+| VOLUME               | 35   |       Yes        | Yes |       Yes        | Yes      ||
+| AVG_DAILY_VOL_3MONTH | 39   |       Yes        | Yes |||
+| BETA                 | 67   |       Yes        | Yes |||
+| EPS                  | 68   |       Yes        | Yes |||
+| PE_RATIO             | 69   |       Yes        | Yes |||
+| DIV                  | 70   |       Yes        | Yes |||
+| DIV_YIELD            | 71   |       Yes        | Yes |||
+| EX_DIV_DATE          | 72   |       Yes        | Yes |                  |          | yyyy-mm-dd |
+| PAYOUT_RATIO         | 73   |       Yes        | No  |||
+| EXPIRY_DATE          | 74   | Yes (on options) | No  |                  |          | yyyy-mm-dd |
+| SHARES_OUT           | 75   |       Yes        | Yes |                  |          ||
+| FREE_FLOAT           | 76   |       Yes        | Yes |                  |          ||
+| SETTLEMENT_DATE      | 77   |       Yes        |     |                  |          ||
+| CLOSE                | 90   |        No        | No  |       Yes        |          ||
+| ADJ_CLOSE            | 91   |        No        | No  |       Yes        |          ||
+| SECTOR               | 98   |       Yes        | Yes |||
+| INDUSTRY             | 99   |       Yes        | Yes |||
+| TICKER               | 101  |       Yes        | Yes |                  | Yes      |
+| EXCHANGE             | 102  |       Yes        | No  |||
+| CURRENCY             | 103  |       Yes        | Yes |                  | Yes      |
+| NAME                 | 104  |       Yes        | Yes |||
+| TIMEZONE             | 105  |       Yes        | Yes |||

 ### Dealing with missing data:

@@ -131,8 +137,8 @@ python3 -m unittest discover src

 ### Tested with:
 - Windows 10 / LibreOffice Calc 7.1.2.2 / Python 3.8.8
- Ubuntu 20.04 / LibreOffice Calc 6.4.3.2 / Python 3.8.2
- MacOS 10.15.7 / LibeOffice Calc 7.2.0.4 / Python 3.8.10
+- Ubuntu 20.04.5 / LibreOffice Calc 6.4.7.2 / Python 3.8.10
+- MacOS 10.15.7 / LibreOffice Calc 7.2.0.4 / Python 3.8.10

 (Previous versions)
 - Debian 10.3  / LibreOffice Calc 6.1.5.2 / Python 3.7.3
@@ -5,8 +5,20 @@ set -o nounset
 set -o pipefail
 set -o noclobber

-export PATH=$PATH:/usr/lib/libreoffice/sdk/bin
-export PATH=$PATH:/usr/lib/libreoffice/program
+if [[ $OSTYPE == "darwin"* ]]; then
+  echo MacOS
+  # Assuming both are installed in the applications folder
+  # Required some steps to make it work for MacOS M1, mind the `find` call which could return more than one (shouldn't)
+  # install_name_tool -change @__VIA_LIBRARY_PATH__/libreglo.dylib $(find /Applications -name "libreglo.dylib") /Applications/LibreOffice7.4_SDK/bin/idlc
+  # install_name_tool -change @__VIA_LIBRARY_PATH__/libuno_sal.dylib.3 $(find /Applications -name "libuno_sal.dylib.3") /Applications/LibreOffice7.4_SDK/bin/idlc
+  # install_name_tool -change @__VIA_LIBRARY_PATH__/libuno_salhelpergcc3.dylib.3 $(find /Applications -name "libuno_salhelpergcc3.dylib.3") /Applications/LibreOffice7.4_SDK/bin/idlc
+  # codesign --force -s - $(find /Applications -name "idlc")
+  export PATH=$PATH:/Applications/LibreOffice7.4_SDK/bin
+  export PATH=$PATH:/Applications/LibreOffice.app/Contents/MacOS
+else
+  export PATH=$PATH:/usr/lib/libreoffice/sdk/bin
+  export PATH=$PATH:/usr/lib/libreoffice/program
+fi

 # Setup build directories

@@ -37,6 +49,7 @@ cp -f "${PWD}"/src/tz.py "${PWD}"/build/
 cp -f "${PWD}"/src/financials_ft.py "${PWD}"/build/
 cp -f "${PWD}"/src/financials_google.py "${PWD}"/build/
 cp -f "${PWD}"/src/financials_yahoo.py "${PWD}"/build/
+cp -f "${PWD}"/src/financials_coinbase.py "${PWD}"/build/

 # this copies python modules dateutil, pytz, pyparsing to extension so it doesn't have to be installed by user

@@ -48,18 +48,17 @@ class BaseClient:
        os.makedirs(self.basedir, exist_ok=True)

        user_agents = [
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0',
-            'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0',
-            'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0',
-            'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0',
-            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/96.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/97.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/97.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/97.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/97.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/97.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/97.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/97.0',
+            'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/97.0',

            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36',
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
@@ -195,6 +194,8 @@ class BaseClient:
        tick[Datacode.EXCHANGE] = None
        tick[Datacode.EXPIRY_DATE] = None
        tick[Datacode.EX_DIV_DATE] = None
+        tick[Datacode.FREE_FLOAT] = None
+        tick[Datacode.SETTLEMENT_DATE] = None
        tick[Datacode.HIGH] = None
        tick[Datacode.HIGH_52_WEEK] = None
        tick[Datacode.INDUSTRY] = None
@@ -210,10 +211,16 @@ class BaseClient:
        tick[Datacode.PE_RATIO] = None
        tick[Datacode.PREV_CLOSE] = None
        tick[Datacode.SECTOR] = None
+        tick[Datacode.SHARES_OUT] = None
        tick[Datacode.TICKER] = None
        tick[Datacode.TIMEZONE] = None
        tick[Datacode.VOLUME] = None

+        tick[Datacode.YAHOO_SUMMARY_RECEIVED] = False
+        tick[Datacode.YAHOO_STATISTIC_RECEIVED] = False
+        tick[Datacode.YAHOO_PROFILE_RECEIVED] = False
+        tick[Datacode.TIMESTAMP] = None
+
        return tick

    def _return_value(self, data: dict, datacode: int):
@@ -320,6 +327,18 @@ class BaseClient:
                else:
                    return data[Datacode.EXPIRY_DATE]

+            elif datacode == Datacode.FREE_FLOAT.value and Datacode.FREE_FLOAT in data:
+                return data[Datacode.FREE_FLOAT]
+
+            elif datacode == Datacode.SETTLEMENT_DATE.value and Datacode.SETTLEMENT_DATE in data:
+                if data[Datacode.SETTLEMENT_DATE]:
+                    return data[Datacode.SETTLEMENT_DATE].isoformat()
+                else:
+                    return data[Datacode.SETTLEMENT_DATE]
+
+            elif datacode == Datacode.SHARES_OUT.value and Datacode.SHARES_OUT in data:
+                return data[Datacode.SHARES_OUT]
+
            elif datacode == Datacode.CLOSE.value and Datacode.CLOSE in data:
                return data[Datacode.CLOSE]

@@ -44,6 +44,9 @@ class Datacode(Enum):
    EX_DIV_DATE = 72
    PAYOUT_RATIO = 73
    EXPIRY_DATE = 74
+    SHARES_OUT = 75
+    FREE_FLOAT = 76
+    SETTLEMENT_DATE = 77

    CLOSE = 90
    ADJ_CLOSE = 91
@@ -57,6 +60,9 @@ class Datacode(Enum):
    NAME = 104
    TIMEZONE = 105

+    YAHOO_SUMMARY_RECEIVED = 996
+    YAHOO_STATISTIC_RECEIVED = 997
+    YAHOO_PROFILE_RECEIVED = 998
    TIMESTAMP = 999

    @classmethod
@@ -56,6 +56,7 @@ import six
 from datacode import Datacode
 import financials_google as google
 import financials_yahoo as yahoo
+import financials_coinbase as coinbase
 import financials_ft as ft

 implementation_name = "com.financials.getinfo.python.FinancialsImpl"  # as defined in Financials.xcu
@@ -96,6 +97,7 @@ class FinancialsImpl(unohelper.Base, Financials):
        self.ctx = ctx
        self.google = google.createInstance(ctx)
        self.yahoo = yahoo.createInstance(ctx)
+        self.coinbase = coinbase.createInstance(ctx)
        self.ft = ft.createInstance(ctx)

    @profile
@@ -141,6 +143,8 @@ class FinancialsImpl(unohelper.Base, Financials):
                s = self.yahoo.getRealtime(ticker, datacode)
            elif source == 'FT':
                s = self.ft.getRealtime(ticker, datacode)
+            elif source == 'COINBASE':
+                s = self.coinbase.getRealtime(ticker, datacode)
            else:
                s = 'Source \'{}\' not supported'.format(source)

@@ -0,0 +1,109 @@
+#  financials_coinbase.py
+#
+#  license: GNU LGPL
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 3 of the License, or (at your option) any later version.
+
+
+import csv
+import datetime
+import logging
+import os
+import pprint
+import re
+import time
+import json
+
+import dateutil.parser
+import pytz
+
+import jsonParser
+from baseclient import BaseClient, HttpException
+from datacode import Datacode
+
+logger = logging.getLogger(__name__)
+
+
+# logger.setLevel(logging.DEBUG)
+
+class Coinbase(BaseClient):
+    def __init__(self, ctx):
+        super().__init__()
+
+        self.crumb = None
+        self.realtime = {}
+        self.js = jsonParser.jsonObject
+
+    def getRealtime(self, ticker, datacode):
+
+        """
+        Retrieve realtime data for ticker from Coinbase and cache it for further lookups
+
+        :param ticker: the ticker symbol e.g. ETH-EUR
+        :param datacode: the requested datacode, not all are supported
+        :return:
+        """
+
+        # remove white space
+        ticker = "".join(ticker.split())
+
+        # use cached value for up to 60 seconds
+        if ticker in self.realtime:
+            tick = self.realtime[ticker]
+            if time.time() - 60 < tick[Datacode.TIMESTAMP]:
+                return self._return_value(tick, datacode)
+            else:
+                del self.realtime[ticker]
+
+        url = 'https://api.exchange.coinbase.com/products/{}/stats'.format(ticker)
+
+        try:
+            text = self.urlopen(url, redirect=True, data=None, headers=None)
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Coinbase.getRealtime({}, {}) - urlopen: {}'.format(ticker, datacode, e)
+
+        try:
+            with open(os.path.join(self.basedir, 'coinbase-{}.json'.format(ticker)), "w", encoding="utf-8") as text_file:
+                print(f"<!-- '{url}' -->\r\n\r\n{text}", file=text_file)
+        except BaseException:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+
+        try:
+            results = json.loads(text)
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Coinbase.getRealtime({}, {}) - crumb: {}'.format(ticker, datacode, e)
+
+        try:
+            price = results['last']
+
+            if not price:
+                return 'Could not find price for \'{}\''.format(ticker)
+
+            if ticker not in self.realtime:
+                self.realtime[ticker] =  self.get_ticker()
+
+            tick = self.realtime[ticker]
+
+            tick[Datacode.TIMESTAMP] = time.time()
+            tick[Datacode.LAST_PRICE] = float(price)
+            tick[Datacode.OPEN] = float(results['open'])
+            tick[Datacode.HIGH] = float(results['high'])
+            tick[Datacode.LOW] = float(results['low'])
+            tick[Datacode.VOLUME] =  float(results['volume'])
+            tick[Datacode.TICKER] = ticker.split('-', 1)[0]
+            tick[Datacode.CURRENCY] = ticker.split('-', 1)[1]
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Coinbase.getRealtime({}, {}) - process: {}'.format(ticker, datacode, e)
+
+        return self._return_value(self.realtime[ticker], datacode)
+
+def createInstance(ctx):
+    return Coinbase(ctx)
@@ -241,6 +241,18 @@ class FT(BaseClient):
                tick[Datacode.AVG_DAILY_VOL_3MONTH] = self.save_wrapper(
                    lambda: handle_abbreviations(html.unescape(match.group(1))))

+            r = r'<th>\s*Shares outstanding\s*</th><td>\s*([0-9,\.btnmk]+)\s*</td>'
+            match = re.compile(r, flags=re.DOTALL).search(text, start)
+            if match:
+                tick[Datacode.SHARES_OUT] = self.save_wrapper(
+                    lambda: handle_abbreviations(html.unescape(match.group(1))))
+
+            r = r'<th>\s*Free float\s*</th><td>\s*([0-9,\.btnmk]+)\s*</td>'
+            match = re.compile(r, flags=re.DOTALL).search(text, start)
+            if match:
+                tick[Datacode.FREE_FLOAT] = self.save_wrapper(
+                    lambda: handle_abbreviations(html.unescape(match.group(1))))
+
            r = r'<th>\s*P/E.*?</th><td>\s*([0-9,\.\-]+)\s*<'
            match = re.compile(r, flags=re.DOTALL).search(text, start)
            if match:
@@ -9,22 +9,20 @@


 import csv
-import datetime
 import html
 import logging
 import os
-import pprint
 import re
 import time
 import urllib.parse
 from http import cookiejar

 import dateutil.parser
-import pytz

 import jsonParser
 from baseclient import BaseClient, HttpException
 from datacode import Datacode
+from naivehtmlparser import NaiveHTMLParser

 logger = logging.getLogger(__name__)

@@ -32,22 +30,38 @@ logger = logging.getLogger(__name__)
 # logger.setLevel(logging.DEBUG)


-def raw(m, key, default=0.0):
+def default(obj, prop, fallback=''):
    try:
-        return m[key]['raw']
+        if obj is None or property is None:
+            return fallback
+
+        v = None
+        if hasattr(obj, prop):
+            v = getattr(obj, prop)
+        elif prop in obj:
+            v = obj[prop]
+        return v if v is not None else fallback
    except:
        pass
-
-    return default
+    return fallback


-def fmt(m, key, default=0.0):
-    try:
-        return m[key]['fmt']
-    except:
-        pass
+def handle_abbreviations(s):
+    s = str(s).strip()
+    if s.endswith('M'):
+        return float(s[:-1]) * 1000000
+    elif s.endswith('B'):
+        return float(s[:-1]) * 1000000000
+    elif s.endswith('T'):
+        return float(s[:-1]) * 1000000000000
+    return float(s)

-    return default
+
+def cookie(name, value):
+    return cookiejar.Cookie(version=0, name=name, value=value,
+                            port=None, port_specified=False, domain=".yahoo.com", domain_specified=True,
+                            domain_initial_dot=True, path="/", path_specified=True, secure=True, expires=None,
+                            discard=False, comment=None, comment_url=None, rest=dict())


 class Yahoo(BaseClient):
@@ -72,7 +86,7 @@ class Yahoo(BaseClient):
            ticks = {}

            for row in reader:
-                tick  = self.get_ticker()
+                tick = self.get_ticker()
                try:
                    tick[Datacode.OPEN] = float(row['Open'])
                    tick[Datacode.LOW] = float(row['Low'])
@@ -88,6 +102,18 @@ class Yahoo(BaseClient):

            self.historicdata[ticker] = ticks

+    def get_cookies(self):
+        return [
+            cookie("A1", "d=AQABBDcIZWMCEHYhFYqQ7qyTvvD2eAT87mcFEgABCAGDlGPBY_bPb2UB9qMAAAcILwhlY6iIogg&S=AQAAAjZvTuAn1nH4h71eKJtCEHk"),
+            cookie("A1S", "d=AQABBDcIZWMCEHYhFYqQ7qyTvvD2eAT87mcFEgABCAGDlGPBY_bPb2UB9qMAAAcILwhlY6iIogg&S=AQAAAjZvTuAn1nH4h71eKJtCEHk&j=GDPR"),
+            cookie("A3", "d=AQABBDcIZWMCEHYhFYqQ7qyTvvD2eAT87mcFEgABCAGDlGPBY_bPb2UB9qMAAAcILwhlY6iIogg&S=AQAAAjZvTuAn1nH4h71eKJtCEHk"),
+            cookie("GUC", "AQABCAFjlINjwUIcFQQQ&s=AQAAAFOQKXn7&g=Y5M5Jg"),
+            cookie("GUCS", "ASHFadZS"),
+            cookie("maex", "{\"v2\":{}}"),
+            cookie("PRF", "t=TQQQ%2BASTO.L%2BCHMI%2BVFIAX%2BIBM%2BXMR-USD%2BMVV%2BSECU-B.ST%2BMSFT"),
+            cookie("thamba", "1")
+        ]
+
    def getRealtime(self, ticker, datacode):

        """
@@ -101,35 +127,46 @@ class Yahoo(BaseClient):
        # remove white space
        ticker = "".join(ticker.split())

+        needStatistics = datacode in [Datacode.SHARES_OUT.value, Datacode.FREE_FLOAT.value, Datacode.PAYOUT_RATIO.value]
+        needProfile = datacode in [Datacode.SECTOR.value, Datacode.INDUSTRY.value]
+
        # use cached value for up to 60 seconds
        if ticker in self.realtime:
            tick = self.realtime[ticker]
            if time.time() - 60 < tick[Datacode.TIMESTAMP]:
-                return self._return_value(tick, datacode)
+                if (tick[Datacode.YAHOO_STATISTIC_RECEIVED] or not needStatistics) and (
+                        tick[Datacode.YAHOO_PROFILE_RECEIVED] or not needProfile) and (
+                        tick[Datacode.YAHOO_SUMMARY_RECEIVED]):
+                    return self._return_value(tick, datacode)
            else:
                del self.realtime[ticker]

+        if ticker not in self.realtime:
+            self.realtime[ticker] = self.get_ticker()
+
+        if needStatistics:
+            return self.getRealtimeStatistics(ticker, datacode)
+
+        if needProfile:
+            return self.getRealtimeProfile(ticker, datacode)
+
+        return self.getRealtimeSummary(ticker, datacode)
+
+    def getRealtimeSummary(self, ticker, datacode):
+
+        """
+        Retrieve realtime data from Yahoo Finance - Summary tab
+        """
+
+        tick = self.realtime[ticker]
+
        url = 'https://finance.yahoo.com/quote/{}?p={}'.format(ticker, ticker)

-        cookies = [cookiejar.Cookie(version=0,
-                                    name="B",
-                                    value="er7g22lg35od5&b=3&s=8p",
-                                    port=None, port_specified=False,
-                                    domain=".yahoo.com", domain_specified=True, domain_initial_dot=True,
-                                    path="/", path_specified=True,
-                                    secure=True,
-                                    expires=None,
-                                    discard=False,
-                                    comment=None,
-                                    comment_url=None,
-                                    rest=dict())
-                   ]
-
        try:
-            text = self.urlopen(url, redirect=True, data=None, headers=None, cookies=cookies)
+            text = self.urlopen(url, redirect=True, data=None, headers=None, cookies=self.get_cookies())
        except BaseException as e:
            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getRealtime({}, {}) - urlopen: {}'.format(ticker, datacode, e)
+            return 'Yahoo.getRealtimeSummary({}, {}) - urlopen: {}'.format(ticker, datacode, e)

        try:
            with open(os.path.join(self.basedir, 'yahoo-{}.html'.format(ticker)), "w", encoding="utf-8") as text_file:
@@ -138,122 +175,261 @@ class Yahoo(BaseClient):
            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)

        try:
-            text = urllib.parse.unquote(text)
-            text = text.replace('\\u002F', '/')
-
-            r = '"CrumbStore":{"crumb":"([^"]{11})"'
+            r = '"crumb":"([^"]{11})"'
            pattern = re.compile(r)
            match = pattern.search(text)
-
-            if  match:
+            if match:
                self.crumb = match.group(1)
-
        except BaseException as e:
            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getRealtime({}, {}) - crumb: {}'.format(ticker, datacode, e)
+            return 'Yahoo.getRealtimeSummary({}, {}) - crumb: {}'.format(ticker, datacode, e)
+
+        tick[Datacode.TIMESTAMP] = time.time()
+        tick[Datacode.YAHOO_SUMMARY_RECEIVED] = True

        try:
-            start = text.find('"QuoteSummaryStore":{')
-
-            if start < 0:
-                return None
-
-            start = start + len('"QuoteSummaryStore":')
-            results = self.js.parseString(text[start:])
-
-            if not results:
-                return None
-
+            parser = NaiveHTMLParser()
+            root = parser.feed(text)
+            parser.close()
        except BaseException as e:
            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getRealtime({}, {}) - parsing: {}'.format(ticker, datacode, e)
-
-        with open(os.path.join(self.basedir, 'yahoo-{}.js'.format(ticker)), "w", encoding="utf-8") as text_file:
-            print(f"// '{url}' QuoteSummaryStore:\n", file=text_file)
-            pprint.pprint(results.asList(), stream=text_file)
+            return 'Yahoo.getRealtimeSummary({}, {}) - HTML parsing: {}'.format(ticker, datacode, e)

        try:
-            price = results['price']
-            quoteType = results['quoteType']
-            summaryDetail = results['summaryDetail']
+            parsed = {}

-            if not price:
-                return 'Could not find price for \'{}\''.format(ticker)
+            found = root.findall(f".//fin-streamer[@data-symbol='{ticker}']")
+            for d in found:
+                if hasattr(d, 'attrib') and 'data-field' in d.attrib:
+                    parsed[d.attrib['data-field']] = default(d.attrib, 'value').replace('−', '-').replace(',', '').strip()

-            if ticker not in self.realtime:
-                self.realtime[ticker] =  self.get_ticker()
+            found = root.findall(f".//td[@data-test]")
+            for d in found:
+                if d:
+                    span = d.find('./span')
+                    if hasattr(d, 'attrib') and hasattr(span, 'text'):
+                        parsed[d.attrib['data-test']] = default(span, 'text').replace('−', '-').replace(',', '').strip()
+                else:
+                    if hasattr(d, 'attrib') and hasattr(d, 'text'):
+                        parsed[d.attrib['data-test']] = default(d, 'text').replace('−', '-').replace(',', '').strip()

-            tick = self.realtime[ticker]
+            if 'regularMarketPrice' not in parsed:
+                return None

-            tick[Datacode.TIMESTAMP] = time.time()
+            tick[Datacode.PREV_CLOSE] = self.save_wrapper(lambda: float(parsed['PREV_CLOSE-value']))
+            tick[Datacode.OPEN] = self.save_wrapper(lambda: float(parsed['OPEN-value']))
+            tick[Datacode.CHANGE] = self.save_wrapper(lambda: float(parsed['regularMarketChange']))
+            tick[Datacode.CHANGE_IN_PERCENT] = self.save_wrapper(lambda: float(parsed['regularMarketChangePercent']))
+
+            t = default(parsed, 'DAYS_RANGE-value').split(' - ')
+            tick[Datacode.LOW] = self.save_wrapper(lambda: float(t[0]))
+            tick[Datacode.HIGH] = self.save_wrapper(lambda: float(t[1]))
+
+            tick[Datacode.LAST_PRICE] = self.save_wrapper(lambda: float(parsed['regularMarketPrice']))
+            tick[Datacode.VOLUME] = self.save_wrapper(lambda: float(parsed['regularMarketVolume']))
+            tick[Datacode.AVG_DAILY_VOL_3MONTH] = self.save_wrapper(lambda: float(parsed['AVERAGE_VOLUME_3MONTH-value']))
+            tick[Datacode.BETA] = self.save_wrapper(lambda: float(parsed['BETA_5Y-value']))
+            tick[Datacode.EPS] = self.save_wrapper(lambda: float(parsed['EPS_RATIO-value']))
+            tick[Datacode.PE_RATIO] = self.save_wrapper(lambda: float(parsed['PE_RATIO-value']))
+
+            t = default(parsed, 'DIVIDEND_AND_YIELD-value').replace('(', '').replace(')', '').replace('%', '').strip().split(' ')
+            tick[Datacode.DIV] = self.save_wrapper(lambda: float(t[0]))
+            tick[Datacode.DIV_YIELD] = self.save_wrapper(lambda: float(t[1])/100.0)

-            tick[Datacode.PREV_CLOSE] = float(raw(price, 'regularMarketPreviousClose'))
-            tick[Datacode.OPEN] = float(raw(price, 'regularMarketOpen'))
-            tick[Datacode.CHANGE] = float(raw(price, 'regularMarketChange'))
-            tick[Datacode.CHANGE_IN_PERCENT] = 100 * float(raw(price, 'regularMarketChangePercent'))
-            tick[Datacode.LOW] = float(raw(price, 'regularMarketDayLow'))
-            tick[Datacode.HIGH] = float(raw(price, 'regularMarketDayHigh'))
-            tick[Datacode.LAST_PRICE] = float(raw(price, 'regularMarketPrice'))
-            tick[Datacode.VOLUME] = float(raw(price, 'regularMarketVolume'))
-            tick[Datacode.AVG_DAILY_VOL_3MONTH] = float(raw(price, 'averageDailyVolume3Month'))
-            tick[Datacode.BETA] = float(raw(summaryDetail, 'beta'))
-            tick[Datacode.EPS] = self.save_wrapper(lambda: float(raw(results['defaultKeyStatistics'], 'trailingEps')))
-            tick[Datacode.PE_RATIO] = float(raw(summaryDetail, 'trailingPE'))
-            tick[Datacode.DIV] = float(raw(summaryDetail, 'dividendRate'))
-            tick[Datacode.DIV_YIELD] = float(raw(summaryDetail, 'dividendYield'))
            tick[Datacode.EX_DIV_DATE] = self.save_wrapper(
-                lambda: dateutil.parser.parse(str(fmt(summaryDetail, 'exDividendDate')), yearfirst=True, dayfirst=False).date())
+                lambda: dateutil.parser.parse(parsed['EX_DIVIDEND_DATE-value'], yearfirst=True, dayfirst=False).date())

-            tick[Datacode.PAYOUT_RATIO] = float(raw(summaryDetail, 'payoutRatio'))
-            tick[Datacode.LOW_52_WEEK] = float(raw(summaryDetail, 'fiftyTwoWeekLow'))
-            tick[Datacode.HIGH_52_WEEK] = float(raw(summaryDetail, 'fiftyTwoWeekHigh'))
-            tick[Datacode.MARKET_CAP] = float(raw(summaryDetail, 'marketCap'))
+            t = default(parsed, 'FIFTY_TWO_WK_RANGE-value').split(' - ')
+            tick[Datacode.LOW_52_WEEK] = self.save_wrapper(lambda: float(t[0]))
+            tick[Datacode.HIGH_52_WEEK] = self.save_wrapper(lambda: float(t[1]))

-            tick[Datacode.BID] = float(raw(summaryDetail, 'bid'))
-            tick[Datacode.ASK] = float(raw(summaryDetail, 'ask'))
-            tick[Datacode.BIDSIZE] = float(raw(summaryDetail, 'bidSize'))
-            tick[Datacode.ASKSIZE] = float(raw(summaryDetail, 'askSize'))
+            tick[Datacode.MARKET_CAP] = self.save_wrapper(lambda: float(handle_abbreviations(parsed['MARKET_CAP-value'])))
+
+            t = default(parsed, 'BID-value').split(' x ')
+            tick[Datacode.BID] = self.save_wrapper(lambda: float(t[0]))
+            tick[Datacode.BIDSIZE] = self.save_wrapper(lambda: float(t[1]))
+
+            t = default(parsed, 'ASK-value').split(' x ')
+            tick[Datacode.ASK] = self.save_wrapper(lambda: float(t[0]))
+            tick[Datacode.ASKSIZE] = self.save_wrapper(lambda: float(t[1]))

            tick[Datacode.EXPIRY_DATE] = self.save_wrapper(
-                lambda: dateutil.parser.parse(str(fmt(summaryDetail, 'expireDate')), yearfirst=True, dayfirst=False).date())
+                lambda: dateutil.parser.parse(parsed['EXPIRE_DATE-value'], yearfirst=True, dayfirst=False).date())

-            if quoteType:
-                t = int(price['regularMarketTime'])
-                tz = pytz.timezone(quoteType['exchangeTimezoneName'])
+            tick[Datacode.SETTLEMENT_DATE] = self.save_wrapper(
+                lambda: dateutil.parser.parse(parsed['SETTLEMENT_DATE-value'], yearfirst=True, dayfirst=False).date())

-                tick[Datacode.TIMEZONE] = tz
-                dt = datetime.datetime.fromtimestamp(t, tz)
+            r = '<div id="quote-market-notice"[^>]*><span>([^>]*?)(. Market open.)?</span></div>'
+            match = re.compile(r, flags=re.DOTALL).search(text)
+            if match:
+                t = html.unescape(match.group(1)).strip().split(' ')
+                tick[Datacode.TIMEZONE] = self.save_wrapper(lambda: t[-1])

-                tick[Datacode.LAST_PRICE_DATE] = dt.date()
-                tick[Datacode.LAST_PRICE_TIME] = dt.time()
+            # if quoteType:
+            #     t = int(price['regularMarketTime'])
+            #     tz = pytz.timezone(quoteType['exchangeTimezoneName'])
+            #
+            #     tick[Datacode.TIMEZONE] = tz
+            #     dt = datetime.datetime.fromtimestamp(t, tz)
+            #
+            #     tick[Datacode.LAST_PRICE_DATE] = dt.date()
+            #     tick[Datacode.LAST_PRICE_TIME] = dt.time()

-            tick[Datacode.TICKER] = self.save_wrapper(lambda: str(price['symbol']))
-            tick[Datacode.EXCHANGE] = self.save_wrapper(lambda: str(price['exchange']))
-            tick[Datacode.CURRENCY] = self.save_wrapper(lambda: str(price['currency']))
+            tick[Datacode.TICKER] = ticker

-            # some Moscow symbols miss currency in data block but show it in text e.g. VTBBA.ME, TBIOA.ME
-            if not tick[Datacode.CURRENCY]:
-                r = r'Currency in ([A-Z]{3})\b'
-                match = re.compile(r, flags=re.DOTALL).search(text)
-                if match:
-                    tick[Datacode.CURRENCY] = match.group(1)
+            r = '<span>([ \\w]+?) - [^>]*Currency in ([\\w]+)[^>]*</span>'
+            match = re.compile(r, flags=re.DOTALL).search(text)
+            if match:
+                tick[Datacode.EXCHANGE] = self.save_wrapper(lambda: html.unescape(match.group(1)).strip())
+                tick[Datacode.CURRENCY] = self.save_wrapper(lambda: html.unescape(match.group(2)).strip())

-            # fallback for yield on US mutual funds and ETFs, which is in different field
+            # fallback for dividend/yield on mutual funds and ETFs
+            if not tick[Datacode.DIV]:
+                tick[Datacode.DIV] = self.save_wrapper(lambda: float(parsed['LAST_DIVIDEND-value']))
            if not tick[Datacode.DIV_YIELD]:
-                tick[Datacode.DIV_YIELD] = float(raw(summaryDetail, 'yield'))
+                tick[Datacode.DIV_YIELD] = self.save_wrapper(lambda: float(parsed['TD_YIELD-value'].replace('%', '').strip())/100.0)

-            name = price['longName'] or price['shortName']
-            if name:
-                tick[Datacode.NAME] = html.unescape(str(name))
-            else:
+            tick[Datacode.NAME] = self.save_wrapper(
+                lambda: html.unescape(root.find('.//h1').text).strip())
+
+            if not tick[Datacode.NAME]:
                tick[Datacode.NAME] = tick[Datacode.TICKER]

-            tick[Datacode.SECTOR] = self.save_wrapper(lambda: str(results['summaryProfile']['sector']))
-            tick[Datacode.INDUSTRY] = self.save_wrapper(lambda: str(results['summaryProfile']['industry']))
-
        except BaseException as e:
            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
-            return 'Yahoo.getRealtime({}, {}) - process: {}'.format(ticker, datacode, e)
+            return 'Yahoo.getRealtimeSummary({}, {}) - process: {}'.format(ticker, datacode, e)
+
+        return self._return_value(self.realtime[ticker], datacode)
+
+    def getRealtimeStatistics(self, ticker, datacode):
+
+        """
+        Retrieve realtime data from Yahoo Finance - Statistics tab
+        """
+
+        tick = self.realtime[ticker]
+
+        url = 'https://finance.yahoo.com/quote/{}/key-statistics?p={}'.format(ticker, ticker)
+
+        try:
+            text = self.urlopen(url, redirect=True, data=None, headers=None, cookies=self.get_cookies())
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getRealtimeStatistics({}, {}) - urlopen: {}'.format(ticker, datacode, e)
+
+        try:
+            with open(os.path.join(self.basedir, 'yahoo-{}-statistics.html'.format(ticker)), "w", encoding="utf-8") as text_file:
+                print(f"<!-- '{url}' -->\r\n\r\n{text}", file=text_file)
+        except BaseException:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+
+        try:
+            parser = NaiveHTMLParser()
+            root = parser.feed(text)
+            parser.close()
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getRealtimeStatistics({}, {}) - HTML parsing: {}'.format(ticker, datacode, e)
+
+        tick[Datacode.TIMESTAMP] = time.time()
+        tick[Datacode.YAHOO_STATISTIC_RECEIVED] = True
+
+        try:
+
+            parsed = {}
+
+            statistics = root.find(".//section[@data-test='qsp-statistics']")
+
+            if statistics is None:
+                return None
+
+            # Valuation Measures
+            found = statistics.find('./div[2]/div[1]//table').findall('.//tr')
+            for d in found:
+                key = d.find('./td[1]/span').text
+                if key is not None:
+                    parsed[key] = d.find('./td[2]').text
+
+            # Stock Price History
+            found = statistics.find('./div[2]/div[2]/div[1]/div[1]//table').findall('.//tr')
+            for d in found:
+                key = d.find('./td[1]/span').text
+                if key is not None:
+                    parsed[key] = d.find('./td[2]').text
+
+            # Share Statistics
+            found = statistics.find('./div[2]/div[2]/div[1]/div[2]//table').findall('.//tr')
+            for d in found:
+                key = d.find('./td[1]/span').text
+                if key is not None:
+                    parsed[key] = d.find('./td[2]').text
+
+            # Dividends & Splits
+            found = statistics.find('./div[2]/div[2]/div[1]/div[3]//table').findall('.//tr')
+            for d in found:
+                key = d.find('./td[1]/span').text
+                if key is not None:
+                    parsed[key] = d.find('./td[2]').text
+
+            tick[Datacode.SHARES_OUT] = self.save_wrapper(
+                lambda: float(handle_abbreviations(parsed['Shares Outstanding'])))
+            tick[Datacode.FREE_FLOAT] = self.save_wrapper(
+                lambda: float(handle_abbreviations(parsed['Float'])))
+            tick[Datacode.PAYOUT_RATIO] = self.save_wrapper(
+                lambda: float(handle_abbreviations(parsed['Payout Ratio'].replace('%', '').strip()))/100.0)
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getRealtimeStatistics({}, {}) - process: {}'.format(ticker, datacode, e)
+
+        return self._return_value(self.realtime[ticker], datacode)
+
+    def getRealtimeProfile(self, ticker, datacode):
+
+        """
+        Retrieve realtime data from Yahoo Finance - Profile tab
+        """
+
+        tick = self.realtime[ticker]
+
+        url = 'https://finance.yahoo.com/quote/{}/profile?p={}'.format(ticker, ticker)
+
+        try:
+            text = self.urlopen(url, redirect=True, data=None, headers=None, cookies=self.get_cookies())
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getRealtimeProfile({}, {}) - urlopen: {}'.format(ticker, datacode, e)
+
+        try:
+            with open(os.path.join(self.basedir, 'yahoo-{}-profile.html'.format(ticker)), "w", encoding="utf-8") as text_file:
+                print(f"<!-- '{url}' -->\r\n\r\n{text}", file=text_file)
+        except BaseException:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+
+        try:
+            parser = NaiveHTMLParser()
+            root = parser.feed(text)
+            parser.close()
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getRealtimeProfile({}, {}) - HTML parsing: {}'.format(ticker, datacode, e)
+
+        tick[Datacode.TIMESTAMP] = time.time()
+        tick[Datacode.YAHOO_PROFILE_RECEIVED] = True
+
+        try:
+
+            p = root.find(".//*[span='Sector(s)']")
+
+            if p is None:
+                return None
+
+            tick[Datacode.SECTOR] = self.save_wrapper(lambda: p.find("./span[2]").text)
+            tick[Datacode.INDUSTRY] = self.save_wrapper(lambda: p.find("./span[4]").text)
+
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getRealtimeProfile({}, {}) - process: {}'.format(ticker, datacode, e)

        return self._return_value(self.realtime[ticker], datacode)

@@ -304,10 +480,10 @@ class Yahoo(BaseClient):
                min_tick_date = int(dateutil.parser.parse(min(ticks), yearfirst=True, dayfirst=False).timestamp())  # remember current earliest date

        if not self.crumb:
-            self.getRealtime(ticker, datacode)
+            self.getRealtime(ticker, Datacode.LAST_PRICE)

        if not self.crumb:
-            return 'Yahoo.getHistoric({}, {}, {}) - crumb'.format(ticker, datacode, date)
+            return 'Yahoo.getHistoric({}, {}, {}) - crumb missing'.format(ticker, datacode, date)

        try:
            t1 = int(date_as_dt.timestamp())
@@ -14,7 +14,7 @@ import os
 cur_dir = os.getcwd()

 addin_id = "com.financials.getinfo"
-addin_version = "3.0.4"
+addin_version = "3.2.1"
 addin_displayname = "Financial Market Extension"
 addin_publisher_link = "https://github.com/cmallwitz/Financials-Extension"
 addin_publisher_name = "The Publisher"
@@ -0,0 +1,37 @@
+#  test_yahoo.py
+#
+#  license: GNU LGPL
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 3 of the License, or (at your option) any later version.
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+import unittest
+
+logging.basicConfig(level=logging.ERROR, format="%(asctime)s %(name)s %(levelname)s %(message)s")
+
+import financials
+from datacode import Datacode
+import testutils
+
+financials = financials.createInstance(None)
+
+
+class Test(unittest.TestCase):
+
+    def test_currency(self):
+        s = financials.getRealtime('ETH-EUR', Datacode.LAST_PRICE.value, 'COINBASE')
+        self.assertEqual(float, type(s), 'test_currency LAST_PRICE')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('unittest_args', nargs='*')
+    args = parser.parse_args()
+    unit_argv = [sys.argv[0]] + args.unittest_args
+    unittest.main(argv=unit_argv)
@@ -15,6 +15,7 @@ import unittest
 logging.basicConfig(level=logging.ERROR, format="%(asctime)s %(name)s %(levelname)s %(message)s")

 import financials
+from datacode import Datacode
 import testutils

 financials = financials.createInstance(None)
@@ -70,6 +71,12 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('IBM:NYQ', 'EPS', 'FT')
        self.assertEqual(float, type(s), 'test_US_equity EPS {}'.format(s))

+        s = financials.getRealtime('IBM:NYQ', 'SHARES_OUT', 'FT')
+        self.assertEqual(float, type(s), 'test_US_equity SHARES_OUT {}'.format(s))
+
+        s = financials.getRealtime('IBM:NYQ', 'FREE_FLOAT', 'FT')
+        self.assertEqual(float, type(s), 'test_US_equity FREE_FLOAT {}'.format(s))
+
        s = financials.getRealtime('IBM:NYQ', 'PE_RATIO', 'FT')
        self.assertEqual(float, type(s), 'test_US_equity PE_RATIO {}'.format(s))

@@ -96,7 +103,7 @@ class Test(unittest.TestCase):

        s = financials.getRealtime('IBM:NYQ', 'INDUSTRY', 'FT')
        self.assertEqual(str, type(s), 'test_US_equity INDUSTRY {}'.format(s))
-        self.assertEqual(s, 'Software & Computer Services', 'test_US_equity INDUSTRY {}'.format(s))
+        self.assertEqual(s, 'Technology', 'test_US_equity INDUSTRY {}'.format(s))

        # may fail (s is None) on weekends when date/time displayed doesn't have time component with TZ
        s = financials.getRealtime('IBM:NYQ', 'TIMEZONE', 'FT')
@@ -128,6 +135,33 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('VFIAX', 'LAST_PRICE_TIME', 'FT')
        self.assertEqual(str, type(s), 'test_US_mutuals LAST_PRICE_TIME {}'.format(s))

+    def test_US_futures(self):
+
+        s = financials.getRealtime('ESH3:IOM', Datacode.NAME.value, 'FT')
+        self.assertEqual(str, type(s), 'test_realtime_US_futures NAME {}'.format(s))
+        self.assertEqual('EMINI S&P MAR3', s, 'test_US_futures NAME {}'.format(s))
+
+        s = financials.getRealtime('ESH3:IOM', Datacode.LAST_PRICE.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures LAST_PRICE {}'.format(s))
+
+        # s = financials.getRealtime('ESH3:IOM', Datacode.OPEN.value, 'FT')
+        # self.assertEqual(float, type(s), 'test_US_futures OPEN {}'.format(s))
+
+        s = financials.getRealtime('ESH3:IOM', Datacode.VOLUME.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures VOLUME {}'.format(s))
+
+        s = financials.getRealtime('ESH3:IOM', Datacode.LOW_52_WEEK.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures LOW_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('ESH3:IOM', Datacode.HIGH_52_WEEK.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures HIGH_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('ESH3:IOM', Datacode.CHANGE.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures CHANGE {}'.format(s))
+
+        s = financials.getRealtime('ESH3:IOM', Datacode.CHANGE_IN_PERCENT.value, 'FT')
+        self.assertEqual(float, type(s), 'test_US_futures CHANGE_IN_PERCENT {}'.format(s))
+
    def test_UK_ETF(self):
        s = financials.getRealtime('CSP1:LSE:GBX', 'NAME', 'FT')
        self.assertEqual(str, type(s), 'test_UK_ETF NAME {}'.format(s))
@@ -202,7 +236,7 @@ class Test(unittest.TestCase):

        s = financials.getRealtime('SAPX:GER', 'INDUSTRY', 'FT')
        self.assertEqual(str, type(s), 'test_DE_equity INDUSTRY {}'.format(s))
-        self.assertEqual('Software & Computer Services', s, 'test_DE_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Technology', s, 'test_DE_equity INDUSTRY {}'.format(s))

        s = financials.getRealtime('SAPX:GER', 'LAST_PRICE_DATE', 'FT')
        self.assertEqual(str, type(s), 'test_DE_equity LAST_PRICE_DATE {}'.format(s))
@@ -217,6 +251,12 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('SAPX:GER', 'EPS', 'FT')
        self.assertEqual(float, type(s), 'test_DE_equity EPS {}'.format(s))

+        s = financials.getRealtime('SAPX:GER', 'SHARES_OUT', 'FT')
+        self.assertEqual(float, type(s), 'test_DE_equity SHARES_OUT {}'.format(s))
+
+        s = financials.getRealtime('SAPX:GER', 'FREE_FLOAT', 'FT')
+        self.assertEqual(float, type(s), 'test_DE_equity FREE_FLOAT {}'.format(s))
+
        s = financials.getRealtime('SAPX:GER', 'PE_RATIO', 'FT')
        self.assertEqual(float, type(s), 'test_DE_equity PE_RATIO {}'.format(s))

@@ -231,7 +271,7 @@ class Test(unittest.TestCase):
        self.assertTrue(testutils.is_date(s), 'test_DE_equity EX_DIV_DATE {}'.format(s))

        s = financials.getRealtime('ISHAX:GER', 'NAME', 'FT')
-        self.assertEqual('Intershop Communications AG', s, 'test_DE_equity NAME {}'.format(s))
+        self.assertEqual('INTERSHOP Communications AG', s, 'test_DE_equity NAME {}'.format(s))

        s = financials.getRealtime('ISHAX:GER', 'BETA', 'FT')
        self.assertTrue(testutils.is_positive_float(s), 'test_DE_equity BETA {}'.format(s))
@@ -263,7 +303,7 @@ class Test(unittest.TestCase):

        s = financials.getRealtime('NOVO B:CPH', 'industry', 'FT')
        self.assertEqual(str, type(s), 'test_DK_equity INDUSTRY {}'.format(s))
-        self.assertEqual('Pharmaceuticals & Biotechnology', s, 'test_DK_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Pharmaceuticals and Biotechnology', s, 'test_DK_equity INDUSTRY {}'.format(s))

    def test_TY_equity(self):
        s = financials.getRealtime('6503:TYO', 'OPEN', 'FT')
@@ -299,40 +339,40 @@ class Test(unittest.TestCase):
        self.assertEqual('General Industrials', s, 'test_TY_equity INDUSTRY {}'.format(s))

    def test_TLV_equity(self):
-        s = financials.getRealtime('DELT:TLV', 'LAST_PRICE', 'FT')
-        self.assertEqual(float, type(s), 'test_TlV_equity LAST_PRICE {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'LAST_PRICE', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity LAST_PRICE {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'OPEN', 'FT')
-        self.assertEqual(float, type(s), 'test_TlV_equity OPEN {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'OPEN', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity OPEN {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'LOW', 'FT')
-        self.assertEqual(float, type(s), 'test_TlV_equity LOW {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'LOW', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity LOW {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'HIGH', 'FT')
-        self.assertEqual(float, type(s), 'test_TlV_equity HIGH {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'HIGH', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity HIGH {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'LOW_52_WEEK', 'FT')
-        self.assertEqual(float, type(s), 'test_TlV_equity LOW_52_WEEK {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'LOW_52_WEEK', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity LOW_52_WEEK {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'HIGH_52_WEEK', 'FT')
-        self.assertEqual(float, type(s), 'test_TlV_equity HIGH_52_WEEK {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'HIGH_52_WEEK', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity HIGH_52_WEEK {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'MARKET_CAP', 'FT')
-        self.assertEqual(float, type(s), 'test_TlV_equity MARKET_CAP {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'MARKET_CAP', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity MARKET_CAP {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'VOLUME', 'FT')
-        self.assertEqual(float, type(s), 'test_TlV_equity VOLUME {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'VOLUME', 'FT')
+        self.assertEqual(float, type(s), 'test_TLV_equity VOLUME {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'CURRENCY', 'FT')
-        self.assertEqual('ILa', s, 'test_TlV_equity CURRENCY')
+        s = financials.getRealtime('LUMI:TLV', 'CURRENCY', 'FT')
+        self.assertEqual('ILa', s, 'test_TLV_equity CURRENCY')

-        s = financials.getRealtime('DELT:TLV', 'SECTOR', 'FT')
-        self.assertEqual(str, type(s), 'test_TlV_equity SECTOR {}'.format(s))
-        self.assertEqual('Consumer Goods', s, 'test_TY_equity SECTOR {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'SECTOR', 'FT')
+        self.assertEqual(str, type(s), 'test_TLV_equity SECTOR {}'.format(s))
+        self.assertEqual('Financials', s, 'test_TY_equity SECTOR {}'.format(s))

-        s = financials.getRealtime('DELT:TLV', 'INDUSTRY', 'FT')
-        self.assertEqual(str, type(s), 'test_TlV_equity INDUSTRY {}'.format(s))
-        self.assertEqual('Personal Goods', s, 'test_TY_equity INDUSTRY {}'.format(s))
+        s = financials.getRealtime('LUMI:TLV', 'INDUSTRY', 'FT')
+        self.assertEqual(str, type(s), 'test_TLV_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Banks', s, 'test_TY_equity INDUSTRY {}'.format(s))

    def test_index(self):
        s = financials.getRealtime('INX:IOM', 'ticker', 'FT')
@@ -32,6 +32,17 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('EURGBP=X', Datacode.LAST_PRICE.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_currency LAST_PRICE')

+    def test_realtime_US_ZVZZT(self):
+
+        s = financials.getRealtime('ZVZZT', Datacode.PAYOUT_RATIO.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_ZVZZT PAYOUT_RATIO {}'.format(s))
+
+        s = financials.getRealtime('ZVZZT', Datacode.SECTOR.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_ZVZZT SECTOR {}'.format(s))
+
+        s = financials.getRealtime('ZVZZT', Datacode.LAST_PRICE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_ZVZZT LAST_PRICE {}'.format(s))
+
    def test_realtime_US_equity(self):

        s = financials.getRealtime('^GSPC', Datacode.NAME.value, 'YAHOO')
@@ -70,7 +81,7 @@ class Test(unittest.TestCase):

        s = financials.getRealtime('IBM', Datacode.NAME.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_US_equity NAME {}'.format(s))
-        self.assertEqual(s, 'International Business Machines Corporation',
+        self.assertEqual(s, 'International Business Machines Corporation (IBM)',
                         'test_realtime_US_equity NAME {}'.format(s))

        s = financials.getRealtime('IBM', Datacode.SECTOR.value, 'YAHOO')
@@ -82,7 +93,7 @@ class Test(unittest.TestCase):
        self.assertEqual(s, 'Information Technology Services', 'test_realtime_US_equity INDUSTRY {}'.format(s))

        s = financials.getRealtime('IBM', Datacode.TIMEZONE.value, 'YAHOO')
-        self.assertEqual(s, 'America/New_York', 'test_realtime_US_equity TIMEZONE {}'.format(s))
+        self.assertEqual(s, 'EST', 'test_realtime_US_equity TIMEZONE {}'.format(s))

        s = financials.getRealtime('IBM', Datacode.BETA.value, 'YAHOO')
        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_equity BETA {}'.format(s))
@@ -106,8 +117,14 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('IBM', Datacode.PAYOUT_RATIO.value, 'YAHOO')
        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_equity PAYOUT_RATIO {}'.format(s))

+        s = financials.getRealtime('IBM', Datacode.SHARES_OUT.value, 'YAHOO')
+        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_equity SHARES_OUT {}'.format(s))
+
+        s = financials.getRealtime('IBM', Datacode.FREE_FLOAT.value, 'YAHOO')
+        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_equity FREE_FLOAT {}'.format(s))
+
        s = financials.getRealtime('IBM', Datacode.EXCHANGE.value, 'YAHOO')
-        self.assertEqual(s, 'NYQ', 'test_realtime_US_equity EXCHANGE')
+        self.assertEqual(s, 'NYSE', 'test_realtime_US_equity EXCHANGE')

    def test_realtime_US_mutuals(self):

@@ -117,51 +134,98 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('VFIAX', Datacode.LAST_PRICE.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_realtime_US_mutuals LAST_PRICE {}'.format(s))

-        s = financials.getRealtime('VFIAX', Datacode.LAST_PRICE_DATE.value, 'YAHOO')
-        self.assertEqual(str, type(s), 'test_realtime_US_mutuals LAST_PRICE_DATE {}'.format(s))
-
-        s = financials.getRealtime('VFIAX', Datacode.LAST_PRICE_TIME.value, 'YAHOO')
-        self.assertEqual(str, type(s), 'test_realtime_US_mutuals LAST_PRICE_TIME {}'.format(s))
+        s = financials.getRealtime('VFIAX', Datacode.DIV.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_mutuals DIV {}'.format(s))

        s = financials.getRealtime('VFIAX', Datacode.DIV_YIELD.value, 'YAHOO')
-        self.assertTrue(testutils.is_positive_float(s), 'test_realtime_US_mutuals DIV_YIELD {}'.format(s))
+        self.assertIsNone(s, 'test_realtime_US_mutuals DIV_YIELD {}'.format(s)) # no yield
+
+        s = financials.getRealtime('SHRAX', Datacode.DIV.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_mutuals DIV {}'.format(s))
+
+        s = financials.getRealtime('SHRAX', Datacode.DIV_YIELD.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_mutuals DIV_YIELD {}'.format(s)) # no yield
+
+        # s = financials.getRealtime('VERX.L', Datacode.DIV.value, 'YAHOO')
+        # self.assertIsNone(s, 'test_realtime_US_mutuals DIV {}'.format(s)) # no dividend
+
+        s = financials.getRealtime('VERX.L', Datacode.DIV_YIELD.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_mutuals DIV_YIELD {}'.format(s))

    def test_realtime_US_options(self):

        # symbol from https://finance.yahoo.com/quote/IBM/options?p=IBM
        
-        s = financials.getRealtime('IBM230120C00130000', Datacode.PREV_CLOSE.value, 'YAHOO')
+        s = financials.getRealtime('IBM240119C00120000', Datacode.PREV_CLOSE.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_realtime_US_options PREV_CLOSE {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.NAME.value, 'YAHOO')
+        s = financials.getRealtime('IBM240119C00120000', Datacode.NAME.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_US_options NAME {}'.format(s))
-        self.assertEqual('IBM Jan 2023 130.000 call', s, 'test_realtime_US_options NAME {}'.format(s))
+        self.assertEqual('IBM Jan 2024 120.000 call', s, 'test_realtime_US_options NAME {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.EXPIRY_DATE.value, 'YAHOO')
+        s = financials.getRealtime('IBM240119C00120000', Datacode.EXPIRY_DATE.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
        self.assertTrue(testutils.is_date(s), 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
-        self.assertEqual("2023-01-20", s, 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
+        self.assertEqual("2024-01-19", s, 'test_realtime_US_options EXPIRY_DATE {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.LAST_PRICE.value, 'YAHOO')
+        s = financials.getRealtime('IBM240119C00120000', Datacode.LAST_PRICE.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_realtime_US_options LAST_PRICE {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.OPEN.value, 'YAHOO')
+        s = financials.getRealtime('IBM240119C00120000', Datacode.OPEN.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_realtime_US_options OPEN {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.VOLUME.value, 'YAHOO')
+        s = financials.getRealtime('IBM240119C00120000', Datacode.VOLUME.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_realtime_US_options VOLUME {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.BID.value, 'YAHOO')
+        s = financials.getRealtime('IBM240119C00120000', Datacode.BID.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_realtime_US_options BID {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.ASK.value, 'YAHOO')
+        s = financials.getRealtime('IBM240119C00120000', Datacode.ASK.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_realtime_US_options ASK {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.BIDSIZE.value, 'YAHOO')
-        self.assertEqual(float, type(s), 'test_realtime_US_options BIDSIZE {}'.format(s))
+        s = financials.getRealtime('IBM240119C00120000', Datacode.PAYOUT_RATIO.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_options PAYOUT_RATIO {}'.format(s))

-        s = financials.getRealtime('IBM230120C00130000', Datacode.ASKSIZE.value, 'YAHOO')
-        self.assertEqual(float, type(s), 'test_realtime_US_options ASKSIZE {}'.format(s))
+        s = financials.getRealtime('IBM230120C00130000', Datacode.SECTOR.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_US_options SECTOR {}'.format(s))
+
+    def test_realtime_US_futures(self):
+
+        s = financials.getRealtime('ESH23.CME', Datacode.NAME.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_futures NAME {}'.format(s))
+        self.assertEqual('E-Mini S&P 500 Mar 23 (ESH23.CME)', s, 'test_realtime_US_options NAME {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.SETTLEMENT_DATE.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_futures SETTLEMENT_DATE {}'.format(s))
+        self.assertTrue(testutils.is_date(s), 'test_realtime_US_futures SETTLEMENT_DATE {}'.format(s))
+        self.assertEqual("2023-03-17", s, 'test_realtime_US_futures SETTLEMENT_DATE {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.LAST_PRICE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.OPEN.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures OPEN {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.VOLUME.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures VOLUME {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.BID.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures BID {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.ASK.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures ASK {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.CHANGE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures CHANGE {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.CHANGE_IN_PERCENT.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures CHANGE_IN_PERCENT {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.LOW.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures LOW {}'.format(s))
+
+        s = financials.getRealtime('ESH23.CME', Datacode.HIGH.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_futures HIGH {}'.format(s))

    def test_realtime_UK_ETF(self):

@@ -169,12 +233,11 @@ class Test(unittest.TestCase):
        self.assertEqual(float, type(s), 'test_realtime_UK_ETF LAST_PRICE {}'.format(s))

        s = financials.getRealtime('VERX.L', Datacode.TIMEZONE.value, 'YAHOO')
-        self.assertEqual(s, 'Europe/London', 'test_realtime_UK_ETF TIMEZONE {}'.format(s))
+        self.assertEqual('GMT', s, 'test_realtime_UK_ETF TIMEZONE {}'.format(s))

        s = financials.getRealtime('CSP1.L', Datacode.NAME.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_UK_ETF NAME {}'.format(s))
-        self.assertEqual(s, 'iShares VII Public Limited Company - iShares Core S&P 500 UCITS ETF',
-                         'test_realtime_UK_ETF NAME {}'.format(s))
+        self.assertEqual('iShares Core S&P 500 UCITS ETF USD (Acc) (CSP1.L)', s, 'test_realtime_UK_ETF NAME {}'.format(s))

    def test_realtime_DE_equity(self):

@@ -182,11 +245,11 @@ class Test(unittest.TestCase):
        self.assertEqual(float, type(s), 'test_realtime_DE_equity LAST_PRICE {}'.format(s))

        s = financials.getRealtime('SAP.DE', Datacode.TIMEZONE.value, 'YAHOO')
-        self.assertEqual(s, 'Europe/Berlin', 'test_realtime_DE_equity TIMEZONE {}'.format(s))
+        self.assertEqual('CET', s, 'test_realtime_DE_equity TIMEZONE {}'.format(s))

        s = financials.getRealtime('SAP.DE', Datacode.SECTOR.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_DE_equity SECTOR {}'.format(s))
-        self.assertEqual(s, 'Technology', 'test_realtime_DE_equity SECTOR {}'.format(s))
+        self.assertEqual('Technology', s, 'test_realtime_DE_equity SECTOR {}'.format(s))

        s = financials.getRealtime('SAP.DE', Datacode.INDUSTRY.value, 'YAHOO')
        self.assertEqual(str, type(s), 'test_realtime_DE_equity INDUSTRY {}'.format(s))
@@ -196,11 +259,51 @@ class Test(unittest.TestCase):
        self.assertEqual(str, type(s), 'test_realtime_DE_equity NAME {}'.format(s))

        s = financials.getRealtime('C060.DE', Datacode.EXCHANGE.value, 'YAHOO')
-        self.assertEqual(s, 'GER', 'test_realtime_DE_equity EXCHANGE')
+        self.assertEqual(s, 'XETRA', 'test_realtime_DE_equity EXCHANGE')
+
+    def test_TA_equity(self):
+
+        s = financials.getRealtime('LUMI.TA', 'LAST_PRICE', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'OPEN', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity OPEN {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'LOW', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity LOW {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'HIGH', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity HIGH {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'LOW_52_WEEK', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity LOW_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'HIGH_52_WEEK', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity HIGH_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'MARKET_CAP', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity MARKET_CAP {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'VOLUME', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_TA_equity VOLUME {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'CURRENCY', 'YAHOO')
+        self.assertEqual('ILA', s, 'test_TA_equity CURRENCY')
+
+        s = financials.getRealtime('LUMI.TA', 'SECTOR', 'YAHOO')
+        self.assertEqual(str, type(s), 'test_TLV_equity SECTOR {}'.format(s))
+        self.assertEqual('Financial Services', s, 'test_TA_equity SECTOR {}'.format(s))
+
+        s = financials.getRealtime('LUMI.TA', 'INDUSTRY', 'YAHOO')
+        self.assertEqual(str, type(s), 'test_TLV_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Banks—Regional', s, 'test_TA_equity INDUSTRY {}'.format(s))

    def test_DK_equity(self):
+        s = financials.getRealtime('NOVO-B.CO', 'last_price', 'YAHOO')
+        self.assertEqual(float, type(s), 'test_DK_equity LAST_PRICE {}'.format(s))
+
        s = financials.getRealtime('NOVO-B.CO', 'name', 'YAHOO')
-        self.assertEqual('Novo Nordisk A/S', s, 'test_DK_equity NAME {}'.format(s))
+        self.assertEqual('Novo Nordisk A/S (NOVO-B.CO)', s, 'test_DK_equity NAME {}'.format(s))

        s = financials.getRealtime('NOVO-B.CO', 'currency', 'YAHOO')
        self.assertEqual('DKK', s, 'test_DK_equity CURRENCY {}'.format(s))
@@ -210,6 +313,14 @@ class Test(unittest.TestCase):
        self.assertEqual('Biotechnology', s, 'test_DK_equity INDUSTRY {}'.format(s))

    def test_realtime_TY_equity(self):
+        s = financials.getRealtime('6503.T', Datacode.SECTOR.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_TY_equity SECTOR {}'.format(s))
+        self.assertEqual(s, 'Industrials', 'test_TY_equity SECTOR {}'.format(s))
+
+        s = financials.getRealtime('6503.T', Datacode.INDUSTRY.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_TY_equity INDUSTRY {}'.format(s))
+        self.assertEqual(s, 'Electrical Equipment & Parts', 'test_TY_equity INDUSTRY {}'.format(s))
+
        s = financials.getRealtime('6503.T', Datacode.OPEN.value, 'YAHOO')
        self.assertEqual(float, type(s), 'test_TY_equity OPEN {}'.format(s))

@@ -234,13 +345,8 @@ class Test(unittest.TestCase):
        s = financials.getRealtime('6503.T', Datacode.CURRENCY.value, 'YAHOO')
        self.assertEqual(s, 'JPY', 'test_TY_equity CURRENCY')

-        s = financials.getRealtime('6503.T', Datacode.SECTOR.value, 'YAHOO')
-        self.assertEqual(str, type(s), 'test_TY_equity SECTOR {}'.format(s))
-        self.assertEqual(s, 'Industrials', 'test_TY_equity SECTOR {}'.format(s))
-
-        s = financials.getRealtime('6503.T', Datacode.INDUSTRY.value, 'YAHOO')
-        self.assertEqual(str, type(s), 'test_TY_equity INDUSTRY {}'.format(s))
-        self.assertEqual(s, 'Electrical Equipment & Parts', 'test_TY_equity INDUSTRY {}'.format(s))
+        s = financials.getRealtime('6503.T', Datacode.TIMEZONE.value, 'YAHOO')
+        self.assertEqual(s, 'JST', 'test_TY_equity TIMEZONE')

    def test_historic_US_equity(self):

@@ -254,12 +360,12 @@ class Test(unittest.TestCase):
        self.assertIsNone(s, 'test_historic_US_equity LAST_PRICE {}'.format(s))

        s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(167.190002, s, 'test_historic_US_equity CLOSE {}'.format(s))
+        self.assertEqual(159.837479, s, 'test_historic_US_equity CLOSE {}'.format(s))

        financials.yahoo.historicdata = {}

        s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(167.190002, s, 'test_historic_US_equity CLOSE {}'.format(s))
+        self.assertEqual(159.837479, s, 'test_historic_US_equity CLOSE {}'.format(s))

        directory = os.path.join(str(pathlib.Path.home()), '.financials-extension')
        ibm = os.path.join(directory, 'yahoo-IBM.csv')
@@ -271,11 +377,10 @@ class Test(unittest.TestCase):
        financials.yahoo.historicdata = {}

        s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(167.190002, s, 'test_historic_US_equity CLOSE {}'.format(s))
+        self.assertEqual(159.837479, s, 'test_historic_US_equity CLOSE {}'.format(s))

-        # Note: quarterly dividend and splits will change past adjusted prices - will fail after the next dividend
        s = financials.getHistoric('IBM', Datacode.ADJ_CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(134.699081, s, 'test_historic_US_equity ADJ_CLOSE {}'.format(s))
+        self.assertEqual(float, type(s), 'test_historic_US_equity ADJ_CLOSE {}'.format(s))

    def test_historic_UK_ETF(self):

@@ -323,13 +428,19 @@ class Test(unittest.TestCase):
        self.assertEqual(s, 82.889999, 'test_historic_DE_equity CLOSE {}'.format(s))

        s = financials.getHistoric('C060.DE', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(s, 68.209999, 'test_historic_DE_equity CLOSE {}'.format(s))
+        self.assertEqual(s, 72.870003, 'test_historic_DE_equity CLOSE {}'.format(s))

    def test_realtime_errors(self):

        s = financials.getRealtime('NO_NAME', Datacode.LAST_PRICE.value, 'YAHOO')
        self.assertIsNone(s, 'test_realtime_errors LAST_PRICE {}'.format(s))

+        s = financials.getRealtime('NO_NAME', Datacode.PAYOUT_RATIO.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_errors PAYOUT_RATIO {}'.format(s))
+
+        s = financials.getRealtime('NO_NAME', Datacode.SECTOR.value, 'YAHOO')
+        self.assertIsNone(s, 'test_realtime_errors SECTOR {}'.format(s))
+
        s = financials.getRealtime('C060.DE', -1, 'YAHOO')
        self.assertEqual('Datacode -1 not supported', s, 'test_realtime_errors -1 {}'.format(s))
Author	SHA1	Message	Date
cmallwitz	a89edcc5fc	Rework Yahoo after website change (done)	2022-12-20 15:17:54 +00:00
cmallwitz	c71825143e	Rework Yahoo after website change (phase 1)	2022-12-18 18:58:59 +00:00
cmallwitz	587b391393	Update to README.md	2022-12-18 17:32:58 +00:00
cmallwitz	605e9ef185	Updated examples.ods for COINBASE	2022-12-13 13:52:29 +00:00
cmallwitz	e0491318c3	Fixing Coinbase merge	2022-12-13 13:27:52 +00:00
cmallwitz	24b1406772	Merge branch 'TheSerapher-coinbase-and-macos'	2022-12-13 13:25:08 +00:00
cmallwitz	1d9c4e824f	Merge branch 'coinbase-and-macos' of github.com:TheSerapher/Financials-Extension into TheSerapher-coinbase-and-macos	2022-12-13 13:24:24 +00:00
Sebastian Grewe	c3a6d8d69a	[WIP] Examples.ods	2022-12-13 10:29:00 +01:00
Sebastian Grewe	d0fc0ae2bf	Revert "[VERSION] 3.1.0" This reverts commit `e4d395240d`.	2022-12-09 15:44:48 +01:00
Sebastian Grewe	e4d395240d	[VERSION] 3.1.0	2022-12-09 15:44:15 +01:00
Sebastian Grewe	c3dc5c35fc	[ADDED] Super simple test	2022-12-09 15:43:47 +01:00
cmallwitzandSebastian Grewe	b356a1bca4	Updated Yahoo EU consent cookies	2022-12-09 15:43:47 +01:00
Sebastian Grewe	befd0956f0	[FIX] Compile for MacOS	2022-12-09 15:14:14 +01:00
cmallwitz	9e842b30de	Updated Yahoo EU consent cookies	2022-12-09 13:51:19 +00:00
Sebastian Grewe	6d0f56c87d	[ADDED] Coinbase Exchange for Crypto symbols	2022-12-09 14:43:35 +01:00
Sebastian Grewe	cd41959559	[UPDATE] added support for MacOS to compile.sh	2022-12-09 14:18:48 +01:00
cmallwitz	52e7119081	Updated Yahoo EU consent cookies	2022-11-04 13:25:40 +00:00
cmallwitz	520d2441cb	Refresh Yahoo cookies	2022-06-02 21:19:49 +01:00
cmallwitz	8b370dfc73	Updated version	2022-01-07 21:42:07 +00:00
cmallwitz	af82f16740	Updated Yahoo EU consent	2022-01-07 21:33:55 +00:00
cmallwitz	4e626c272e	Add shares outstanding and free float information	2021-09-28 20:28:00 +01:00