Added BID/ASK/BIDSIZE/ASKSIZE/EXPIRY_DATE fields for FT/YAHOO and minor ccy fix

README.md

@@ -6,7 +6,7 @@ using old-fashioned web scraping.
 
 ### Feedback requested:
 
-Please provide about using the extension here: https://github.com/cmallwitz/Financials-Extension/issues/10
+Please provide feedback about using the extension here: https://github.com/cmallwitz/Financials-Extension/issues/10
 
 ### System requirements:
 
@@ -19,13 +19,23 @@ On Ubuntu 18 and 20, dateutil and pytz may already be installed, but you can get
 
 On Debian 10, the following steps have worked for me (as root)
 - apt install python3-pip
-- pip3 install python-dateutil
-- pip3 install pytz
-- pip3 install pyparsing
+- pip3 install python-dateutil pytz pyparsing
+
+Users on Windows 10 have reported this to work - as LibreOffice on Windows ships with its own, but minimal 
+Python runtime a few more steps are required
+- Download the script https://bootstrap.pypa.io/get-pip.py to your computer
+- Start a Command Prompt (CMD) as Administrator
+- on this command prompt run (change path as required)
+  
+  "c:\Program Files\LibreOffice\program\python.exe" c:\temp\get-pip.py
+
+- and then
+
+   "c:\Program Files\LibreOffice\program\python.exe" -m pip install python-dateutil pytz pyparsing
 
 ### Usage:
 
-Have a look at the 'releases' tab on GitHub and download a prebuilt **Financials-Extension.oxt** file and load it into Calc 
+Under 'Releases' on GitHub there is downloadable **Financials-Extension.oxt** file - load it into Calc 
 under menu item: Tools, Extension Manager...
 
 Getting data should be a simple as having this in a cell: 
@@ -46,42 +56,55 @@ assets in other regions, drop me a line (best to include full URLs and possibly
 site for comparison). While data for last price is most likely consistent across sites, they may differ for other data 
 points.     
 
+### LibreOffice: using , (comma) vs ; (semicolon) to separate arguments in formula
+
+There is a setting in "Tools" / "Options..." / "LibreOffice Calc" / "Formula" called "Functions". Here the user can specify the character used to separate arguments in formula.
+
+Mine is set to , (comma) - when I enter ; (semicolon) in a formula (Ubuntu / UK English), no error is reported but the semicolon is converted to , (comma)
+
+Depending on your system's language and default LibreOffice settings, you maybe better off using ; instead of , in your formulas.
+
 ### List of supported data points
 
 You can either specify numbers or names (lower or upper case) - not all bits are available from all sources and they are not necessarily consistent across sources either.
 
-|Name|Code|YAHOO|FT|GOOGLE|YAHOO (historic)|
-| :--- | :--- | :---: | :---: | :---: | :---: |
-|PREV_CLOSE|5|Yes|Yes|Yes||
-|OPEN|6|Yes|Yes|Yes|Yes|
-|CHANGE|7|Yes|Yes|Yes||
-|LAST_PRICE_DATE|8|Yes|Yes|Yes||
-|LAST_PRICE_TIME|10|Yes|Yes|Yes||
-|CHANGE_IN_PERCENT|11|Yes|Yes|Yes||
-|LOW|14|Yes|Yes|Yes|Yes|
-|HIGH|16|Yes|Yes|Yes|Yes|
-|LAST_PRICE|21|Yes|Yes|Yes||
-|HIGH_52_WEEK|24|Yes|Yes|Yes||
-|LOW_52_WEEK|26|Yes|Yes|Yes||
-|MARKET_CAP|27|Yes|Yes|Yes||
-|VOLUME|35|Yes|Yes|No|Yes|
-|AVG_DAILY_VOL_3MONTH|39|Yes|Yes|No||
-|BETA|67|Yes|Yes|No||
-|EPS|68|Yes|Yes|No||
-|PE_RATIO|69|Yes|Yes|No||
-|DIV|70|Yes|Yes|No||
-|DIV_YIELD|71|Yes|Yes|No||
-|EX_DIV_DATE|72|Yes|Yes|No||
-|PAYOUT_RATIO|73|Yes|No|No||
-|CLOSE|90|No|No|No|Yes|
-|ADJ_CLOSE|91|No|No|No|Yes|
-|SECTOR|98|Yes|Yes|No||
-|INDUSTRY|99|Yes|Yes|No||
-|TICKER|101|Yes|Yes|Yes||
-|EXCHANGE|102|Yes|No|Yes||
-|CURRENCY|103|Yes|Yes|Yes||
-|NAME|104|Yes|Yes|Yes||
-|TIMEZONE|105|Yes|Yes|Yes||
+|Name|Code|YAHOO|FT|GOOGLE|YAHOO (historic)|Notes|
+| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
+|PREV_CLOSE|5|Yes|Yes|Yes|||
+|OPEN|6|Yes|Yes|Yes|Yes||
+|CHANGE|7|Yes|Yes|Yes|||
+|LAST_PRICE_DATE|8|Yes|Yes|Yes| |yyyy-mm-dd|
+|LAST_PRICE_TIME|10|Yes|Yes|Yes|||
+|CHANGE_IN_PERCENT|11|Yes|Yes|Yes|||
+|LOW|14|Yes|Yes|Yes|Yes||
+|HIGH|16|Yes|Yes|Yes|Yes||
+|LAST_PRICE|21|Yes|Yes|Yes|||
+|BID|22|Yes|Yes|No|||
+|ASK|25|Yes|Yes|No|||
+|HIGH_52_WEEK|24|Yes|Yes|Yes|||
+|LOW_52_WEEK|26|Yes|Yes|Yes|||
+|MARKET_CAP|27|Yes|Yes|Yes|||
+|BIDSIZE|30|Yes|Yes|No|||
+|ASKSIZE|31|Yes|Yes|No|||
+|VOLUME|35|Yes|Yes|No|Yes||
+|AVG_DAILY_VOL_3MONTH|39|Yes|Yes|No|||
+|BETA|67|Yes|Yes|No|||
+|EPS|68|Yes|Yes|No|||
+|PE_RATIO|69|Yes|Yes|No|||
+|DIV|70|Yes|Yes|No|||
+|DIV_YIELD|71|Yes|Yes|No|||
+|EX_DIV_DATE|72|Yes|Yes|No| |yyyy-mm-dd|
+|PAYOUT_RATIO|73|Yes|No|No|||
+|EXPIRY_DATE|74|Yes (on options)|No|No| |yyyy-mm-dd|
+|CLOSE|90|No|No|No|Yes||
+|ADJ_CLOSE|91|No|No|No|Yes||
+|SECTOR|98|Yes|Yes|No|||
+|INDUSTRY|99|Yes|Yes|No|||
+|TICKER|101|Yes|Yes|Yes|||
+|EXCHANGE|102|Yes|No|Yes|||
+|CURRENCY|103|Yes|Yes|Yes|||
+|NAME|104|Yes|Yes|Yes|||
+|TIMEZONE|105|Yes|Yes|Yes|||
 
 ### Dealing with missing data:
 
@@ -94,6 +117,10 @@ your favorite web browser (or other tools) to check if the page actually contain
 If it does, the file trace.log has a record of all calls to the extension with the value returned to LibreOffice. 
 Otherwise, the file extension.log in the same location might have more details about errors or exceptions.  
 
+In general, web scraping can't be compared to using a stable API - the websites might have issues - from a technical or
+data perspective. I have found especially on the weekend it can sometimes be "flaky" and closing/reopening LibreCalc can 
+refresh things. 
+
 ### Build:
 
 You will need the LibreOffice SDK installed. 
@@ -104,9 +131,7 @@ On my system (Ubuntu) I installed packages: libreoffice-dev libreoffice-java-com
 
 cd ~/tech/IdeaProjects/Financials-Extension/
 
-python3 src/test_yahoo.py
-
-python3 src/test_google.py
+python3 -m unittest discover src
 
 \# This builds file **Financials-Extension.oxt**
 

src/baseclient.py

@@ -142,9 +142,12 @@ class BaseClient:
         self.response = self.request('POST' if data else 'GET', url, data, headers, cookies, **kwargs)
         text = self.response.read()
 
-        # Allow two redirects: used by Yahoo for some cookie based consent
+        # Allow redirects - used by Yahoo for some cookie based consent
+        redirect_count = 3
 
-        if 300 <= self.response.status < 400:
+        while 300 <= self.response.status < 400 and redirect_count >= 0:
+
+            redirect_count -= 1
             location = self.response.getheader('Location')
 
             if location and redirect:
@@ -156,21 +159,6 @@ class BaseClient:
                 self.response = self.request('POST' if data else 'GET', location, data, headers, cookies, **kwargs)
                 text = self.response.read()
 
-                if 300 <= self.response.status < 400:
-                    location = self.response.getheader('Location')
-
-                    if location and redirect:
-
-                        if location.startswith('/'):
-                            scheme, _, host, path = url.split('/', 3)
-                            location = '{}//{}{}'.format(scheme, host, location)
-
-                        self.response = self.request('POST' if data else 'GET', location, data, headers, cookies,
-                                                     **kwargs)
-                        text = self.response.read()
-                    else:
-                        raise RedirectException(location)
-
             else:
                 raise RedirectException(location)
 
@@ -187,6 +175,47 @@ class BaseClient:
 
         return text
 
+    def get_ticker(self):
+
+        tick = {}
+
+        tick[Datacode.ADJ_CLOSE] = None
+        tick[Datacode.ASKSIZE] = None
+        tick[Datacode.ASK] = None
+        tick[Datacode.AVG_DAILY_VOL_3MONTH] = None
+        tick[Datacode.BETA] = None
+        tick[Datacode.BIDSIZE] = None
+        tick[Datacode.BID] = None
+        tick[Datacode.CHANGE] = None
+        tick[Datacode.CHANGE_IN_PERCENT] = None
+        tick[Datacode.CURRENCY] = None
+        tick[Datacode.DIV] = None
+        tick[Datacode.DIV_YIELD] = None
+        tick[Datacode.EPS] = None
+        tick[Datacode.EXCHANGE] = None
+        tick[Datacode.EXPIRY_DATE] = None
+        tick[Datacode.EX_DIV_DATE] = None
+        tick[Datacode.HIGH] = None
+        tick[Datacode.HIGH_52_WEEK] = None
+        tick[Datacode.INDUSTRY] = None
+        tick[Datacode.LAST_PRICE] = None
+        tick[Datacode.LAST_PRICE_DATE] = None
+        tick[Datacode.LAST_PRICE_TIME] = None
+        tick[Datacode.LOW] = None
+        tick[Datacode.LOW_52_WEEK] = None
+        tick[Datacode.MARKET_CAP] = None
+        tick[Datacode.NAME] = None
+        tick[Datacode.OPEN] = None
+        tick[Datacode.PAYOUT_RATIO] = None
+        tick[Datacode.PE_RATIO] = None
+        tick[Datacode.PREV_CLOSE] = None
+        tick[Datacode.SECTOR] = None
+        tick[Datacode.TICKER] = None
+        tick[Datacode.TIMEZONE] = None
+        tick[Datacode.VOLUME] = None
+
+        return tick
+
     def _return_value(self, data: dict, datacode: int):
 
         """
@@ -234,6 +263,18 @@ class BaseClient:
             elif datacode == Datacode.LAST_PRICE.value and Datacode.LAST_PRICE in data:
                 return data[Datacode.LAST_PRICE]
 
+            elif datacode == Datacode.BID.value and Datacode.BID in data:
+                return data[Datacode.BID]
+
+            elif datacode == Datacode.ASK.value and Datacode.ASK in data:
+                return data[Datacode.ASK]
+
+            elif datacode == Datacode.BIDSIZE.value and Datacode.BIDSIZE in data:
+                return data[Datacode.BIDSIZE]
+
+            elif datacode == Datacode.ASKSIZE.value and Datacode.ASKSIZE in data:
+                return data[Datacode.ASKSIZE]
+
             elif datacode == Datacode.LOW_52_WEEK.value and Datacode.LOW_52_WEEK in data:
                 return data[Datacode.LOW_52_WEEK]
 
@@ -273,6 +314,12 @@ class BaseClient:
             elif datacode == Datacode.PAYOUT_RATIO.value and Datacode.PAYOUT_RATIO in data:
                 return data[Datacode.PAYOUT_RATIO]
 
+            elif datacode == Datacode.EXPIRY_DATE.value and Datacode.EXPIRY_DATE in data:
+                if data[Datacode.EXPIRY_DATE]:
+                    return data[Datacode.EXPIRY_DATE].isoformat()
+                else:
+                    return data[Datacode.EXPIRY_DATE]
+
             elif datacode == Datacode.CLOSE.value and Datacode.CLOSE in data:
                 return data[Datacode.CLOSE]
 

src/datacode.py

@@ -24,6 +24,11 @@ class Datacode(Enum):
 
     LAST_PRICE = 21
 
+    BID = 22
+    ASK = 25
+    BIDSIZE = 30
+    ASKSIZE = 31
+
     HIGH_52_WEEK = 24
     LOW_52_WEEK = 26
     MARKET_CAP = 27
@@ -38,6 +43,7 @@ class Datacode(Enum):
     DIV_YIELD = 71
     EX_DIV_DATE = 72
     PAYOUT_RATIO = 73
+    EXPIRY_DATE = 74
 
     CLOSE = 90
     ADJ_CLOSE = 91

src/financials.py

@@ -19,7 +19,6 @@ import time
 from functools import wraps
 from importlib import util
 
-import dateutil.parser
 import unohelper
 from com.financials.getinfo import Financials
 
@@ -47,6 +46,8 @@ if dateutil_missing or pyparsing_missing or pytz_missing:
     msg += ' pytz' if pytz_missing else ''
     raise Exception("THIS EXTENSION NEEDS THE FOLLOWING PYTHON 3 LIBRARIES INSTALLED:" + msg)
 
+import dateutil.parser
+
 from datacode import Datacode
 import financials_google as google
 import financials_yahoo as yahoo
@@ -181,7 +182,7 @@ class FinancialsImpl(unohelper.Base, Financials):
 
                 try:
                     offset = int(date)  # offset for 1899-12-30
-                    d = dateutil.parser.parse('1899-12-30') + datetime.timedelta(days=offset)
+                    d = dateutil.parser.parse('1899-12-30', yearfirst=True, dayfirst=False) + datetime.timedelta(days=offset)
                     d = d.date().isoformat()
                 except:
                     return 'Date format not supported: {}'.format(date)
@@ -190,7 +191,7 @@ class FinancialsImpl(unohelper.Base, Financials):
             elif type(date) == str:
 
                 try:
-                    int(dateutil.parser.parse(date).strftime('%s'))
+                    int(dateutil.parser.parse(date, yearfirst=True, dayfirst=False).timestamp())
                 except:
                     return 'Date format not supported: \'{}\''.format(date)
 
@@ -218,7 +219,7 @@ class FinancialsImpl(unohelper.Base, Financials):
     @profile
     def support(self, datacode):
 
-        s = 'ctx={}\nid(self)={}\nversion={}\nfile={}\ncwd={}\nhome={}\nuname={}\npid={}\nsys.executable={}\nsys.version={}\nlocale={}'.format(
+        s = 'ctx={}\nid(self)={}\nversion={}\nfile={}\ncwd={}\nhome={}\nuname={}\npid={}\nsys.executable={}\nsys.version={}\nlocale={}\ndefaultlocale={}'.format(
             self.ctx,
             id(self),
             version,
@@ -229,7 +230,8 @@ class FinancialsImpl(unohelper.Base, Financials):
             os.getpid(),
             sys.executable,
             sys.version.replace("\n", " "),
-            locale.getlocale())
+            locale.getlocale(),
+            locale.getdefaultlocale())
 
         if datacode:
             s = '{}\ntype(datacode)={}\nstr(datacode)={}'.format(

src/financials_ft.py

@@ -52,9 +52,9 @@ class FT(BaseClient):
     def getRealtime(self, ticker: str, datacode: int):
 
         """
-        Retrieve data for ticker from Alpha Vantage and cache it for further lookups
+        Retrieve data for ticker from Financial Times and cache it for further lookups
 
-        :param ticker: the ticker symbol e.g. VOD.LON
+        :param ticker: the ticker symbol e.g. VOD:LSE
         :param datacode: the requested datacode
         :return:
         """
@@ -71,7 +71,7 @@ class FT(BaseClient):
                 del self.realtime[ticker]
 
         if ticker not in self.realtime:
-            self.realtime[ticker] = {}
+            self.realtime[ticker] = self.get_ticker()
 
         tick = self.realtime[ticker]
 
@@ -88,40 +88,11 @@ class FT(BaseClient):
         try:
             with open(os.path.join(self.basedir, f'ft-{ticker}.html'), "w", encoding="utf-8") as text_file:
                 print(f"<!-- '{self.last_url}' -->\r\n\r\n{text}", file=text_file)
-        except BaseException as e:
-            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+        except BaseException:
+            logger.exception("BaseException ticker=%s datacode=%s %s", ticker, datacode)
 
         tick[Datacode.TIMESTAMP] = time.time()
 
-        tick[Datacode.NAME] = None
-        tick[Datacode.TICKER] = None
-        tick[Datacode.CURRENCY] = None
-        tick[Datacode.LAST_PRICE] = None
-        tick[Datacode.CHANGE] = None
-        tick[Datacode.CHANGE_IN_PERCENT] = None
-        tick[Datacode.VOLUME] = None
-        tick[Datacode.LOW_52_WEEK] = None
-        tick[Datacode.HIGH_52_WEEK] = None
-        tick[Datacode.LAST_PRICE_DATE] = None
-        tick[Datacode.LAST_PRICE_TIME] = None
-        tick[Datacode.TIMEZONE] = None
-
-        tick[Datacode.OPEN] = None
-        tick[Datacode.HIGH] = None
-        tick[Datacode.LOW] = None
-        tick[Datacode.PREV_CLOSE] = None
-        tick[Datacode.MARKET_CAP] = None
-
-        tick[Datacode.EXCHANGE] = None
-        tick[Datacode.AVG_DAILY_VOL_3MONTH] = None
-        tick[Datacode.BETA] = None
-        tick[Datacode.EPS] = None
-        tick[Datacode.PE_RATIO] = None
-        tick[Datacode.DIV] = None
-        tick[Datacode.DIV_YIELD] = None
-        tick[Datacode.EX_DIV_DATE] = None
-        tick[Datacode.PAYOUT_RATIO] = None
-
         try:
             r = '<h1 class="mod-tearsheet-overview__header__name mod-tearsheet-overview__header__name--large">(.*?)</h1>'
             match = re.compile(r, flags=re.DOTALL).search(text)
@@ -148,7 +119,7 @@ class FT(BaseClient):
                 tick[Datacode.INDUSTRY] = self.save_wrapper(
                     lambda: html.unescape(match.group(2)).strip())
 
-            r = r'<span [^>]*>Price \(([A-Z]+|--)\)</span><span [^>]*>([0-9,\.]+)</span>'
+            r = r'<span [^>]*>Price \(([A-Za-z]{3}|--)\)</span><span [^>]*>([0-9,\.]+)</span>'
             match = re.compile(r, flags=re.DOTALL).search(text, start)
             if match:
                 start = match.span(0)[1]
@@ -197,7 +168,7 @@ class FT(BaseClient):
 
                 try:
                     value = html.unescape(match.group(1)).strip()
-                    dt = dateutil.parser.parse(value, tzinfos=whois_timezone_info)
+                    dt = dateutil.parser.parse(value, yearfirst=True, dayfirst=False, tzinfos=whois_timezone_info)
                     tick[Datacode.LAST_PRICE_DATE] = dt.date()
                     tick[Datacode.LAST_PRICE_TIME] = dt.time()
 
@@ -205,7 +176,7 @@ class FT(BaseClient):
                     if len(time_bits) >= 4:
                         tick[Datacode.TIMEZONE] = time_bits[-1]
 
-                except BaseException as e:
+                except BaseException:
                     pass
 
             # second attempt at 52 week range
@@ -247,6 +218,18 @@ class FT(BaseClient):
                 tick[Datacode.LOW] = self.save_wrapper(
                     lambda: float(html.unescape(match.group(1)).replace(',', '').strip()))
 
+            r = r'<th>\s*Bid\s*</th><td>([0-9,\.]+)</td>'
+            match = re.compile(r, flags=re.DOTALL).search(text, start)
+            if match:
+                tick[Datacode.BID] = self.save_wrapper(
+                    lambda: float(html.unescape(match.group(1)).replace(',', '').strip()))
+
+            r = r'<th>\s*Offer\s*</th><td>([0-9,\.]+)</td>'
+            match = re.compile(r, flags=re.DOTALL).search(text, start)
+            if match:
+                tick[Datacode.ASK] = self.save_wrapper(
+                    lambda: float(html.unescape(match.group(1)).replace(',', '').strip()))
+
             r = r'<th>\s*Previous close\s*</th><td>\s*([0-9,\.]+)\s*</td>'
             match = re.compile(r, flags=re.DOTALL).search(text, start)
             if match:
@@ -298,9 +281,9 @@ class FT(BaseClient):
             if match:
                 try:
                     value = html.unescape(match.group(1)).strip()
-                    dt = dateutil.parser.parse(value, tzinfos=whois_timezone_info)
+                    dt = dateutil.parser.parse(value, yearfirst=True, dayfirst=False, tzinfos=whois_timezone_info)
                     tick[Datacode.EX_DIV_DATE] = dt.date()
-                except BaseException as e:
+                except BaseException:
                     pass
 
         except BaseException as e:

src/financials_google.py

@@ -102,34 +102,12 @@ class Google(BaseClient):
             logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
 
         if ticker not in self.realtime:
-            self.realtime[ticker] = {}
+            self.realtime[ticker] =  self.get_ticker()
 
         tick = self.realtime[ticker]
 
         tick[Datacode.TIMESTAMP] = time.time()
 
-        tick[Datacode.NAME] = None
-        tick[Datacode.TICKER] = None
-        tick[Datacode.CURRENCY] = None
-        tick[Datacode.LAST_PRICE] = None
-        tick[Datacode.CHANGE] = None
-        tick[Datacode.CHANGE_IN_PERCENT] = None
-        tick[Datacode.VOLUME] = None
-        tick[Datacode.LOW_52_WEEK] = None
-        tick[Datacode.HIGH_52_WEEK] = None
-        tick[Datacode.LAST_PRICE_DATE] = None
-        tick[Datacode.LAST_PRICE_TIME] = None
-        tick[Datacode.TIMEZONE] = None
-
-        tick[Datacode.OPEN] = None
-        tick[Datacode.HIGH] = None
-        tick[Datacode.LOW] = None
-        tick[Datacode.PREV_CLOSE] = None
-        tick[Datacode.MARKET_CAP] = None
-
-        tick[Datacode.EXCHANGE] = None
-        tick[Datacode.AVG_DAILY_VOL_3MONTH] = None
-
         try:
             r = '<span[^>]+role="heading"[^>]+>(.*?)</span>'
             pattern = re.compile(r)
@@ -200,7 +178,7 @@ class Google(BaseClient):
             try:
                 value = html.unescape(header.find('./div[1]/span[1]/span[2]').text).replace('·', '').strip()
                 logger.debug(value)
-                dt = dateutil.parser.parse(value, tzinfos=whois_timezone_info)
+                dt = dateutil.parser.parse(value, yearfirst=True, dayfirst=False, tzinfos=whois_timezone_info)
                 tick[Datacode.LAST_PRICE_DATE] = dt.date()
                 tick[Datacode.LAST_PRICE_TIME] = dt.time()
 

src/financials_yahoo.py

@@ -72,7 +72,7 @@ class Yahoo(BaseClient):
             ticks = {}
 
             for row in reader:
-                tick = {}
+                tick  = self.get_ticker()
                 try:
                     tick[Datacode.OPEN] = float(row['Open'])
                     tick[Datacode.LOW] = float(row['Low'])
@@ -113,7 +113,7 @@ class Yahoo(BaseClient):
 
         cookies = [cookiejar.Cookie(version=0,
                                     name="B",
-                                    value="7pbfivtfkl00m&b=3&s=if",
+                                    value="er7g22lg35od5&b=3&s=8p",
                                     port=None, port_specified=False,
                                     domain=".yahoo.com", domain_specified=True, domain_initial_dot=True,
                                     path="/", path_specified=True,
@@ -134,7 +134,7 @@ class Yahoo(BaseClient):
         try:
             with open(os.path.join(self.basedir, 'yahoo-{}.html'.format(ticker)), "w", encoding="utf-8") as text_file:
                 print(f"<!-- '{url}' -->\r\n\r\n{text}", file=text_file)
-        except BaseException as e:
+        except BaseException:
             logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
 
         try:
@@ -145,7 +145,7 @@ class Yahoo(BaseClient):
             pattern = re.compile(r)
             match = pattern.search(text)
 
-            if match:
+            if  match:
                 self.crumb = match.group(1)
 
         except BaseException as e:
@@ -181,7 +181,7 @@ class Yahoo(BaseClient):
                 return 'Could not find price for \'{}\''.format(ticker)
 
             if ticker not in self.realtime:
-                self.realtime[ticker] = {}
+                self.realtime[ticker] =  self.get_ticker()
 
             tick = self.realtime[ticker]
 
@@ -202,15 +202,20 @@ class Yahoo(BaseClient):
             tick[Datacode.DIV] = float(raw(summaryDetail, 'dividendRate'))
             tick[Datacode.DIV_YIELD] = float(raw(summaryDetail, 'dividendYield'))
             tick[Datacode.EX_DIV_DATE] = self.save_wrapper(
-                lambda: dateutil.parser.parse(str(fmt(summaryDetail, 'exDividendDate'))).date())
+                lambda: dateutil.parser.parse(str(fmt(summaryDetail, 'exDividendDate')), yearfirst=True, dayfirst=False).date())
+
             tick[Datacode.PAYOUT_RATIO] = float(raw(summaryDetail, 'payoutRatio'))
             tick[Datacode.LOW_52_WEEK] = float(raw(summaryDetail, 'fiftyTwoWeekLow'))
             tick[Datacode.HIGH_52_WEEK] = float(raw(summaryDetail, 'fiftyTwoWeekHigh'))
             tick[Datacode.MARKET_CAP] = float(raw(summaryDetail, 'marketCap'))
 
-            tick[Datacode.TIMEZONE] = None
-            tick[Datacode.LAST_PRICE_DATE] = None
-            tick[Datacode.LAST_PRICE_TIME] = None
+            tick[Datacode.BID] = float(raw(summaryDetail, 'bid'))
+            tick[Datacode.ASK] = float(raw(summaryDetail, 'ask'))
+            tick[Datacode.BIDSIZE] = float(raw(summaryDetail, 'bidSize'))
+            tick[Datacode.ASKSIZE] = float(raw(summaryDetail, 'askSize'))
+
+            tick[Datacode.EXPIRY_DATE] = self.save_wrapper(
+                lambda: dateutil.parser.parse(str(fmt(summaryDetail, 'expireDate')), yearfirst=True, dayfirst=False).date())
 
             if quoteType:
                 t = int(price['regularMarketTime'])
@@ -222,9 +227,16 @@ class Yahoo(BaseClient):
                 tick[Datacode.LAST_PRICE_DATE] = dt.date()
                 tick[Datacode.LAST_PRICE_TIME] = dt.time()
 
-            tick[Datacode.TICKER] = str(price['symbol'])
-            tick[Datacode.EXCHANGE] = str(price['exchange'])
-            tick[Datacode.CURRENCY] = str(price['currency'])
+            tick[Datacode.TICKER] = self.save_wrapper(lambda: str(price['symbol']))
+            tick[Datacode.EXCHANGE] = self.save_wrapper(lambda: str(price['exchange']))
+            tick[Datacode.CURRENCY] = self.save_wrapper(lambda: str(price['currency']))
+
+            # some Moscow symbols miss currency in data block but show it in text e.g. VTBBA.ME, TBIOA.ME
+            if not tick[Datacode.CURRENCY]:
+                r = r'Currency in ([A-Z]{3})\b'
+                match = re.compile(r, flags=re.DOTALL).search(text)
+                if match:
+                    tick[Datacode.CURRENCY] = match.group(1)
 
             name = price['longName'] or price['shortName']
             if name:
@@ -232,13 +244,13 @@ class Yahoo(BaseClient):
             else:
                 tick[Datacode.NAME] = tick[Datacode.TICKER]
 
+            tick[Datacode.SECTOR] = self.save_wrapper(lambda: str(results['summaryProfile']['sector']))
+            tick[Datacode.INDUSTRY] = self.save_wrapper(lambda: str(results['summaryProfile']['industry']))
+
         except BaseException as e:
             logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
             return 'Yahoo.getRealtime({}, {}) - process: {}'.format(ticker, datacode, e)
 
-        tick[Datacode.SECTOR] = self.save_wrapper(lambda: str(results['summaryProfile']['sector']))
-        tick[Datacode.INDUSTRY] = self.save_wrapper(lambda: str(results['summaryProfile']['industry']))
-
         return self._return_value(self.realtime[ticker], datacode)
 
     def getHistoric(self, ticker: str, datacode: int, date):
@@ -262,6 +274,12 @@ class Yahoo(BaseClient):
         if Datacode.ADJ_CLOSE != datacode and ticker not in self.historicdata:
             self._read_ticker_csv_file(ticker)
 
+        try:
+            date_as_dt = dateutil.parser.parse(date, yearfirst=True, dayfirst=False)
+        except BaseException as e:
+            logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
+            return 'Yahoo.getHistoric({}, {}, {}) - date_as_dt: {}'.format(ticker, datacode, date, e)
+
         if ticker in self.historicdata:
             ticks = self.historicdata[ticker]
 
@@ -274,12 +292,12 @@ class Yahoo(BaseClient):
 
             # (potentially) future date
             if date > max(ticks):
-                t1 = int(dateutil.parser.parse(date).strftime('%s'))
+                t1 = int(date_as_dt.timestamp())
                 t2 = int(time.time())
                 if t1 > t2:
                     return 'Future date \'{}\''.format(date)
 
-                min_tick_date = int(dateutil.parser.parse(min(ticks)).strftime('%s'))  # remember current earliest date
+                min_tick_date = int(dateutil.parser.parse(min(ticks), yearfirst=True, dayfirst=False).timestamp())  # remember current earliest date
 
         if not self.crumb:
             self.getRealtime(ticker, datacode)
@@ -288,7 +306,7 @@ class Yahoo(BaseClient):
             return 'Yahoo.getHistoric({}, {}, {}) - crumb'.format(ticker, datacode, date)
 
         try:
-            t1 = int(dateutil.parser.parse(date).strftime('%s'))
+            t1 = int(date_as_dt.timestamp())
             t2 = int(time.time())
 
             if min_tick_date:
@@ -297,7 +315,7 @@ class Yahoo(BaseClient):
             if t1 >= t2:
                 return 'Future date \'{}\''.format(date)
 
-            if t1 < int(dateutil.parser.parse('2000-01-01').strftime('%s')):
+            if t1 < int(dateutil.parser.parse('2000-01-01', yearfirst=True, dayfirst=False).timestamp()):
                 return 'Date before 2000 \'{}\''.format(date)
 
             t1 = t1 - 2682000  # pad with extra month

src/generate_metainfo.py

@@ -14,7 +14,7 @@ import os
 cur_dir = os.getcwd()
 
 addin_id = "com.financials.getinfo"
-addin_version = "2.4.0"
+addin_version = "2.6.0"
 addin_displayname = "Financial Market Extension"
 addin_publisher_link = "https://github.com/cmallwitz/Financials-Extension"
 addin_publisher_name = "The Publisher"

src/test_ft.py

@@ -134,7 +134,7 @@ class Test(unittest.TestCase):
         self.assertEqual('iShares Core S&P 500 UCITS ETF USD (Acc)', s, 'test_UK_ETF NAME {}'.format(s))
 
         s = financials.getRealtime('C060:GER:EUR', 'NAME', 'FT')
-        self.assertEqual(str, type(s), 't_UK_ETF NAME {}'.format(s))
+        self.assertEqual(str, type(s), 'test_UK_ETF NAME {}'.format(s))
 
         s = financials.getRealtime('VERX:LSE:GBP', 'LAST_PRICE', 'FT')
         self.assertEqual(float, type(s), 'test_UK_ETF LAST_PRICE {}'.format(s))
@@ -142,6 +142,23 @@ class Test(unittest.TestCase):
         s = financials.getRealtime('VERX:LSE:GBP', 'TIMEZONE', 'FT')
         self.assertEqual(str, type(s), 'test_UK_ETF TIMEZONE {}'.format(s))
 
+    def test_UK_equity(self):
+        s = financials.getRealtime('VOD:LSE', 'NAME', 'FT')
+        self.assertEqual(str, type(s), 'test_UK_equity NAME {}'.format(s))
+        self.assertEqual('Vodafone Group PLC', s, 'test_UK_ETF NAME {}'.format(s))
+
+        s = financials.getRealtime('VOD:LSE', 'BID', 'FT')
+        self.assertEqual(float, type(s), 'test_UK_equity BID {}'.format(s))
+
+        s = financials.getRealtime('VOD:LSE', 'ASK', 'FT')
+        self.assertEqual(float, type(s), 'test_UK_equity ASK {}'.format(s))
+
+        s = financials.getRealtime('VOD:LSE', 'LAST_PRICE', 'FT')
+        self.assertEqual(float, type(s), 'test_UK_equity LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('VOD:LSE', 'TIMEZONE', 'FT')
+        self.assertEqual(str, type(s), 'test_UK_equity TIMEZONE {}'.format(s))
+
     def test_DE_equity(self):
         s = financials.getRealtime('SAPX:GER', 'NAME', 'FT')
         self.assertEqual('SAP SE', s, 'test_DE_equity NAME {}'.format(s))
@@ -270,6 +287,42 @@ class Test(unittest.TestCase):
         self.assertEqual(str, type(s), 'test_TY_equity INDUSTRY {}'.format(s))
         self.assertEqual('General Industrials', s, 'test_TY_equity INDUSTRY {}'.format(s))
 
+    def test_TLV_equity(self):
+        s = financials.getRealtime('DELT:TLV', 'LAST_PRICE', 'FT')
+        self.assertEqual(float, type(s), 'test_TlV_equity LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'OPEN', 'FT')
+        self.assertEqual(float, type(s), 'test_TlV_equity OPEN {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'LOW', 'FT')
+        self.assertEqual(float, type(s), 'test_TlV_equity LOW {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'HIGH', 'FT')
+        self.assertEqual(float, type(s), 'test_TlV_equity HIGH {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'LOW_52_WEEK', 'FT')
+        self.assertEqual(float, type(s), 'test_TlV_equity LOW_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'HIGH_52_WEEK', 'FT')
+        self.assertEqual(float, type(s), 'test_TlV_equity HIGH_52_WEEK {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'MARKET_CAP', 'FT')
+        self.assertEqual(float, type(s), 'test_TlV_equity MARKET_CAP {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'VOLUME', 'FT')
+        self.assertEqual(float, type(s), 'test_TlV_equity VOLUME {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'CURRENCY', 'FT')
+        self.assertEqual('ILa', s, 'test_TlV_equity CURRENCY')
+
+        s = financials.getRealtime('DELT:TLV', 'SECTOR', 'FT')
+        self.assertEqual(str, type(s), 'test_TlV_equity SECTOR {}'.format(s))
+        self.assertEqual('Consumer Goods', s, 'test_TY_equity SECTOR {}'.format(s))
+
+        s = financials.getRealtime('DELT:TLV', 'INDUSTRY', 'FT')
+        self.assertEqual(str, type(s), 'test_TlV_equity INDUSTRY {}'.format(s))
+        self.assertEqual('Personal Goods', s, 'test_TY_equity INDUSTRY {}'.format(s))
+
     def test_index(self):
         s = financials.getRealtime('INX:IOM', 'ticker', 'FT')
         self.assertEqual('INX:IOM', s, 'test_index TICKER {}'.format(s))

src/test_yahoo.py

@@ -123,6 +123,41 @@ class Test(unittest.TestCase):
         s = financials.getRealtime('VFIAX', Datacode.LAST_PRICE_TIME.value, 'YAHOO')
         self.assertEqual(str, type(s), 'test_realtime_US_mutuals LAST_PRICE_TIME {}'.format(s))
 
+    def test_realtime_US_options(self):
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.PREV_CLOSE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options PREV_CLOSE {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.NAME.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_options NAME {}'.format(s))
+        self.assertEqual('IBM Jun 2021 90.000 call', s, 'test_realtime_US_options NAME {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.EXPIRY_DATE.value, 'YAHOO')
+        self.assertEqual(str, type(s), 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
+        self.assertTrue(testutils.is_date(s), 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
+        self.assertEqual("2021-06-18", s, 'test_realtime_US_options EXPIRY_DATE {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.LAST_PRICE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options LAST_PRICE {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.OPEN.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options OPEN {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.VOLUME.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options VOLUME {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.BID.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options BID {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.ASK.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options ASK {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.BIDSIZE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options BIDSIZE {}'.format(s))
+
+        s = financials.getRealtime('IBM210618C00090000', Datacode.ASKSIZE.value, 'YAHOO')
+        self.assertEqual(float, type(s), 'test_realtime_US_options ASKSIZE {}'.format(s))
+
     def test_realtime_UK_ETF(self):
 
         s = financials.getRealtime('VERX.L', Datacode.LAST_PRICE.value, 'YAHOO')
@@ -200,7 +235,7 @@ class Test(unittest.TestCase):
         self.assertEqual('Not a trading day \'2017-01-01\'', s, 'test_historic_US_equity CLOSE {}'.format(s))
 
         s = financials.getHistoric('IBM', Datacode.LAST_PRICE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual('Data doesn\'t exist - 21', s, 'test_historic_US_equity LAST_PRICE {}'.format(s))
+        self.assertIsNone(s, 'test_historic_US_equity LAST_PRICE {}'.format(s))
 
         s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2017-01-03', 'YAHOO')
         self.assertEqual(167.190002, s, 'test_historic_US_equity CLOSE {}'.format(s))
@@ -224,7 +259,7 @@ class Test(unittest.TestCase):
 
         # Note: quarterly dividend and splits will change past adjusted prices - will fail after the next dividend
         s = financials.getHistoric('IBM', Datacode.ADJ_CLOSE.value, '2017-01-03', 'YAHOO')
-        self.assertEqual(139.61322, s, 'test_historic_US_equity ADJ_CLOSE {}'.format(s))
+        self.assertEqual(137.772202, s, 'test_historic_US_equity ADJ_CLOSE {}'.format(s))
 
     def test_historic_UK_ETF(self):
 
@@ -287,6 +322,9 @@ class Test(unittest.TestCase):
         s = financials.getRealtime('NO_NAME', Datacode.LAST_PRICE.value, 'YAHOO')
         self.assertIsNone(s, 'test_realtime_errors LAST_PRICE {}'.format(s))
 
+        s = financials.getRealtime('C060.DE', -1, 'YAHOO')
+        self.assertEqual('Datacode -1 not supported', s, 'test_realtime_errors -1 {}'.format(s))
+
     def test_historic_errors(self):
 
         s = financials.getHistoric('NO_NAME', Datacode.LAST_PRICE.value, '2018-01-08', 'YAHOO')
@@ -299,7 +337,7 @@ class Test(unittest.TestCase):
         self.assertEqual(s, 'Datacode 9999 not supported', 'test_historic_errors 9999')
 
         s = financials.getRealtime('IBM', Datacode.ADJ_CLOSE.value, 'YAHOO')
-        self.assertEqual(s, 'Data doesn\'t exist - 91', 'test_historic_errors ADJ_CLOSE {}'.format(s))
+        self.assertIsNone(s, 'test_historic_errors ADJ_CLOSE {}'.format(s))
 
         s = financials.getHistoric('IBM', Datacode.CLOSE.value, '2030-01-01', 'YAHOO')
         self.assertEqual(s, 'Future date \'2030-01-01\'', 'test_historic_errors CLOSE {}'.format(s))