{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Daily" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can specify devices for storage and calculation, such as the CPU or GPU. By default, data are created in the main memory and then use the CPU for calculations.\n", "\n", "The deep learning framework requires all input data for calculation to be on the same device, be it CPU or the same GPU.\n", "\n", "You can lose significant performance by moving data without care. A typical mistake is as follows: computing the loss for every minibatch on the GPU and reporting it back to the user on the command line (or logging it in a NumPy ndarray) will trigger a global interpreter lock which stalls all GPUs. It is much better to allocate memory for logging inside the GPU and only move larger logs.\n", "\n", "- for Tensorflow-2: You can just use LSTM with no activation specified (ied default to tanh) function and it will automatically use the CuDNN version\n", "- Gradient clipping is a technique to prevent exploding gradients in very deep networks, usually in recurrent neural networks. ... This prevents any gradient to have norm greater than the threshold and thus the gradients are clipped.\n", "\n", "### Tips for Activation Functions:\n", "- When using the ReLU function for hidden layers, it is a good practice to use a “He Normal” or “He Uniform” weight initialization and scale input data to the range 0-1 (normalize) prior to training.\n", "- When using the Sigmoid function for hidden layers, it is a good practice to use a “Xavier Normal” or “Xavier Uniform” weight initialization (also referred to Glorot initialization, named for Xavier Glorot) and scale input data to the range 0-1 (e.g. the range of the activation function) prior to training.\n", "- When using the TanH function for hidden layers, it is a good practice to use a “Xavier Normal” or “Xavier Uniform” weight initialization (also referred to Glorot initialization, named for Xavier Glorot) and scale input data to the range -1 to 1 (e.g. the range of the activation function) prior to training." ] }, { "cell_type": "code", "execution_count": 90, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "import tensorflow.keras\n", "from tensorflow.keras.utils import to_categorical, plot_model\n", "from tensorflow.keras.preprocessing.sequence import pad_sequences\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import LSTM, Dense, GRU, Embedding, Bidirectional, TimeDistributed, BatchNormalization, Flatten\n", "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard\n", "from tensorflow.keras.regularizers import l2\n", "from tqdm import tqdm\n", "# from keras_tqdm import TQDMNotebookCallback\n", "import tensorflow.keras.backend as K\n", "import os\n", "import time\n", "import pandas as pd\n", "import numpy as np\n", "import psutil\n", "# Ignore harmless warnings\n", "import warnings\n", "warnings.filterwarnings(\"ignore\")\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "sns.set_style('darkgrid')\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 91, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2.3.1\n", "2.4.0\n" ] } ], "source": [ "# tf.debugging.set_log_device_placement(True)\n", "print(tf.__version__)\n", "print(tensorflow.keras.__version__)" ] }, { "cell_type": "code", "execution_count": 92, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'/home/ec2-user/Models'" ] }, "execution_count": 92, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pwd" ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using GPU\n" ] } ], "source": [ "gpu_devices = tf.config.list_physical_devices('GPU')\n", "\n", "if gpu_devices:\n", " print('Using GPU')\n", " for gpu in gpu_devices[0:2]:\n", " tf.config.experimental.set_memory_growth(gpu, True)\n", "else:\n", " print('Using CPU')\n", " tf.config.optimizer.set_jit(True)\n", " print('used: {}% free: {:.2f}GB'.format(psutil.virtual_memory().percent, float(psutil.virtual_memory().free)/1024**3))#@ " ] }, { "cell_type": "code", "execution_count": 94, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "8 Physical GPU, 2 Logical GPUs\n" ] } ], "source": [ "gpus = tf.config.list_physical_devices('GPU')\n", "if gpus:\n", " # Restrict TensorFlow to only use some GPUs\n", " try:\n", " tf.config.experimental.set_visible_devices(gpus[0:2], 'GPU')\n", " logical_gpus = tf.config.experimental.list_logical_devices('GPU')\n", " print(len(gpus), \"Physical GPU,\", len(logical_gpus), \"Logical GPUs\")\n", " except RuntimeError as e:\n", " # Visible devices must be set at program startup\n", " print(e)" ] }, { "cell_type": "code", "execution_count": 95, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[LogicalDevice(name='/device:GPU:0', device_type='GPU'),\n", " LogicalDevice(name='/device:GPU:1', device_type='GPU')]" ] }, "execution_count": 95, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tf.config.experimental.list_logical_devices('GPU')" ] }, { "cell_type": "code", "execution_count": 96, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "used: 2.8% free: 351.21GB\n" ] } ], "source": [ "import psutil\n", "print('used: {}% free: {:.2f}GB'.format(psutil.virtual_memory().percent, float(psutil.virtual_memory().free)/1024**3))" ] }, { "cell_type": "code", "execution_count": 99, "metadata": {}, "outputs": [], "source": [ "# Prepare News headlines\n", "def clean_text(df, column):\n", " import re \n", " #(\"\".join(headline)).strip()\n", " headline = []\n", " for i in df[column].apply(lambda x: ''+x+'<\\s>'):\n", " headline.append(i)\n", " return headline\n", "\n", "#get sequences of equal length to ensure <\\s is at the end\n", "def extract_end(char_seq, seq_len):\n", " if len(char_seq) > seq_len:\n", " char_seq = char_seq[:seq_len] #char_seq[-seq_len:]\n", " return char_seq\n", "\n", "# Encode to integers by using ascii 128\n", "def encode2bytes(text):\n", " #text = tf.strings.unicode_split(text, 'UTF-8').to_list()\n", " final_list = []\n", " for sent in text:\n", " temp_list = []\n", " for char in sent:\n", " if ord(char) < 128 :\n", " temp_list.append(ord(char))\n", " final_list.append(temp_list)\n", " return final_list\n", "\n", "def split_X_y(text):\n", " X = []\n", " y = []\n", " for i in text:\n", " X.append(i[0:-1])\n", " y.append(i[1:])\n", " return X,y\n", "\n", "def to_bytes(sequence):\n", " byte_text = []\n", " for i in data.headline:\n", " i = i.encode('utf-8')\n", " byte_text.append(i)\n", " return byte_text" ] }, { "cell_type": "code", "execution_count": 100, "metadata": {}, "outputs": [], "source": [ "# fix random seed for reproducibility\n", "K.clear_session()\n", "tf.keras.backend.clear_session()\n", "np.random.seed(42)\n", "tf.random.set_seed(42)" ] }, { "cell_type": "code", "execution_count": 101, "metadata": {}, "outputs": [], "source": [ "idx = pd.IndexSlice\n", "max_length = 1000" ] }, { "cell_type": "code", "execution_count": 179, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['/model_data/15_min', '/model_data/daily', '/model_data/hourly']\n", "[1. 0.]\n" ] } ], "source": [ "# Get data\n", "with pd.HDFStore('./model_data.h5', mode = 'r') as data:\n", " print(data.keys())\n", " data = data['model_data/daily']\n", " data.headline = data.headline.apply(lambda x: extract_end(x, max_length))\n", " data['headline'] = data.headline.apply(lambda x: '' + x + '<\\s')\n", " X = data.loc[:,'headline']\n", " y = data.loc[:, 'label']\n", " y[y<0] = 0\n", " print(y.unique())" ] }, { "cell_type": "code", "execution_count": 180, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
headlineOpenClosereturnslabel
tickertime
A2019-07-11<s>Delivery Of Ionization Auxiliary Equipment ...1146.1600341144.0799560.0011011.0
2019-07-12<s>Epa Do Agilent 7250 Gc/q-tof Mass Spectrome...1142.9300541145.3399660.0045141.0
2019-07-16<s>AGILENT TECHNOLOGIES INC <A.N>: EVERCORE IS...1146.7299801153.459961-0.0058260.0
2019-07-17<s>Procurement Of Spares And Consumables For T...1150.9200441146.7399900.0004361.0
2019-07-18<s>NYSE ORDER IMBALANCE <A.N> 68900.0 SHARES O...1142.0000001147.239990-0.0136760.0
ZTS2020-09-28<s>NYSE ORDER IMBALANCE <ZTS.N> 59715.0 SHARES...162.380005161.3200070.0071911.0
2020-09-30<s>Zoetis to Host Webcast and Conference Call ...162.919998165.369995-0.0081030.0
2020-10-07<s>Sachem Head takes $1.2 bln position in Elan...163.350006159.9100040.0203861.0
2020-10-09<s>NYSE ORDER IMBALANCE <ZTS.N> 230640.0 SHARE...163.979996165.4299930.0188601.0
2020-10-12<s>ZOETIS INC <ZTS.N>: CREDIT SUISSE RAISES PR...167.080002168.550003-0.0191630.0
\n", "
" ], "text/plain": [ " headline \\\n", "ticker time \n", "A 2019-07-11 Delivery Of Ionization Auxiliary Equipment ... \n", " 2019-07-12 Epa Do Agilent 7250 Gc/q-tof Mass Spectrome... \n", " 2019-07-16 AGILENT TECHNOLOGIES INC : EVERCORE IS... \n", " 2019-07-17 Procurement Of Spares And Consumables For T... \n", " 2019-07-18 NYSE ORDER IMBALANCE 68900.0 SHARES O... \n", "ZTS 2020-09-28 NYSE ORDER IMBALANCE 59715.0 SHARES... \n", " 2020-09-30 Zoetis to Host Webcast and Conference Call ... \n", " 2020-10-07 Sachem Head takes $1.2 bln position in Elan... \n", " 2020-10-09 NYSE ORDER IMBALANCE 230640.0 SHARE... \n", " 2020-10-12 ZOETIS INC : CREDIT SUISSE RAISES PR... \n", "\n", " Open Close returns label \n", "ticker time \n", "A 2019-07-11 1146.160034 1144.079956 0.001101 1.0 \n", " 2019-07-12 1142.930054 1145.339966 0.004514 1.0 \n", " 2019-07-16 1146.729980 1153.459961 -0.005826 0.0 \n", " 2019-07-17 1150.920044 1146.739990 0.000436 1.0 \n", " 2019-07-18 1142.000000 1147.239990 -0.013676 0.0 \n", "ZTS 2020-09-28 162.380005 161.320007 0.007191 1.0 \n", " 2020-09-30 162.919998 165.369995 -0.008103 0.0 \n", " 2020-10-07 163.350006 159.910004 0.020386 1.0 \n", " 2020-10-09 163.979996 165.429993 0.018860 1.0 \n", " 2020-10-12 167.080002 168.550003 -0.019163 0.0 " ] }, "execution_count": 180, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.head().append(data.tail())" ] }, { "cell_type": "code", "execution_count": 183, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
headlineOpenClosereturnslabel
tickertime
A2020-10-01<s>AGILENT TECHNOLOGIES, INC. -- S-8<\\s101.769997101.220001-0.0119540.0
2020-10-02<s>The Replacement Of Sampler Handler Assembly...100.209999100.0100020.0310971.0
2020-10-08<s>Agilent Measurement Suite celebrates early ...104.199997104.1600040.0153611.0
2020-10-09<s>Spare Parts For Equipment Manufactured By A...104.830002105.760002-0.0031200.0
2020-10-13<s>NYSE ORDER IMBALANCE <A.N> 50070.0 SHARES O...105.720001105.419998-0.0034150.0
AAL2020-10-01<s>American Airlines says it will begin furlou...12.41000012.5800000.0333861.0
2020-10-02<s>Refinitiv Newscasts - Airlines push ahead w...12.05000013.0000000.0092311.0
2020-10-05<s>American Airlines Adds Costa Rica to Prefli...13.09000013.120000-0.0449700.0
2020-10-06<s>AMERICAN AIRLINES DELAYS PLAN TO START BOEI...13.27000012.5300000.0430971.0
2020-10-07<s>BUZZ-U.S. airlines rebound as Trump pushes ...12.96000013.0700000.0061211.0
2020-10-08<s>BUZZ-U.S. STOCKS ON THE MOVE-Eaton Vance, I...13.37000013.1500000.0038021.0
2020-10-09<s>MCCONNELL SAYS AIRLINE AID SHOULD BE PART O...13.25000013.200000-0.0212120.0
2020-10-12<s>Refinitiv Newscasts - Flying during the COV...13.14000012.920000-0.0541800.0
AAP2020-10-01<s>Hot Shot’s Secret Everyday Diesel Treatment...154.639999154.8500060.0041981.0
2020-10-02<s>Advance Auto Parts Names Mann + Hummel 2020...152.240005155.5000000.0023151.0
2020-10-06<s>ADVANCE AUTO PARTS INC <AAP.N>: JP MORGAN R...160.259995155.1600040.0106341.0
AAPL2020-10-01<s>Zuckerberg eyes augmented reality as the fu...117.629997116.790001-0.0322800.0
2020-10-02<s>Apple working on foldable iPhone with 'self...112.889999113.0199970.0307911.0
2020-10-05<s>Apple, Google tracing apps limited Google, ...113.910004116.500000-0.0286690.0
2020-10-06<s>BUZZ-Logitech shares slip on reports of App...115.800003113.1600040.0169671.0
2020-10-07<s>UPDATE 6-U.S. lawmakers detail Big Tech's m...114.699997115.080002-0.0009560.0
2020-10-08<s>Pomerantz Law Firm Achieves Victory on Beha...116.199997114.9700010.0173961.0
2020-10-09<s>Refinitiv Newscasts - Blockchain Interviews...115.279999116.9700010.0635211.0
2020-10-12<s>PRESS DIGEST- Financial Times - Oct. 12 APP...120.019997124.400002-0.0265270.0
2020-10-13<s>APPLE INC SAYS EVERY NEW IPHONE WILL FEATUR...125.320000121.0999980.0007431.0
ABBV2020-10-01<s>Public coverage of AbbVie's SKYRIZI® for th...88.00000087.139999-0.0117050.0
2020-10-02<s>AbbVie to Present New Data From 15 Abstract...86.50000086.1200030.0210171.0
2020-10-05<s>SHAREHOLDER ACTION REMINDER: The Schall Law...86.48999887.930000-0.0232000.0
2020-10-06<s>AbbVie to Host Third-Quarter 2020 Earnings ...87.97000185.8899990.0137391.0
2020-10-07<s>Allergan Aesthetics, an AbbVie Company, Acq...86.11000187.0700000.0031011.0
2020-10-08<s>Allergan, an AbbVie Company, and Von Miller...87.27999987.3399960.0041221.0
2020-10-09<s>Allergan Aesthetics To Present Data From 4 ...87.41000487.6999970.0070701.0
2020-10-12<s>AbbVie - Allergan Aesthetics to Present Dat...88.19999788.320000-0.0055480.0
2020-10-13<s>NIH SAYS TRIAL WILL TEST RISANKIZUMAB, IN C...87.91999887.830002-0.0200390.0
ABC2020-10-01<s>BRIEF-HHS Says Gilead Anticipates Producing...97.12999795.3600010.0002101.0
2020-10-05<s>AmerisourceBergen Corporation - ION & IPN S...95.76999796.389999-0.0078850.0
2020-10-06<s>AmerisourceBergen Announces Date and Time f...97.19999795.629997-0.0004180.0
2020-10-12<s>AMERISOURCEBERGEN CORP <ABC.N> : EVERCORE I...97.37999796.8600010.0042331.0
2020-10-13<s>MWI Animal Health Selects CVP Impack Automa...96.19999797.2699970.0120281.0
ABMD2020-10-08<s>TCT Connect to Highlight How Impella Enable...271.000000271.2000120.0087021.0
2020-10-09<s>ABIOMED INC -- SC 13G/A ABIOMED - TCT Conne...273.299988273.5599980.0156821.0
2020-10-12<s>ABIOMED INC <ABMD.O> : SVB LEERINK CUTS TAR...274.799988277.850006-0.0032390.0
ABT2020-10-01<s>Refinitiv Newscasts - Navigating the econom...109.180000108.639999-0.0196980.0
2020-10-02<s>Abbott's Libre 3 Receives CE Mark, Boosts C...107.680000106.5000000.0193431.0
2020-10-05<s>NYSE ORDER IMBALANCE <ABT.N> 227300.0 SHARE...107.180000108.559998-0.0212790.0
2020-10-06<s>CANADA HAS SIGNED A DEAL WITH ABBOTT RAPID ...108.559998106.2500000.0140241.0
2020-10-07<s>ABBOTT LABORATORIES <ABT.N> : WELLS FARGO R...107.360001107.7399980.0074251.0
2020-10-08<s>Allergy Immunotherapies Market - Actionable...108.199997108.5400010.0102271.0
2020-10-09<s>ABBOTT LABORATORIES <ABT.N> : JP MORGAN RAI...109.339996109.6500020.0127681.0
2020-10-12<s>REG - Allergy Therapeutics - Holding(s) in...110.250000111.050003-0.0241330.0
\n", "
" ], "text/plain": [ " headline \\\n", "ticker time \n", "A 2020-10-01 AGILENT TECHNOLOGIES, INC. -- S-8<\\s \n", " 2020-10-02 The Replacement Of Sampler Handler Assembly... \n", " 2020-10-08 Agilent Measurement Suite celebrates early ... \n", " 2020-10-09 Spare Parts For Equipment Manufactured By A... \n", " 2020-10-13 NYSE ORDER IMBALANCE 50070.0 SHARES O... \n", "AAL 2020-10-01 American Airlines says it will begin furlou... \n", " 2020-10-02 Refinitiv Newscasts - Airlines push ahead w... \n", " 2020-10-05 American Airlines Adds Costa Rica to Prefli... \n", " 2020-10-06 AMERICAN AIRLINES DELAYS PLAN TO START BOEI... \n", " 2020-10-07 BUZZ-U.S. airlines rebound as Trump pushes ... \n", " 2020-10-08 BUZZ-U.S. STOCKS ON THE MOVE-Eaton Vance, I... \n", " 2020-10-09 MCCONNELL SAYS AIRLINE AID SHOULD BE PART O... \n", " 2020-10-12 Refinitiv Newscasts - Flying during the COV... \n", "AAP 2020-10-01 Hot Shot’s Secret Everyday Diesel Treatment... \n", " 2020-10-02 Advance Auto Parts Names Mann + Hummel 2020... \n", " 2020-10-06 ADVANCE AUTO PARTS INC : JP MORGAN R... \n", "AAPL 2020-10-01 Zuckerberg eyes augmented reality as the fu... \n", " 2020-10-02 Apple working on foldable iPhone with 'self... \n", " 2020-10-05 Apple, Google tracing apps limited Google, ... \n", " 2020-10-06 BUZZ-Logitech shares slip on reports of App... \n", " 2020-10-07 UPDATE 6-U.S. lawmakers detail Big Tech's m... \n", " 2020-10-08 Pomerantz Law Firm Achieves Victory on Beha... \n", " 2020-10-09 Refinitiv Newscasts - Blockchain Interviews... \n", " 2020-10-12 PRESS DIGEST- Financial Times - Oct. 12 APP... \n", " 2020-10-13 APPLE INC SAYS EVERY NEW IPHONE WILL FEATUR... \n", "ABBV 2020-10-01 Public coverage of AbbVie's SKYRIZI® for th... \n", " 2020-10-02 AbbVie to Present New Data From 15 Abstract... \n", " 2020-10-05 SHAREHOLDER ACTION REMINDER: The Schall Law... \n", " 2020-10-06 AbbVie to Host Third-Quarter 2020 Earnings ... \n", " 2020-10-07 Allergan Aesthetics, an AbbVie Company, Acq... \n", " 2020-10-08 Allergan, an AbbVie Company, and Von Miller... \n", " 2020-10-09 Allergan Aesthetics To Present Data From 4 ... \n", " 2020-10-12 AbbVie - Allergan Aesthetics to Present Dat... \n", " 2020-10-13 NIH SAYS TRIAL WILL TEST RISANKIZUMAB, IN C... \n", "ABC 2020-10-01 BRIEF-HHS Says Gilead Anticipates Producing... \n", " 2020-10-05 AmerisourceBergen Corporation - ION & IPN S... \n", " 2020-10-06 AmerisourceBergen Announces Date and Time f... \n", " 2020-10-12 AMERISOURCEBERGEN CORP : EVERCORE I... \n", " 2020-10-13 MWI Animal Health Selects CVP Impack Automa... \n", "ABMD 2020-10-08 TCT Connect to Highlight How Impella Enable... \n", " 2020-10-09 ABIOMED INC -- SC 13G/A ABIOMED - TCT Conne... \n", " 2020-10-12 ABIOMED INC : SVB LEERINK CUTS TAR... \n", "ABT 2020-10-01 Refinitiv Newscasts - Navigating the econom... \n", " 2020-10-02 Abbott's Libre 3 Receives CE Mark, Boosts C... \n", " 2020-10-05 NYSE ORDER IMBALANCE 227300.0 SHARE... \n", " 2020-10-06 CANADA HAS SIGNED A DEAL WITH ABBOTT RAPID ... \n", " 2020-10-07 ABBOTT LABORATORIES : WELLS FARGO R... \n", " 2020-10-08 Allergy Immunotherapies Market - Actionable... \n", " 2020-10-09 ABBOTT LABORATORIES : JP MORGAN RAI... \n", " 2020-10-12 REG - Allergy Therapeutics - Holding(s) in... \n", "\n", " Open Close returns label \n", "ticker time \n", "A 2020-10-01 101.769997 101.220001 -0.011954 0.0 \n", " 2020-10-02 100.209999 100.010002 0.031097 1.0 \n", " 2020-10-08 104.199997 104.160004 0.015361 1.0 \n", " 2020-10-09 104.830002 105.760002 -0.003120 0.0 \n", " 2020-10-13 105.720001 105.419998 -0.003415 0.0 \n", "AAL 2020-10-01 12.410000 12.580000 0.033386 1.0 \n", " 2020-10-02 12.050000 13.000000 0.009231 1.0 \n", " 2020-10-05 13.090000 13.120000 -0.044970 0.0 \n", " 2020-10-06 13.270000 12.530000 0.043097 1.0 \n", " 2020-10-07 12.960000 13.070000 0.006121 1.0 \n", " 2020-10-08 13.370000 13.150000 0.003802 1.0 \n", " 2020-10-09 13.250000 13.200000 -0.021212 0.0 \n", " 2020-10-12 13.140000 12.920000 -0.054180 0.0 \n", "AAP 2020-10-01 154.639999 154.850006 0.004198 1.0 \n", " 2020-10-02 152.240005 155.500000 0.002315 1.0 \n", " 2020-10-06 160.259995 155.160004 0.010634 1.0 \n", "AAPL 2020-10-01 117.629997 116.790001 -0.032280 0.0 \n", " 2020-10-02 112.889999 113.019997 0.030791 1.0 \n", " 2020-10-05 113.910004 116.500000 -0.028669 0.0 \n", " 2020-10-06 115.800003 113.160004 0.016967 1.0 \n", " 2020-10-07 114.699997 115.080002 -0.000956 0.0 \n", " 2020-10-08 116.199997 114.970001 0.017396 1.0 \n", " 2020-10-09 115.279999 116.970001 0.063521 1.0 \n", " 2020-10-12 120.019997 124.400002 -0.026527 0.0 \n", " 2020-10-13 125.320000 121.099998 0.000743 1.0 \n", "ABBV 2020-10-01 88.000000 87.139999 -0.011705 0.0 \n", " 2020-10-02 86.500000 86.120003 0.021017 1.0 \n", " 2020-10-05 86.489998 87.930000 -0.023200 0.0 \n", " 2020-10-06 87.970001 85.889999 0.013739 1.0 \n", " 2020-10-07 86.110001 87.070000 0.003101 1.0 \n", " 2020-10-08 87.279999 87.339996 0.004122 1.0 \n", " 2020-10-09 87.410004 87.699997 0.007070 1.0 \n", " 2020-10-12 88.199997 88.320000 -0.005548 0.0 \n", " 2020-10-13 87.919998 87.830002 -0.020039 0.0 \n", "ABC 2020-10-01 97.129997 95.360001 0.000210 1.0 \n", " 2020-10-05 95.769997 96.389999 -0.007885 0.0 \n", " 2020-10-06 97.199997 95.629997 -0.000418 0.0 \n", " 2020-10-12 97.379997 96.860001 0.004233 1.0 \n", " 2020-10-13 96.199997 97.269997 0.012028 1.0 \n", "ABMD 2020-10-08 271.000000 271.200012 0.008702 1.0 \n", " 2020-10-09 273.299988 273.559998 0.015682 1.0 \n", " 2020-10-12 274.799988 277.850006 -0.003239 0.0 \n", "ABT 2020-10-01 109.180000 108.639999 -0.019698 0.0 \n", " 2020-10-02 107.680000 106.500000 0.019343 1.0 \n", " 2020-10-05 107.180000 108.559998 -0.021279 0.0 \n", " 2020-10-06 108.559998 106.250000 0.014024 1.0 \n", " 2020-10-07 107.360001 107.739998 0.007425 1.0 \n", " 2020-10-08 108.199997 108.540001 0.010227 1.0 \n", " 2020-10-09 109.339996 109.650002 0.012768 1.0 \n", " 2020-10-12 110.250000 111.050003 -0.024133 0.0 " ] }, "execution_count": 183, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.loc[idx[:,'2020-10-01':], ].head(50)" ] }, { "cell_type": "code", "execution_count": 104, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "MultiIndex: 110905 entries, ('A', Timestamp('2019-07-11 00:00:00')) to ('ZTS', Timestamp('2020-10-12 00:00:00'))\n", "Data columns (total 5 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 headline 110905 non-null object \n", " 1 Open 110905 non-null float64\n", " 2 Close 110905 non-null float64\n", " 3 returns 110905 non-null float64\n", " 4 label 110905 non-null float64\n", "dtypes: float64(4), object(1)\n", "memory usage: 4.7+ MB\n" ] } ], "source": [ "data.info()" ] }, { "cell_type": "code", "execution_count": 105, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
Open110905.0179.9496531195.9368223.22000041.50999880.650002141.72999629440.470703
Close110905.0179.9708241195.7482593.02000041.47000180.570000141.72999629423.310547
returns110905.00.0005670.032472-0.998369-0.0113050.0009740.0125530.429971
label110905.00.5270460.4992700.0000000.0000001.0000001.0000001.000000
n_Characters110905.0297.054542298.79255611.00000078.000000164.000000390.0000001006.000000
\n", "
" ], "text/plain": [ " count mean std min 25% \\\n", "Open 110905.0 179.949653 1195.936822 3.220000 41.509998 \n", "Close 110905.0 179.970824 1195.748259 3.020000 41.470001 \n", "returns 110905.0 0.000567 0.032472 -0.998369 -0.011305 \n", "label 110905.0 0.527046 0.499270 0.000000 0.000000 \n", "n_Characters 110905.0 297.054542 298.792556 11.000000 78.000000 \n", "\n", " 50% 75% max \n", "Open 80.650002 141.729996 29440.470703 \n", "Close 80.570000 141.729996 29423.310547 \n", "returns 0.000974 0.012553 0.429971 \n", "label 1.000000 1.000000 1.000000 \n", "n_Characters 164.000000 390.000000 1006.000000 " ] }, "execution_count": 105, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['n_Characters'] = data['headline'].str.len()\n", "data.describe().T" ] }, { "cell_type": "code", "execution_count": 106, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((110905,), (110905,))" ] }, "execution_count": 106, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X.shape, y.shape" ] }, { "cell_type": "code", "execution_count": 107, "metadata": {}, "outputs": [], "source": [ "# Train / Test / Infer: Test which is Validation here: 2019-07-08--->2019-08-01, Infer on \n", "X_train, X_test, X_infer = X.loc[idx[:,'2019-08-01':'2020-10-01'], ], X.loc[idx[:,:'2019-08-01'], ], X.loc[idx[:,'2020-10-01':], ]\n", "y_train, y_test, y_infer = y.loc[idx[:,'2019-08-01':'2020-10-01'], ], y.loc[idx[:,:'2019-08-01'], ], y.loc[idx[:,'2020-10-01':], ]" ] }, { "cell_type": "code", "execution_count": 108, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((103065,), (5697,), (103065,), (5697,), (2874,), (2874,))" ] }, "execution_count": 108, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train.shape, X_test.shape, y_train.shape, y_test.shape, X_infer.shape, y_infer.shape" ] }, { "cell_type": "code", "execution_count": 109, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "y labels:\n", " 1.0 58452\n", "0.0 52453\n", "Name: label, dtype: int64\n", "Train Labels:\n", " 1.0 54331\n", "0.0 48734\n", "Name: label, dtype: int64\n", "Test Labels:\n", " 0.0 3009\n", "1.0 2688\n", "Name: label, dtype: int64\n", "Inference Labels:\n", " 1.0 1753\n", "0.0 1121\n", "Name: label, dtype: int64\n", "Train Ratio 0.93\n" ] } ], "source": [ "print('y labels:\\n', y.value_counts())\n", "print('Train Labels:\\n', y_train.value_counts())\n", "print('Test Labels:\\n', y_test.value_counts())\n", "print('Inference Labels:\\n', y_infer.value_counts())\n", "print(\"Train Ratio %.2f\" % (len(X_train)/ len(X)))" ] }, { "cell_type": "code", "execution_count": 110, "metadata": {}, "outputs": [], "source": [ "# ENCODE ORDINAL\n", "X_train = [i.encode('utf-8') for i in X_train]\n", "X_test = [i.encode('utf-8') for i in X_test]\n", "X_infer = [i.encode('utf-8') for i in X_infer]\n", "# X_encode = encode2bytes(X)" ] }, { "cell_type": "code", "execution_count": 111, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[b'Agilent Settles Key Intellectual Property Case in China AGILENT SETTLES KEY INTELLECTUAL PROPERTY CASE IN CHINA Agilent Settles Key Intellectual Property Case in China EChrom and Pannatek Admit to using Agilent Technology Without Permission, Agree to Pay Damages and Cease Using Agilent Technology AGILENT TECHNOLOGIES- CO REACHED AGREEMENT WITH ECHROM AND PANNATEK, AS WELL AS CERTAIN FORMER AGILENT EMPLOYEES, REGARDING AN INTELLECTUAL PROPERTY DISPUTE BRIEF-Agilent Settles Key Intellectual Property Case In China Agilent Companion Diagnostic Gains Expanded FDA Approval in Esophageal Squamous Cell Carcinoma<\\\\s',\n", " b'NYSE ORDER IMBALANCE 65800.0 SHARES ON BUY SIDE<\\\\s',\n", " b'AGILENT TECHNOLOGIES INC SEC Filings files Form -- 4<\\\\s',\n", " b'Turbo Pump Controller (model No. X3506 64002 Make Agilent NYSE ORDER IMBALANCE 51900.0 SHARES ON SELL SIDE<\\\\s',\n", " b'Supplying Of Spares With Fitting And Fixing For 70 Nos. Tri-cycle Paddle Van In Ward Nos.-63,64,65,66 & 67, Under A.d / Br.-vii / Swm-i. Agilent Technologies (A) Earnings Expected to Grow: Should You Buy?<\\\\s']" ] }, "execution_count": 111, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train[0:5]" ] }, { "cell_type": "code", "execution_count": 112, "metadata": {}, "outputs": [], "source": [ "#Tokenize\n", "import json\n", "with open('Tokenizer.json', encoding='utf-8') as f:\n", " data = json.load(f)\n", " tokenizer = tf.keras.preprocessing.text.tokenizer_from_json(data)\n", "with open('index2char.json', encoding='utf-8') as f:\n", " index2char = json.load(f)\n", "char2index = dict((int(v),int(k)) for k,v in index2char.items())\n", "tokenizer.word_index = char2index\n", "# with open('index2char.json', 'w', encoding='utf-8') as f:\n", "# json.dump(index2char, f, ensure_ascii=False, indent=4) \n" ] }, { "cell_type": "code", "execution_count": 125, "metadata": {}, "outputs": [], "source": [ "X_train = tokenizer.texts_to_sequences(X_train)\n", "X_test = tokenizer.texts_to_sequences(X_test)\n", "X_infer = tokenizer.texts_to_sequences(X_infer)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### How to convert back to readable form:" ] }, { "cell_type": "code", "execution_count": 158, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ 0 0 0 ... 31 63 86]\n", "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000Agilent Settles Key Intellectual Property Case in China AGILENT SETTLES KEY INTELLECTUAL PROPERTY CASE IN CHINA Agilent Settles Key Intellectual Property Case in China EChrom and Pannatek Admit to using Agilent Technology Without Permission, Agree to Pay Damages and Cease Using Agilent Technology AGILENT TECHNOLOGIES- CO REACHED AGREEMENT WITH ECHROM AND PANNATEK, AS WELL AS CERTAIN FORMER AGILENT EMPLOYEES, REGARDING AN INTELLECTUAL PROPERTY DISPUTE BRIEF-Agilent Settles Key Intellectual Property Case In China Agilent Companion Diagnostic Gains Expanded FDA Approval in Esophageal Squamous Cell Carcinoma<\\s\n" ] } ], "source": [ "print(X_train[0])\n", "print(bytes(list(map(index2char.get, X_train[0]))).decode('utf-8'))" ] }, { "cell_type": "code", "execution_count": 130, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Headlines length: \n", "Mean 297.05 words (Std 298.79) max 1006\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABIoAAAI/CAYAAAAYxjIJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZoklEQVR4nO3dYajvd33Y8ffpva7b7XJ71J7jQpJNR9OvilC2hioriNTRaSaLD6p1GxpdhlC6rpuD1Y2BsO2BhTGXB0UopmuEUnW2YB7ISokdYQ8MXcJA1vgtobPJNZnXNTlmcB8Yr2cP8k130XNv3Dkn539PfL3gcv//7+/3P/9PDoH758339/tv7e/vBwAAAAA/sOkBAAAAALg+CEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVHV20wNcy7e//e39y5f3Nz0GAMALOnNmK59bAIDT4GUvO/O/q52Djl3Xoejy5f329i5tegwAgBe0vX3O5xYA4FTY2bnhT652zKVnAAAAAFRCEQAAAACLUAQAAABAJRQBAAAAsAhFAAAAAFRCEQAAAACLUAQAAABAJRQBAAAAsAhFAAAAAFRCEQAAAACLUAQAAABAJRQBAAAAsAhFAAAAAFRCEQAAAACLUAQAAABAJRQBAAAAsAhFAAAAAFRCEQAAAACLUAQAAABAJRQBAAAAsJzd9AAAAKfZ7u7571q7ePGZDUwCAHB0dhQBABzSQZHoWusAANe7F9xRNMb49eod1cU55xvW2iuqT1evrr5SvXvO+fQYY6u6u7q9ulS9f8758HrNndW/Wj/238457z3e/xQAAAAAjuJ72VH0G9XbvmPtw9X9c85bq/vX86q3V7euPx+sPl5/FpY+Ur2x+snqI2OMlx91eAAAAACOzwuGojnnA9VT37F8R/X8jqB7q3desf7JOef+nPOL1fYY48bqb1W/N+d8as75dPV7fXd8AgAAAGCDDnuPolfNOZ+sWn/vrvWbqsevOO/CWrvaOgAAAADXieP+1rOtA9b2r7F+TWfObLW9fe7IQwEAnDSfYQCA0+iwoehrY4wb55xPrkvLLq71C9UtV5x3c/XEWn/Ld6z/lxd6k8uX99vbu3TIEQEANsdnGADgerWzc8NVjx320rP7qjvX4zurz12x/r4xxtYY403VN9alab9b/cwY4+XrJtY/s9YAAAAAuE684I6iMcZv9dxuoB8ZY1zouW8v+2j1mTHGXdVj1bvW6Z+vbq8erS5VH6iacz41xvg31R+s8/71nPM7b5ANAAAAwAZt7e+/4K2CNubZZy/v27YNAFyvdnfPX/XYxYvPnOAkAADfu52dGx6qbjvo2GEvPQMAAADgJUYoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgOXspgcAALiWN7/5jX35y49seoz/b7u75zc9woFe+9rX9cADD256DADgOiUUAQDXtes5alwrBl28+MwJTgIAcDxcegYAAABAJRQBABza1XYN2U0EAJxWLj0DADiC56PQ7u55gQgAOPXsKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIDl7FFePMb4p9U/rParL1UfqG6sPlW9onq4eu+c85tjjB+sPln9RPWn1c/NOb9ylPcHAAAA4PgcekfRGOOm6h9Xt80531Cdqd5T/Ur1sTnnrdXT1V3rJXdVT885f7T62DoPAAAAgOvEUS89O1v9hTHG2epc9WT109Vn1/F7q3eux3es563jbx1jbB3x/QEAAAA4JocORXPOr1b/rnqs5wLRN6qHqr0557fWaReqm9bjm6rH12u/tc5/5WHfHwAAAIDjdeh7FI0xXt5zu4ReU+1V/6l6+wGn7q+/D9o9tH/A2p85c2ar7e1zhx0RAOBE+dwCAJx2R7mZ9d+s/uec8+tVY4zfqf5GtT3GOLt2Dd1cPbHOv1DdUl1Yl6r9cPXUtd7g8uX99vYuHWFEAICT43MLAHAa7OzccNVjR7lH0WPVm8YY59a9ht5a/WH1+9XPrnPurD63Ht+3nreOf2HOec0dRQAAAACcnKPco+jBnrsp9cPVl9bP+rXql6sPjTEe7bl7EN2zXnJP9cq1/qHqw0eYGwAAAIBjtrW/f/1u6nn22cv7tnADAKfB7u75Ll58ZtNjAAC8oJ2dGx6qbjvo2FEuPQMAAADgJUQoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYzh7lxWOM7eoT1Ruq/eofVLP6dPXq6ivVu+ecT48xtqq7q9urS9X755wPH+X9AQAAADg+R91RdHf1n+ecr61+vHqk+nB1/5zz1ur+9bzq7dWt688Hq48f8b0BAAAAOEaHDkVjjPPVm6t7quac35xz7lV3VPeu0+6t3rke31F9cs65P+f8YrU9xrjx0JMDAAAAcKyOcunZX62+Xv3HMcaPVw9Vv1S9as75ZNWc88kxxu46/6bq8Stef2GtPXmEGQAAAAA4JkcJRWerv1794pzzwTHG3f2/y8wOsnXA2v613uDMma22t88dYUQAgJPjcwsAcNodJRRdqC7MOR9czz/bc6Hoa2OMG9duohuri1ecf8sVr7+5euJab3D58n57e5eOMCIAwMnxuQUAOA12dm646rFD36Nozvm/qsfHGGMtvbX6w+q+6s61dmf1ufX4vup9Y4ytMcabqm88f4kaAAAAAJt3lB1FVb9Y/eYY489Vf1x9oOfi02fGGHdVj1XvWud+vrq9erS6tM4FAAAA4Dqxtb9/zdsEbdSzz17et4UbADgNdnfPd/HiM5seAwDgBe3s3PBQddtBxw596RkAAAAALy1CEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAACVUAQAAADAIhQBAAAAUAlFAAAAACxCEQAAAABVnT3qDxhjnKn+W/XVOec7xhivqT5VvaJ6uHrvnPObY4wfrD5Z/UT1p9XPzTm/ctT3BwAAAOB4HMeOol+qHrni+a9UH5tz3lo9Xd211u+qnp5z/mj1sXUeAAAAANeJI4WiMcbN1d+uPrGeb1U/XX12nXJv9c71+I71vHX8ret8AAAAAK4DR91R9B+qf159ez1/ZbU35/zWen6humk9vql6vGod/8Y6HwAAAIDrwKHvUTTGeEd1cc750BjjLWv5oB1C+9/DsQOdObPV9va5w44IAHCifG4BAE67o9zM+qeqvzPGuL3689X5ntthtD3GOLt2Dd1cPbHOv1DdUl0YY5ytfrh66lpvcPnyfnt7l44wIgDAyfG5BQA4DXZ2brjqsUNfejbn/BdzzpvnnK+u3lN9Yc7596vfr352nXZn9bn1+L71vHX8C3POa+4oAgAAAODkHMe3nn2nX64+NMZ4tOfuQXTPWr+neuVa/1D14RfhvQEAAAA4pK39/et3U8+zz17et4UbADgNdnfPd/HiM5seAwDgBe3s3PBQddtBx45yjyIA4JT7sR/7y+3t7W16jJeM3d3zmx7hJWN7e7s/+qPHNj0GAHzfEYoA4PvY3t6eXTDHZHv7nJtZHyPRDQA248W4RxEAAAAAp5BQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVHX2sC8cY9xSfbL6S9W3q1+bc949xnhF9enq1dVXqnfPOZ8eY2xVd1e3V5eq9885Hz7a+AAAAAAcl6PsKPpW9c/mnK+r3lT9whjj9dWHq/vnnLdW96/nVW+vbl1/Plh9/AjvDQAAAMAxO3QomnM++fyOoDnn/6keqW6q7qjuXafdW71zPb6j+uScc3/O+cVqe4xx46EnBwAAAOBYHcs9isYYr67+WvVg9ao555P1XEyqdtdpN1WPX/GyC2sNAAAAgOvAoe9R9Lwxxl+sfrv6J3POZ8YYVzt164C1/Wv97DNnttrePnfECQGAa/Fv7fE4c+YH/C6Pmd8nAJy8I4WiMcbLei4S/eac83fW8tfGGDfOOZ9cl5ZdXOsXqluuePnN1RPX+vmXL++3t3fpKCMCAC/Av7XHY3v7nN/lMfP7BIAXx87ODVc9duhLz9a3mN1TPTLn/PdXHLqvunM9vrP63BXr7xtjbI0x3lR94/lL1AAAAADYvKPsKPqp6r3Vl8YY/32t/cvqo9Vnxhh3VY9V71rHPl/dXj1aXao+cIT3BgAAAOCYHToUzTn/awffd6jqrQecv1/9wmHfDwAAAIAX17F86xkAAAAAp59QBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAItQBAAAAEAlFAEAAACwCEUAAAAAVEIRAAAAAMvZTQ8AAGzOl37+h9r51Zs3PcZLxs6mB3gJ+dLP/9CmRwCA70tb+/v7m57hqp599vL+3t6lTY8BAC9Zu7vnu3jxmU2P8ZKwvX0un1uOj/83AeDFs7Nzw0PVbQcdc+kZAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAJVQBAAAAMAiFAEAAABQCUUAAAAALEIRAAAAAFWd3fQAAMBm7e6e3/QI8F22t7c3PQIAfF8SigDg+9jFi89seoSXjN3d836fAMCp59IzAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgEooAAAAAWIQiAAAAACqhCAAAAIBFKAIAAACgqrObHgAA4Fre/OY39uUvP7LpMb4nu7vnNz3CC3rta1/XAw88uOkxAIDrlFAEAFzXTkvU2N4+197epU2PAQBwJCceisYYb6vurs5Un5hzfvSkZwAAAADgu53oPYrGGGeqX63eXr2++rtjjNef5AwAAAAAHOykb2b9k9Wjc84/nnN+s/pUdccJzwAAAADAAU46FN1UPX7F8wtrDQAAAIANO+l7FG0dsLZ/tZPPnNlqe/vcizgOAMDxOHPmB3xuAQBOvZMORReqW654fnP1xNVOvnx537eHAACngm89AwBOi52dG6567KRD0R9Ut44xXlN9tXpP9fdOeAYAAAAADnCi9yiac36r+kfV71aPVJ+Zc/6Pk5wBAAAAgIOd9I6i5pyfrz5/0u8LAAAAwLWd9LeeAQAAAHCdEooAAAAAqIQiAAAAABahCAAAAIBKKAIAAABgEYoAAAAAqIQiAAAAABahCAAAAIBKKAIAAABgEYoAAAAAqIQiAAAAABahCAAAAIBKKAIAAABgEYoAAAAAqGprf39/0zNcy9erP9n0EAAAAAAvIX+l2jnowPUeigAAAAA4IS49AwAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKCqs5seAADgNBtj/Hr1jurinPMNm54HAOAo7CgCADia36jetukhAACOg1AEAHAEc84Hqqc2PQcAwHEQigAAAACohCIAAAAAFqEIAAAAgEooAgAAAGDZ2t/f3/QMAACn1hjjt6q3VD9Sfa36yJzzno0OBQBwSEIRAAAAAJVLzwAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKASigAAAABYhCIAAAAAKqEIAAAAgEUoAgAAAKCq/wuK/ytS/i3V9wAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Summarize headlines length\n", "print(\"Headlines length: \")\n", "result = [len(sentence) for sentence in X]\n", "print(\"Mean %.2f words (Std %.2f) max %d\" % (np.mean(result), np.std(result), max(result)))\n", "# plot review length\n", "plt.figure(figsize=(20,10));\n", "plt.boxplot(result)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 131, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((103065, 1116), (5697, 1023), (2874, 1032))" ] }, "execution_count": 131, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# prepadded to solve for forward propagation not masking\n", "max_sentence_len = max(map(len, X))\n", "X_train = pad_sequences(X_train, maxlen = max(map(len, X_train)), padding = 'pre', truncating='pre')\n", "X_test = pad_sequences(X_test, maxlen = max(map(len, X_test)), padding = 'pre', truncating='pre')\n", "X_infer = pad_sequences(X_infer, maxlen = max(map(len, X_infer)), padding = 'pre', truncating='pre')\n", "X_train.shape, X_test.shape, X_infer.shape\n", "# X_padded = pad_sequences(X_encode, maxlen = max_sentence_len, padding = 'pre', truncating='pre')" ] }, { "cell_type": "code", "execution_count": 132, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((103065, 1), (5697, 1), (2874, 1))" ] }, "execution_count": 132, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Our vectorized labels\n", "train_labels = np.asarray(y_train).astype('float32').reshape((-1,1))\n", "test_labels = np.asarray(y_test).astype('float32').reshape((-1,1))\n", "infer_labels = np.asarray(y_infer).astype('float32').reshape((-1,1))\n", "train_labels.shape, test_labels.shape, infer_labels.shape" ] }, { "cell_type": "code", "execution_count": 133, "metadata": {}, "outputs": [], "source": [ "train_seq_data = tf.data.Dataset.from_tensor_slices((X_train,train_labels))\n", "test_seq_data = tf.data.Dataset.from_tensor_slices((X_test,test_labels))\n", "infer_seq_data = tf.data.Dataset.from_tensor_slices((X_infer,infer_labels))" ] }, { "cell_type": "code", "execution_count": 134, "metadata": {}, "outputs": [], "source": [ "AUTOTUNE = tf.data.experimental.AUTOTUNE\n", "\n", "def configure_dataset(dataset):\n", " return dataset.cache().prefetch(buffer_size=AUTOTUNE)" ] }, { "cell_type": "code", "execution_count": 135, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train Set Shape: \n", "Test Set Shape: \n" ] } ], "source": [ "batch_size = 64\n", "\n", "train_seq_data = train_seq_data.batch(batch_size, drop_remainder=True)\n", "test_seq_data = test_seq_data.batch(batch_size, drop_remainder=True)\n", "infer_seq_data = infer_seq_data.batch(64, drop_remainder=True)\n", "print('Train Set Shape: ', train_seq_data, '\\nTest Set Shape: ', test_seq_data)" ] }, { "cell_type": "code", "execution_count": 136, "metadata": {}, "outputs": [], "source": [ "# with tf.device('CPU'):\n", "train_seq_data = configure_dataset(train_seq_data)\n", "test_seq_data = configure_dataset(test_seq_data)\n", "infer_seq_data = configure_dataset(infer_seq_data)" ] }, { "cell_type": "code", "execution_count": 137, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(,\n", " ,\n", " )" ] }, "execution_count": 137, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_seq_data, test_seq_data, infer_seq_data" ] }, { "cell_type": "code", "execution_count": 138, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"CharLSTM\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "EmbedLayer (Embedding) (128, None, 256) 47360 \n", "_________________________________________________________________\n", "BiLSTM (Bidirectional) (128, None, 2048) 10493952 \n", "_________________________________________________________________\n", "time_distributed (TimeDistri (None, None, 185) 379065 \n", "=================================================================\n", "Total params: 10,920,377\n", "Trainable params: 10,920,377\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "# def create_language_model(batch_size):\n", "# model = Sequential(name = 'CharLSTM')\n", "# model.add(Embedding(127, 256,batch_input_shape=[batch_size, None], \n", "# mask_zero=True, name ='EmbedLayer'))\n", "# model.add(Bidirectional(LSTM(1024, return_sequences=True,stateful=False,\n", "# recurrent_initializer='glorot_uniform'), merge_mode ='ave',name = 'BiLSTM'))\n", "# model.add(TimeDistributed(Dense(127, name = 'TimeDistDense')))\n", "# model.compile(optimizer=tf.optimizers.SGD(learning_rate=1e-1), \n", "# loss = tf.losses.SparseCategoricalCrossentropy(from_logits = True))\n", "# return model\n", "# #Compile then load weights\n", "# checkpoint_dir = './training_checkpoints_CharWeights'\n", "\n", "# ChaRmodel = create_language_model(batch_size=None)\n", "\n", "# print(tf.train.latest_checkpoint(checkpoint_dir))\n", "# ChaRmodel.load_weights(tf.train.latest_checkpoint(checkpoint_dir))\n", "\n", "# ChaRmodel.build(tf.TensorShape([1,None]))\n", "\n", "# # Get layers to intialize classification model\n", "# embeddings = ChaRmodel.layers[0].get_weights()[0]\n", "# lstm = ChaRmodel.layers[1].get_weights()[0]\n", "# print(embeddings.shape, lstm.shape)\n", "\n", "ChaRmodel = tf.keras.models.load_model('CharLM.h5')\n", "ChaRmodel.summary()" ] }, { "cell_type": "code", "execution_count": 139, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "185" ] }, "execution_count": 139, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(char2index)" ] }, { "cell_type": "raw", "metadata": {}, "source": [] }, { "cell_type": "code", "execution_count": 161, "metadata": {}, "outputs": [], "source": [ "def direction_model():\n", " model = Sequential(name = 'RNNStocks')\n", " model.add(Embedding(input_dim = 185, output_dim = 256,batch_input_shape=[None, None],\n", " mask_zero = True, name ='EmbedLayer'))\n", " model.add(Bidirectional(LSTM(1024,\n", " return_sequences=False,stateful=False, \n", " recurrent_initializer='glorot_uniform'), merge_mode ='concat',name = 'BiLSTM'))\n", " #final state encodes full representation of a single passed headine\n", " model.add(BatchNormalization(name='BatchNormal')) #After RNN(S-shape activation-f(x) / Before ReLU(Non-Gaussian))\n", "# model.add(tf.keras.layers.Masking(mask_value=0))\n", " model.add(Dense(512, name = 'FullConnected', kernel_initializer='he_normal')) \n", " model.add(tf.keras.layers.LeakyReLU()) #controls vanishing gradients:f(x) = a * (exp(x) - 1.) for x < 0 ; f(x) = x for x >= 0\n", " model.add(BatchNormalization(name='BatchNormal2'))\n", " model.add(Dense(1, activation='sigmoid',name='Output'))\n", " model.compile(optimizer=tf.optimizers.Adadelta(learning_rate = 1e-04), loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n", " metrics=['accuracy', tf.keras.metrics.AUC(name='AUC')])\n", " return model" ] }, { "cell_type": "code", "execution_count": 141, "metadata": {}, "outputs": [], "source": [ "# previous_training = tf.keras.models.load_model('daily.h5').get_weights()" ] }, { "cell_type": "code", "execution_count": 162, "metadata": {}, "outputs": [], "source": [ " # batch size of 128 headlines is used to space out weight updates for our large data\n", "epochs = 100\n", "# # with strategy.scope():\n", "# gpus = tf.config.experimental.list_logical_devices('GPU')\n", "# if gpus:\n", "# # Replicate your computation on multiple GPUs\n", "# for gpu in gpus:\n", "with tf.device('GPU:1'):\n", " model = direction_model()\n", " checkpoint_dir = './training_checkpoints_CharWeights_V2'\n", " # print(tf.train.latest_checkpoint(checkpoint_dir))\n", " # model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))\n", " model.layers[0].set_weights(ChaRmodel.layers[0].get_weights())\n", " model.layers[1].set_weights(ChaRmodel.layers[1].get_weights())" ] }, { "cell_type": "code", "execution_count": 163, "metadata": {}, "outputs": [], "source": [ "#Freeze Language Model Layers for initial training of Added Layers:\n", "# Freeze all the layers before the `fine_tune_at` layer\n", "for layer in model.layers[:2]:\n", " layer.trainable = False" ] }, { "cell_type": "code", "execution_count": 164, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"RNNStocks\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "EmbedLayer (Embedding) (None, None, 256) 47360 \n", "_________________________________________________________________\n", "BiLSTM (Bidirectional) (None, 2048) 10493952 \n", "_________________________________________________________________\n", "BatchNormal (BatchNormalizat (None, 2048) 8192 \n", "_________________________________________________________________\n", "FullConnected (Dense) (None, 512) 1049088 \n", "_________________________________________________________________\n", "leaky_re_lu (LeakyReLU) (None, 512) 0 \n", "_________________________________________________________________\n", "BatchNormal2 (BatchNormaliza (None, 512) 2048 \n", "_________________________________________________________________\n", "Output (Dense) (None, 1) 513 \n", "=================================================================\n", "Total params: 11,601,153\n", "Trainable params: 1,054,721\n", "Non-trainable params: 10,546,432\n", "_________________________________________________________________\n" ] } ], "source": [ "model.summary()" ] }, { "cell_type": "code", "execution_count": 165, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n", "True\n", "True\n", "True\n", "True\n", "True\n", "True\n" ] } ], "source": [ "print(np.all(model.get_layer('EmbedLayer').get_weights()[0] == ChaRmodel.get_layer('EmbedLayer').get_weights()[0] ))\n", "for i in range(0, len(model.get_layer('BiLSTM').get_weights())):\n", " print(np.all(model.get_layer('BiLSTM').get_weights()[i] == ChaRmodel.get_layer('BiLSTM').get_weights()[i]))" ] }, { "cell_type": "code", "execution_count": 166, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[True, True, True, True, True, True, True]\n", "[True, True, True]\n" ] } ], "source": [ "print([layer.supports_masking for layer in model.layers])\n", "print([layer.supports_masking for layer in ChaRmodel.layers])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Test Model Pre training it" ] }, { "cell_type": "code", "execution_count": 168, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MSFT rallying after last earnings call<\\s\n", "[[31], [86], [33], [48], [54], [41], [55], [3], [85], [68], [79], [79], [92], [76], [81], [74], [3], [68], [73], [87], [72], [85], [3], [79], [68], [86], [87], [3], [72], [68], [85], [81], [76], [81], [74], [86], [3], [70], [68], [79], [79], [31], [63], [86]]\n" ] }, { "data": { "text/plain": [ "(1, 44)" ] }, "execution_count": 168, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample = 'MSFT rallying after last earnings call' \n", "sample = '' + sample + '<\\s' \n", "print(sample)\n", "sample = [i.encode('utf-8') for i in sample]\n", "sample = tokenizer.texts_to_sequences(sample)\n", "print(sample)\n", "sample = tf.squeeze(sample)\n", "sample = tf.expand_dims(sample, 0).numpy()\n", "sample.shape" ] }, { "cell_type": "code", "execution_count": 169, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MSFT rallying after last earnings call<\\s\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n", "[[31], [86], [33], [48], [54], [41], [55], [3], [85], [68], [79], [79], [92], [76], [81], [74], [3], [68], [73], [87], [72], [85], [3], [79], [68], [86], [87], [3], [72], [68], [85], [81], [76], [81], [74], [86], [3], [70], [68], [79], [79], [31], [63], [86], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]]\n" ] }, { "data": { "text/plain": [ "(1, 64)" ] }, "execution_count": 169, "metadata": {}, "output_type": "execute_result" } ], "source": [ "padded_sample = 'MSFT rallying after last earnings call' \n", "padded_sample = '' + padded_sample + '<\\s' + chr(0) * 20\n", "print(padded_sample)\n", "padded_sample = [i.encode('utf-8') for i in padded_sample]\n", "padded_sample = tokenizer.texts_to_sequences(padded_sample)\n", "print(padded_sample)\n", "padded_sample = tf.squeeze(padded_sample)\n", "padded_sample = tf.expand_dims(padded_sample, 0).numpy()\n", "padded_sample.shape" ] }, { "cell_type": "code", "execution_count": 170, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0.3163721]]\n", "vs\n", "[[0.3163721]]\n" ] } ], "source": [ "print(model(sample).numpy())\n", "print('vs')\n", "print(model(padded_sample).numpy())" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.6294831119673949" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "1/(1+np.exp(-0.53))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Compute PreTraining Baseline Metrics" ] }, { "cell_type": "code", "execution_count": 171, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "89/89 [==============================] - 50s 560ms/step - loss: 0.7115 - accuracy: 0.5283 - AUC: 0.5113\n" ] } ], "source": [ "# TEST DATA\n", "initial_epochs = 10\n", "validation_steps= int(X_test.shape[0] / batch_size)\n", "\n", "initial_loss, initial_accuracy, intial_AUC = model.evaluate(test_seq_data, steps = validation_steps)" ] }, { "cell_type": "code", "execution_count": 172, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initial loss: 0.71 | initial Accuracy : 52.83% | Initial AUC: 51.13%\n" ] } ], "source": [ "print(f'Initial loss: {initial_loss:.2f} | initial Accuracy : {initial_accuracy:.2%} | Initial AUC: {intial_AUC:.2%}')" ] }, { "cell_type": "code", "execution_count": 175, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "44/44 [==============================] - 24s 556ms/step - loss: 0.6721 - accuracy: 0.3899 - AUC: 0.5058\n" ] } ], "source": [ "# VALIDATION DATA\n", "validation_step_2 = int(X_infer.shape[0] / batch_size)\n", "\n", "initial_loss, initial_accuracy, intial_AUC = model.evaluate(infer_seq_data, steps = validation_step_2)" ] }, { "cell_type": "code", "execution_count": 176, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initial loss: 0.67 | initial Accuracy : 38.99% | Initial AUC: 50.58%\n" ] } ], "source": [ "print(f'Initial loss: {initial_loss:.2f} | initial Accuracy : {initial_accuracy:.2%} | Initial AUC: {intial_AUC:.2%}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Warm Start Training\n", "* Pretrained weights should be freezed, for the intial tuning of new added final layers to avoid large gradient updates that can eliminate the pretrained results from the language model." ] }, { "cell_type": "code", "execution_count": 184, "metadata": {}, "outputs": [], "source": [ "early_stopping = EarlyStopping(monitor='val_accuracy',\n", " patience=5,\n", " mode='max',\n", " restore_best_weights=True)\n", "csv_logs = tf.keras.callbacks.CSVLogger('./daily_log.csv', separator=\",\", append=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/30\n", "1610/1610 [==============================] - 4016s 2s/step - loss: 0.7166 - accuracy: 0.5018 - AUC: 0.5018 - val_loss: 0.7476 - val_accuracy: 0.5065 - val_AUC: 0.5058\n", "Epoch 2/30\n", "1610/1610 [==============================] - 3998s 2s/step - loss: 0.7149 - accuracy: 0.5005 - AUC: 0.5017 - val_loss: 0.7449 - val_accuracy: 0.5056 - val_AUC: 0.5013\n", "Epoch 3/30\n", "1610/1610 [==============================] - 4019s 2s/step - loss: 0.7138 - accuracy: 0.5011 - AUC: 0.5027 - val_loss: 0.7428 - val_accuracy: 0.4989 - val_AUC: 0.5009\n", "Epoch 4/30\n", "1610/1610 [==============================] - 3975s 2s/step - loss: 0.7132 - accuracy: 0.5004 - AUC: 0.5032 - val_loss: 0.7414 - val_accuracy: 0.5037 - val_AUC: 0.5032\n", "Epoch 5/30\n", " 530/1610 [========>.....................] - ETA: 43:40 - loss: 0.7106 - accuracy: 0.5076 - AUC: 0.5121" ] } ], "source": [ "history = model.fit(train_seq_data,\n", " epochs=30,\n", " validation_data = test_seq_data,\n", " callbacks=[early_stopping, csv_logs])" ] }, { "cell_type": "code", "execution_count": 186, "metadata": {}, "outputs": [], "source": [ "def plot_learning_curves(df):\n", " fig, axes = plt.subplots(ncols=2, figsize=(15, 4))\n", " df[['loss', 'val_loss']].plot(ax=axes[0], title='Cross-Entropy')\n", " df[['accuracy', 'val_accuracy']].plot(ax=axes[1], title='Accuracy')\n", " for ax in axes:\n", " ax.legend(['Training', 'Validation'])\n", " sns.despine() \n", " fig.tight_layout();" ] }, { "cell_type": "code", "execution_count": 187, "metadata": {}, "outputs": [], "source": [ "metrics = pd.DataFrame(history.history)" ] }, { "cell_type": "code", "execution_count": 188, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABDAAAAEYCAYAAACqUwbqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXxU1fnH8c8smSSTBBI2WQREwMOirNadRXHXautWd63WWhS12mq1aq2i1tZatWp/taIVN3BrrVZbxSrYutUVF+C4QQngwg5ZJzNzf3/cmzAJWSYwN5Pl+3698mLmbvPcY2QOzz3nOQHHcRARERERERERac+C2Q5ARERERERERKQlSmCIiIiIiIiISLunBIaIiIiIiIiItHtKYIiIiIiIiIhIu6cEhoiIiIiIiIi0e0pgiIiIiIiIiEi7pwSGiIiIiIiIiLR74WwHICL+MMacDFwCjAA2A+8DN1hr/5OFWHYClgLlDXadba19NI3zHWC4tfYzH8ITERERnxlj5gNjgb7W2uoshyMiHZRGYIh0QsaYS4DbgBuBHYBBwB+Aoxs5ti0TmcXW2sKUnxaTF+lo43sQERGRVvAeZEwCHOCoNvxc9Q9EOhn9Ty3SyRhjugPXAd+31v4lZdczwDPGmF8CuwJVuJ2IS4wxDwK/Bk7wjn0M+Jm1ttoY0wu4H9gPSAIfA1OstUljzM+AC4FuwCrgPGvtv7Yh5vtxR2fsBEwGFgEnW2s/N8a84h220BuJcTbwNfAQcAdwMTAPOM0Ycw7wM6AH8B/gR9baVd5nOMBFwI+9eP/sHZsDfOnd04fesX2A/wGDrLWrW3s/IiIiUs/pwBvAm8AZwOMAxph84HrgOKAY+BA4yFpbaYzZD/gNMAp3JOnV1tr7vZEcD1lrZ3nXOBP4gbV2P++9A8zA/b4PA0OMMbcDxwDdgU+BH1tr/+0dH8LtD5wN9AE+Ab4DXA5UWWt/UnsTxphngH9Za2/zoY1EJA0agSHS+ewN5AF/beaYo4EncDsLDwNXAnsB43CHd+4BXOUd+xNgBdAbdzTHzwHHGGNwOwjfstYWAYcAy7Yj7pOAa4ES4DPgBgBr7WRv/9gGozb64iYqBgM/NMYcAPwKNwnTDzcBMbfBZ3wX2B2Y4LXBWd4w1rnAqQ1ieVHJCxERkYw4Hbe/8TBwiDFmB2/7b4GJwD643+mXAUljzCDgH7gPKnrj9k/eb8XnfQfYEzf5AfCWd40ewCPA48aYPG/fJbjf+4fjPuA4C6gAZgMnGWOCAN4DnWnAnNbcuIhklkZgiHQ+PYE11tp4M8e8bq19yntdaYw5BbjAWvsNgDHmWuBu4GqgBjchMNirQVH7xCIB5AKjjDGrrbXL0ohtjZv3qLO3tXax9/ov1tr/etd+GPhdC9dKAtfUzqP17uE+a+273vsrgPXGmJ1SYvu1tXYdsM4Ycxtuh2UWbiflCWPMFdbaJHAa7lMfERER2Q7eSIrBwGPW2jXGmM+Bk71REWcBe1lrV3qHv+adcwrug4TaZMFa7yddv/K+7wGw1j6Usu8WY8xVgAEWAj8ALrPWWm//wtrPNMZsxE1azANOBOZba79uRRwikmFKYIh0PmuBXsaYcDNJjNIG7/vjjlio9T9vG8DNwC+BF7zkw5+stTdZaz8zxvzY2zfaGPM8cIm1dpUxpizlWqNSXvdqJqavUl5XAIVNHFdrtbW2qsE9vFv7xlpbZoxZCwxgy8iQ1Puuu0dr7ZvGmHJgijHmS2AY8HQLny8iIiItOwN4wVq7xnv/iLftYdwRo583cs7AJranq14/xxjzE9xERX/cOhzdgF5pfNZs3BGa87w/b9+OmEQkA5TAEOl8Xsetb/Ed3GkijXEavF+F+3TkY+/9IG8b1trNuNNIfmKMGQ28bIx5y1r7L2vtI8AjxphuuCM2fg2cZq2tl3zwindlWlP3UPuZBbijUVamHDOQRu7RU9tJ+Qp4okFyRERERFrJq3FxAhAyxtQ+qMjFncLaD7e/MpQtox5qleJOZ21MORBNed+3kWPq+gjGmEm4NS6mAR97NbzWA4GUzxoKfNTIdR4CPjLGjAVGAk81coyItCElMEQ6GWvtRmPML4C7jDFx4AXcaSAHAvvjjm5oaA5wlTHmLdwv/V/gfmljjDkSWIL7dGITkAASXg2MAcCruB2QSvyrq/M1sDNubYymPALMNcY8AizGXYHlzQZTWy41xryJO7rjIupPU3kQ+AC3UNhpmQtdRESky/oObr9hNyCWsv0x3LoY9wG/M8achvtdvwfuaMqHgZ8bY04A/oJbfHOgtfZ93FoYxxhjZuGOqKgt7t2UIiAOrAbCxpjLcUdg1JoFzDTGLMLtZ+wGrLTWrrXWrvD6Rg8CT1prK7e9KUQkE1TEU6QTstb+Drco1VW4X9iluAU3m3pycD3wNu4/4D/E7Txc7+0bDrwIlOGO7viDtXY+7hOUm4A1uKMW+uAW+GzOBmNMWcrPJWne0i+B2caYDV5nZive6idXA0/irioyFHe+aqq/Ae/gdn6eBe5NOX8F7n07eHU+REREZLucAfzZWrvcWvtV7Q9wJ3AK7kofH+IW2VyHO5IzaK1djltU8yfe9vdxi4wD3IqbDPkad/Tkwy3E8DxuQdBPcKePVlF/isnvcBMqL+A+qLkXyE/ZPxs3qfFga29eRDIv4DgNR2GLiHQ+3rJqw71CpE0dcx+wylp7VVPHiIiISNdhjJmMOyp1J6/Qt4hkkaaQiIhQV6fjGGB8lkMRERGRdsAYk4M75XSWkhci7YOmkIhIl2eMmYlbvOtma+3SbMcjIiIi2WWMGQlswC02eluWwxERj6aQiIiIiIiIiEi7pxEYIiIiIiIiItLudagaGLFY3Nm4MfOrFxUW5lJWVp3x68oWamP/qY39pfb1n9rYf361ce/eRYGMX7QN+NWvAP0++03t6z+1sf/Uxv5TG/vLz/Ztqm/RoUZgBAL+9I/C4ZAv15Ut1Mb+Uxv7S+3rP7Wx/9TG9fnVrwC1td/Uvv5TG/tPbew/tbG/stG+HSqBISIiIiIiIiJdkxIYIiIiIiIiItLuKYEhIiIiIiIiIu1ehyriKSIikq5EIs769auJx2PZDqXd+PrrANuzfHo4HKGkpDehkLoPIiLStahfsbXt7VdA6/sW6oGIiEintH79avLyohQU9PW1WGNHEgoFSSSS23Su4ziUl29i/frV9OrVL8ORiYiItG/qV2xte/oVsG19C00hERGRTikej1FQ0E2djAwJBAIUFHTTkycREemS1K/IvG3pWyiBISIinZY6GZml9hQRka5M34OZ19o2VQJDRERERERERNq9Ll8DI/fTvxEqfYFo0XDivUYR7zmSZNGOoOyaiIhsh40bN3DRRecBsG7dWoLBIMXFJQDcc89scnJyWrzGjTdey6mnnsGgQTs1ecyTTz5GUVERBx98WEbilu2UqKbo5csI9tqJcJ99ifedAMEu390SEZHtpH6FK7C9VUPbUk1NwtmwoSKj18xd8jhF7/6ewPqldduSkSISPUcS7zWSeM+RxHuOIt5zBOREM/rZXUlxcZRM/7eT+tTG/lL7+i/TbfzVV/+jb9/BGbve9rj33rvJz49y8smn1dvuOA6O4xAMts2AyO0ttgWNt2vv3kUdMuvvR7+CWDnd//lDclb8m4CTJBkpombHfYkNmkps4FSS3XbM7Od1Ufo72X9qY/+pjf2XyTZWv2JrmehXQOv6Fl3+kUD1iOPJ3+sMNnyzmvC6JYTXLCa8dhHhtYvJXfIk+TVlADgESHTfiYQ3SkOjNUREZFusWFHKFVf8hDFjxrFo0Uf85je3cd999/DJJ0uorq5m2rSD+P73zwFg+vSzueSSyxgyZChHHnkgRx99LG+88Rp5eXncdNMtlJT04E9/+gPFxcWccMLJTJ9+NmPGjOPdd9+irKyMn//8GnbbbSyVlZVcf/0vWLlyBYMHD2HFilIuv/wqhg83WW6NTihSwMajHqY4N0bFxy8QKV1AZPkCcr/4JwDx4qHEBk2hZuAUYgP2gZz8LAcsIiIdWVfrV3T5BEadSAHxvhOJ9524ZZuTJLiplPDaxYTXuEmN8OqPyP382bpDkpFuxHuOJNFrhDdSY6RGa4iItDPPfvw1T3/0VUavedSufTli9A7bdO6yZUv5+c+v4dJLfw7A9Okz6NatO/F4nAsv/BFTp05jyJCd651TVlbGuHETmD79Au6443f8/e9Pc9ppZ251bcdxuOeeB/jPfxbw5z/P4ne/u4MnnniUHj16cdNNt7BkyRLOPvvUbYpbWiG/mNiwI4kNOxIch9D6z7xkxnzyP36Y6Af34QQj1PTfk9igKcQGTSHRY4QeioiIdADqV2SvX6EERnMCQZLdBxPrPpjYzodu2R4rrz9aY80icpc8QX5NOdDUaI1RJIsGqGMiIiIMGLAjI0eOrns/b97zPPvs30gkEqxZs5ply77YqqORm5vL3nvvC4AxI1m48L1Grz1lygF1x3z11SoAPvzwfU455QwAhg/fZatri88CARI9hlPZYziVY38A8UpyVv2XyPIFREoXUPja9fDa9SQKdnBHZgyaQmzgZJy8kmxHLiIiHUBX6lcogbEtNFpDRKRDOWL0Dtv8VMMPeXlbpg2Uli7n8cfncs89sykqKuK6664mFtt6PfTU4lzBYJBEItHotSORnK2O6Uj1rrqEcD41g6ZQM2gK5UCwbBWR5a+QU7qAyNLnyVvyGA4B4n3GeqMzphLfYbyKgYqItBPqV2SvX6FvwkxpYrRGIFZGaJ3dktRobLRG8RCvaOiouqKhGq0hItI1lJeXE41GKSgoYM2aNfz3v6+z5557Z/QzxowZx0svzWPChIl8/vlnLFu2tOWTpM0kC/tTNepEqkadCMkE4W/erxudEX3nDgrevp1kpJtXDHSKioGKiEiTOnu/QgkMnzmRwmZGayzaMg2lydEaI7dMQ+lhNFpDRKSTMWYEQ4YM4fTTv0f//gPYbbexGf+MY4/9Htdffw2nnnoCu+wygiFDhlJQUJjxz5EMCIbq+g0Ve1xCoGoDOSv+Q2T5fCKlC8j94h8AxEuGERvojuKI9d9bxUBFRATo/P2KLr+MKrSfJYwaG60RWruYYJOjNWpXQmn/ozXaSxt3Zmpjf6l9/deZl1HNtng8TiKRIBrNZ9myZVxyyQzmzPkL4XDrn2NoGdX0+PJ3huMQWv+pNzpjPjkr3yCQqMYJ5VLTb48uVQxUfyf7T23sP7Wx/zrrMqrZlsl+BWgZ1Q4rs6M1RuhpjIiIAFBZWclFF00nmUzgOA6XXvrzbe5kSBYFAiR67EJlj12oHHeOVwz0TTehsbx+MdDYwKnu6IyBk1QMVEREMiqb/Qr1Xtq7erU1DtuyOXW0hjdiI3fJ41vV1oj3HOUlNjrOaA0REcmsoqIi7rvvIUKhIIlEMtvhSKaE86kZNJWaQVPdYqCbVxEpXUDO8gXkLv0n+UsebVAMdH/iO4xTMVAREdku2exX6Busg0p3tEbO6g/J+/zvdYfUG61RWzRUozVEREQ6vGRRf6pGnUTVqJMgGSf8zUK3dsbyBsVAB+5HbKC7ukmyaEC2wxYREUmbEhidSXOjNdYuqbfEq0ZriIhIthhjDgVuB0LALGvtTQ32nwncDKz0Nt1prZ3l7TsDuMrbfr21dra3PQLcCUwFksCV1ton/b2TdiwYTikG+hMCVevJWfGql9CYT+7nzwFbioHGBk2lpv9eeqAhIiLtmhIYXYATKSTeb3fi/XZP2ZgkuGl5vaTGVqM1crsT7zmi/hKvGq0hIiLbwRgTAu4CDgJWAG8ZY5621i5qcOij1toZDc7tAVwD7A44wDveueuBK4FvrLW7GGOCQA+/76UjcfJKiA07ktiwI91ioOs+IVLq1s7I//ghoh/c6xYD7b+nl9CYQqKH0YMMERFpV9JKYKTxpORWYH/vbRToY60tTtnfDVgM/LW2M2KMmQ/0Ayq9ww621n6z7bcirRIIkuy+E7HuO7UwWmNRM6M1RtUVDU0W9lcnR0RE0rEH8Jm19gsAY8xc4GigYQKjMYcA86y167xz5wGHAnOAs4ARANbaJLAm86F3EoEAiZ6Gyp6GynE/TCkGOt8rBjoTXptJoqAvsUFTqBk4ldjA/VQMVEREsq7FBEY6T0qstRenHH8BML7BZWYCCxq5/CnW2re3JXDxR+ZGa4wi3sNotIaIdFkzZvyQ0077PnvuuXfdtscee4Tly5fz059e3ug5Bx00iXnz/s2aNau57babuf763zR63RkzfsyIEaOa/OzHHnuEo446hry8PAB++tMLueaaGygu7r6dd5URA4DSlPcrgD0bOe5YY8xk4BPgYmttaRPnDjDG1D40mWmMmQp8Dsyw1n7dXCChUIDi4ui23UULQqGgb9fOvCj0OhzGHI4D1GxaQeDzlwh+8RJ5S/9J/uJHcQJBnH7jcXY+AGfoATj9J2a1GGjHat+OSW3sP7Wx/zLZxl9/HSAUCmbkWtvivPPO4fTTv89ee+1Tt23u3IcpLV3OpZde0eg5BxywLy+99CqrV6/m1lt/w4033tzodS+44GJGjmy6XzF37sN85zvHkJfn/tvukksu4Nprb6SoqCgjbRIIpP99nM43T2uflJyEO7wT7/iJwA7AP3GHfEpH0+JojS1FQ+uN1ggESXQfQrznSIL9RpKbN5BE8RAS3Yfg5BU39WkiIp3CgQcewr/+9UK9BMaLL77A+edf1OK5vXr1bjR5ka7HHpvDwQcfXpfA+O1vf7/N1/JBY8P1nAbvnwHmWGurjTE/AmYDBzRzbhjYEXjVWnuJMeYS4LfAac0Fkkg4bNhQ0dr401JcHPXt2v7rAUOOc39qi4H+72UipQsIv3oLgf/cTDK3OzU77ueubjJwKsmi/m0aYcdu345Bbew/tbH/MtnGjuNkdSWvadMOZt685/nWt/aq2zZv3vOcf/5FzcaVSCTp0aMnM2f+utHjHMchmUw2e41HH32Egw46jJycXABuvvn2etffXo6z9fdx795FjR6bTgIj3SclGGMGA0OAl7z3QeAW3A7EtEZO+bMxJgE8iVuIq2EHph6/npQo+7mtotCnDzC5bkvCSZLY8D8CX39M4JuPCH6ziNyvP4LPn6VbSv/UifbEKdkZeg7DKdkZp+dQnJKh0GMIRAqzcC8dn36P/aX29V+m2zjbT0oOPPAgZs36PxKJOJFIhC+/XMXatWswZgQXXTSdzZs3E4/HOffc85g8eWrdeaFQkC+/XMVPf3oRDz/8OFVVVdxwwy9ZuvQLdtppCLFYjGAwSCgU5De/uZHFiz+murqa/fefxjnnTOexx+awZs1qLrzwRxQXF3PXXX/iu989gj//+SGKi0t47LFH+Pvf/wbAt7/9HU488RS+/HIVF188g7Fjx/Phhwvp3bsPv/717+oSIKla85SkCSuAgSnvdwRWpR5grV2b8vYe4Ncp505N2bcjMB9YC1QAf/W2Pw6cvT1Biie1GOiePyVQtZ5I6X/IKXWnm+R+/iwA8ZLhXjJjCjUD9oKwRmCKiGTS/vtPY9as/yMWi9X1K9asWc3w4bt4/YpNxONxzjlnOpMmTa137pdfruKyy37Mgw8+RnV1FTfeeC3Lli1l8OAhVFdX1x3329/+isWLF9X1K84++1wef3yu1684l+7di7njjrs57rhvM2vWg/Ts2YO5cx/i2WefBtx+xQknnOz1Yy5kzJhxfPjhB/Tu3ZubbrqF3Nyt+xWtlU4CI50nJbVOBJ6w1ia89+cBz1lrS40xDY89xVq70hhThJvAOA14oLlA/HpSouxnhgV2gL47QN8D6jYVF4bYvHwJoQ1fENq4dMufn79MqHxOvdMTBTuQ6D6ERPHO3p/e626DILz9v/SdlX6P/aX29V+m2zj1SUnukifIWzw3Y9cGqBp5ItUjjmtyf2FhN0aOHM1rr/2HSZOm8vzz/+SAAw4iHM7hxhtvpqCgkA0bNnDuuWeyzz6TCHh1hBIJ9ylIbfxPPvkYkUgus2fP5bPPPuXss0+te1JyzjnT6datO4lEgosumo61lmOP/R5z5jzE73//R4qLi+vaIJFwWLJkEX//+9+4++77cRyHH/7wTMaOHU9RUTdWrCjll7+8gcsuu5Krr76cl156kUMOObzRdk33KUkT3gKGG2OG4K4yciJwcuoBxph+1tovvbdH4dbRAngeuNEYU1uM4WDgCmutY4x5Bje58RLuQ5N0ampIKzl5JVQP/zbVw7/tFQO1RJYvIFK6gPyPHiS6cJZXDHSvuoRGoscuqpMlIp1KNvoV3bsXM3LkaN588zUmTZrKiy++wLRpBxOJ5G7Vr9hvvyl1/YqG/vrXJ8jNzavXr6j1wx+eV69f8dlnn3L88Sfy6KMP8/vf301xcf1R9EuWLOK5557hT3+aXdevGDduQr1+xc9+dhVXX3058+e/1Gi/orXSSWC0+KQkxYnA+Snv9wYmGWPOAwqBiDGmzFp7ubV2JYC1drMx5hHcqSrNJjCkAwvnkugxnESP4Vvvq6kgtHEZoQ1fEN6wtC7Bkbv0eYKVWx7COQRIFu1YNw2l9s948c4kuw3M6lxcEZHGHHjgIbz44gtMmjSVf/3rBa644hcA3H33XSxc+B6BQJDVq1ezbt1aevbs1eg1Fi58j+OOOxGAYcOGM3TosLp9L700j6ef/iuJRIK1a9ewbNkXDBvWyN+zddd6n8mT9yc/3306PmXK/ixc+D777TeZfv36M3y4+7DBmBF8+WVTX/Xbx1obN8bMwE1GhID7rLUfG2OuA9621j4NXGiMOQqIA+uAM71z1xljZuImQQCuqy3oCfwMeNAYcxuwGvi+LzcgWwQCJHqOoLLnCCrHnws1leSsesNb3WQ+ha9eB0CisB+xgZNVDFREZDupX5FeAqPFJyVuUMYAJcDrtdustaek7D8T2N1ae7kxJgwUW2vXGGNygCOBF7fnRqQDy4mS6OWuahJrsCtQvZFQSlKj9nXuJ38lGNtUd5wTDJMoGuiO1ChOGb3RfYg7LzeQvWHkIpJ91SOOa/aphl8mTZrKHXfcirVLqK6uwpgRPPfcM2zYsIF7732IcDjMccd9m1is4d9+9TX2FGXVqpXMmfMQ99zzAN26deOGG37Z4nUcp+mZmjk5OXWvg8EQiUR1k8duL2vtc8BzDbb9IuX1FUCjFcmstfcB9zWy/X+kzmmUtpeTT83g/akZvD/lQHDzSndlk9IF5H7+j7pioPE+49zRGYOmEu8zVg8gRKTDUb/ClY1+RYvfGGk+KQG3eOfclupYeHKB573kRQg3eXHPNt2BdGpObnfiO4wjvsO4BjscAlXr6pIa4bqpKUuJrHyNQLxyy6GhXBLdd6o/csNLcCSjfTSsVUR8E41GGT9+Ir/61XUceOAhAJSVlVFSUkI4HObdd9/mq6++bPYaY8eO54UX/sGECbvzxRef8fnnnwFQXl5OXl4+hYWFrFu3ljfeeI3x4yfWfW5FRflWQz3Hj5/AzJnXcOqpZ+I4Dq+88jJXX32dD3cuAsmiAVSNPoWq0ae4xUC/ft9bqnU+0bduo+CtW7NeDFREpCNRvyK9ERgtPinx3v+yhWvcD9zvvS4HJqYfpkgDgQBOfk/i+T2J9/sW9fJ5jkOw/KuUehvez/rPiSx7iUBySyYxmVOwVVKj9rWGuIpIJhx44CFceeWlXHvtjQAcfPBh/OxnF3P22acxfPguDB68U7Pnf/e7x3HjjddyxhknMmzYLowcORqA4cN3YZddDKeddgL9+w9gt93G1p1z1FHf5ac/vZCePXtxxx131203ZiSHHXYk55xzOuAW29plF/+mi4jUCYbrlmnfUgz03+QsX0CkdH5KMdBdvNEZU6jpv6eKgYqINNDV+xWB5oZ9tDc1NQlHRTw7pnbTxskEwbKVXlJjS4IjvOELgptXEHASWw7N7Z6S1Ni53ggOJ9KqgnVtot20cSel9vVfptv4q6/+R9++gzN2vc4gFApu93JnjbVr795FHXIom1/9CtDfGa1SVwzUXdkk58v/EkhUpxQDnUps0BQSJcPrRk2qff2nNvaf2th/mWxj9Su2lol+BbSub6FJh9K1BEMkuw0i2W0QNYOm1N+XiBHaVJoyasNNcOSseoO8T/5S79Bkfm/ixTuTKN6pQYJjJz0tEhERaY16xUB/BDWVRFa97o3OWEDhq9fCq7XFQN3aGYw+GIhkO3IREWljSmCI1ApFSJQMJVEydOt9NZWENi1rUFB0GbnLXiJYubreoYnC/vWTGrWjOLoNhJA6WyIiIs3KySc2+ABigw9wi4FuWkGk1B2dkfv5c+QvnovzSgnB780jWdA329GKiEgbUgJDJB05+SR6jiTRc+RWuwKxzXXLwKYmOHI/e5pg9ca645xAqG4Z2HiDqSnJwgEQDLXlHYl0CY7jNLkOurReR5p2Kp1HstuOVI0+larRp0IyTs7K1+j+zKnkv38P5ftene3wRKQLUb8i81rbt1ACQ2Q7OZEi4r13I957t632BarWb11MdMMX5H35FsGa8i3XCEZIdB+cUkR0SF2CI1nQVyuliGyDcDhCefkmCgq6qbORAY7jUF6+iXBYI8kki4JhagZOxhn1XfI+foiKiTNUdFtE2oT6FZm3LX0LJTBEfOTklRDvO5F43waL7jgOgYrVhDd+sVVB0UjpAgIp6yQ74fwtiY3u3ugN77WT31PJDZEmlJT0Zv361ZSVbch2KO1GIBDYrlEU4XCEkpLeGYxIZNsk9v4xOR8/Sf6H91PxrYuzHY6IdAHqV2xte/sV0Pq+hRIYItkQCOAU9KGmoA81/feqv89JEiz7ssHIjS8IrV1MZOnzBJLxukOTkW51xUODPQeSHywhGe2d8tMHJ68YAsE2vkGR7AuFwvTq1S/bYbQrqngvncYOo6kePI38D+6jYty5kBPNdkQi0smpX7G1bPQrlMAQaW8CQZJFA0gWDaBm4KT6+5JxgptKCW+sP2oj5+v3CS59nsJ41VaXc4Jhkvm9SEb7eEmN1Nd9cFKTHTkFGtEhIiIdQsXECyj5y3fIX/QIlWN/kO1wRESkDSiBIdKRBMMki4cQKx4Cgw+ot1T7m1QAACAASURBVKu4ez4bV39DsGI1wYpvCJa7fwYq17jvve3hNR8RrFhDwElsdXknnJeS3Ohd/3V+6rZeEM5rq7sWERHZSrzf7sT670n++3dTuevpWulLRKQLUAJDpLMIBHAiRSQiRSSKd27+WCdJoGq9l9hITXCsrnsd2rCUnFVvEqxa3+glkrnd6yc68ns1SHq4fzr5PbXCioiI+KJiwgyK/34aefYvVI06MdvhiIiIz5TAEOmKAkGc/J4k8nuS6NnCsYkagpVrtkpwpCY9wt984L6uKdvqdCcQxMnrsfWIjnqv3R8nt1hTWEREJG01g6ZS02tX8t/7A1UjjlfCXESkk1MCQ0SaF8ohWdiPZGEaRYtqKppIdKype52z/jOCFasJJGNbne4Ec5pNcKSO7lDBNhERIRCgcsL5dHthOpEv/kFs2JHZjkhERHykBIaIZE5OlGT3wSS7D27+OMchUL1xS4Kjcs1WSY/Q5pXkfP0+gaq1BJzkVpdI5hTg5PciWdDYqI4+KXU7emletIhIJ1Y99HDi3YcQfedOYkOP0Eg+EZFOTAkMEWl7gQBOXjGJvGISPYY3f2wyQaBqXZPTV4IV3xBa9yk5K14lWL2x8UvkFjcyfaXXVtucvBINPxYR6WiCISonnEfRy5eSU7qAmkFTsx2RiIj4RAkMEWnfgiGcaG8S0d4kGNX8sYnqBkVJa//csi3n6/cIVnxNoLElZwMhrxhpL2952S0JjkCPvuRW1vh0k83J0pPELDzBDBQVkBML4+QUeD9RnJxCd3nfcJ6eqopIk6rMsUT/ewvRd+5koxIYIiKdlhIYItJ5hHJJFg0gWTSg+eMch0BNeV1SI1A3mqP+6I6ctUsIVq4hkIwD0K0NbqGrK25iu0OgQWKjideRwq2208zxBIJten8i4pNQhMpx51L46rWEv3ybeL/dsx2RiIj4QAkMEel6AgGcSCGJSGGaS85uoHukis2btx614SvHadvP2/LBWfnUomiQsnXrCNSUez8VKa8bvq8gUFNGsGodgc2lW7bFygg4ibQ/0wlH6yc2Is0lSBr+NHZcFIL6ahXJhspRJxN9+3ai797JpiPuz3Y4IiLiA/WyRESaEwji5PeA4iiJUEW2o+nciqPEc7ezjR0HkjEvmdEw+VHWRCKkwevqTQTLvnJfx8rc7Y2smtNkCKFcLxlS2Pxoka1eNzx+yz5COdvXLu2MMeZQ4HYgBMyy1t7UYP+ZwM3ASm/TndbaWd6+M4CrvO3XW2tne9vnA/2ASm/fwdbab3y8DWlvIgVUjj2bgv/eQmjtYhI9R2Y7IhERyTAlMEREpPMIBCCU6yYR8koyd92ElxRpcVRIuZco2Xp7sOKb+tsbqcPSFCcYSWu6TPOvC6H72My1yTYyxoSAu4CDgBXAW8aYp621ixoc+qi1dkaDc3sA1wC74w4Vesc7d713yCnW2rf9vQNpzyp3O5P89/5I9J272HzwndkOR0REMkwJDBERkZaEIjihCE5eU1U6tkEyQSDuJTS2Gi3SUpLE/QlWrqu/Pd78CJbE1Kth9LmZu4dtswfwmbX2CwBjzFzgaKBhAqMxhwDzrLXrvHPnAYcCc3yKVToYJ6+EqtGnkr/wHsr3vLTlZb1FRKRDSSuBkcZQz1uB/b23UaCPtbY4ZX83YDHw19qnKcaYicD9QD7wHHCRtTZbE75FRETaVjCEEynCiRRBQYau6SShprLxhEe8guiog6A6Q5+17QYApSnvVwB7NnLcscaYycAnwMXW2tImzk2t2vtnY0wCeBJ3ekmz/YpQKEBxcXQbbqFloVDQt2tLC+07+UL48M8UL5pF8rBb2jawTkS/w/5TG/tPbeyvbLRviwmMdIZ6WmsvTjn+AmB8g8vMBBY02PZ/wA+BN3ATGIcC/9iGexARERFwV1WJeMVIG9kdzY9CddZruTS2Hm7DcJ8B5lhrq40xPwJmAwe0cO4p1tqVxpgi3ATGacADzQWSSDhs2OBPexQXR327trTUvt0pNMeR9/7DbBgzg2TBDm0aW2eh32H/qY39pzb2l5/t27t3UaPb01k/rm6op7U2BtQO9WzKSaQM5fRGWuwAvJCyrR/QzVr7uvd05AHgO2nEIiIiIh3bCmBgyvsdgVWpB1hr11pra8eK3ANMbOlca+1K78/NwCO4/RfpoirGTwcnTv7Ce7IdioiIZFA6CYyWhmvWMcYMBoYAL3nvg8AtwKWNXHNFOtcUERGRTuUtYLgxZogxJgKcCDydeoD3oKPWUbjTUAGeBw42xpQYY0qAg4HnjTFhY0wv79wc4EjgI5/vQ9qxZPEQqod9m7yPHiRQtSHb4YiISIakUwMjnaGetU4EnrDWJrz35wHPWWtLjTHbes06fs1V1dwo/6mN/ac29pfa139qY/+1hza21saNMTNwkxEh4D5r7cfGmOuAt621TwMXGmOOAuLAOuBM79x1xpiZuEkQgOu8bQW4iYwc75ov4o7ckC6sYsL55H36N/I/mk3F7hdlOxwREcmAdBIYLQ71THEicH7K+72BScaY84BCIGKMKcMtCLpjmtes49dcVc2N8p/a2H9qY3+pff2nNvafX23c1DzVplhrn8Otf5W67Rcpr68Armji3PuA+xpsK2fLNBMRABK9RlE9+ADyF95LxdhzIEcJUhGRji6dBEbdUE9gJW6S4uSGBxl3iEUJ8HrtNmvtKSn7zwR2t9Ze7r3fbIzZC3gTOB24Y9tvQ0RERESkvooJMyj56zHkL3qEyrE/yHY4IiKynVqsgWGtjQO1Qz0XA4/VDvX0hnfWOgmY24qlUKcDs4DPgM/RCiQiIiIikkHx/nsQ67cn+e/fDYlYtsMREZHtFHCcdPMN2VdTk3A0haRjUhv7T23sL7Wv/9TG/vNxCkljta3aPb/6FaDfZ7+1pn0j/3uJ7n8/nU0H3EL1yO/5HFnnod9h/6mN/ac29pfPy6g22rdIZxUSEREREZEOKTZof2p6jSb67h8gmWj5BBERabeUwBARERGRzisQoHLC+YQ3fE5k6T+zHY2IiGwHJTBEREREpFOrHnoE8e47EX3nTuhA06dFRKQ+JTBEREREpHMLhqiccB45qz8kp/SVbEcjIiLbSAkMEREREen0qsyxJAp2IPrundkORUREtpESGCIiIiLS+YVyqRx3LpGVrxP+6p1sRyMiIttACQwRERER6RIqR51CMreY6Dt3ZTsUERHZBkpgiIiIiEjXECmgcsz3yV32AqG1S7IdjYiItJISGCIiIiLSZVSOOQsnHCX6rkZhiIh0NEpgiIiIiEiX4eSVUDn6VHI/fZrgpuXZDkdERFpBCQwRERER6VIqx50DgSDR9/6Y7VBERKQVlMAQERERkS4lWdiPqhHHkbf4UQLl32Q7HBERSZMSGCIiIiLS5VSOnw7JGqIL78l2KCIikiYlMERERESky0kU70z10CPJ++hBAtUbsx2OiIikQQkMEREREemSKiacT7CmjPwPZ2c7FBERSYMSGCIiIiLSJSV6j6Z60P7kL5wFNZXZDkdERFoQznYAIiIi0vUYYw4FbgdCwCxr7U0N9p8J3Ays9Dbdaa2d5e07A7jK2369tXZ2g3OfBna21u7q3x1IZ1E5cQa5fz2WvMVzqBpzVrbDERGRZmgEhoiIiLQpY0wIuAs4DBgFnGSMGdXIoY9aa8d5P7XJix7ANcCewB7ANcaYkpRrHwOU+X0P0nnU9N+Tmn7fcpdUTcSyHY6IiDRDCQwRERFpa3sAn1lrv7DWxoC5wNFpnnsIMM9au85aux6YBxwKYIwpBC4BrvchZunEKibMIFS2itxPnsp2KCIi0gxNIREREZG2NgAoTXm/AndERUPHGmMmA58AF1trS5s4d4D3eiZwC1CRThChUIDi4mgrQ09PKBT07driQ/uOPRLnrdEULfw/8vc6HQJ6xqffYf+pjf2nNvZXNtpXCQwRERFpa4FGtjkN3j8DzLHWVhtjfgTMBg5o6lxjzDhgmLX2YmPMTukEkUg4bNiQVq6j1YqLo75dW/xp39xx59HthfOpePcvxIYentFrd0T6Hfaf2th/amN/+dm+vXsXNbo9rQRGGoW2bgX2995GgT7W2mJjzGDgL955OcAd1to/eufMB/oBtSWfD7bWftOKexIREZGOaQUwMOX9jsCq1AOstWtT3t4D/Drl3KkNzp0P7A1MNMYsw+3f9DHGzLfWph4r0qTqoUeQ6PYbou/eRWznwyDQWK5MRESyqcUERkqhrYNwOw1vGWOettYuqj3GWntxyvEXAOO9t18C+3hPTwqBj7xzazspp1hr387QvYiIiEjH8BYw3BgzBHeVkROBk1MPMMb0s9Z+6b09CljsvX4euDGlcOfBwBXW2nXA/3nn7gT8XckLaZVgmIoJ0ymafzk5K/5DzcBJ2Y5IREQaSGeCX2sLbZ0EzAGw1sastdXe9tw0P09EREQ6MWttHJiBm4xYDDxmrf3YGHOdMeYo77ALjTEfG2MWAhcCZ3rnrsOtdfGW93Odt01ku1WNOJ5EdAei79yR7VBERKQR6UwhSbfQFt6UkSHASynbBgLPAsOAS1NGXwD82RiTAJ7EXce94fxXERER6YSstc8BzzXY9ouU11cAVzRx7n3Afc1cexmwa0YCla4llEvluB9S+NpMwl+9S7zvhGxHJCIiKdJJYKRTaKvWicAT1tpE7QavYvgYY0x/4CljzBPW2q9xp4+sNMYU4SYwTgMeaC4Qv6qFqzqt/9TG/lMb+0vt6z+1sf/UxiItqxp9CtF3fk/03bvYdPi92Q5HRERSpJPAaLHQVooTgfMb22GtXWWM+RiYhJvkWOlt32yMeQR3qkqzCQy/qoWrOq3/1Mb+Uxv7S+3rP7Wx//xq46YqhYt0RE6kkMrdvk/B27cRWmtJ9DTZDklERDzp1KSoK7RljIngJimebniQMcYAJcDrKdt2NMbke69LgH0Ba4wJG2N6edtzgCOBj7b3ZkREREREtlfl2LNxwvlE3/tDtkORzshxKHzlKoIvXw81StyLtEaLCYw0C22BW7xzboM6FiOBN70CXAuA31prP8Qt6Pm8MeYD4H3cCuT3ZOSORERERES2g5NXQuXoU8j95CmCm5ZnOxzpZPIWPUz+h/cTeu139HhkfyJf/BMclQIUSUfA6UD/s9TUJBxNIemY1Mb+Uxv7S+3rP7Wx/3ycQtJYvax2z69+Bej32W9t0b7BslX0eHBfqkadTNmUG3z9rPZIv8P+CG5aQcncacT7jCN4wM/h2UsIr7NU73QgZZNmkuw2sOWLSNr0e+wvP9u3qb6FljUVEREREWkgWdifKnMseYvnEqhYne1wpDNwHIpevhSAzQf8FmfQ3qw/4Z+U7XM1kRWv0WPO/kTfvgMSsSwHKtJ+KYEhIiIiItKIygnnQSJGdOGsbIcinUDeooeJrPg35ftctWWkRSiHyvHnsu7k+cQG7U/Bm7+m5NGDyVnxanaDFWmnlMAQEREREWlEonhnqoceQd5HDxCo3pjtcKQDC25aQcGrM4kN2Jeq0adstT9Z1J9Nh93DxiNmE0jUUPy371E07wIC5d9kIVqR9ksJDBERERGRJlROnEEwtpn8Dx/IdijSUTkORfMvI+A4bD7gtxBo+p9gsZ2mse6kFynf/SJyP3uWHo9MJe/D+yGZaLt4RdoxJTBERERERJoQ770rsUFTyf9gFtRUZjsc6YDyFj1CpPQVyva9Kr0ineF8Kva8lPUnvUi8zxiKXrmK4ie+Tfjr9/0PVqSdUwJDRERERKQZFRNnEKxcS97iudkORTqY4OaVKVNHTm3VuYnindl41Bw2HXwXwfKvKH7i2xQu+LmmM0mXpgSGiIiIiEgzavrtSU3f3Ym+90dI1GQ7HOkovFVHAk6yxakjTQoEqB5+NOtPnk/lmO+T9/FD9Hh4Crn2SXCczMcs0s4pgSEiIiIi0pxAgIqJMwiVrST3079lOxrpIFo9daQZTm43yiddx4bjnyPRbSDdXryI7k8dT2jdJxmKVqRjUAJDRERERKQFscHTiPccQfTdu8BJZjscaee2Z+pIc+K9d2XDsX9j89SbCK9dTMmjB1Pw+q9Un0W6DCUwRERERERaEghQMWEG4fWfEln6fLajkfas3tSRm7dt6khzAkGqRp/KupMXUL3LMUTfvYsec/YnsvSFzH6OSDukBIaIiIiISBqqhx1Jottgou/cqfoD0qT6U0cG+fY5TrQXm6f9jg3ffRInp4Duz51Ft2fPIrip1LfPFMk2JTBERERERNIRDFMxfjo53ywkZ8Wr2Y5G2qEtU0f2yejUkebU9N+T9Sf8k7K9rySy4t/0mLM/+e/cCYlYm3y+SFtSAkNEREREJE1VI44jEe1D9N07sx2KtDeZWHVkW4VyqJwwnXUnzyc2aCqFb9xEyaOHkLPytbaLQaQNKIEhIiIiIpKucB6VY88hsuI/hL9+L9vRSDvSVlNHmpMsGsCmw2ax8YjZBBLVFD91AkXzLiRQsTor8YhkWjjbAYiIiEjXY4w5FLgdCAGzrLU3Ndh/JnAzsNLbdKe1dpa37wzgKm/79dba2d72fwL9cPs3/wbOt9YmfL4V6YKqdj2N6Lt3En3nTjYdfm+2w5F2IBtTR5oT22ka6wbsQ/SdO4i+939Elr1I+V4/c2MLhrIdnsg20wgMERERaVPGmBBwF3AYMAo4yRgzqpFDH7XWjvN+apMXPYBrgD2BPYBrjDEl3vEnWGvHArsCvYHjfb4V6aKcSCGVu51J7tLnCa37JNvhSLZlc+pIc3LyqdjrMtaf+CLx3rtR9MqVFD95FOFvPsh2ZCLbrJ383yUiIiJdyB7AZ9baL6y1MWAucHSa5x4CzLPWrrPWrgfmAYcCWGs3eceEgQigZSLEN5VjzsYJ5xN99w/ZDkWyLG/xHHfqyD5XZm3qSHMSJUPZePRcNh10J8GyLyl+/AgKX7mSQPXGbIcm0mqaQiIiIiJtbQCQus7fCtwRFQ0da4yZDHwCXGytLW3i3AG1b4wxz+MmSP4BPNFcEKFQgOLi6DbdQEtCoaBv15Z20r7FUZLjTyf37VmEDrwKitvfP1y3R7to445g4wrCr15HcvAk8vY7l7xWjL5o8zbe42SSY46EBTeS984s8r74B4kDZ+KMPg4CgbaLow3p99hf2WhfJTBERESkrTXWU244WuIZYI61ttoY8yNgNnBAS+daaw8xxuQBD3vHz2sqiETCYcOGitbGnpbi4qhv15b2077BkWfT4537iL9yK2WTb8h2OBnVXtq4XXMcuj9zASSTrJ/8a5Ibq1p1enbaOAx7/oLwkO9QuOAKcv52LrG3Z1M25UYSJcPaOBb/6ffYX362b+/eRY1u1xQSERERaWsrgIEp73cEVqUeYK1da62t9t7eA0xsxblVwNOkPy1FZJski/pTZY4hb9FcrfLQBblTRxa026kjzYn3GcOGY59m85RfEV7zMSVzDyL6xq+hpjLboYk0SwkMERERaWtvAcONMUOMMRHgRNyEQx1jTL+Ut0cBi73XzwMHG2NKvOKdBwPPG2MKa88xxoSBw4ElPt+HCJXjz4NEjOhCrUbSlQQ3r6TgP9e5q47selq2w9k2wRBVu57GupMXUD38aAreuYMecw4gsuzFbEcm0qS0ppCksdTZrcD+3tso0MdaW2yMGQz8xTsvB7jDWvtH75yJwP1APvAccJG1VsW2REREOjlrbdwYMwM3GREC7rPWfmyMuQ5421r7NHChMeYoIA6sA870zl1njJmJmwQBuM7btgPwtDEm17vmS8Af2/TGpEtKlAwlNvRw8j6aTcWE83Byu2U7JPGb41D08mXtb9WRbeREe7H5wNuoGvk9ChdcSfdnz6R6yCGUTbqOZNGAli8g0oYCjtN8zsBb6uwT4CDcYZtvASdZaxc1cfwFwHhr7VneU5WAN3+1EPgI2Mdau8oY81/gIuAN3ATG7621/2gulpqahOPHHBvNjfKf2th/amN/qX39pzb2n19t3Lt3UYes/uZXvwL0++y39ta+4dUfUvLYYZTtdTmVE2dkO5yMaG9t3J7kLXqEopcvY/PkG6ja7Yxtvk67bONEjPyFsyh461YAyr/1YyrHngOhSJYD2zbtso07EZ9rYDTat0gnXdjapc5OAuYAWGtjKfNXc2s/zxvi2c1a+7o36uIB4Dtp3YmIiIiISDsS770bsYFTiC6cBXHVEOjMtkwd2bvjTh1pTihC5YTzWHfSy8QGTqbw9V9R8uih5Kx8PduRiQDpTSFJd6kzvCkjQ3CHbdZuGwg8CwwDLvVGX+zuXSf1mi2OT/JruTMtr+M/tbH/1Mb+Uvv6T23sP7WxiH8qJs6g+KnjyVv8KFW7nZntcMQPjkPR/NqpI7d0+KkjzUl225FNh99LZOk8Cv/9C4qfOp4qcxxl+1yFE+2V7fCkC0sngZHOUme1TgSesNYmajd4a7aPMcb0B54yxjzRymvW8Wu5Mw0t8p/a2H9qY3+pff2nNvafj1NIMn5NkY6mpv9e1PSdSPS9P1I16hQI5WQ7JMmwvMVziSxfwObJ13e4VUe2VWzIQazbcT+i7/ye6Ht/JLJsHuV7XU7VqJMhGMp2eNIFpZM2bHG5shQn4k0fachauwr4GJjkXXPHNK8pIiIiItK+BQJUTJhBaPMKcj/7W7ajkQwLbl5Fwau1U0dOz3Y4bSsnn4q9fsb6771AvNdoihZcQfGTRxNe/WG2I5NsStRAxbo2/9h0EhgtLnUGYIwxQAnwesq2HY0x+d7rEmBfwFprvwQ2G2P2MsYEgNMB/U0vIiIiIh1WbKdpxHsYou/8AZxktsORTHEciuZfSiCZ6PRTR5qT6DGcjUc/yqYDf09o80qKHz+CwleuIlC9KduhSRsJblpO3kcP0u25s+l5726EbzNQ07ajZ1ucQpLmUmfgFu+c22Ap1JHALcYYB3fayG+ttbWpuulsWUb1H96PiIiIiEjHFAhSMeF8ur14IZGl84jtfEi2I5IM6IpTR5oUCFBtjiG20zQK3vwNeR/OJvL5c5Tv+wuqhx8NgQ65KJU0JVZOZNXrRJbPJ2f5AsIblwKQKNqR6uFHkzP2GMhp29paLS6j2p5oGdWOS23sP7Wxv9S+/lMb+0/LqNanZVQ7rnbdvsk4PR6eTDKvBxuOe6bD/oOuXbdxGwpuXkXJ3GnEe+/KxqMfzejoi87QxuFvPqBwwRXkfLOQ2I77UTb5BhIlQ7MdVp3O0MZtynEIrV1MZPl8N2nx5VsEkjU44TxiA/ahZuAUYoOmkijeGQKBrCyjmk4RTxERERERSUcwTMX46RQtuIKcla9Rs+O+2Y5ItlXq1JH9f9tlp440J95nDBuOfZq8RQ9T8PpNlMw9kIrx06nY/QII52c7PElDoHItkdJXiCxfQE7pK4QqvgEg3nMElWPOIjZoKjX9vgXhvCxH6lICQ0REREQkg6pGHE/0rVuJvnMnG5XA6LDqTR3pPjjb4bRfwRBVu55O9c6HUfja9RS883vyPn2Kskkzie00LdvRSUOJGnK+fpec5QuILJ9PePWHBHBI5pUQGziZ8oFTqBk0mWRB32xH2iglMEREREREMimcR+XYH1D4+o2Ev1lIvM/YbEckrdSlVx3ZRk60N5sPvJ2qkd+jcMGVdH/2DKp3PpSy/a4lWTQg2+F1acFNy4l4CYucla8RjG3GCYSI951AxR4/ITZoCvHeYzrE0rhKYIiIiIiIZFjVrqcRffcuou/cyabD7sl2ONIamjqyXWoG7MP67z1P/vt/ouDt2+jxyFTKv3UJlWN/AKGcbIfXNdRUEFn5OjnL5xMpXUB4wxcAJAoHUD3sKGKDplCz4744ud2zHGjrKYEhIiIiIpJhTqSIyt3OpODt2wmt+5REj+HZDknSlLf4UU0d2V6hCJUTZ1A9/GgK/30Nha/fQJ59grIpN1LTf89sR9f51Cu+ucArvhlzi2/2d0cRucU3h3bYwsK1lMAQEREREfFB5ZiziL5/N9H3/sDmabdmOxxJgzt15FpNHcmQZLeBbDriPiJLX6Dw37+g+K/HUjXieMr2uQonv2e2w+vQApXr3OKbpQvIWf4KoYqvAYj3MFSO+X67K76ZKUpgiIiIiIj4wMnvSeWok8n/6AHK9/ip6gC0d3VTR+KaOpJhsSEHs27H/Sh4+3by37+byNIXKN/rCqpGn6x2TleihvDX77mjLEoXEP7mA7f4Zm6xW3xz0BRqBk4mWdgv25H6SgkMERERERGfVI47l/yPHiD/vT9SPnlmtsORZtRNHZk0U1NH/JATpXzvK6gyx1L4ypUULbicvCWPUjblV8R775rt6Nql4KZSt/hm6XxyVry6pfjmDuM7XPHNTFECQ0RERETEJ8miAVTvcgz5i+dQsftFONFe2Q5JGlE3daT/XlTtdka2w+nUEj12YePRj5H7yV8ofHUmxY8fTuVuZ1Kx56U4kaJsh5ddtcU3S70lTusV3/y2V3xzvw5ZfDNTlMAQEREREfFRxYTzyF3yOPkf3EvFXj/LdjjSkONQNP8yd+rIAbdoSkNbCASoNscSGzyNgjdvJv+DP5P72bOU7/cLqocd1eELTaatrvjmAreWxar/1hXfrOm/F2WdqPhmpiiBISIiIiLio0TJMGJDDyP/w9lUTjhPT5nbGXfqyHxNHckCJ6+Ysik3UDXieAoX/JxuL5xPbNFcyqbcQKJ452yH54tA1Xq3+ObyBeQsX9BI8c0p1PTbo9MV38wUJTBERERERHxWMWEGJZ8/R95HD1A54fxshyOeYJmmjrQH8R3GseG4Z8j7+EEK3vgNJXMOpGLCdComzoBwfrbD2z7JOOGv3iVSOy2krvhmd6/45tQuUXwzU5TAEBERkTZljDkUuB0IAbOstTc12H8mcDOw0tt0p7V2lrfvDOAqb/v11trZxpgo8DgwFEgAz1hrL/f9RkRaId5nDLGBpYQHAQAAIABJREFUk4m+P4vKMWd1/H+UdQaOQ9HLmjrSbgRDVO12JtU7H07hazMpePt28j55irLJM4kNPiDb0bVKcNMKIqXz3VEWK14lGNuEEwgS32ECFXtcQmzgFOJ9xnap4puZogSGiIiItBljTAi4CzgIWAG8ZYx52lq7qMGh/9/encfHXdX7H3/NlmWyp9mTrrQ9bSldoRSQRSqbVwoqS8EroHCVK7hd9SJXBS9evXjV6wZ6RVYV2VGKooBgAX/Q0pXSlJ5Slm5J0zVp9szy/f0xk3SSJm3a5ptJJu/n45FH5vud8/3OmdNpevLp+ZzPI9baG3tcWwjcCpwIOMBKY8xioB34obX278aYNOAFY8wF1tq/uP1+RI5Ey5wbyH/qcjI2PEbb9KuS3Z0RT6kjQ5OTVULjOT+nbeoisl/+Bnl/uor24z5M02nfJppTkezu9S7USlrNawS2xIIW/vp3AIhkV9A+8Z/oGB3ffDMjP8kdHf4UwBAREZHBNA/YZK19F8AY8zBwEdAzgNGb84DnrbV749c+D5xvrX0I+DuAtbbDGLMKqHKj8yLHIlR5KqHSOQRX/ZK2aVeCV1PxZFHqyNAXqjqNfZc/R+aau8ha8RMKNy+hed6/0TrjWvAFkts5x8G3d0Ns880tLxGofR1PpB3Hl06o8hSapn+SjtFnEimYqM03B5h+aoqIiMhgqgS2JhxvA07upd3HjTFnABuBL1trt/ZxbWXiRcaYfOBCYikqh+TzecjPDx5Z7/vJ5/O6dm8Z3uPrOeMr+B77BAXb/4pzwmXJ7k6fhvMYH5bj4PvLzXicCJ6L7yS/IDsp3UjpMR4wQVjw74TnXo7v2a+T/ep/kfX2E0Qu+BHO6PmHvXpAx7hlL573l+B95wU87y7B01QLgFM8heiJ1+JMOBtn9Cl4ApmkA+kD86pDWjI+wwpgiIiIyGDq7b+inB7HTwMPWWvbjTHXAw8AZx/uWmOMH3gI+FnnCo9DiUQc6utb+t3xI5GfH3Tt3jLMx7f4dAoKDfzjx9RXfXjI7rswrMf4MNLfeoTcd1+g8fTbaPOUQpLeZyqP8cArhnPvJm3Sc2S/cgv+33yY1imX03zqN3AyC7u13NfSwYMrt/PkG7W0hiL4vB58Hk/su9eD1wN+74HjxOf8CecCnihTIpbZoVXM7FjFhNBGvDg0e3N4K2MOG4quZGP2STSmleCr9+Bb48G3dlO3e/T2Gl2v1cs5n6ezD/R+bR997euc1wMeF1eAuPkZLi7uvVqTAhgiIiIymLYBoxOOq4CaxAbW2j0Jh78Gvp9w7Vk9rl2ScHwX8La19icD1FeRgefx0jLnc+T+7Yukvf83Osafm+wejSjephqy//HteOrINcnujhwJj4eOCeexd/TpZK34CZlr7iL9vWdpPuVm2qZdwe7mEL9dsY0n36ilPRzl7MlFTC7PpbklRCTqEHGc2Pf4V7jHcSTqkB/awfS2FcxoX8n0jjfIcpqJ4MX6JvO7tEW87pvFeo4jFPUS2e8QrneIOHu73yd+33C0Z2w+OXweDgqSdAt29BIE8XoOE2zxxIIsY4uz+dTcSvy+wQvEKoAhIiIig2k5MMkYM55YlZFFwJWJDYwx5dba2vjhQuCt+ONnge8ZYwrix+cCN8ev+S8gD7jO3e6LHLv2SRcRWfZDgit/Tse4c5QjP1gch+y/3xSvOvLDIbv6RQ4jEKT5lP+gbfLHyX75P8hZchN7lz7A1xs/ybroWM6bWsKn5o1h3Kjg4VcIdG2++RJpW1/Cv28TENt8s+O4hTTEN98szsjnAuCCI+xqNCFIEu4R4OjrXOyYwwZbIk7C9b0ET7q/Nv0O4nTeN9r5OBKlvY++7moJcdWcykENKiiAISIiIoPGWhs2xtxILBjhA+611lYbY24DVlhrFwNfMMYsBMLAXuCa+LV7jTHfIRYEAbgtfq4K+AawAVhljIGE0qsiQ47XT8ucfyXnpf8gsP1VQlWnJbtHI0L6hkdJ3/J3Gk+/jWjeuGR3R47RVv9YHsj6b3zhx/h664P8IfANdk+5Cu/pN+Gk9bEvg+Pg22vjm28u6bH55nyapn2CjjFnEimYNCCBRa/Hg9fnIZCi1VKTkQblcZyhsbSlP0KhiOPGACn/zH0aY/dpjN2l8XWfxth9bo1xcXHOsPzvYzfmFeGow69f28zJE4uYWZyFzzssh2bIS4mfF+E2Rv3mFMJFU2lY+Ptk9+YgKTHGCbxNNRQ8tIBw0TQaLn5sSKy+SLUxHixb97Vy37ItPPPWTrweWDi9jE/PzGVC9U/IqP4d0awSmk/7Nu0TP0J+QRYNO7aTtvWVrlUWvuYdAIQLJtMx5iw6xpxJqGIe+DOT/M6GH5f3wOj1H9B+rcAwxpxPbDdvH3C3tfb2Hs//GPhg/DAIlFhr840xs4BfArlABPiutfaR+DX3A2cCDfHrrrHWrjmSNyUiIiIylLSFIjz71k7uXbqFirwMLplZzsLpZeRlJrnknww9/gxaZl1H9mv/jX/nWsIlM5Ldo9TVLXXkR0MieCFH7r09Ldy7bAvPbdhJwOflkpnlXHXSaEpyYvU+ms76b9qmXkb2S/9B7nP/Sse6+/ERZlTNKjw4RNPz6Kg6nZYxZ9Ix+kyiORVJfkdyNA4bwDDG+IA7gXOIbZ613Biz2FrbVa/dWvvlhPafB2bHD1uAq6y1bxtjKoCVxphnrbX18ee/Zq19fIDei4iIiEhSZaf7efxTJ7K8ton7/997/Ozl9/jVq5s5b0oxl86qYEpp77uqy8jUNv0qgivvJLjqDvaff1eyu5OylDoyvL29q4l7l27hhY27yQh4uXJuFZ84sYqirLSD2oZLZ1N/yZ/IWPcbgqvugPzRtJz0JTrGnEW4ZBZ4UzSXYwTpzwqMecCmznJkxpiHgYuA9X20vwK4FcBau7HzpLW2xhizEygG6vu4VkRERGRY8/u8XDC9jFOqctm0q5nH1tTwzPo6Fq+r44TyXC6bXcGCyUUEBnHXdhmanLQcWk+4muDKO/Dt20SkYGKyu5RyYlVH/pOOipNVdWSYeauukXte28JL7+whK83HNSeP5so5VeQHD7OizeujbcanaJvxKfLzg7QoTSel9CeAUQlsTTjeBpzcW0NjzFhgPPBiL8/NA9KAdxJOf9cYcwvwAvB1a237oTri83nIz+9jQ5Zj4PN5XbmvHKAxdp/G2F0aX/dpjN2nMR58E4uzuPmcSdx4+niert7BE2/U8q1nNvDjJQE+OqOcj80o71r+LCNT64xrCb7xa4KrfkHjgv9NdndSS1fqSEipI8PI2pr93LN0M6++t4+cdD+fOWUsl8+pIDdDqXjSvwBGb5tn9LXz5yLgcWttJPGkMaYc+C1wtbU2Gj99M7CDWFDjLuAm4LZDdSQScVzZJEQb6LhPY+w+jbG7NL7u0xi7z8VNPAf8nqkmJ8PPlXOrWDSnkmWb9/Ho6hruXbqF+5dt4axJRVw6q4I5VXl4VE5zxHGCRbROvYLM6t/SPO8rRHMqk92llJG+4TGljgwjK7fWc8/SLSzfUk9+ZoDPfWAcl86qIDtdhTPlgP58GrYBoxOOq4CaPtouAm5IPGGMyQX+DHzTWru083xCffd2Y8x9wFf722kRERGR4cjr8XDKuEJOGVfItvpWnnijlsXrdvDCxt0cVxTkslkVnD+1lGCa8rRHktbZ15NZ/Vsy1/yK5tMP+f950k/eplqy//FtpY4McY7j8PqWWOBi9bYGCoMBvnjmBD4+s5zMVK09KsekPwGM5cAkY8x4YDuxIMWVPRuZWNH1AuC1hHNpwB+A31hrH+vRvtxaW2uM8QAXA+uO+l2IiIiIDDNV+Zl88cwJfPbUsTy3YRePrN7Of/9tEz97+T0unF7GJTPLGVuolJ+RIJpTSfvkj5K5/ve0nPhFnMxRye7S8OY4ZP/93/FEO5Q6MkQ5jsOr7+3jnqWbebO2kZLsNL76weO46IQyMhS4kEM4bADDWhs2xtwIPEusjOq91tpqY8xtwApr7eJ40yuAh621iekllwFnAKOMMdfEz3WWS33QGFNMLEVlDXD9gLwjERERkWEkI+Bj4QllXDi9lLU1+3lsTQ2Pr6nh4VXbmT+ugMtmVXDq+EJ8XqWXpLKW2Z8jfcPjZK69l5aTv5bs7gxrnakjTR/4T6WODDFRx+HlTXu4d9kW3qprojw3na9/aCIXHl9Gml+BJjk8j+P0tZ3F0BMKRRztgTE8aYzdpzF2l8bXfRpj97m4B8aw/M3arXkFHPtY727u4I9ra3lybS27mjqoyE3nklkVXDi9jPxMbWSXqj8vcv9yHYHtr7H3qqU4acndW2a4jrG3qZaChxYQHjWVho8+NqRXXwzXMT4akajDi2/v5t6lW9i0u5mq/Aw+NW8MH55Wgt/FikwjaYyTwc3x7WtuoR1RRERERIaYoqw0rjtlLNfMG82STXt4bE0NP3v5PX716mbOm1LMpbMqmFKqzVNTTcucGyl4969krPstrXM+l+zuDD+OQ/aSm+KpIz8c0sGLkSIcdXhuw07uW7aF9/e2Mq4wk/+8wHDulBL8WlUmR0EBDBEREZEhyu/z8iFTzIdMMZt2NfPYmhqeWV/H4nV1nFCey2WzK1gwuYiAi/+DKYMnXDqLjqrTyXzjblpnfBr8Gcnu0rCSvuEx0je/GEsdyR+f7O6MaOFIlGfW7+T+17ewtb6NiUVZfO8jUzl7UpHS4eSYKIAhIiIiMgxMLM7i5nMmcePp43m6egdPvFHLt57ZwI+XBLh4Rjkfm1FOaU56srspx6hl7o3kP3U5GRseo236J5PdnWGjq+pI+cm0zvhUsrszYnWEozxdvYMHXt9K7f52ppRk84OF0zhj4ii8KhMtA0ABDBEREZFhJCfDz5Vzq1g0p5Jlm/fx6Ooa7lu6hQeWbeGsSUVcOquCOVV5ePTLwrAUqjyVUMksgqt/Sdu0K8Cr6fphKXUk6dpCEZ56cwe/Wb6VnU0dnFCew00LJnHq+AL9LJIBpZ+IIiIiIsOQ1+PhlHGFnDKukO0NrTyxppan1u3ghY27mTAqyGWzK7hgainBNJUkHFY8Hlrm3kjeX64jfdPTtE/+aLJ7NOQpdSR5WkMRnnijlt8u38relhCzq/K45XzDvDH5ClyIKxTAEBERERnmKvMy+cKZE/jMqWN5bsMuHl1Tw+1/28TPX36PC6eXccnMcsYWBpPdTemnjvHnEi6YTHDlHbRPukgrCg5BqSPJ0dQe5rE1Nfx+5XbqW0OcNCaf780fw9zR+cnumqQ4BTBEREREUkRGwMfCE8q4cHopb9Y28ujq7Ty+poaHV21n/tgCLp1dwWnjC7WJ3lDn8dIy93Pk/u1LpL3/Ah3jz0l2j4YmpY4Muv1tIR5ZVcPDq7ezvy3MqeMLuHb+WGZU5Ca7azJCKIAhIiIikmI8Hg8zKnKZUZHLl87q4I9ra3lybS1f+WM1FbnpXDKrggunl5GfGUh2V6UP7RMvIrLshwRX3UHHuA+BluMfJN0+rtSRQVLfEuL3q7bx6OoamjsinHncKD49fwzTylTOWQaXAhgiIiIiKawoK43rThnLNfNG89I7e3h0dQ0/e/k9fvXqZs41xVw2u4IppfolZMjxBWiZfT05L3+TQM1SQpWnJLtHQ4q3qZbsV25V6ojL9jR38LsV23jijRraQlEWTC7i0/PHMKk4O9ldkxFKAQwRERGREcDv87JgcjELJhezaVczj62p4Zn1dTxdXccJ5blcNruCBZOLCPi0DH+oaJt6OVnLf0Jw1R00KIBxgOOQveTrSh1x0c7Gdn6zfCt/fHMHoUiUc6eU8KmTRzNhVFayuyYjnAIYIiIiMqiMMecDPwV8wN3W2tt7PH8N8ANge/zUHdbau+PPXQ18M37+v6y1D8TPfxe4Ciiw1uq/Bg9jYnEWN58ziRtPH8+f1tfx+JoavvXMBn68JMDFM8r52IxySnPSk91N8WfSMvM6spfejn/nWsIlM1x9uajjsGVfKxt3NuFPDzAuN41xhUG8Qyx9JZY68gJNH/i2UkcGWO3+Nh54fSuL1+0g6sCHp5ZwzcljGFOQmeyuiQAKYIiIiMggMsb4gDuBc4BtwHJjzGJr7foeTR+x1t7Y49pC4FbgRMABVsav3Qc8DdwBvO32e0glORl+rphTyeWzK1i2eR+Prq7hvqVbeGDZFs6aVMSlsyqYU5WncohJ1Db9KoKr7iS46k72n/+rAbtvOBLl3T0tbNjZxMadTWyoa+LtXc20hCLd2mWl+ZhalsPx8a/p5TkUZycvuNU9deTTSetHqtlW38r9y7byp/V1eICF08u4et5oKvIykt01kW4UwBAREZHBNA/YZK19F8AY8zBwEdAzgNGb84DnrbV749c+D5wPPGStXRo/50qnU53X4+GUcYWcMq6Q7Q2tPLGmlqfW7eCFjbuZMCrIZbMruGBqKcE0X7K7OuI46bm0Tb+azFV34tv3DpGC4474Hm2hCJt2N7Ohrgm7M/a1aXczoYgDQGbAy+TibC6cXoopycaUZFOYH2TZ27tYV7uf6h2NPLhiG+ForH1JdhrTugIauUwpzSY7fRB+rVDqyIB7f08L972+hWff2onP6+HjM8r55ElVlOUqcCFDkwIYIiIiMpgqga0Jx9uAk3tp93FjzBnARuDL1tqtfVxbebQd8fk85OcHj/byw9zb69q93ZafH+T4saP42gVT+dObtfxu2RZu/9sm7njlfT42u5JPzBvNhCRv4Decx/eonPF5WHs3+dV3EfnIzw/ZtLEtxPraRtbX7qe6Zj/ra/fzzq4m4rEH8jMDTKvI5epJxRxfnsu08hzGjcrC26O0rs/nZWLJgT/n9lCEt3Y08sa2etZua2Dt9gaWbNoDxAqkHFeUzczRecyozGNmVR6TS3MGfD8Vz9qH8G9+gcg53yN33PEDeu9kSObneGNdI79Y8g7PVO8gw+/j6lPGce0HxlGSk1qBixH3s2KQJWN8FcAQERGRwdRbLoLT4/hpYqsq2o0x1wMPAGf389p+i0Qc6utbjvbyQ8rPD7p278F0znGFfGhCAW/WNvLo6u38/vUt/GbpZuaPLeDS2RWcNr4Qn3fw00tSZXz7L4vsqZeT8ebv2T/zi0RzKgDY29LRbVWF3dnEtvq2rquKs9MwJdmcMWEMU0qymVKaTWlO+kEpQfv3tx70ir2N8bicNMZNLeGiqSUANLSGWF/XSHVtI9U7GnnxrZ08sSq2dU2634spye5KPTm+PIfKvIyjTkfyNtVS8OzNdJSfTMOkf4YU+PNPxufY1jVx99LNLNm0h2DAx1UnjeYTcyspCKZBJJpyf69G3s+KweXm+BYX914dSwEMERERGUzbgNEJx1VATWIDa+2ehMNfA99PuPasHtcuGfAeSjcej4cZFbnMqMjly2d18Mc3a3nyjVq+8sdqKnLT+fjMChaeUEZ+ZiDZXU1ZjuOweeKnmbzuQTY98wN+nnYtdmcTu5o6utpU5WdgSrJZOL2sKw1kVFaaq/3Kywx0pR519rN2fzvVOzqDGvt5cm0tD8WDGnkZfo4v79xPI5fjy3LID/bjc6PUkWO2rnY/9yzdwj/e3Ut2uo/r5o9h0ZxK8vT3VoYZBTBERERkMC0HJhljxhOrMrIIuDKxgTGm3FpbGz9cCLwVf/ws8D1jTEH8+FzgZve7LJ1GZaVx7fyxXH3SaF56Zw+Prq7h56+8x12vbeZcU8xlsyuYUtr7/5pJ/yRWAklcXdHQFuZHgVO4YNdimrIu5MTRFUwpjQUqJhdnk5OR/Gm9x+OhIi+DirwMzjHFAISjDu/ubmbdjkbWx1dqLH1/S1dKS2VeRtcKjePLcjAl2WQEuu+1km6fUNWRo7R6WwP3LN3Mss315GX4uf60sVw+u3Jw9iwRcYE+uSIiIjJorLVhY8yNxIIRPuBea221MeY2YIW1djHwBWPMQiAM7AWuiV+71xjzHWJBEIDbEjb0/B9igZCgMWYbsfKs3x7Etzai+H1eFkwuZsHkYjbtbubxNTX8ubqOp6vrOKE8l0tnl7NgUjFpfv1P+aF0VgLpDFL0rAQS8HmYWJTFWZOKmFKSTVXm1wj+bSEPTltJy8lnJ7n3/eP3ephcks3kkmw+NqMcgJaOCG/VNbJ+Ryyg8UbNfp6zuwDweWPveXp5DtPKcpiV18LsV24lVD5PVUf6yXEcVmyt556lW1i5tYHCYIDPnz6ej88qJytNv/7J8OZxnKNOHR10oVDEcSPHRrlR7tMYu09j7C6Nr/s0xu5za4yLi3OGZY1Nt+YVMDI/z41tYf60vo7H19SwZV8rhcEAF88o52MzyinNGdiym8NxfPtbCWRKaeyX/Skl2YwfFTxoI8zcZ64lULOUvVctw0lzbzPVwR7j3U3tVO9oYv2O/ayrbWR9XSNN7WHuCfyQU73VfK3wDgqrpnJ8eQ7Ty3IoGeDPVDIM9Bg7jsNr7+/jnqVbWFuzn6KsND55UhUfm1F+0KqWkWI4/qwYTlzeA6PXuYVCcCIiIiJyzHIy/Fwxp5LLZ1fw+uZ9PLq6hvuWbuGBZVs4c2IRl82uYE5V3lFv4jicNLWHu22suaGuiff3tnSlTeRl+Jlcks2i2ZWx/SpKsxmdn9mvDVFb5t5IwXvPklH9O1pnX+/yOxk8RdnpnDkxnTMnjgJiqTTNq37PhKWrearkBt7tKOevKw+Uci3OTuP4stgqjenlOUwtzRmxaRGO4/DyO3u5Z+lm3qprojQnnX9fMJGF08tI1yooSTEj82+5iIiIiLjC6/Ewf1wh88cVsr2hlSfW1LJ43Q5efHs3E0YFuXRWBR+eVkowLTX+R3hvS0dXkOJQlUA+GE8DMaXZlPVSCaS/wqWz6aj6AJlr7qL1hGvAn1plLzv5W+oYt/p7hMrncepHb+JUj5eOcJSNu5q6qp5U72g8UMoVGFcYZFp8hcbx5TlMLMoa8FKuQ0nUcfj727u5Z+kW3t7VTGVeBt84ZxL/dHxpSr9vGdn6FcAwxpwP/JRYrurd1trbezz/Y+CD8cMgUGKtzTfGzAJ+CeQCEeC71tpH4teMBx4GCoFVwCettR2IiIiISEqozMvkC2dO4DOnjuU5u4tHV9fw/Rc2cccr7/GR40u5ZFYF4wqDye5mvziOQ11je1egYsPOJjbubGJnQiWQyrwMppS6XwmkZc6N5C9eRMaGx2mb/s8Dfv+kcxyy/34Tnkh7t6ojaX4v08tzmV6e29W0oTXEW3WNXZVPXntvL3+urou193kwJTkJlU9yqMo/+lKuQ0Uk6vC83cW9y7bw3p4WxhRk8u3zDedNLcGfhLLGIoPpsAEMY4wPuBM4h1j5suXGmMXW2vWdbay1X05o/3lgdvywBbjKWvu2MaYCWGmMedZaW0+sJNqPrbUPG2P+D7iWWLBDRERERFJIRsDHwullXHh8KW/WNvLo6u088UYtj6yuYf7YAi6ZVcEHJhT2K4ViMEQdh637WrulgHRWAgHwemBsYZC5o/MxJfF9KwaxEkio6jRCJTMJrv4lbdMWgTe1FlUnVh2J5E84ZNu8zEDXih+IBZp2NLYfWKVRu58/rq3l4YRSrtPiwYzOwEZB0N1yswMlHInyl7d2cv/rW9myr5UJo4J895+msGBy8ZD5uyPitv78tJsHbLLWvgtgjHkYuAhY30f7K4BbAay1GztPWmtrjDE7gWJjTANwNgfKpj0AfBsFMERERERSlsfjYUZFLjMqcvnyWR388c1annyjlq8+VU15bjqXzKxg4Qll5GcGBq1PPSuB2J1NbNzZdyUQU5LNpOKs5G6K6PHQMvdG8v7yL6Rv+hPtky9OXl8GmLd5B9n/OPqqIx6Ph/LcDMpzM/hQQinX9/Y0sy4e1Fi/o5F7lx0o5VoRL+U6/RClXJMpFInydHUdD7y+lZqGNiYXZ/H9C6dy1qQivMN8NYnIkepPAKMS2JpwvA04ubeGxpixwHjgxV6emwekAe8Ao4B6a2044Z6Vh+uIz+chP3/glxn6fF5X7isHaIzdpzF2l8bXfRpj92mMZSgZlZXGtfPHcvVJo3npnT08urqGn7/yHne9tplzTTGXzq5gamnOgL5mZyWQxFUV7+xupqNHJZCPHF+KKY0FKyb0UglkKOgYfx7hgkkEV91B+6SLIBV+kXUcspd8/aDUkWPl93qYVJzNpOJsPhov5doaipVyra6NBTTW1e7n+c5Srh44riiL6eW5sY1Cy3MYXxgc9FUO7eEoT725gwde38LOpg6mleXwlQ8ex+kTCod9GozI0epPAKO3vx191V5dBDxurY0knjTGlAO/Ba621kaNMUdyzy6RiONKmRaV13Gfxth9GmN3aXzdpzF2n4tlVAf8njJy+H1eFkwuZsHkYjbtbubxNTU8s76Op6vrOKE8h0tnV7BgUjFpR1hNobdKIJv3thCPVZCb4ceUZHP5UVQCGRI8Xlrm3EDuC18ibfMLdIz7ULJ7dMzSNz5B+vt/61fqyLHKDPiYU5XPnKr8rnO7mztYv+NA6slzdidPrq0FIBjwMbUsu2svjePLcynJTnMlkNAWivDk2lp+u3wbu5s7mFmRyzfPm8z8sQUKXMiI158AxjZgdMJxFVDTR9tFwA2JJ4wxucCfgW9aa5fGT+8G8o0x/vgqjEPdU0RERERGgIlFWXz9Q5O48fTx/Km6jsfW1HDLM5afBN/l4hPK+NjMCkpz0g+6LrESyMb4Bpt9VQLp3LPiWCqBDBXtky4i8voPCa68g46xC4b1Kgxv8w6yX7mVUPlJtJ7wqaT0oSgrjTOOG8UZxx0o5bp1X2vXBqHVOxp5aNV2QvEoWFFWWre9NKaVHVsp1+aOMI+vqeXBFdvY1xrixNF5fOfDU5g7emSUHxbpj/78DVsOTIpXDdlOLEhxZc9GxhgDFACvJZxLA/4A/MZa+1jneWutY4z5O3AJsUokVwP0sc45AAAZgklEQVRPHcP7EBEREZEUkZ3uZ9GcSi6bXcHrm/fx6Ooa7lu2lQde38qZE4v48IwKNmyv71pd0bMSiClxvxLIkOAL0DLrs+S88i0CtcsIVcxPdo+OTrfUkR+Bd2jsP+H1eBhbGGRsYZAPTysFoCMc5e1dTV1lXKtrG3npnT1d14wrzOxaoXF8WQ6Tig9fyrWxLcwjq7fz8KrtNLSFmT+ugOvmj2FmZZ6r709kODpsAMNaGzbG3Ag8S6yM6r3W2mpjzG3ACmvt4njTK4CHrbWJqSCXAWcAo4wx18TPXWOtXQPcBDxsjPkvYDVwz4C8IxERERFJCV6Pp6vCxPaGVp58o5an3tzBi2/vTnolkKGibeoislb8hODKO2gYpgGMwUwdOVZpfm8sOJFQynV/W4i3dsSCGutq9/Pa+/v48/qdQGwTWFOSnbBSI5fR8VKu9S0d/N//e59HVm+nqT3C6RMKuXb+mG73FpHuPI5z2K0nhoxQKOJoD4zhSWPsPo2xuzS+7tMYu8/FPTCG5dpmt+YVoM+zm9pCEfaGHQr9niFVKSKZgit+Ttay77Pvsr8SLp4+IPccrM+wt3kHBQ8tIFI4mfqLHx8yqy+OheM41DW2xwMasZUab+1opC0cBWL7r0wuyeatHY00d0Q4e1IRn54/BlOSneSepx79LHaXm+Pb19xiZIWoRURERGRYywj4mFasX0oStZ5wFZmrf0HmqjtpPO+Xye5O/3WmjoTbhlTqyLHyeDyU5WZQlpvBgskHSrm+v6eFdbX7qd7RyIa6Js6eUsKVsyuYWJSV5B6LDB8KYIiIiIiIDGNOeh5t068ic9UvaDn5a0M+DaNTV+rIabcOmz4fLb/Xw8TiLCYWZ3FxvJSrVgeIHLmhV9RaRERERESOSMvM68CXRuaqXyS7K/3iba47UHVkxqeT3R0RGSYUwBARERERGeacYDFtUxeRYZ/A21ST7O4cWoqmjoiI+xTAEBERERFJAS2zrwcnSuaaXye7K4cUSx15nub5X0/51BERGVgKYIiIiIiIpIBo7mjaJ19MZvXv8LTtS3Z3eqXUERE5FgpgiIiIiIikiJbZn8MTbiXzjXuS3ZWDKXVERI6RAhgiIiIiIikiMsrQPv48Mt+8D09HU7K70036xidR6oiIHAuVURUREZFBZ4w5H/gp4APuttbe3uP5a4AfANvjp+6w1t4df+5q4Jvx8/9lrX0gfn4ucD+QCTwDfNFa67j7TkSGnpY5N1Dw3rNkVD9I6+zPJrs7QGfqyC1KHRGRY6IVGCIiIjKojDE+4E7gAmAacIUxZlovTR+x1s6Kf3UGLwqBW4GTgXnArcaYgnj7XwKfASbFv853952IDE3hsjl0VJ5K5ht3QaQ92d1R6oiIDBgFMERERGSwzQM2WWvftdZ2AA8DF/Xz2vOA5621e621+4DngfONMeVArrX2tfiqi98AF7vReZHhoGXu5/E115Gx4fFkdyUhdeQmpY6IyDFRAENEREQGWyWwNeF4W/xcTx83xqw1xjxujBl9mGsr448Pd0+RESFU9QFCJTMJrvoFRMNJ60f31JFrk9YPEUkN2gNDREREBpunl3M996p4GnjIWttujLkeeAA4+xDX9uee3fh8HvLzg/3o7pHz+byu3Vs0vv3lOf3f8D1xNQW1z+Mc//EjunZAxthx8D33DTyRdrjoTvILc47tfilGn2P3aYzdlYzxHfEBjEjUYVdjOz7Hwevpbe4jIiIiA2wbMDrhuAqoSWxgrd2TcPhr4PsJ157V49ol8fNVh7pnT5GIQ319yxF0u//y84Ou3Vs0vv1W+kEKCibCKz+mvuJ8OIK57kCMcbp9gty3/0rTabfQ6qsA/Zl1o8+x+zTG7nJzfIuLew94jvgAxv+8sIkn19bi93ooyUmnNCedss7vubHvsXMZZKf78CjIISIicqyWA5OMMeOJVRlZBFyZ2MAYU26trY0fLgTeij9+Fvhewsad5wI3W2v3GmMajTHzgWXAVcDPXX4fIkObx0vL7M+R++K/kbb5RTrGLRi0l+5KHSk7UakjIjJgRnwA49PzxzBjTAHv1jVS19hGXWM7a7Y3sLOxnUiPhafBgI/S3L6CHBmU5qST7te2IiIiIodirQ0bY24kFozwAfdaa6uNMbcBK6y1i4EvGGMWAmFgL3BN/Nq9xpjvEAuCANxmrd0bf/yvHCij+pf4l8iI1j75o0Re/xHBVXcOXgDDcchecnOs6siC/1XVEREZMB7HGT7l0UOhiOPGEpXelr5Eog57mjvY0dhOXfxrx/62rsd1je3sbQkddK+CzEC3lRuxIEdG1+OirDR83pG3ikPLt9ynMXaXxtd9GmP3uTXGxcU5w/IfNrfmFaDPs9s0vkcmY+295LxyC/UffYJQxcn9uuZYxjjdPknu374QSx2Z9ZmjusdIoM+x+zTG7nI5haTXucWIX4HRF188paQkJ73PNu3hKDs7gxvx1Rs79seON+9rZfmWepo7Igfdtzgr7aCVG4lBj7wMv1JVRERERGRAtE29gqwVPyVz5R39DmAcrVjqyLeUOiIirlAA4xik+72MLshkdEFmn22a2sNdQY2eQY51tY28sHE34Wj3VTAZfm+3oEZZzoEVHKW5sfSVjICW4omIiIhIPwQyaZ1xLVnL/gffrmoixce78zpKHRERlymA4bLsdD8Ti/1MLM7q9fmo47C3JURdPD2le8pKO+/s3see5o6D6sDlZfi7paeUdUtZSacoOx3/CExVEREREZGDtZ5wNZmrfkFw1Z00nvcLV14jfeMfSH//OZpOu4VI/gRXXkNERjYFMJLM6/FQlJVGUVYax5f33iYUibKz6cDKjcSvmoY2Vm9roLE93OO+UJSVRmlORsJKju6rOPIzA0pVERERERkBnPQ82k64iszV/0dL/VcHPMCg1BERGQz9CmAYY84Hfkpsp/C7rbW393j+x8AH44dBoMRamx9/7q/AfOAf1tqPJFxzP3Am0BA/dY21ds3Rv5XUFfB5qczLpDKv71SV5o5wt/SUxNUcG+oaeWnTbjp6lFVJj6eqlCRWVYkHODpTV4JpWvonIiIikgpaZlxH5hv3kLn6lzR98AcDd+PE1JGzf6TUERFxzWEDGMYYH3AncA6wDVhujFlsrV3f2cZa++WE9p8HZifc4gfEghqf7eX2X7PWPn6UfZcEWWl+JozyM2FU76kqjuOwrzUUC27sPxDc6Ax4vL55H7ubO+ixHQc56f6Dqqok7s9Rkp1OwKfSsSIiIiJDnZNVQtvUy8lY/xAtJ/0b0ew+lv8eoa7UkVO/RaTguAG5p4hIb/qzAmMesMla+y6AMeZh4CJgfR/trwBu7Tyw1r5gjDnrGPspx8jj8VAYTKMwmMbU0pxe24QjUXY3dyRsOtq9fOybNftpaOuequIBRnWrqpK4kiO2P0dhMDAI71BEREREDqdl9vVkVD9I5ppf0/yBW475fp7mnfHUkbm0zrxuAHooItK3/gQwKoGtCcfbgF7rLxljxgLjgRf7+frfNcbcArwAfN1a236oxj6fh/z8YD9v3X8+n9eV+w5HRaNgyiGeb+kIs6OhjZqGNmob2uKPW6ltaOO9va28+t4+WkPdS8cGfB5GZcX23CjICpCfmUZ+MEB+MEBBMI38zED8OI2C+LmcdD9ebUJ6RPQ5dpfG130aY/dpjEUkmjuG9kkLyaz+HS0nfh4no+Dob+Y45LzUmTqiqiMi4r7+BDB6+y2yZ1GMTouAx621kT6eT3QzsANIA+4CbgJuO9QFkYhDfX1LP259ZPLzg67cN1UVBrwUFgWZXnTwJNhxHBrawt0qqdQ1ttMcibKroY2G1hC19a00tIZpaAsdlLLSyeuB3IwAeRl+8jITvwfIy/STnxnodj4//n0kp7Poc+wuja/7NMbuc2uMi4t7X9knIkNTy5wbyNj4BzLX3kfLvH876vukb/wD6e89q9QRERk0/QlgbANGJxxXATV9tF0E3NCfF7bW1sYfthtj7gO+2p/rZGjzeDyxFRWZAUxJdtf53ibNUcehqT3cFczo/F7fGqKhLUxDa4iG1hD1bWF2NLZjdzbR0BamPRzt8/WDAR95mX7yMgLxIIe/K+gR+979XH5mgGDAp2osIiIiMmJERk2hfdy5ZK69h5ZZn4W03vdQOxSljohIMvQngLEcmGSMGQ9sJxakuLJnI2OMAQqA1/rzwsaYcmttrTHGA1wMrOt3ryUleD0ecjMC5GYEGE3fFVZ6agtFaGgLxwIdicGOhCBI5/dtDbHVHj3LzCbyez09VnkcWO2Rn9njcTzwkZMRwK8UFxERERmmWubeQMETz5G5/kFaZ33myC5W6oiIJMlhAxjW2rAx5kbgWWJlVO+11lYbY24DVlhrF8ebXgE8bK3tlhRgjHmF2LYK2caYbcC11tpngQeNMcXEUlTWANcP2LuSlJYR8JER8FGak97va8JRh8Zuqzw6Ax3dHze0hdmyr5WG2kYaWkOE+8pxIVah5bCrPTpTXOKPMwL6B15ERESSL1w2l47KU8hc8ytaT7gafP2fV6W//UeljohIUngcp+9f0IaaUCjiaA+M4Wk4jrHjODR3RA5e2ZGw2qO+l5UfLaG+t4BJ93sPrPBI2Luje6CjezAkO92Ptx8pLsNxjIcTja/7NMbuc3EPjGG5JM2teQXo8+w2je/ACGx5ifynP0HjB/+HtmndF1j3Ncae5p0UPnQ2kYLjqP/ok1p9cQz0OXafxthdbo5vX3OL/qSQiIxIHo+H7PRYAKEyr//XdYSj7G+L7d2RuI9H7HH31R5v72qmvjVEY3u43xua5veW7pIZoLggSLQ9RIbfR0bAS2bAR2Yg9ngkb24qIiIivQuNPoNQ8QwyV/2CtimXHz4Y0ZU60qrUERFJCgUwRAZYmt9LUXY6Rdn9X4oZdRwa28K97udR3yPwUbu/jQ11ocNuaJrI5/WQGQ9qZPi9ZMSDG5kBb68Bj8x4mk6m39t1ruc1mQnnfNoPREREZPjxeGiZewN5f/0s6e/8mfZJCw/Z/EDqyDeVOiIiSaEAhsgQ4PV4ulZSUHBkG5p2Vm3xpwfYta+Z1lCU1lCEtnCUtlCE1lCE1lDscVv8udZQhNZwlJaOCHuaQ13n2sOx5w+x9UevAj5PV3CkK/iREODISAiEJB53D5j0uNZ/4Pn+pNCIiIjIkeuYcAHh/OMIrryD9okXQh//5nqad5L9cmfVkX8Z5F6KiMQogCEyjGUEfJQFfJTlxnPQ8vq/6qMvjuPQEXFiQZDOoEc4Ej/uDIocCJJ0tmtNeK4t3n5/W5idje0HXXOk0v3eg4IenYGOzqDHgedj39P9Bx73dk1mfOVJut+rMroiIjJyeby0zPkcuS9+hbQtf6dj7NkHt1HqiIgMEQpgiEg3Ho+HdL+HdL8XMgMDfn/HcbpWehwIdhxYLXJglUh81Ug4IfiREDRpDUXZ1xKiNtTetYKkLRztd1pN1/uFbitDuqXKBHxd6TL52el4otFuwZID1x0IkPR8Lt2vFSQiIjK0tU/+KJHXf0jmyjt7DWAodUREhgoFMERkUHk8nq5SuAUu3D/qON1SZfqTStPzXOc1u5o6ugIn7RGH1o4wHZEjr9yUuIKkZ7CkZ9Cj23Ff53vsX6JNWkVE5Jj40mid9Vmy//Ft/LXLCZef1PVUV+pI6RyljohI0imAISIpxevxEEzzEUwb2OWtnWWiwlGH9nD3YEhbYopNV1pN9xUkbfFgSXvCKpL9bWHqGtu7AiadQZQjDZH4vJ6+AyQHrQg5eC+SvlaPdB6naRWJDDBjzPnATwEfcLe19vY+2l0CPAacZK1dYYxJA34FnAhEgS9aa5fE214OfCN+zz9ba//d9TcikkJap11JcMXPCK68g/0feSB2MjF1ZIFSR0Qk+RTAEBE5An6vB3+an6w0d+5/0B4kicGN8IEgR8+gR+JKk/aEIEpDa+hA2/j50FGsIuktINJXykxvG7f2FTjpPPZrFcmIYYzxAXcC5wDbgOXGmMXW2vU92uUAXwCWJZz+FwBr7QnGmBLgL8aYk4AC4AfAXGvtLmPMA8aYBdbaFwbhLYmkhkCQ1pnXkrXsB/h2r4f8E0l/+6mE1JGJye6hiIgCGCIiQ4nbe5AAh11F0jNA0tsqks79SxoGaBWJ3xtLLQp4PQR8HgI+L2k+LwGfhzS/N37c/Xy3Nj4vAX+sTexc7LE/3qbbtf7ENon36n5Pbe7qmnnAJmvtuwDGmIeBi4D1Pdp9B/gf4KsJ56YBLwBYa3caY+qJrcZwgI3W2l3xdn8DPt7ZVkT6p3X61WSu+gXBVXdC2e1kv/xNpY6IyJCiAIaIyAgz6KtIEoMgh1hFgs9LY0sHoUiUjohDKBIlFHHoiEQJRaK0dEQJRcJdx51tOuLtQpHoEZcAPpRAr8GQA0GOrsf+g9skXncgcHLguOe9D9zzQIDF740FbxLbpUhQpRLYmnC8DTg5sYExZjYw2lr7J2NMYgDjDeCieNBjNDA3/v1FYIoxZlz8fhcDLn3CRVKXk5FP2/RPkrnmLpwndyp1RESGHAUwRERkQB3tKpLOfUaORTgaD2qEDwQ5ugU8wlFC0QOPO3oESbqCIuEooW736t6m83FrKMr+tjChaLxNL/ccyKCK3+uh58qUgLfvVSqdgZOAN/bn8clTx1OakfRfRHqLwnSNkjHGC/wYuKaXdvcCU4EVwGbgVSBsrd1njPlX4BFie2O8Ckw4XEd8Pg/5+cEj7X+/+Hxe1+4tGl9Xnf4FWHsf3q2vEVnwn+SMn5HsHqUsfY7dpzF2VzLGVwEMERFJGX6vB783Vgp3qOgWVOkzKHJwUOXgVSY9Aixd13dv0x6O0tQePhBIid8zEnWYM34UpePdqP9zRLYRWzXRqQqoSTjOAaYDS4wxAGXAYmPMQmvtCuDLnQ2NMa8CbwNYa58Gno6f/wwQOVxHIhHnmINmfRmIgJz0TePrplyCJ36JzIb17J18DWicXaPPsfs0xu5yc3yLi3N6Pa8AhoiIiIuGUlBliEzklgOTjDHjge3AIuDKziettQ1AUeexMWYJ8NV4FZIg4LHWNhtjziG2+mJ9vF1JfF+MAuBzwGWD9o5EUkzLiZ8nLT+o4IWIDDkKYIiIiMigsdaGjTE3As8SK3l6r7W22hhzG7DCWrv4EJeXAM8aY6LEgh+fTHjup8aYmfHHt1lrN7rRfxEREUkeBTBERERkUFlrnwGe6XHulj7anpXw+H3A9NHuioHroYiIiAxF3mR3QERERERERETkcBTAEBEREREREZEhTwEMERERERERERnyFMAQERERERERkSFPAQwRERERERERGfIUwBARERERERGRIU8BDBEREREREREZ8jyO4yS7D0diF7A52Z0QERGRbnYD5ye7E0dB8woREZGhqde5xXALYIiIiIiIiIjICKQUEhEREREREREZ8hTAEBEREREREZEhTwEMERERERERERnyFMAQERERERERkSFPAQwRERERERERGfIUwBARERERERGRIc+f7A4kmzHmfOCngA+421p7e5K7lFKMMfcCHwF2WmunJ7s/qcYYMxr4DVAGRIG7rLU/TW6vUosxJgN4GUgn9jPzcWvtrcntVeoxxviAFcB2a+1Hkt2fVGOMeR9oBCJA2Fp7YlI7lMI0r3CX5hXu09zCXZpXDB7NLdyVrLnFiF6BEf9Q3wlcAEwDrjDGTEtur1LO/cD5ye5ECgsDX7HWTgXmAzfoMzzg2oGzrbUzgVnA+caY+UnuUyr6IvBWsjuR4j5orZ2l4IV7NK8YFPejeYXbNLdwl+YVg0dzC/cN+txiRAcwgHnAJmvtu9baDuBh4KIk9ymlWGtfBvYmux+pylpba61dFX/cSOyHdGVye5VarLWOtbYpfhiIfzlJ7FLKMcZUAf8E3J3svogcI80rXKZ5hfs0t3CX5hWDQ3OL1DXSAxiVwNaE423oB7QMU8aYccBsYFmSu5JyjDE+Y8waYCfwvLVWYzywfgL8O7GlyuIOB3jOGLPSGPOZZHcmhWleISlFcwt3aF4xKDS3cF9S5hYjPYDh6eWcIqAy7BhjsoEngC9Za/cnuz+pxlobsdbOAqqAecYY5V0PEGNMZy77ymT3JcWdZq2dQyy14QZjzBnJ7lCK0rxCUobmFu7RvMJdmlsMmqTMLUZ6AGMbMDrhuAqoSVJfRI6KMSZAbILxoLX2yWT3J5VZa+uBJSj/eiCdBiyMbwT1MHC2MeZ3Se1RCrLW1sS/7wT+QCzVQQae5hWSEjS3GByaV7hGc4tBkKy5xUgPYCwHJhljxhtj0oBFwOIk90mk34wxHuAe4C1r7f8muz+pyBhTbIzJjz/OBD4EbEhur1KHtfZma22VtXYcsZ/BL1pr/znJ3UopxpgsY0xO52PgXGBdcnuVsjSvkGFPcwt3aV7hPs0t3JfMucWIDmBYa8PAjcCzxDYoetRaW53cXqUWY8xDwGuxh2abMebaZPcpxZwGfJJYZHlN/OvDye5UiikH/m6MWUvsl5PnrbV/SnKfRI5EKfAPY8wbwOvAn621f01yn1KS5hXu07xiUGhu4S7NKyQVJG1u4XEcpWaKiIiIiIiIyNA2oldgiIiIiIiIiMjwoACGiIiIiIiIiAx5CmCIiIiIiIiIyJCnAIaIiIiIiIiIDHkKYIiIiIiIiIjIkKcAhoiIiIiIiIgMeQpgiIiIiIiIiMiQ9/8B751yjLpsD9cAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plot_learning_curves(metrics)" ] }, { "cell_type": "code", "execution_count": 189, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "89/89 [==============================] - 51s 568ms/step - loss: 0.7476 - accuracy: 0.5065 - AUC: 0.5058\n" ] } ], "source": [ "second_loss, second_accuracy, second_AUC = model.evaluate(test_seq_data, steps = validation_steps)" ] }, { "cell_type": "code", "execution_count": 190, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Transfer loss: 0.75 | Transfer Accuracy : 50.65% | Transfer AUC: 50.58%\n" ] } ], "source": [ "print(f'Transfer loss: {second_loss:.2f} | Transfer Accuracy : {second_accuracy:.2%} | Transfer AUC: {second_AUC:.2%}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fine Tune all Layers for Target Task" ] }, { "cell_type": "code", "execution_count": 191, "metadata": {}, "outputs": [], "source": [ "model.trainable = True" ] }, { "cell_type": "code", "execution_count": 192, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"RNNStocks\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "EmbedLayer (Embedding) (None, None, 256) 47360 \n", "_________________________________________________________________\n", "BiLSTM (Bidirectional) (None, 2048) 10493952 \n", "_________________________________________________________________\n", "BatchNormal (BatchNormalizat (None, 2048) 8192 \n", "_________________________________________________________________\n", "FullConnected (Dense) (None, 512) 1049088 \n", "_________________________________________________________________\n", "leaky_re_lu (LeakyReLU) (None, 512) 0 \n", "_________________________________________________________________\n", "BatchNormal2 (BatchNormaliza (None, 512) 2048 \n", "_________________________________________________________________\n", "Output (Dense) (None, 1) 513 \n", "=================================================================\n", "Total params: 11,601,153\n", "Trainable params: 11,596,033\n", "Non-trainable params: 5,120\n", "_________________________________________________________________\n" ] } ], "source": [ "model.summary()" ] }, { "cell_type": "code", "execution_count": 197, "metadata": {}, "outputs": [], "source": [ "# Name of the checkpoint files and save each weights at each epoch\n", "# checkpoint_dir = './training_Daily'\n", "# checkpoint_prefix = os.path.join(checkpoint_dir, \"daily.h5\")\n", "checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n", " filepath='daily.h5',\n", " verbose=1,\n", " monitor='val_accuracy',\n", " mode='max',\n", " save_best_only=True)\n", "\n", "early_stopping = EarlyStopping(monitor='val_AUC',\n", " patience = 20,\n", " mode='max',\n", " restore_best_weights=True)\n", "csv_logs = tf.keras.callbacks.CSVLogger('./daily_log.csv', separator=\",\", append=True)" ] }, { "cell_type": "code", "execution_count": 198, "metadata": {}, "outputs": [], "source": [ "base_learning_rate = 0.0001\n", "model.compile(optimizer =tf.keras.optimizers.Adam(lr=base_learning_rate / 10), loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n", " metrics=['accuracy', tf.keras.metrics.AUC(name='AUC')])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 7/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.7074 - accuracy: 0.4907 - AUC: 0.5021\n", "Epoch 00007: val_accuracy improved from -inf to 0.52844, saving model to daily.h5\n", "1610/1610 [==============================] - 3972s 2s/step - loss: 0.7074 - accuracy: 0.4907 - AUC: 0.5021 - val_loss: 0.6993 - val_accuracy: 0.5284 - val_AUC: 0.5046\n", "Epoch 8/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.7042 - accuracy: 0.4879 - AUC: 0.5028\n", "Epoch 00008: val_accuracy did not improve from 0.52844\n", "1610/1610 [==============================] - 3991s 2s/step - loss: 0.7042 - accuracy: 0.4879 - AUC: 0.5028 - val_loss: 0.7096 - val_accuracy: 0.5219 - val_AUC: 0.5014\n", "Epoch 9/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.6987 - accuracy: 0.4828 - AUC: 0.5115\n", "Epoch 00009: val_accuracy did not improve from 0.52844\n", "1610/1610 [==============================] - 3969s 2s/step - loss: 0.6987 - accuracy: 0.4828 - AUC: 0.5115 - val_loss: 0.7056 - val_accuracy: 0.5246 - val_AUC: 0.4984\n", "Epoch 10/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.6947 - accuracy: 0.4804 - AUC: 0.5213\n", "Epoch 00010: val_accuracy did not improve from 0.52844\n", "1610/1610 [==============================] - 4006s 2s/step - loss: 0.6947 - accuracy: 0.4804 - AUC: 0.5213 - val_loss: 0.7006 - val_accuracy: 0.5281 - val_AUC: 0.4954\n", "Epoch 11/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.6925 - accuracy: 0.4790 - AUC: 0.5279\n", "Epoch 00011: val_accuracy did not improve from 0.52844\n", "1610/1610 [==============================] - 3941s 2s/step - loss: 0.6925 - accuracy: 0.4790 - AUC: 0.5279 - val_loss: 0.6978 - val_accuracy: 0.5283 - val_AUC: 0.4954\n", "Epoch 12/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.6912 - accuracy: 0.4786 - AUC: 0.5326\n", "Epoch 00012: val_accuracy did not improve from 0.52844\n", "1610/1610 [==============================] - 3968s 2s/step - loss: 0.6912 - accuracy: 0.4786 - AUC: 0.5326 - val_loss: 0.6960 - val_accuracy: 0.5283 - val_AUC: 0.4930\n", "Epoch 13/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.6903 - accuracy: 0.4785 - AUC: 0.5368\n", "Epoch 00013: val_accuracy did not improve from 0.52844\n", "1610/1610 [==============================] - 3994s 2s/step - loss: 0.6903 - accuracy: 0.4785 - AUC: 0.5368 - val_loss: 0.6998 - val_accuracy: 0.5283 - val_AUC: 0.4959\n", "Epoch 14/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.6896 - accuracy: 0.4791 - AUC: 0.5404\n", "Epoch 00014: val_accuracy did not improve from 0.52844\n", "1610/1610 [==============================] - 4013s 2s/step - loss: 0.6896 - accuracy: 0.4791 - AUC: 0.5404 - val_loss: 0.7024 - val_accuracy: 0.5283 - val_AUC: 0.4965\n", "Epoch 15/94\n", "1610/1610 [==============================] - ETA: 0s - loss: 0.6889 - accuracy: 0.4805 - AUC: 0.5438\n", "Epoch 00015: val_accuracy improved from 0.52844 to 0.53020, saving model to daily.h5\n", "1610/1610 [==============================] - 3971s 2s/step - loss: 0.6889 - accuracy: 0.4805 - AUC: 0.5438 - val_loss: 0.7009 - val_accuracy: 0.5302 - val_AUC: 0.5010\n", "Epoch 16/94\n", "1036/1610 [==================>...........] - ETA: 23:07 - loss: 0.6882 - accuracy: 0.4822 - AUC: 0.5472" ] } ], "source": [ "epochs = 100 - (history.epoch[-1] +1) \n", "start = time.time()\n", "fine_tuned_history = model.fit(train_seq_data, epochs=epochs, initial_epoch=(history.epoch[-1] + 1),\n", " verbose = 1, validation_data=(test_seq_data),\n", " callbacks=[checkpoint_callback, early_stopping, csv_logs])\n", "end = time.time()\n", "print(\"Time took {:3.1f} min\".format((end-start)/60))" ] }, { "cell_type": "code", "execution_count": 200, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "89/89 [==============================] - 50s 564ms/step - loss: 0.6948 - accuracy: 0.5318 - AUC: 0.5123\n", "Test loss: 0.6948203444480896\n", "Test accuracy: 0.5317766666412354\n", "Test AUC: 0.5122969746589661\n" ] } ], "source": [ "score = model.evaluate((test_seq_data), verbose=1)\n", "print('Test loss:', score[0])\n", "print('Test accuracy:', score[1])\n", "print('Test AUC:', score[2])" ] }, { "cell_type": "code", "execution_count": 201, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "44/44 [==============================] - 25s 569ms/step - loss: 0.6873 - accuracy: 0.3913 - AUC: 0.5152\n", "Test loss: 0.6873216032981873\n", "Test accuracy: 0.39133521914482117\n", "Test AUC: 0.5152168273925781\n" ] } ], "source": [ "score = model.evaluate((infer_seq_data), verbose=1)\n", "print('Test loss:', score[0])\n", "print('Test accuracy:', score[1])\n", "print('Test AUC:', score[2])" ] }, { "cell_type": "code", "execution_count": 202, "metadata": {}, "outputs": [], "source": [ "fine_tuned = pd.DataFrame(fine_tuned_history.history)\n", "df = metrics.append(fine_tuned).reset_index(drop = 1)\n", "df.index +=1" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, axes = plt.subplots(ncols=3, figsize=(16, 4))\n", "#ACCURACY\n", "df1 = (df[['accuracy', 'val_accuracy']]\n", " .rename(columns={'accuracy': 'Training',\n", " 'val_accuracy': 'Validation'}))\n", "df1.plot(ax=axes[0], title='Accuracy', xlim=(1, len(df)))\n", "axes[0].axvline(df.val_accuracy.idxmax(), ls='--', lw=1, c='k')\n", "axes[0].axvline(len(metrics), ls='-', lw=1, c='k')\n", "#AUC\n", "df2 = (df[['AUC', 'val_AUC']]\n", " .rename(columns={'AUC': 'Training',\n", " 'val_AUC': 'Validation'}))\n", "df2.plot(ax=axes[1], title='Area under the ROC Curve', xlim=(1, len(df)))\n", "\n", "axes[1].axvline(df.val_AUC.idxmax(), ls='--', lw=1, c='k')\n", "axes[1].axvline(len(metrics), ls='-', lw=1, c='k')\n", "#LOSS\n", "df2 = (df[['loss', 'val_loss']]\n", " .rename(columns={'loss': 'Training',\n", " 'val_loss': 'Validation'}))\n", "df2.plot(ax=axes[2], title='Loss', xlim=(1, len(df)))\n", "\n", "axes[2].axvline(df.val_loss.idxmin(), ls='--', lw=1, c='k')\n", "axes[2].axvline(len(metrics), ls='-', lw=1, c='k')\n", "for i in [0, 1, 2]:\n", " axes[i].set_xlabel('Epoch')\n", "\n", "sns.despine()\n", "fig.tight_layout()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# for ax in axes:\n", "# ax.axvline(10, ls='--', lw=1, c='k')\n", "# ax.legend(['Training', 'Validation', 'Start Fine Tuning'])\n", "# ax.set_xlabel('Epoch')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Refit on other seq length" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" } }, "nbformat": 4, "nbformat_minor": 4 }