{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# LSTM networks module" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "LSTM networks are complex Deep Learning models that are extremely powerful at detecting patterns from various time series, and especially good at finding relationships between various timeseries. This is perfect in a hydrological modelling context, as these models can learn patterns between meteorological variables (such as precipitation, temperature and wind speed timeseries) and streamflow. Furthermore, when we also add catchment descriptors, LSTM models can learn hydrograph dynamics according to catchment attributes such as the area, slope, land-use and other such descriptors.\n", "\n", "This page demonstrates how to use `xhydro` to perform simple LSTM modelling on local (one) and regional (multiple) catchments. However, LSTM models are infinitely flexible and it would be a monumental task to expose all possible hyperparameters and modelling decisions through an interface. Therefore, this package can be seen as a starting point to develop custom models on custom data. Eventually, users will want to add codes, models and methods to the package. Official models from research projects could be added, and any model used to generate operational datasets could be implemented as well. For the time being, we will use extremely simple models to show how the code works, and we can then let users explore and add functionalities as needed." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-07-03 14:26:44.776201: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", "2024-07-03 14:26:44.801639: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2024-07-03 14:26:45.225103: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" ] } ], "source": [ "import os\n", "\n", "import pooch\n", "import tensorflow.keras.backend as K\n", "import xarray as xr\n", "\n", "from xhydro_lstm.lstm_controller import (\n", " control_local_lstm_training,\n", " control_regional_lstm_training,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example for application on a single catchment\n", "\n", "For this example, we will explore some existing data for a single catchment and see how we can use it for LSTM modelling. We will first get the data from the `xhydro-testdata` test data repository, and we will then process the data while explaining what is going on in the backend. This package is optimized for use with .nc data, and more precisely xarray.Dataset formats." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 1MB\n",
              "Dimensions:        (time: 12419, watershed: 1)\n",
              "Coordinates:\n",
              "  * time           (time) datetime64[ns] 99kB 1979-01-01 ... 2012-12-31\n",
              "  * watershed      (watershed) <U7 28B '01BG005'\n",
              "Data variables: (12/19)\n",
              "    t2m            (time, watershed) float64 99kB ...\n",
              "    tp             (time, watershed) float64 99kB ...\n",
              "    sd             (time, watershed) float64 99kB ...\n",
              "    sp             (time, watershed) float64 99kB ...\n",
              "    d2m            (time, watershed) float64 99kB ...\n",
              "    tcc            (time, watershed) float64 99kB ...\n",
              "    ...             ...\n",
              "    latitude       (watershed) float64 8B ...\n",
              "    forest         (watershed) float64 8B ...\n",
              "    crops          (watershed) float64 8B ...\n",
              "    shrubs         (watershed) float64 8B ...\n",
              "    elevation      (watershed) float64 8B ...\n",
              "    slope          (watershed) float64 8B ...
" ], "text/plain": [ " Size: 1MB\n", "Dimensions: (time: 12419, watershed: 1)\n", "Coordinates:\n", " * time (time) datetime64[ns] 99kB 1979-01-01 ... 2012-12-31\n", " * watershed (watershed) \n", "IMPORTANT
\n", "- This file MUST contain the observed streamflow (Target variable) under the name `qobs`.
\n", "- This file MUST contain the drainage area of the catchment under the name `drainage_area`.\n", "\n", "\n", "\n", "For completeness, let's take a look at the regional (i.e. multi-basin) dataset that we will use later:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 3MB\n",
              "Dimensions:        (time: 12419, watershed: 3)\n",
              "Coordinates:\n",
              "  * time           (time) datetime64[ns] 99kB 1979-01-01 ... 2012-12-31\n",
              "  * watershed      (watershed) <U7 84B '01BG005' '01BG009' '02QC009'\n",
              "Data variables: (12/19)\n",
              "    t2m            (time, watershed) float64 298kB ...\n",
              "    tp             (time, watershed) float64 298kB ...\n",
              "    sd             (time, watershed) float64 298kB ...\n",
              "    sp             (time, watershed) float64 298kB ...\n",
              "    d2m            (time, watershed) float64 298kB ...\n",
              "    tcc            (time, watershed) float64 298kB ...\n",
              "    ...             ...\n",
              "    latitude       (watershed) float64 24B ...\n",
              "    forest         (watershed) float64 24B ...\n",
              "    crops          (watershed) float64 24B ...\n",
              "    shrubs         (watershed) float64 24B ...\n",
              "    elevation      (watershed) float64 24B ...\n",
              "    slope          (watershed) float64 24B ...
" ], "text/plain": [ " Size: 3MB\n", "Dimensions: (time: 12419, watershed: 3)\n", "Coordinates:\n", " * time (time) datetime64[ns] 99kB 1979-01-01 ... 2012-12-31\n", " * watershed (watershed) \n", "Path to the netcdf file containing the required input and target data for the LSTM. The ncfile must contain a dataset named \"qobs\" and \"drainage_area\" for the code to work, as these are required as target and scaling for training, respectively.\n", " \n", "* **dynamic_var_tags** *[list of str]*
\n", "List of dataset variables to use in the LSTM model training. Must be part of the input_data_filename dataset.\n", "\n", "* **qsim_pos** *[list of bool]*
\n", "List of same length as dynamic_var_tags. Should be set to all False EXCEPT where the dynamic_var_tags refer to flow simulations (ex: simulations from a hydrological model). Those should be set to True, because we will scale them according to the catchment size.\n", "\n", "* **batch_size** *[int, Optional]*
\n", "Number of data points to use in training. Datasets are often way too big to train in a single batch on a single GPU or CPU, meaning that the dataset must be divided into smaller batches. This has an impact on the training performance and final model skill, and should be handled accordingly. Defaults to `32` if unspecified by the user.\n", "\n", "* **epochs** *[int, Optional]*
\n", "Number of training evaluations. Larger number of epochs means more model iterations and deeper training. At some point, training will stop due to a stop in validation skill improvement. Defaults to `200` if unspecified by the user.\n", "\n", "* **window_size** *[int, Optional]*
\n", "Number of days of look-back for training and model simulation. LSTM requires a large backwards-looking window to allow the model to learn from long-term weather patterns and history to predict the next day's streamflow. Usually set to 365 days to get one year of previous data. This makes the model heavier and longer to train but can improve results. Defaults to `365` if unspecified by the user.\n", "\n", "* **train_pct** *[int, Optional]*
\n", "Percentage of days from the dataset to use as training. The higher, the better for model training skill, but it is important to keep a decent amount for validation and testing. Defaults to `60` if unspecified by the user.\n", " \n", "* **valid_pct** *[int, Optional]*
\n", "Percentage of days from the dataset to use as validation. The sum of train_pct and valid_pct needs to be less than 100, such that the remainder can be used for testing. A good starting value is 20%. Validation is used as the stopping criteria during training. When validation stops improving, then the model is overfitting and training is stopped. Defaults to `20` if unspecified by the user.\n", "\n", "* **use_cpu** *[bool, Optional]*
\n", "Flag to force the training and simulations to be performed on the CPU rather than on the GPU(s). Must be performed on a CPU that has AVX and AVX2 instruction sets, or tensorflow will fail. CPU training is very slow and should only be used as a last resort (such as for CI testing and debugging). Defaults to `True` if unspecified by the user, to maximize compatibility.\n", " \n", "* **use_parallel** *[bool, Optional]*
\n", "Flag to make use of multiple GPUs to accelerate training further. Models trained on multiple GPUs can have larger batch_size values as different batches can be run on different GPUs in parallel. Speedup is not linear as there is overhead related to the management of datasets, batches, the gradient merging and other steps. Still very useful and should be used when possible. Defaults to `False` if unspecified by the user.\n", "\n", "* **do_train** *[bool, Optional]*
\n", "Indicate that the code should perform the training step. This is not required as a pre-trained model could be used to perform a simulation by passing an existing model in \"name_of_saved_model\". Defaults to `True` if unspecified by the user.\n", "\n", "* **model_structure** *[str, Optional]*
\n", "This is where we can define which LSTM model structure we desire from within the LSTM_static.py function, which contains all the available model structures. Users that want to contribute new model structures should do so here. By default, we use dummy models, but we can recommend some simple starter models such as `[\"simple_local_lstm\", \"simple_regional_lstm\"]` depending on if the model is local or regional.\n", "\n", "* **do_simulation** *[bool, Optional]*
\n", "Indicate that simulations should be performed to obtain simulated streamflow and KGE metrics on the watersheds of interest, using the \"name_of_saved_model\" pre-trained model. If set to True and 'do_train' is True, then the new trained model will be used instead. Defaults to `True` if unspecified by the user.\n", "\n", "* **training_func** *[str, Optional]*
\n", "Name of the objective function used for training. For a regional model, it is highly recommended to use the scaled nse_loss variable that uses the standard deviation of streamflow as inputs. For a local model, the `kge` function is preferred. Defaults to `kge` if unspecified by the user. Can be one of `[\"kge\", \"nse_scaled\"]`.\n", "\n", "* **filename_base** *[str]*
\n", "Name of the trained model that will be trained if it does not already exist. Do not add the \".h5\" extension, it will be added automatically. Defaults to `LSTM_results` if unspecified by the user.\n", "\n", "* **simulation_phases** *[list of str]*
\n", "List of periods to generate the simulations. Can contain `['train','valid','test','full']`, corresponding to the training, validation, testing and complete periods, respectively.\n", "\n", "* **name_of_saved_model** *[str]*
\n", "Path to the model that has been pre-trained if required for simulations.\n", "\n", "Returns\n", "-------\n", "* **kge_results** *[array-like]*
\n", "Kling-Gupta Efficiency metric values for each of the watersheds in the input_data_filename ncfile after running in simulation mode (thus after training). Contains n_watersheds items, each containing 4 values representing the KGE values in training, validation, testing and full period, respectively. If one or more simulation phases are not requested, the items will be set to None.\n", "\n", "* **flow_results** *[array-like]*
\n", "Streamflow simulation values for each of the watersheds in the input_data_filename ncfile after running in simulation mode (thus after training). Contains n_watersheds items, each containing 4x 2D-arrays representing the observed and simulation series in training, validation, testing and full period, respectively. If one or more simulation phases are not requested, the items will be set to None.\n", "\n", "* **name_of_saved_model** *[str]*
\n", "Path to the model that has been trained, or to the pre-trained model if it already existed.\n", "\n", "
\n", "We will now define these inputs for use in the local LSTM model." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Define some of the basic required inputs.\n", "\n", "# The path to the input data file containing flow, weather data (dynamic features) and catchment descriptors\n", "# (static features).\n", "input_data_filename = input_data_filename_local\n", "\n", "# Names of the dynamic features to be used to train the data, from the input_data_filename dataset. We are only\n", "# using two here to keep things simple and to match the LSTM model complexity.\n", "dynamic_var_tags = [\"t2m\", \"tp\"]\n", "\n", "# Scale variable according to area. Used for simulated flow inputs. Ensure the same order as dynamic_var_tags.\n", "qsim_pos = [False, False]\n", "\n", "# Perform the training of the model. We will select \"True\" for this first step.\n", "do_train = True\n", "\n", "# Define the model structure. We do not want to use the dummy model setup here, so let's use a more appropriate model\n", "model_structure = \"simple_local_lstm\"\n", "\n", "# Also perform a simulation after the model is trained. We will select \"True\" to simulate flows on the periods\n", "# identified in the \"simulation_phases\" parameter.\n", "do_simulation = True\n", "\n", "# The phases on which we want to perform the simulations. Can be a list containing any combination of\n", "# ['train','valid','test','full'].\n", "simulation_phases = [\"test\"]\n", "\n", "# Optional, and only needed if we intend to do a simulation using a pre-trained LSTM model. We can then provide\n", "# the name of the trained model to use for simulation.\n", "name_of_saved_model = None\n", "\n", "# batch size used in the training - multiple of 32 is ideal for efficiency but not critical.\n", "batch_size = 256\n", "\n", "# Number of epoch to train the LSTM model. Keeping this very short for this example and the model will not converge.\n", "epochs = 5\n", "\n", "# Number of time steps (days) to use in the LSTM model\n", "window_size = 180\n", "\n", "# Percentage of days to use from the training data for training the LSTM model.\n", "train_pct = 70\n", "\n", "# Percentage of days to use from the training data for training the LSTM model.\n", "valid_pct = 15\n", "\n", "# Use CPU as GPU is not guaranteed to be installed with CUDA/CuDNN etc. Much slower, but more compatible.\n", "use_cpu = True\n", "\n", "# Use multiple GPUs in parallel if available. Highly suggested for larger models, but useless if using CPU.\n", "use_parallel = False\n", "\n", "# The objective function to use for training. Since we are using a local model, use the KGE directly\n", "training_func = \"kge\"\n", "\n", "# The name of the output files to be generated after simulations\n", "filename_base = \"./LSTM_results_local_\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " We can now run the model and get results:\n", "\n", "
\n", " WARNING:
\n", "There is a high likelihood that the model will not converge and return nan values for the loss and val_loss metrics. If this happens, the model with restart, and do so until the model converges (or run indefinitely). When it does converge, there is a good chance that the results will be poor. This is because this test example has too few data to properly converge, to keep these examples small and easy to run.\n", "
\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "USING CPU\n", "Epoch 1/5\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2024-07-03 14:26:46.560537: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:282] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n", "/home/richard/miniconda3/envs/xhydrolstm/lib/python3.11/site-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.\n", " warnings.warn(\n", "/home/richard/miniconda3/envs/xhydrolstm/lib/python3.11/site-packages/keras/src/trainers/data_adapters/py_dataset_adapter.py:121: UserWarning: Your `PyDataset` class should call `super().__init__(**kwargs)` in its constructor. `**kwargs` can include `workers`, `use_multiprocessing`, `max_queue_size`. Do not pass these arguments to `fit()`, as they will be ignored.\n", " self._warn_if_super_not_called()\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 111ms/step - loss: nan\n", "Epoch 1: val_loss did not improve from inf\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m6s\u001b[0m 125ms/step - loss: nan - val_loss: nan - learning_rate: 5.0000e-04\n", "Epoch 2/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 108ms/step - loss: nan\n", "Epoch 2: val_loss did not improve from inf\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 116ms/step - loss: nan - val_loss: nan - learning_rate: 5.0000e-04\n", "Epoch 3/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 105ms/step - loss: nan\n", "Epoch 3: val_loss did not improve from inf\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 114ms/step - loss: nan - val_loss: nan - learning_rate: 5.0000e-04\n", "Epoch 4/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 106ms/step - loss: nan\n", "Epoch 4: val_loss did not improve from inf\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 115ms/step - loss: nan - val_loss: nan - learning_rate: 5.0000e-04\n", "Epoch 5/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 107ms/step - loss: nan\n", "Epoch 5: val_loss did not improve from inf\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 116ms/step - loss: nan - val_loss: nan - learning_rate: 5.0000e-04\n", "USING CPU\n", "Epoch 1/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 112ms/step - loss: 1.3023\n", "Epoch 1: val_loss improved from inf to 1.10500, saving model to /tmp/tmpi9opkmt_/LSTM_results_local_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m6s\u001b[0m 128ms/step - loss: 1.3011 - val_loss: 1.1050 - learning_rate: 5.0000e-04\n", "Epoch 2/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 111ms/step - loss: 1.0814\n", "Epoch 2: val_loss improved from 1.10500 to 1.03765, saving model to /tmp/tmpi9opkmt_/LSTM_results_local_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 120ms/step - loss: 1.0807 - val_loss: 1.0376 - learning_rate: 5.0000e-04\n", "Epoch 3/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 108ms/step - loss: 1.0267\n", "Epoch 3: val_loss improved from 1.03765 to 1.02908, saving model to /tmp/tmpi9opkmt_/LSTM_results_local_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 117ms/step - loss: 1.0266 - val_loss: 1.0291 - learning_rate: 5.0000e-04\n", "Epoch 4/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 112ms/step - loss: 1.0166\n", "Epoch 4: val_loss improved from 1.02908 to 1.02367, saving model to /tmp/tmpi9opkmt_/LSTM_results_local_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 121ms/step - loss: 1.0166 - val_loss: 1.0237 - learning_rate: 5.0000e-04\n", "Epoch 5/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 105ms/step - loss: 1.0150\n", "Epoch 5: val_loss improved from 1.02367 to 1.02104, saving model to /tmp/tmpi9opkmt_/LSTM_results_local_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 114ms/step - loss: 1.0150 - val_loss: 1.0210 - learning_rate: 5.0000e-04\n", "\u001b[1m7/7\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 55ms/step\n" ] } ], "source": [ "K.clear_session()\n", "\n", "KGE_results, flow_results, name_of_saved_model = control_local_lstm_training(\n", " input_data_filename=input_data_filename,\n", " dynamic_var_tags=dynamic_var_tags,\n", " qsim_pos=qsim_pos,\n", " batch_size=batch_size,\n", " epochs=epochs,\n", " window_size=window_size,\n", " train_pct=train_pct,\n", " valid_pct=valid_pct,\n", " use_cpu=use_cpu,\n", " use_parallel=use_parallel,\n", " do_train=do_train,\n", " model_structure=model_structure,\n", " do_simulation=do_simulation,\n", " training_func=training_func,\n", " filename_base=filename_base,\n", " simulation_phases=simulation_phases,\n", " name_of_saved_model=name_of_saved_model,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now explore the results. A quick reminder: The KGE_results and flow_results are lists of size 4, reflecting results for the training, validation, testing and full periods, respectively. Since we only asked for the \"test\" simulation to be performed, we will only receive results in the 3rd element in the list:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The KGE results are:\n" ] }, { "data": { "text/plain": [ "[None, None, -0.0144405131655152, None]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "The simulated flows are:\n" ] }, { "data": { "text/plain": [ "[None,\n", " None,\n", " array([[218. , 191. , 180. , ..., 12.1 ,\n", " 12.6 , 12.7 ],\n", " [ 0.34784153, 0.34113994, 0.3369055 , ..., 0. ,\n", " 0. , 0. ]], dtype=float32),\n", " None]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "The trained model has been saved at the following location:\n" ] }, { "data": { "text/plain": [ "'/tmp/tmpi9opkmt_/LSTM_results_local_.keras'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Show the KGE results\n", "print(\"The KGE results are:\")\n", "display(KGE_results)\n", "\n", "print(\"\\nThe simulated flows are:\")\n", "display(flow_results)\n", "\n", "print(\"\\nThe trained model has been saved at the following location:\")\n", "display(name_of_saved_model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Now that the model has been trained, we can use the controller to perform a simulation by changing just a few elements to the inputs:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m 1/33\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m6s\u001b[0m 205ms/step" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/richard/miniconda3/envs/xhydrolstm/lib/python3.11/site-packages/keras/src/trainers/data_adapters/py_dataset_adapter.py:121: UserWarning: Your `PyDataset` class should call `super().__init__(**kwargs)` in its constructor. `**kwargs` can include `workers`, `use_multiprocessing`, `max_queue_size`. Do not pass these arguments to `fit()`, as they will be ignored.\n", " self._warn_if_super_not_called()\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 44ms/step\n", "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 53ms/step\n", "WARNING:tensorflow:5 out of the last 42 calls to .one_step_on_data_distributed at 0x7fde48c5fce0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n", "\u001b[1m7/7\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 57ms/step\n", "WARNING:tensorflow:5 out of the last 16 calls to .one_step_on_data_distributed at 0x7fde48c5d8a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n", "\u001b[1m48/48\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 44ms/step\n", "The KGE results are:\n" ] }, { "data": { "text/plain": [ "[-0.009339690893735542,\n", " -0.017701832438825615,\n", " -0.0144405131655152,\n", " -0.011388125559591344]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "The simulated flows are:\n" ] }, { "data": { "text/plain": [ "[array([[2.7400000e+01, 2.6600000e+01, 2.5500000e+01, ..., 1.0600000e+02,\n", " 1.2000000e+02, 1.0600000e+02],\n", " [5.3376455e-02, 5.4626308e-02, 5.4637779e-02, ..., 3.1778562e-01,\n", " 2.7539933e-01, 2.3393394e-01]], dtype=float32),\n", " array([[12.6 , 12.3 , 12. , ..., 57.3 ,\n", " 51.7 , 47.3 ],\n", " [ 0. , 0. , 0. , ..., 0.15638022,\n", " 0.14065447, 0.11017577]], dtype=float32),\n", " array([[218. , 191. , 180. , ..., 12.1 ,\n", " 12.6 , 12.7 ],\n", " [ 0.34784153, 0.34113994, 0.3369055 , ..., 0. ,\n", " 0. , 0. ]], dtype=float32),\n", " array([[27.4 , 26.6 , 25.5 , ..., 12.6 ,\n", " 12.7 , 12.3 ],\n", " [ 0.05337645, 0.05462631, 0.05463778, ..., 0. ,\n", " 0. , 0. ]], dtype=float32)]" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Define the variables that need to be changed:\n", "\n", "# Perform the training of the model. We will select \"False\" now that we have a saved trained model\n", "do_train = False\n", "\n", "# The phases on which we want to perform the simulations. Can be a list containing any combination of\n", "# ['train','valid','test','full'].\n", "simulation_phases = [\"train\", \"valid\", \"test\", \"full\"]\n", "\n", "# The name of the saved model from the previous training.\n", "name_of_saved_model = name_of_saved_model\n", "\n", "# And run the model\n", "KGE_results_sim, flow_results_sim, _ = control_local_lstm_training(\n", " input_data_filename=input_data_filename,\n", " dynamic_var_tags=dynamic_var_tags,\n", " qsim_pos=qsim_pos,\n", " batch_size=batch_size,\n", " epochs=epochs,\n", " window_size=window_size,\n", " train_pct=train_pct,\n", " valid_pct=valid_pct,\n", " use_cpu=use_cpu,\n", " use_parallel=use_parallel,\n", " do_train=do_train,\n", " model_structure=model_structure,\n", " do_simulation=do_simulation,\n", " training_func=training_func,\n", " filename_base=filename_base,\n", " simulation_phases=simulation_phases,\n", " name_of_saved_model=name_of_saved_model,\n", ")\n", "\n", "print(\"The KGE results are:\")\n", "display(KGE_results_sim)\n", "\n", "print(\"\\nThe simulated flows are:\")\n", "display(flow_results_sim)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Regional model\n", "\n", "We can now apply the exact same concept to the regional model dataset that contained multiple watersheds. We will first define the input parameters just like for the local model:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# Define some of the basic required inputs.\n", "\n", "# The path to the input data file containing flow, weather data (dynamic features) and catchment descriptors\n", "# (static features).\n", "input_data_filename = input_data_filename_regional\n", "\n", "# Names of the dynamic features to be used to train the data, from the input_data_filename dataset. We are only\n", "# using two here to keep things simple and to match the LSTM model complexity.\n", "dynamic_var_tags = [\"t2m\", \"tp\"]\n", "\n", "# Scale variable according to area. Used for simulated flow inputs. Ensure the same order as dynamic_var_tags.\n", "qsim_pos = [False, False]\n", "\n", "# Names of the static features to be used from the input dataset. These are used to regionalize flows between catchments\n", "# based on their attributes. Here we only use 3 to keep things simple. They will not be useful as our dataset only\n", "# contains 3 watersheds as well, so there is not enough data to learn the relationships correctly.\n", "static_var_tags = [\"drainage_area\", \"forest\", \"slope\"]\n", "\n", "# Perform the training of the model. We will select \"True\" for this first step.\n", "do_train = True\n", "\n", "# Define the model structure. We do not want to use the dummy model setup here, so let's use a more appropriate model\n", "model_structure = \"simple_regional_lstm\"\n", "\n", "# Also perform a simulation after the model is trained. We will select \"True\" to simulate flows on the periods\n", "# identified in the \"simulation_phases\" parameter.\n", "do_simulation = True\n", "\n", "# The phases on which we want to perform the simulations. Can be a list containing any combination of\n", "# ['train','valid','test','full'].\n", "simulation_phases = [\"test\"]\n", "\n", "# Optional, and only needed if we intend to do a simulation using a pre-trained LSTM model. We can then provide\n", "# the name of the trained model to use for simulation.\n", "name_of_saved_model = None\n", "\n", "# batch size used in the training - multiple of 32 is ideal for efficiency but not critical.\n", "batch_size = 256\n", "\n", "# Number of epoch to train the LSTM model. Keeping this very short for this example and the model will not converge.\n", "epochs = 5\n", "\n", "# Number of time steps (days) to use in the LSTM model\n", "window_size = 180\n", "\n", "# Percentage of days to use from the training data for training the LSTM model.\n", "train_pct = 70\n", "\n", "# Percentage of days to use from the training data for training the LSTM model.\n", "valid_pct = 15\n", "\n", "# Use CPU as GPU is not guaranteed to be installed with CUDA/CuDNN etc. Much slower, but more compatible.\n", "use_cpu = True\n", "\n", "# Use multiple GPUs in parallel if available. Highly suggested for larger models, but useless if using CPU.\n", "use_parallel = False\n", "\n", "# The objective function to use for training. Since we are using a local model, use the KGE directly\n", "training_func = \"nse_scaled\"\n", "\n", "# The name of the output files to be generated after simulations\n", "filename_base = \"./LSTM_results_regional_\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now we run the model for training:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Currently working on watershed no: 0\n", "Currently working on watershed no: 0\n", "USING CPU\n", "Epoch 1/5\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/richard/miniconda3/envs/xhydrolstm/lib/python3.11/site-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.\n", " warnings.warn(\n", "/home/richard/miniconda3/envs/xhydrolstm/lib/python3.11/site-packages/keras/src/trainers/data_adapters/py_dataset_adapter.py:121: UserWarning: Your `PyDataset` class should call `super().__init__(**kwargs)` in its constructor. `**kwargs` can include `workers`, `use_multiprocessing`, `max_queue_size`. Do not pass these arguments to `fit()`, as they will be ignored.\n", " self._warn_if_super_not_called()\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 407ms/step - loss: 1.6288\n", "Epoch 1: val_loss improved from inf to 0.77263, saving model to /tmp/tmp855g032j/LSTM_results_regional_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m16s\u001b[0m 447ms/step - loss: 1.6132 - val_loss: 0.7726 - learning_rate: 5.0000e-04\n", "Epoch 2/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 400ms/step - loss: 0.7536\n", "Epoch 2: val_loss improved from 0.77263 to 0.49094, saving model to /tmp/tmp855g032j/LSTM_results_regional_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 435ms/step - loss: 0.7507 - val_loss: 0.4909 - learning_rate: 5.0000e-04\n", "Epoch 3/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 399ms/step - loss: 0.3557\n", "Epoch 3: val_loss improved from 0.49094 to 0.41243, saving model to /tmp/tmp855g032j/LSTM_results_regional_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 434ms/step - loss: 0.3551 - val_loss: 0.4124 - learning_rate: 5.0000e-04\n", "Epoch 4/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 397ms/step - loss: 0.2678\n", "Epoch 4: val_loss improved from 0.41243 to 0.36246, saving model to /tmp/tmp855g032j/LSTM_results_regional_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 432ms/step - loss: 0.2681 - val_loss: 0.3625 - learning_rate: 5.0000e-04\n", "Epoch 5/5\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 397ms/step - loss: 0.2566\n", "Epoch 5: val_loss improved from 0.36246 to 0.34656, saving model to /tmp/tmp855g032j/LSTM_results_regional_.keras\n", "\u001b[1m33/33\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 432ms/step - loss: 0.2564 - val_loss: 0.3466 - learning_rate: 5.0000e-04\n", "Currently working on watershed no: 0\n", "\u001b[1m7/7\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 176ms/step\n" ] } ], "source": [ "K.clear_session()\n", "\n", "KGE_results, flow_results, name_of_saved_model = control_regional_lstm_training(\n", " input_data_filename=input_data_filename,\n", " dynamic_var_tags=dynamic_var_tags,\n", " qsim_pos=qsim_pos,\n", " static_var_tags=static_var_tags,\n", " batch_size=batch_size,\n", " epochs=epochs,\n", " window_size=window_size,\n", " train_pct=train_pct,\n", " valid_pct=valid_pct,\n", " use_cpu=use_cpu,\n", " use_parallel=use_parallel,\n", " do_train=do_train,\n", " model_structure=model_structure,\n", " do_simulation=do_simulation,\n", " training_func=training_func,\n", " filename_base=filename_base,\n", " simulation_phases=simulation_phases,\n", " name_of_saved_model=name_of_saved_model,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similarly to the previous model, we can now get the overall results" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The KGE results are:\n" ] }, { "data": { "text/plain": [ "[[None, None, 0.7028731164011819, None]]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "The simulated flows are:\n" ] }, { "data": { "text/plain": [ "[[None,\n", " None,\n", " array([[217.99998446, 190.99999739, 179.9999865 , ..., 12.09999971,\n", " 12.6000001 , 12.6999995 ],\n", " [168.16178894, 163.36073303, 157.63221741, ..., 15.97389889,\n", " 18.29801369, 18.19104004]]),\n", " None]]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "The trained model has been saved at the following location:\n" ] }, { "data": { "text/plain": [ "'/tmp/tmp855g032j/LSTM_results_regional_.keras'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Show the KGE results\n", "print(\"The KGE results are:\")\n", "display(KGE_results)\n", "\n", "print(\"\\nThe simulated flows are:\")\n", "display(flow_results)\n", "\n", "print(\"\\nThe trained model has been saved at the following location:\")\n", "display(name_of_saved_model)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" }, "vscode": { "interpreter": { "hash": "e28391989cdb8b31df72dd917935faad186df3329a743c813090fc6af977a1ca" } } }, "nbformat": 4, "nbformat_minor": 4 }