diff --git a/open-vaccine/.gitkeep b/open-vaccine/.gitkeep
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/open-vaccine/README.md b/open-vaccine/README.md
deleted file mode 100644
index 4743e0383650b12c63121f10bce789cad0736bf4..0000000000000000000000000000000000000000
--- a/open-vaccine/README.md
+++ /dev/null
@@ -1,18 +0,0 @@
-## Example - open-vaccine
-
-### What is it about?
-
-Create and run a pipeline by annotating cells and using KALE Jupyter Lab extension.
-Imported and adjusted from here: https://github.com/kubeflow-kale/kale/tree/master/examples/openvaccine-kaggle-competition
-
-### How to run?
-
-- Open **open-vaccine/open-vaccine.ipynb** in your Notebook server
-- On the left side, select Kubeflow Pipelines Deployment Panel
-- Toggle Enable
-- Select Experiment (existing or new)
-- Write Pipeline name and Pipeline description
-- Untoggle *HP Tuning with Katib*
-- Click Compile and Run at the bottom of the page
-- After successfull compilation, click View
-- Inspect and debug your pipeline via Pipeline log
diff --git a/open-vaccine/open-vaccine.ipynb b/open-vaccine/open-vaccine.ipynb
deleted file mode 100644
index 8338b107bde9c14c96f56f05e9767e51d315eec0..0000000000000000000000000000000000000000
--- a/open-vaccine/open-vaccine.ipynb
+++ /dev/null
@@ -1,807 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "# OpenVaccine: COVID-19 mRNA Vaccine Degradation Prediction\n",
- "\n",
- "In this [Kaggle competition](https://www.kaggle.com/c/stanford-covid-vaccine/overview) we try to develop models and design rules for RNA degradation. As the overview of the competition states:\n",
- "\n",
- ">mRNA vaccines have taken the lead as the fastest vaccine candidates for COVID-19, but currently, they face key potential limitations. One of the biggest challenges right now is how to design super stable messenger RNA molecules (mRNA). Conventional vaccines (like your seasonal flu shots) are packaged in disposable syringes and shipped under refrigeration around the world, but that is not currently possible for mRNA vaccines.\n",
- ">\n",
- ">Researchers have observed that RNA molecules have the tendency to spontaneously degrade. This is a serious limitation--a single cut can render the mRNA vaccine useless. Currently, little is known on the details of where in the backbone of a given RNA is most prone to being affected. Without this knowledge, current mRNA vaccines against COVID-19 must be prepared and shipped under intense refrigeration, and are unlikely to reach more than a tiny fraction of human beings on the planet unless they can be stabilized.\n",
- "\n",
- "<img src=\"images/banner.png\" width=\"1000\" style=\"margin-left: auto; margin-right: auto;\"> \n",
- "\n",
- "The model should predict likely degradation rates at each base of an RNA molecule. The training data set is comprised of over 3000 RNA molecules and their degradation rates at each position."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "# Install necessary packages\n",
- "\n",
- "We can install the necessary package by either running `pip install --user <package_name>` or include everything in a `requirements.txt` file and run `pip install --user -r requirements.txt`. We have put the dependencies in a `requirements.txt` file so we will use the former method.\n",
- "\n",
- "> NOTE: Do not forget to use the `--user` argument. It is necessary if you want to use Kale to transform this notebook into a Kubeflow pipeline"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "tags": [
- "skip"
- ]
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 1)) (2.22.0)\n",
- "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements.txt (line 1)) (2019.11.28)\n",
- "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements.txt (line 1)) (3.0.4)\n",
- "Requirement already satisfied: idna<2.9,>=2.5 in /usr/lib/python3/dist-packages (from requests->-r requirements.txt (line 1)) (2.6)\n",
- "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/jovyan/.local/lib/python3.6/site-packages (from requests->-r requirements.txt (line 1)) (1.24.2)\n"
- ]
- }
- ],
- "source": [
- "!pip install --user -r requirements.txt"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "# Imports\n",
- "\n",
- "In this section we import the packages we need for this example. Make it a habit to gather your imports in a single place. It will make your life easier if you are going to transform this notebook into a Kubeflow pipeline using Kale."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "tags": [
- "imports"
- ]
- },
- "outputs": [],
- "source": [
- "import json\n",
- "import numpy as np\n",
- "import pandas as pd\n",
- "import tensorflow as tf\n",
- "import os\n",
- "\n",
- "from tensorflow.keras.preprocessing.text import Tokenizer"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "# Project hyper-parameters\n",
- "\n",
- "In this cell, we define the different hyper-parameters. Defining them in one place makes it easier to experiment with their values and also facilitates the execution of HP Tuning experiments using Kale and Katib."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "tags": [
- "pipeline-parameters"
- ]
- },
- "outputs": [],
- "source": [
- "# Hyper-parameters\n",
- "LR = 1e-3\n",
- "EPOCHS = 1\n",
- "BATCH_SIZE = 64\n",
- "EMBED_DIM = 100\n",
- "HIDDEN_DIM = 128\n",
- "DROPOUT = .5\n",
- "SP_DROPOUT = .3\n",
- "TRAIN_SEQUENCE_LENGTH = 107"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "Set random seed for reproducibility and ignore warning messages."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "tags": [
- "skip"
- ]
- },
- "outputs": [],
- "source": [
- "tf.random.set_seed(42)\n",
- "np.random.seed(42)\n",
- "\n",
- "tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "# Load and preprocess data\n",
- "\n",
- "In this section, we load and process the dataset to get it in a ready-to-use form by the model. First, let us load and analyze the data."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Load data"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "The data are in `json` format, thus, we use the handy `read_json` pandas method. There is one train data set and two test sets (one public and one private)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "tags": [
- "block:load_data"
- ]
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Cloning into 'kale'...\n",
- "remote: Enumerating objects: 6904, done.\u001b[K\n",
- "remote: Counting objects: 100% (535/535), done.\u001b[K\n",
- "remote: Compressing objects: 100% (272/272), done.\u001b[K\n",
- "remote: Total 6904 (delta 381), reused 356 (delta 260), pack-reused 6369\u001b[K\n",
- "Receiving objects: 100% (6904/6904), 17.28 MiB | 11.12 MiB/s, done.\n",
- "Resolving deltas: 100% (3637/3637), done.\n",
- "Checking out files: 100% (619/619), done.\n"
- ]
- }
- ],
- "source": [
- "os.system(\"git clone https://github.com/kubeflow-kale/kale.git\")\n",
- "os.system(\"cp -r kale/examples/openvaccine-kaggle-competition/. .\")\n",
- "\n",
- "train_df = pd.read_json(\"data/train.json\", lines=True)\n",
- "test_df = pd.read_json(\"data/test.json\", lines=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "We also load the `sample_submission.csv` file, which will prove handy when we will be creating our submission to the competition."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "sample_submission_df = pd.read_csv(\"data/sample_submission.csv\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "Let us now explore the data, their dimensions and what each column mean. To this end, we use the pandas `head` method to visualize a small sample (five rows by default) of our data set."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "train_df.head()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "We see a lot of strange entries, so, let us try to see what they are:\n",
- "\n",
- "* `sequence`: An 107 characters long string in Train and Public Test (130 in Private Test), which describes the RNA sequence, a combination of A, G, U, and C for each sample.\n",
- "* `structure`: An 107 characters long string in Train and Public Test (130 in Private Test), which is a combination of `(`, `)`, and `.` characters that describe whether a base is estimated to be paired or unpaired. Paired bases are denoted by opening and closing parentheses (e.g. (....) means that base 0 is paired to base 5, and bases 1-4 are unpaired).\n",
- "* `predicted_loop_type`: An 107 characters long string, which describes the structural context (also referred to as 'loop type') of each character in sequence. Loop types assigned by bpRNA from Vienna RNAfold 2 structure. From the bpRNA_documentation: `S`: paired \"Stem\" `M`: Multiloop `I`: Internal loop `B`: Bulge `H`: Hairpin loop `E`: dangling End `X`: eXternal loop.\n",
- "\n",
- "Then, we have `signal_to_noise`, which is quality control feature. It records the measurements relative to their errors; the higher value the more confident measurements are.\n",
- "\n",
- "The `*_error_*` columns calculate the errors in experimental values obtained in corresponding `reactivity` and `deg_*` columns.\n",
- "\n",
- "The last five columns (i.e., `recreativity` and `deg_*`) are out depended variables, our targets. Thus, for every base in the molecule we should predict five different values.\n",
- "\n",
- "These are the main columns we care about. For more details, visit the competition [info](https://www.kaggle.com/c/stanford-covid-vaccine/data)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Preprocess data"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "We are now ready to preprocess the data set. First, we define the symbols that encode certain features (e.g. the base symbol or the structure), the features and the target variables."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": [
- "block:preprocess_data",
- "prev:load_data"
- ]
- },
- "outputs": [],
- "source": [
- "symbols = \"().ACGUBEHIMSX\"\n",
- "feat_cols = [\"sequence\", \"structure\", \"predicted_loop_type\"]\n",
- "target_cols = [\"reactivity\", \"deg_Mg_pH10\", \"deg_Mg_50C\", \"deg_pH10\", \"deg_50C\"]\n",
- "error_cols = [\"reactivity_error\", \"deg_error_Mg_pH10\", \"deg_error_Mg_50C\", \"deg_error_pH10\", \"deg_error_50C\"]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "In order to encode values like strings or characters and feed them to the neural network, we need to tokenize them. The `Tokenizer` class will assign a number to each character."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "tokenizer = Tokenizer(char_level=True, filters=\"\")\n",
- "tokenizer.fit_on_texts(symbols)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "Moreover, the tokenizer keeps a dictionary, `word_index`, from which we can get the number of elements in our vocabulary. In this case, we only have a few elements, but if our dataset was a whole book, that function would be handy.\n",
- "\n",
- "> NOTE: We should add `1` to the length of the `word_index` dictionary to get the correct number of elements."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# get the number of elements in the vocabulary\n",
- "vocab_size = len(tokenizer.word_index) + 1"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "We are now ready to process our features. First, we transform each character sequence (i.e., `sequence`, `structure`, `predicted_loop_type`) into number sequences and concatenate them together. The resulting shape should be `(num_examples, 107, 3)`.\n",
- "\n",
- "> Now, we should do this in a way that would permit us to use this processing function with KFServing. Thus, since Numpy arrays are not JSON serializable, this function should accept and return pure Python lists."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def process_features(example):\n",
- " sequence_sentences = example[0]\n",
- " structure_sentences = example[1]\n",
- " loop_sentences = example[2]\n",
- " \n",
- " # transform character sequences into number sequences\n",
- " sequence_tokens = np.array(\n",
- " tokenizer.texts_to_sequences(sequence_sentences)\n",
- " )\n",
- " structure_tokens = np.array(\n",
- " tokenizer.texts_to_sequences(structure_sentences)\n",
- " )\n",
- " loop_tokens = np.array(\n",
- " tokenizer.texts_to_sequences(loop_sentences)\n",
- " )\n",
- " \n",
- " # concatenate the tokenized sequences\n",
- " sequences = np.stack(\n",
- " (sequence_tokens, structure_tokens, loop_tokens),\n",
- " axis=1\n",
- " )\n",
- " sequences = np.transpose(sequences, (2, 0, 1))\n",
- " \n",
- " prepared = sequences.tolist()\n",
- " \n",
- " return prepared[0]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "In the same way we process the labels. We should just extract them and transform them into the correct shape. The resulting shape should be `(num_examples, 68, 5)`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def process_labels(df):\n",
- " df = df.copy()\n",
- " \n",
- " labels = np.array(df[target_cols].values.tolist())\n",
- " labels = np.transpose(labels, (0, 2, 1))\n",
- " \n",
- " return labels"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "public_test_df = test_df.query(\"seq_length == 107\")\n",
- "private_test_df = test_df.query(\"seq_length == 130\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "We are now ready to process the data set and make the features ready to be consumed by the model."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "x_train = [process_features(row.tolist()) for _, row in train_df[feat_cols].iterrows()]\n",
- "y_train = process_labels(train_df)\n",
- "\n",
- "unprocessed_x_public_test = [row.tolist() for _, row in public_test_df[feat_cols].iterrows()]\n",
- "unprocessed_x_private_test = [row.tolist() for _, row in private_test_df[feat_cols].iterrows()]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "# Define and train the model\n",
- "\n",
- "We are now ready to define our model. We have to do with sequences, thus, it makes sense to use RNNs. More specifically, we will use bidirectional Gated Recurrent Units (GRUs) and Long Short Term Memory cells (LSTM). The output layer shoud produce 5 numbers, so we can see this as a regression problem.\n",
- "\n",
- "First let us define two helper functions for GRUs and LSTMs and then, define the body of the full model."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": [
- "block:model_training",
- "prev:preprocess_data"
- ]
- },
- "outputs": [],
- "source": [
- "def gru_layer(hidden_dim, dropout):\n",
- " return tf.keras.layers.Bidirectional(\n",
- " tf.keras.layers.GRU(hidden_dim, dropout=dropout, return_sequences=True, kernel_initializer = 'orthogonal')\n",
- " )"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def lstm_layer(hidden_dim, dropout):\n",
- " return tf.keras.layers.Bidirectional(\n",
- " tf.keras.layers.LSTM(hidden_dim, dropout=dropout, return_sequences=True, kernel_initializer = 'orthogonal')\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "The model has an embedding layer. The embedding layer projects the tokenized categorical input into a high-dimensional latent space. For this example we treat the dimensionality of the embedding space as a hyper-parameter that we can use to fine-tune the model."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def build_model(vocab_size, seq_length=TRAIN_SEQUENCE_LENGTH, pred_len=68,\n",
- " embed_dim=EMBED_DIM,\n",
- " hidden_dim=HIDDEN_DIM, dropout=DROPOUT, sp_dropout=SP_DROPOUT):\n",
- " inputs = tf.keras.layers.Input(shape=(seq_length, 3))\n",
- "\n",
- " embed = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)(inputs)\n",
- " \n",
- " reshaped = tf.reshape(\n",
- " embed, shape=(-1, embed.shape[1], embed.shape[2] * embed.shape[3])\n",
- " )\n",
- " \n",
- " hidden = tf.keras.layers.SpatialDropout1D(sp_dropout)(reshaped)\n",
- " \n",
- " hidden = gru_layer(hidden_dim, dropout)(hidden)\n",
- " hidden = lstm_layer(hidden_dim, dropout)(hidden)\n",
- " \n",
- " truncated = hidden[:, :pred_len]\n",
- " \n",
- " out = tf.keras.layers.Dense(5, activation=\"linear\")(truncated)\n",
- " \n",
- " model = tf.keras.Model(inputs=inputs, outputs=out)\n",
- " \n",
- " return model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "model = build_model(vocab_size)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "model.summary()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "Submissions are scored using MCRMSE (mean columnwise root mean squared error):\n",
- "\n",
- "<img src=\"images/mcrmse.png\" width=\"250\" style=\"margin-left: auto; margin-right: auto;\">\n",
- "\n",
- "Thus, we should code this metric and use it as our objective (loss) function."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "class MeanColumnwiseRMSE(tf.keras.losses.Loss):\n",
- " def __init__(self, name='MeanColumnwiseRMSE'):\n",
- " super().__init__(name=name)\n",
- "\n",
- " def call(self, y_true, y_pred):\n",
- " colwise_mse = tf.reduce_mean(tf.square(y_true - y_pred), axis=1)\n",
- " return tf.reduce_mean(tf.sqrt(colwise_mse), axis=1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "We are now ready to compile and fit the model."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "model.compile(tf.optimizers.Adam(learning_rate=LR), loss=MeanColumnwiseRMSE())"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "history = model.fit(np.array(x_train), np.array(y_train), \n",
- " validation_split=.1, batch_size=BATCH_SIZE, epochs=EPOCHS)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "validation_loss = history.history.get(\"val_loss\")[0]\n",
- "model_weights = model.get_weights()\n",
- "print(model_weights)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Evaluate the model\n",
- "\n",
- "Finally, we are ready to evaluate the model using the two test sets."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": [
- "block:model_evaluation",
- "prev:model_training"
- ]
- },
- "outputs": [],
- "source": [
- "print(model_weights)\n",
- "model_public = build_model(vocab_size, seq_length=107, pred_len=107)\n",
- "model_private = build_model(vocab_size, seq_length=130, pred_len=130)\n",
- "\n",
- "model_public.set_weights(model_weights)\n",
- "model_private.set_weights(model_weights)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": [
- "block:"
- ]
- },
- "outputs": [],
- "source": [
- "public_preds = model_public.predict(np.array([process_features(x) for x in unprocessed_x_public_test]))\n",
- "private_preds = model_private.predict(np.array([process_features(x) for x in unprocessed_x_private_test]))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": [
- "pipeline-metrics"
- ]
- },
- "outputs": [],
- "source": [
- "print(validation_loss)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "kubeflow_notebook": {
- "autosnapshot": false,
- "docker_image": "gitlab-registry.cern.ch/ai-ml/kubeflow_images/tensorflow-notebook-gpu-2.1.0:v0.6.1-27",
- "experiment": {
- "id": "new",
- "name": "dejan-open-vaccine-27"
- },
- "experiment_name": "dejan-open-vaccine-27",
- "katib_metadata": {
- "algorithm": {
- "algorithmName": "random",
- "algorithmSettings": [
- {
- "name": "random_state",
- "value": "10"
- }
- ]
- },
- "maxFailedTrialCount": 1,
- "maxTrialCount": 3,
- "objective": {
- "additionalMetricNames": [],
- "goal": 0,
- "objectiveMetricName": "validation-loss",
- "type": "minimize"
- },
- "parallelTrialCount": 1,
- "parameters": [
- {
- "feasibleSpace": {
- "max": "0.01",
- "min": "0.0001",
- "step": "0.0003"
- },
- "name": "LR",
- "parameterType": "double"
- },
- {
- "feasibleSpace": {
- "max": "256",
- "min": "32",
- "step": "32"
- },
- "name": "BATCH_SIZE",
- "parameterType": "int"
- },
- {
- "feasibleSpace": {
- "max": "100",
- "min": "20",
- "step": "20"
- },
- "name": "EMBED_DIM",
- "parameterType": "int"
- },
- {
- "feasibleSpace": {
- "list": [
- "0.2",
- "0.3",
- "0.4",
- "0.5"
- ]
- },
- "name": "DROPOUT",
- "parameterType": "categorical"
- },
- {
- "feasibleSpace": {
- "list": [
- "0.2",
- "0.3",
- "0.4",
- "0.5"
- ]
- },
- "name": "SP_DROPOUT",
- "parameterType": "categorical"
- }
- ]
- },
- "katib_run": true,
- "pipeline_description": "ML model for the open vaccine Kaggle competition",
- "pipeline_name": "dejan-open-vaccine-model-27",
- "snapshot_volumes": false,
- "steps_defaults": [],
- "volumes": []
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/open-vaccine/requirements.txt b/open-vaccine/requirements.txt
deleted file mode 100644
index c2a72403d12171cef0cd13dde2fbeba354197c55..0000000000000000000000000000000000000000
--- a/open-vaccine/requirements.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-pandas
-requests
-tensorflow==2.3.0
diff --git a/tensorboard/.gitkeep b/tensorboard/.gitkeep
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/tensorboard/keras-tuner-nb.ipynb b/tensorboard/keras-tuner-nb.ipynb
deleted file mode 100644
index 7ce87b0382504c4c7e6e81f57cf2dd805d34de0e..0000000000000000000000000000000000000000
--- a/tensorboard/keras-tuner-nb.ipynb
+++ /dev/null
@@ -1,548 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Keras Tuner KFP example, part II: creating a lightweight \"metrics evaluation\" component\n",
- "\n",
- "This [blog post](https://amygdala.github.io/gcp_blog/ml/kfp/kubeflow/keras/tensorflow/hp_tuning/2020/10/19/keras_tuner.html) and accompanying [tutorial](https://github.com/amygdala/code-snippets/blob/master/ml/kubeflow-pipelines/keras_tuner/README.md) walked through how to build a [Kubeflow Pipelines](https://www.kubeflow.org/docs/pipelines/) (KFP) pipeline that uses the [Keras Tuner](https://blog.tensorflow.org/2020/01/hyperparameter-tuning-with-keras-tuner.html) to build a hyperparameter-tuning workflow that uses distributed HP search.\n",
- "\n",
- "That pipeline does HP tuning, then runs full training on the N best parameter sets identified from the HP search, then deploys the full models to [TF-serving](https://www.tensorflow.org/tfx/guide/serving). \n",
- "One thing that was missing from that pipeline was any check on the quality of the trained models prior to deployment to TF-Serving.\n",
- "\n",
- "This notebook is a follow-on to that tutorial. Here, we show how you can create a KFP \"lightweight component\", built from a python function, to do a simple threshold check on some of the model metrics in order to decide whether to deploy the model. (This is a pretty simple approach, that we're using for illustrative purposes; for production models you'd probably want to do more sophisticated analyses. The [TFMA library](https://www.tensorflow.org/tfx/model_analysis/get_started) might be of interest).\n",
- "\n",
- "We'll also show how to use the KFP SDK to define and run pipelines from a notebook.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup\n",
- "\n",
- "This notebook assumes that you've **done the setup indicated in the [README](https://github.com/amygdala/code-snippets/blob/master/ml/kubeflow-pipelines/keras_tuner/README.md)**, and have an AI Platform Pipelines (Hosted KFP) installation, with GPU node pools added to the cluster.\n",
- "\n",
- "### Create an AI Platform Notebooks instance\n",
- "\n",
- "\n",
- "In addition, create an AI Platform Notebooks instance on which to run this notebook. See setup instructions [here](https://cloud.google.com/ai-platform/notebooks/docs). (You can run this notebook in other environments, e.g. locally, but that requires additional auth setup that we won't go into here).\n",
- "\n",
- "**Once your notebook instance is set up, you should be able to use [this link](https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?name=Create%20a%20new%20KFP%20component%20from%20a%20notebook&download_url=https%3A%2F%2Fraw.githubusercontent.com%2Famygdala%2Fcode-snippets%2Fmaster%2Fml%2Fkubeflow-pipelines%2Fkeras_tuner%2Fnotebooks%2Fmetrics_eval_component.ipynb&url=https%3A%2F%2Fgithub.com%2Famygdala%2Fcode-snippets%2Fblob%2Fmaster%2Fml%2Fkubeflow-pipelines%2Fkeras_tuner%2Fnotebooks%2Fmetrics_eval_component.ipynb) to upload and run the notebook.**"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next we'll do some imports:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 80,
- "metadata": {},
- "outputs": [],
- "source": [
- "import kfp # the Pipelines SDK. \n",
- "from kfp import compiler\n",
- "import kfp.dsl as dsl\n",
- "import kfp.gcp as gcp\n",
- "import kfp.components as comp"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Defining a new 'lightweight component' based on a python function\n",
- "\n",
- "'Lightweight' KFP python components allow you to create a component from a python function definition, and do not require you to build a new container image for every code change. They're helpful for fast iteration in a notebook environment. You can read more [here](https://github.com/kubeflow/pipelines/blob/master/samples/core/lightweight_component/lightweight_component.ipynb).\n",
- "\n",
- "In this section, we'll create a lightweight component that uses training metrics info to decide whether to deploy a model.\n",
- "We'll pass a \"threshold\" dict as a component arg, and compare those thresholds to the metrics values, and use that info to decide whether or not to deploy. Then we'll output a string indicating the decision. \n",
- "\n",
- "(As mentioned above, for production models you'd probably want to do a more substantial analysis. The [TFMA library](https://www.tensorflow.org/tfx/model_analysis/get_started) might be of interest. Stay tuned for a follow-on post that uses TFMA).\n",
- "\n",
- "Then we'll define a pipeline that uses the new component. In the pipeline spec, we'll make the 'serve' step conditional on the \"metrics\" op output."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "First, we'll define the component function, `eval_metrics`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 81,
- "metadata": {},
- "outputs": [],
- "source": [
- "from typing import NamedTuple\n",
- "\n",
- "def eval_metrics(\n",
- " metrics: str,\n",
- " thresholds: str\n",
- ") -> NamedTuple('Outputs', [('deploy', str)]):\n",
- "\n",
- " import json\n",
- " import logging\n",
- "\n",
- " def regression_threshold_check(metrics_info):\n",
- " for k, v in thresholds_dict.items():\n",
- " logging.info('k {}, v {}'.format(k, v))\n",
- " if k in ['root_mean_squared_error', 'mae']:\n",
- " if metrics_info[k][-1] > v:\n",
- " logging.info('{} > {}; returning False'.format(metrics_info[k][0], v))\n",
- " return ('False', )\n",
- " return ('deploy', )\n",
- "\n",
- " logging.getLogger().setLevel(logging.INFO)\n",
- "\n",
- " thresholds_dict = json.loads(thresholds)\n",
- " logging.info('thresholds dict: {}'.format(thresholds_dict))\n",
- " logging.info('metrics: %s', metrics)\n",
- " metrics_dict = json.loads(metrics)\n",
- "\n",
- " logging.info(\"got metrics info: %s\", metrics_dict)\n",
- " res = regression_threshold_check(metrics_dict)\n",
- " logging.info('deploy decision: %s', res)\n",
- " return res\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To keep things simple, we're comparing only RMSE and MAE with given threshold values. (This function is tailored for our Keras regression model). Lower is better, so if a threshold value is higher than the associated model metric, we won't deploy. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next, we'll create a 'container op' from the `eval_metrics` function definition, via the `funct_to_container_op` method. As one of the method args, we specify the base container image that will run the function. \n",
- "Here, we're using one of the [Deep Learning Container images](https://cloud.google.com/ai-platform/deep-learning-containers/docs/). (This container image includes more than is necessary for this simple function, but these DL images can be useful for many ML-related components)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 82,
- "metadata": {},
- "outputs": [],
- "source": [
- "eval_metrics_op = comp.func_to_container_op(eval_metrics, base_image='gcr.io/deeplearning-platform-release/tf2-cpu.2-3:latest')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Define a pipeline that uses the new \"metrics\" op\n",
- "\n",
- "Now, we can define a new pipeline that uses the new op and makes the model serving conditional on the results. \n",
- "\n",
- "The new `eval_metrics_op` takes as an input one of the `train_op` outputs, which outputs a final metrics dict. (We \"cheated\" a bit, as the training component was already designed to output this info; in other cases you might end up defining a new version of such an op that outputs the new info you need).\n",
- "\n",
- "Then, we'll wrap the serving op in a *conditional*; we won't set up a TF-serving service unless the `eval_metrics` op has certified that it is okay.\n",
- "\n",
- "Note that this new version of the pipeline also has a new input parameter— the `thresholds` dict.\n",
- "\n",
- "To keep things simple, we'll first define a pipeline that skips the HP tuning part of the pipeline used [here](https://github.com/amygdala/code-snippets/blob/master/ml/kubeflow-pipelines/keras_tuner/README.md). This will make it easier to test your new op with a pipeline that takes a shorter time to run.\n",
- "\n",
- "Then in a following section we'll show how to augment the full HP tuning pipeline to include the new op.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We'll first instantiate the other pipeline ops from their [reusable components](https://www.kubeflow.org/docs/pipelines/sdk/component-development/) definitions. (And we've defined the `eval_metrics_op` above)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 83,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "<function Serve TF model at 0x7f5a75ace488>\n"
- ]
- }
- ],
- "source": [
- "train_op = comp.load_component_from_url(\n",
- " 'https://gitlab.cern.ch/ai-ml/examples/-/raw/master/tensorboard/training_component/training_comp.yaml'\n",
- " )\n",
- "serve_op = comp.load_component_from_url(\n",
- " 'https://gitlab.cern.ch/ai-ml/examples/-/raw/master/tensorboard/serving_component/serving_comp.yaml'\n",
- " )\n",
- "print(serve_op)\n",
- "tb_op = comp.load_component_from_url(\n",
- " 'https://raw.githubusercontent.com/kubeflow/pipelines/master/components/tensorflow/tensorboard/prepare_tensorboard/component.yaml' \n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next, we'll define the pipeline itself. You might notice that this pipeline has a new parameter, `thresholds`.\n",
- "\n",
- "This pipeline first sets up a TensorBoard visualization for monitoring the training run. Then it starts the training. Once training is finished, the new op checks whether the trained model's final metrics are above the given threshold(s). \n",
- "If so (using the KFP `dsl.Condition` construct), TF-serving is used to set up a prediction service on the Pipelines GKE cluster.\n",
- "\n",
- "You can see that data is being passed between the pipeline ops. [Here's a tutorial](https://gist.github.com/amygdala/bfa0f599a4814b3261367f558a852bfe) that goes into how that works in more detail."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 84,
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsl.pipeline(\n",
- " name='bikes_weather_metrics',\n",
- " description='Model bike rental duration given weather'\n",
- ")\n",
- "def bikes_weather_metrics( \n",
- " train_epochs: int = 2,\n",
- " working_dir: str = 'gs://YOUR/GCS/PATH', # for the full training jobs\n",
- " data_dir: str = 'gs://aju-dev-demos-codelabs/bikes_weather/',\n",
- " steps_per_epoch: int = 2 , # if -1, don't override normal calcs based on dataset size\n",
- " hptune_params: str = '[{\"num_hidden_layers\": %s, \"learning_rate\": %s, \"hidden_size\": %s}]' % (3, 1e-2, 64),\n",
- " thresholds: str = '{\"root_mean_squared_error\": 10000}'\n",
- " ):\n",
- "\n",
- " # create TensorBoard viz for the parent directory of all training runs, so that we can\n",
- " # compare them.\n",
- " tb_viz = tb_op(\n",
- " log_dir_uri='%s/%s' % (working_dir, dsl.RUN_ID_PLACEHOLDER)\n",
- " )\n",
- "\n",
- " train = train_op(\n",
- " data_dir=data_dir,\n",
- " workdir='%s/%s' % (tb_viz.outputs['log_dir_uri'], 0),\n",
- " tb_dir=tb_viz.outputs['log_dir_uri'],\n",
- " epochs=train_epochs, \n",
- " steps_per_epoch=steps_per_epoch,\n",
- " hp_idx=0, \n",
- " hptune_results=hptune_params\n",
- " )\n",
- "\n",
- " eval_metrics = eval_metrics_op(\n",
- " thresholds=thresholds,\n",
- " metrics=train.outputs['metrics_output_path'],\n",
- " )\n",
- "\n",
- " with dsl.Condition(eval_metrics.outputs['deploy'] == 'deploy'): # conditional serving\n",
- " serve = serve_op(\n",
- " model_path=train.outputs['train_output_path'],\n",
- " model_name='bikesw'\n",
- " )\n",
- " #train.set_gpu_limit(1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we can run the pipeline from the notebook. First create a client object to talk to your KFP installation. Using that client, create (or get) an _Experiment_ (which lets you create semantic groupings of pipeline runs).\n",
- "\n",
- "You'll need to set the correct host endpoint for your pipelines installation when you create the client. Visit the [Pipelines panel in the Cloud Console](https://console.cloud.google.com/ai-platform/pipelines/clusters) and click on the **SETTINGS** gear for the desired installation to get its endpoint."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 85,
- "metadata": {},
- "outputs": [],
- "source": [
- "# CHANGE THIS with the info for your KFP cluster installation\n",
- "client = kfp.Client()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 86,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "<a href=\"/pipeline/#/experiments/details/a57d1f5a-4f89-4e6f-9f3b-b27d447d5d9a\" target=\"_blank\" >Experiment details</a>."
- ],
- "text/plain": [
- "<IPython.core.display.HTML object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "exp = client.create_experiment(name='bw_expers') # this is a 'get or create' call"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "(If the `create_experiment` call failed, double check your host endpoint value).\n",
- "\n",
- "Now, we can compile and then run the pipeline. We'll set some vars with pipeline params:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 93,
- "metadata": {},
- "outputs": [],
- "source": [
- "WORKING_DIR = 'gs://dejan_testing_bucket'\n",
- "TRAIN_EPOCHS = 5"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we'll compile and run the pipeline. \n",
- "\n",
- "Note that this pipeline is configured to use a GPU node for the training step, so make sure that you have set up a GPU node pool for the cluster that your KFP installation is running on, as described in this [README](https://github.com/amygdala/code-snippets/blob/master/ml/kubeflow-pipelines/keras_tuner/README.md). Note also that GPU nodes are more expensive. \n",
- "If you want, you can comment out the `train.set_gpu_limit(2)` line in the pipeline definition above to run training on a CPU node."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 94,
- "metadata": {},
- "outputs": [],
- "source": [
- "compiler.Compiler().compile(bikes_weather_metrics, 'bikes_weather_metrics.tar.gz')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 95,
- "metadata": {},
- "outputs": [],
- "source": [
- "#client.upload_pipeline('bikes_weather_metrics.tar.gz', 'pip_bikes_dg_15', 'namespace: dejan-golubovic')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 96,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "<a href=\"/pipeline/#/runs/details/b1a506c9-ad4e-4028-b6f3-7de8ac3cae42\" target=\"_blank\" >Run details</a>."
- ],
- "text/plain": [
- "<IPython.core.display.HTML object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "run = client.run_pipeline(exp.id, 'bw_metrics_test', 'bikes_weather_metrics.tar.gz',\n",
- " params={'working_dir': WORKING_DIR, 'train_epochs': TRAIN_EPOCHS\n",
- " # 'thresholds': THRESHOLDS\n",
- " })"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Once you've kicked off the run, click the generated link to see the pipeline run in the Kubeflow Pipelines dashboard of your pipelines installation. (See the last section of this notebook for more info on how to use your trained and deployed model for prediction).\n",
- "\n",
- "**Note**: It's also possible to start a pipeline run directly from the pipeline function definition, skipping the local compilation, like this:\n",
- "```python\n",
- "kfp.Client(host=kfp_endpoint).create_run_from_pipeline_func(<pipeline_function_name>, arguments={})\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Use the new \"metrics\" op with the full Keras Tuner pipeline\n",
- "\n",
- "To keep things simple, the pipeline above didn't do an HP tuning search.\n",
- "Below is how the full pipeline from [this tutorial](https://github.com/amygdala/code-snippets/blob/master/ml/kubeflow-pipelines/keras_tuner/README.md) would be redefined to use this new op. \n",
- "\n",
- "This definition assumes that you've run the cells above that instantiated the ops from their component specs. This new definition includes an additional `hptune` op (defined \"inline\" using `dsl.ContainerOp()`) that deploys the distributed HP tuning job and then waits for the results. \n",
- "\n",
- "> **Important note**: this example may take a long time to run, and **incur significant charges** in its use of GPUs, depending upon how its parameters are configured."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsl.pipeline(\n",
- " name='bikes_weather_keras_tuner',\n",
- " description='Model bike rental duration given weather, use Keras Tuner'\n",
- ")\n",
- "def bikes_weather_hptune(\n",
- " tune_epochs: int = 2,\n",
- " train_epochs: int = 5,\n",
- " num_tuners: int = 8,\n",
- " bucket_name: str = 'YOUR_BUCKET_NAME', # used for the HP dirs; don't include the 'gs://'\n",
- " tuner_dir_prefix: str = 'hptest',\n",
- " tuner_proj: str = 'p1',\n",
- " max_trials: int = 128,\n",
- " working_dir: str = 'gs://YOUR/GCS/PATH', # for the full training jobs\n",
- " data_dir: str = 'gs://aju-dev-demos-codelabs/bikes_weather/',\n",
- " steps_per_epoch: int = -1 , # if -1, don't override normal calcs based on dataset size\n",
- " num_best_hps: int = 2, # the N best parameter sets for full training\n",
- " # the indices to the best param sets; necessary in addition to the above param because of\n",
- " # how KFP loops work currently. Must be consistent with the above param.\n",
- " num_best_hps_list: list = [0, 1],\n",
- " thresholds: str = '{\"root_mean_squared_error\": 2000}'\n",
- " ):\n",
- "\n",
- " hptune = dsl.ContainerOp(\n",
- " name='ktune',\n",
- " image='gcr.io/google-samples/ml-pipeline-bikes-dep:b97ee76',\n",
- " arguments=['--epochs', tune_epochs, '--num-tuners', num_tuners,\n",
- " '--tuner-dir', '%s/%s' % (tuner_dir_prefix, dsl.RUN_ID_PLACEHOLDER),\n",
- " '--tuner-proj', tuner_proj, '--bucket-name', bucket_name, '--max-trials', max_trials,\n",
- " '--namespace', 'default', '--num-best-hps', num_best_hps, '--executions-per-trial', 2,\n",
- " '--deploy'\n",
- " ],\n",
- " file_outputs={'hps': '/tmp/hps.json'},\n",
- " )\n",
- "\n",
- " # create TensorBoard viz for the parent directory of all training runs, so that we can\n",
- " # compare them.\n",
- " tb_viz = tb_op(\n",
- " log_dir_uri='%s/%s' % (working_dir, dsl.RUN_ID_PLACEHOLDER)\n",
- " )\n",
- "\n",
- " with dsl.ParallelFor(num_best_hps_list) as idx: # start the full training runs in parallel\n",
- "\n",
- " train = train_op(\n",
- " data_dir=data_dir,\n",
- " workdir='%s/%s' % (tb_viz.outputs['log_dir_uri'], idx),\n",
- " tb_dir=tb_viz.outputs['log_dir_uri'],\n",
- " epochs=train_epochs, steps_per_epoch=steps_per_epoch,\n",
- " hp_idx=idx, hptune_results=hptune.outputs['hps']\n",
- " )\n",
- "\n",
- " eval_metrics = eval_metrics_op(\n",
- " thresholds=thresholds,\n",
- " metrics=train.outputs['metrics_output_path'],\n",
- " )\n",
- "\n",
- " with dsl.Condition(eval_metrics.outputs['deploy'] == 'deploy'): # conditional serving\n",
- " serve = serve_op(\n",
- " model_path=train.outputs['train_output_path'],\n",
- " model_name='bikesw',\n",
- " namespace='default'\n",
- " )\n",
- "\n",
- " train.set_gpu_limit(1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "If you want, you can compile and run this pipeline the same way as was done in the previous section. You can also find this pipeline in the example repo [here](https://github.com/amygdala/code-snippets/blob/master/ml/kubeflow-pipelines/keras_tuner/example_pipelines/bw_ktune_metrics.py)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## More detail on the code, and requesting predictions from your model\n",
- "\n",
- "This notebook didn't focus on the details of the pipeline component (step) implementations. The training component uses a Keras model (TF 2.3). The serving component uses [TF-serving](https://www.tensorflow.org/tfx/guide/serving): once the serving service is up and running, you can send prediction requests to your trained model.\n",
- "\n",
- "You can find more detail on these components, and an example of sending a prediction request, [here](https://github.com/amygdala/code-snippets/tree/master/ml/kubeflow-pipelines/keras_tuner).\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "----------------------------\n",
- "Copyright 2020, Google, LLC. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
- "\n",
- "http://www.apache.org/licenses/LICENSE-2.0\n",
- "\n",
- "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
- ]
- }
- ],
- "metadata": {
- "environment": {
- "name": "tf2-2-3-gpu.2-3.m56",
- "type": "gcloud",
- "uri": "gcr.io/deeplearning-platform-release/tf2-2-3-gpu.2-3:m56"
- },
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "kubeflow_notebook": {
- "autosnapshot": false,
- "docker_image": "gitlab-registry.cern.ch/ai-ml/kubeflow_images/tensorflow-notebook-gpu-2.1.0:v0.6.1-30",
- "experiment": {
- "id": "",
- "name": ""
- },
- "experiment_name": "",
- "katib_metadata": {
- "algorithm": {
- "algorithmName": "grid"
- },
- "maxFailedTrialCount": 3,
- "maxTrialCount": 12,
- "objective": {
- "objectiveMetricName": "",
- "type": "minimize"
- },
- "parallelTrialCount": 3,
- "parameters": []
- },
- "katib_run": false,
- "pipeline_description": "",
- "pipeline_name": "",
- "snapshot_volumes": false,
- "steps_defaults": [],
- "volume_access_mode": "rwm",
- "volumes": []
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/tensorboard/serving_component/.gitkeep b/tensorboard/serving_component/.gitkeep
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/tensorboard/serving_component/Dockerfile b/tensorboard/serving_component/Dockerfile
deleted file mode 100644
index f202d36a23671db4f2d8d53bc9fbfb80b7102161..0000000000000000000000000000000000000000
--- a/tensorboard/serving_component/Dockerfile
+++ /dev/null
@@ -1,26 +0,0 @@
-FROM gitlab-registry.cern.ch/cloud/ciadm:209156b743a4f9133b739b3d63130d8c332ceaa4
-
-# Setup kerberos (if model is located on EOS)
-COPY krb5.conf /etc/krb5.conf
-
-# Clone KFP
-WORKDIR /
-RUN git clone -b cern/v1.4.1 https://gitlab.cern.ch/ai-ml/pipelines.git
-
-# Install kfp-server-api
-WORKDIR /pipelines/backend/api/python_http_client
-RUN pip3 install --upgrade "enum34==1.1.8" && \
- pip3 install -U . --upgrade
-
-# Install kfp
-WORKDIR /pipelines/sdk/python
-RUN pip3 install -U . --upgrade
-
-# Install kubernetes python client
-RUN pip3 install kubernetes
-
-# Copy template and serving script
-RUN mkdir -p /ml
-COPY serve_model.py /ml/serve_model.py
-COPY inference_service_template.yaml /ml/inference_service_template.yaml
-WORKDIR /ml
diff --git a/tensorboard/serving_component/inference_service_template.yaml b/tensorboard/serving_component/inference_service_template.yaml
deleted file mode 100644
index 90e52f8a7acf5ed547b15ef9b76eb33bfc77d00e..0000000000000000000000000000000000000000
--- a/tensorboard/serving_component/inference_service_template.yaml
+++ /dev/null
@@ -1,10 +0,0 @@
-apiVersion: "serving.kubeflow.org/v1alpha2"
-kind: "InferenceService"
-metadata:
- name: MODEL_NAME_TO_REPLACE
- namespace: NAMESPACE_TO_REPLACE
-spec:
- default:
- predictor:
- tensorflow:
- storageUri: STORAGE_TO_REPLACE
diff --git a/tensorboard/serving_component/krb5.conf b/tensorboard/serving_component/krb5.conf
deleted file mode 100644
index f1dc2680abf70ab19185ba3d28786d2f6081243b..0000000000000000000000000000000000000000
--- a/tensorboard/serving_component/krb5.conf
+++ /dev/null
@@ -1,89 +0,0 @@
-; AD : This Kerberos configuration is for CERN's Active Directory realm.
-;
-; /etc/krb5.conf
-
-[libdefaults]
- default_realm = CERN.CH
- ticket_lifetime = 25h
- renew_lifetime = 120h
- forwardable = true
- proxiable = true
- default_tkt_enctypes = arcfour-hmac-md5 aes256-cts aes128-cts des3-cbc-sha1 des-cbc-md5 des-cbc-crc
- chpw_prompt = true
- rdns = false
-
-[appdefaults]
-pam = {
- external = true
- krb4_convert = false
- krb4_convert_524 = false
- krb4_use_as_req = false
-}
-
-[domain_realm]
-.cern.ch = CERN.CH
-.fnal.gov = FNAL.GOV
-.hep.man.ac.uk = HEP.MAN.AC.UK
-.in2p3.fr = IN2P3.FR
-# No default domain for KFKI.HU specified.
-
-[realms]
-# Start of puppet output for CERN.CH
- CERN.CH = {
- default_domain = cern.ch
- kpasswd_server = cerndc.cern.ch
- admin_server = cerndc.cern.ch
- kdc = cerndc.cern.ch
- v4_name_convert = {
- host = {
- rcmd = host
- }
- }
- }
-
-
-# Start of puppet output for FNAL.GOV
- FNAL.GOV = {
- default_domain = fnal.gov
- admin_server = krb-fnal-admin.fnal.gov
- kdc = krb-fnal-fcc3.fnal.gov:88
- kdc = krb-fnal-2.fnal.gov:88
- kdc = krb-fnal-3.fnal.gov:88
- kdc = krb-fnal-1.fnal.gov:88
- kdc = krb-fnal-4.fnal.gov:88
- kdc = krb-fnal-enstore.fnal.gov:88
- kdc = krb-fnal-fg2.fnal.gov:88
- kdc = krb-fnal-cms188.fnal.gov:88
- kdc = krb-fnal-cms204.fnal.gov:88
- kdc = krb-fnal-d0online.fnal.gov:88
- }
-
-
-# Start of puppet output for HEP.MAN.AC.UK
- HEP.MAN.AC.UK = {
- default_domain = hep.man.ac.uk
- kpasswd_server = afs4.hep.man.ac.uk
- admin_server = afs4.hep.man.ac.uk
- kdc = afs1.hep.man.ac.uk
- kdc = afs2.hep.man.ac.uk
- kdc = afs3.hep.man.ac.uk
- kdc = afs4.hep.man.ac.uk
- }
-
-
-# Start of puppet output for IN2P3.FR
- IN2P3.FR = {
- default_domain = in2p3.fr
- kpasswd_server = kerberos-admin.in2p3.fr
- admin_server = kerberos-admin.in2p3.fr
- kdc = kerberos-1.in2p3.fr
- kdc = kerberos-2.in2p3.fr
- kdc = kerberos-3.in2p3.fr
- }
-
-
-# Start of puppet output for KFKI.HU
- KFKI.HU = {
- admin_server = kerberos.kfki.hu
- kdc = kerberos.kfki.hu
- }
diff --git a/tensorboard/serving_component/serve_model.py b/tensorboard/serving_component/serve_model.py
deleted file mode 100644
index a28f1fe4208346a44277a2448f589538c3595824..0000000000000000000000000000000000000000
--- a/tensorboard/serving_component/serve_model.py
+++ /dev/null
@@ -1,70 +0,0 @@
-import argparse
-import kfp
-from kubernetes import config
-import yaml
-import kubernetes
-
-
-def get_parser():
- parser = argparse.ArgumentParser(description='Serving Params')
-
- parser.add_argument('--model_name', type=str)
- parser.add_argument('--model_path', type=str)
-
- return parser
-
-def edit_template(src, dst, args, ns):
- with open(src, 'r') as f:
- template = f.read()
-
- template = template.replace('MODEL_NAME_TO_REPLACE', args.model_name)
- template = template.replace('STORAGE_TO_REPLACE', args.model_path)
- template = template.replace('NAMESPACE_TO_REPLACE', ns)
-
- with open(dst, 'w') as f:
- f.write(template)
-
-def create_inferenceservice(client, yaml_filepath, ns):
- CO_GROUP = "serving.kubeflow.org"
- CO_VERSION = "v1alpha2"
- CO_PLURAL = "inferenceservices"
- API_VERSION = "%s/%s" % (CO_GROUP, CO_VERSION)
-
- print('yaml_filepath')
- print(yaml_filepath)
-
- with open(yaml_filepath, 'r') as f:
- infs_spec = yaml.load(f, Loader=yaml.FullLoader)
-
- print('Template loaded for submission')
-
- client.create_namespaced_custom_object(CO_GROUP, CO_VERSION,
- ns, CO_PLURAL,
- infs_spec)
-
-
-parser = get_parser()
-args = parser.parse_args()
-namespace = kfp.Client().get_user_namespace()
-
-print('args:')
-print(args)
-
-print('namespace:')
-print(namespace)
-
-edit_template(src='inference_service_template.yaml', \
- dst='inference_service.yaml', \
- args=args,
- ns=namespace)
-
-print('Template edited')
-
-config.load_incluster_config()
-print('Incluster config loaded')
-
-k8s_co_client = kubernetes.client.CustomObjectsApi()
-print('Client obtained')
-
-create_inferenceservice(k8s_co_client, 'inference_service.yaml', namespace)
-print('CRD created')
diff --git a/tensorboard/serving_component/serving_comp.yaml b/tensorboard/serving_component/serving_comp.yaml
deleted file mode 100644
index d5afe7e2a70c4f0b482d3f50010d6b530cda7aa0..0000000000000000000000000000000000000000
--- a/tensorboard/serving_component/serving_comp.yaml
+++ /dev/null
@@ -1,23 +0,0 @@
-name: Serve TF model
-description: |
- A Kubeflow Pipeline component to deploy a tf-serving service
-metadata:
- labels:
- add-pod-env: 'true'
-inputs:
- - name: model_name
- type: String
- - name: model_path
- type: GCSPath
-implementation:
- container:
- image: gitlab-registry.cern.ch/ai-ml/kubeflow_images/serving:8
- command:
- - python3
- - /ml/serve_model.py
- - --model_name
- - inputValue: model_name
- - --model_path
- - inputValue: model_path
- env:
- KFP_POD_NAME: "{{pod.name}}"
diff --git a/tensorboard/training_component/Dockerfile b/tensorboard/training_component/Dockerfile
deleted file mode 100644
index 8a482aeb3f145f82bf2c2bbbc7ad5ecb82381bbf..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/Dockerfile
+++ /dev/null
@@ -1,9 +0,0 @@
-FROM tensorflow/tensorflow:2.3.0-gpu
-
-RUN pip install --upgrade pip
-RUN pip install pathlib2
-RUN pip install gsutil
-
-ADD ml /ml
-
-ENTRYPOINT ["python", "/ml/bikes_weather_limited.py"]
\ No newline at end of file
diff --git a/tensorboard/training_component/README.md b/tensorboard/training_component/README.md
deleted file mode 100644
index f643b440f591dace1d4675bdcc4c06adc9909421..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/README.md
+++ /dev/null
@@ -1,2 +0,0 @@
-# training_component
-
diff --git a/tensorboard/training_component/ml/bikes_weather_limited.py b/tensorboard/training_component/ml/bikes_weather_limited.py
deleted file mode 100644
index ae9fbc749ea0ccc7e3747888cdf8f2111f0d71dc..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/ml/bikes_weather_limited.py
+++ /dev/null
@@ -1,188 +0,0 @@
-
-# Copyright 2020 Google Inc. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# https://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-import argparse
-import logging
-import json
-import time
-import os
-
-import tensorflow as tf
-import pathlib2
-
-import bwmodel.model as bwmodel
-
-DEVELOP_MODE = False
-NBUCKETS = 5 # for embeddings
-NUM_EXAMPLES = 1000*1000 * 20 # assume 20 million examples
-
-STRATEGY = tf.distribute.MirroredStrategy()
-TRAIN_BATCH_SIZE = 64 * STRATEGY.num_replicas_in_sync
-
-
-def create_model(learning_rate, hidden_size, num_hidden_layers):
-
- inputs, sparse, real = bwmodel.get_layers()
-
- logging.info('sparse keys: %s', sparse.keys())
- logging.info('real keys: %s', real.keys())
-
- model = None
- print('num replicas...')
- print(STRATEGY.num_replicas_in_sync)
-
- with STRATEGY.scope(): # hmmm
- model = bwmodel.wide_and_deep_classifier(
- inputs,
- linear_feature_columns=sparse.values(),
- dnn_feature_columns=real.values(),
- num_hidden_layers=num_hidden_layers,
- dnn_hidden_units1=hidden_size,
- learning_rate=learning_rate)
-
-
- model.summary()
- return model
-
-
-def main():
-
- logging.getLogger().setLevel(logging.INFO)
- parser = argparse.ArgumentParser(description='ML Trainer')
- parser.add_argument(
- '--epochs', type=int, default=1)
- parser.add_argument(
- # e.g. {"num_hidden_layers": 3, "hidden_size": 96, "learning_rate": 0.01}
- '--hptune-results', required=True)
- parser.add_argument(
- '--steps-per-epoch', type=int,
- default=-1) # if set to -1, don't override the normal calcs for this
- parser.add_argument(
- '--hp-idx', type=int,
- default=0)
- parser.add_argument(
- '--workdir', required=True)
- parser.add_argument(
- '--tb-dir', required=True)
- parser.add_argument(
- '--data-dir', default='gs://aju-dev-demos-codelabs/bikes_weather/')
- parser.add_argument(
- '--train-output-path', required=True)
- parser.add_argument(
- '--metrics-output-path', required=True)
-
- args = parser.parse_args()
- logging.info('Tensorflow version %s', tf.__version__)
-
- logging.info('got hptune results: %s', args.hptune_results)
- hptune_info = json.loads(str(args.hptune_results))
- logging.info('hptune_info: %s', hptune_info)
- # extract hptuning best params results
- learning_rate = hptune_info[args.hp_idx]['learning_rate']
- hidden_size = hptune_info[args.hp_idx]['hidden_size']
- num_hidden_layers = hptune_info[args.hp_idx]['num_hidden_layers']
- logging.info('using: learning rate %s, hidden size %s, first hidden layer %s',
- learning_rate, hidden_size, num_hidden_layers)
-
- TRAIN_DATA_PATTERN = args.data_dir + "train*"
- EVAL_DATA_PATTERN = args.data_dir + "test*"
- OUTPUT_DIR = '{}/bwmodel/trained_model'.format(args.workdir)
- logging.info('Writing trained model to %s', OUTPUT_DIR)
-
- train_batch_size = TRAIN_BATCH_SIZE
- eval_batch_size = 1000
- if args.steps_per_epoch == -1: # calc based on dataset size
- steps_per_epoch = NUM_EXAMPLES // train_batch_size
- else:
- steps_per_epoch = args.steps_per_epoch
- logging.info('using %s steps per epoch', steps_per_epoch)
-
- train_dataset = bwmodel.read_dataset(TRAIN_DATA_PATTERN, train_batch_size)
- eval_dataset = bwmodel.read_dataset(EVAL_DATA_PATTERN, eval_batch_size,
- tf.estimator.ModeKeys.EVAL, eval_batch_size * 100 * STRATEGY.num_replicas_in_sync
- )
-
- # Create metadata.json file for Tensorboard 'artifact'
- metadata = {
- 'outputs' : [{
- 'type': 'tensorboard',
- 'source': args.tb_dir
- }]
- }
-
- with open('/mlpipeline-ui-metadata.json', 'w') as f:
- json.dump(metadata, f)
-
- model = create_model(learning_rate, hidden_size, num_hidden_layers)
-
- checkpoint_path = '{}/checkpoints/bikes_weather.cpt'.format(OUTPUT_DIR)
- logging.info("checkpoint path: %s", checkpoint_path)
- cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,
- save_weights_only=True,
- verbose=1)
- tb_callback = tf.keras.callbacks.TensorBoard(log_dir='{}/logs'.format(OUTPUT_DIR),
- update_freq=20000)
-
- logging.info("training model....")
- history = model.fit(train_dataset,
- validation_data=eval_dataset,
- validation_steps=eval_batch_size,
- epochs=args.epochs,
- steps_per_epoch=steps_per_epoch,
- callbacks=[cp_callback, tb_callback]
- )
- logging.info(history.history.keys())
- # write metrics info dict
- metrics_json = json.dumps(history.history)
- print('metrics json: {}'.format(metrics_json))
- try:
- pathlib2.Path(args.metrics_output_path).parent.mkdir(parents=True)
- except FileExistsError as e1:
- logging.info(e1)
- pathlib2.Path(args.metrics_output_path).write_text(metrics_json)
-
- ts = str(int(time.time()))
- export_dir = '{}/export/bikesw/{}'.format(OUTPUT_DIR, ts)
- logging.info('Exporting to %s', export_dir)
-
- try:
- pathlib2.Path(args.train_output_path).parent.mkdir(parents=True)
- except FileExistsError as e2:
- logging.info(e2)
- try:
- logging.info("exporting model....")
- tf.saved_model.save(model, '/tmp/exported_model')
- gsutil_cp = 'gsutil cp /tmp/exported_model ' + export_path
- os.system(gsutil_cp)
- logging.info("train_output_path: %s", args.train_output_path)
- export_path = '{}/export/bikesw'.format(OUTPUT_DIR)
- logging.info('export path: %s', export_path)
- pathlib2.Path(args.train_output_path).write_text(export_path)
- except Exception as e: # retry once if error
- logging.warning(e)
- logging.info("retrying...")
- time.sleep(10)
- logging.info("again ... exporting model....")
- tf.saved_model.save(model, '/tmp/exported_model')
- gsutil_cp = 'gsutil cp /tmp/exported_model ' + export_path
- os.system(gsutil_cp)
- export_path = '{}/export/bikesw'.format(OUTPUT_DIR)
- pathlib2.Path(args.train_output_path).write_text(export_path)
-
-
-
-if __name__ == "__main__":
- main()
diff --git a/tensorboard/training_component/ml/bw_hptune_standalone.py b/tensorboard/training_component/ml/bw_hptune_standalone.py
deleted file mode 100644
index 0b9152e023617bd53b0ba72267d643ebdac8e71d..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/ml/bw_hptune_standalone.py
+++ /dev/null
@@ -1,193 +0,0 @@
-# Copyright 2020 Google Inc. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# https://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import argparse
-import logging
-import json
-import time
-
-import tensorflow as tf
-from google.cloud import storage
-from kerastuner.tuners import RandomSearch # , Hyperband
-
-import bwmodel.model as bwmodel
-
-
-DEVELOP_MODE = False
-NBUCKETS = 5 # for embeddings
-NUM_EXAMPLES = 1000*1000 * 20 # assume 20 million examples
-
-STRATEGY = tf.distribute.MirroredStrategy()
-TRAIN_BATCH_SIZE = 256 * STRATEGY.num_replicas_in_sync
-
-
-def create_model(hp):
-
- inputs, sparse, real = bwmodel.get_layers()
-
- logging.info('sparse keys: %s', sparse.keys())
- logging.info('real keys: %s', real.keys())
-
- model = None
- print('num replicas...')
- print(STRATEGY.num_replicas_in_sync)
-
- model = bwmodel.wide_and_deep_classifier(
- inputs,
- linear_feature_columns=sparse.values(),
- dnn_feature_columns=real.values(),
- num_hidden_layers=hp.Int('num_hidden_layers', 2, 5),
- dnn_hidden_units1=hp.Int('hidden_size', 16, 256, step=32),
- learning_rate=hp.Choice('learning_rate',
- values=[5e-1, 1e-1, 1e-2, 1e-3, 1e-4])
- )
-
- model.summary()
- return model
-
-
-def main():
- logging.getLogger().setLevel(logging.INFO)
- parser = argparse.ArgumentParser(description='Keras Tuner HP search')
- parser.add_argument(
- '--epochs', type=int,
- default=1)
- parser.add_argument(
- '--steps-per-epoch', type=int,
- default=-1) # if set to -1, don't override the normal calcs for this
- parser.add_argument(
- '--tuner-proj',
- required=True)
- parser.add_argument(
- '--bucket-name', required=True)
- parser.add_argument(
- '--tuner-dir',
- required=True)
- parser.add_argument(
- '--tuner-num',
- required=True)
- parser.add_argument(
- '--respath',
- required=True)
- parser.add_argument(
- '--executions-per-trial', type=int,
- default=2)
- parser.add_argument(
- '--max-trials', type=int,
- default=20)
- parser.add_argument(
- '--num-best-hps', type=int,
- default=2)
- parser.add_argument(
- '--data-dir',
- default='gs://aju-dev-demos-codelabs/bikes_weather/')
-
- args = parser.parse_args()
- logging.info('Tensorflow version %s', tf.__version__)
-
- TRAIN_DATA_PATTERN = args.data_dir + "train*"
- EVAL_DATA_PATTERN = args.data_dir + "test*"
-
-
- train_batch_size = TRAIN_BATCH_SIZE
- eval_batch_size = 1000
- if args.steps_per_epoch == -1: # calc based on dataset size
- steps_per_epoch = NUM_EXAMPLES // train_batch_size
- else:
- steps_per_epoch = args.steps_per_epoch
- logging.info('using %s steps per epoch', steps_per_epoch)
-
- logging.info('using train batch size %s', train_batch_size)
- train_dataset = bwmodel.read_dataset(TRAIN_DATA_PATTERN, train_batch_size)
- eval_dataset = bwmodel.read_dataset(EVAL_DATA_PATTERN, eval_batch_size,
- tf.estimator.ModeKeys.EVAL,
- eval_batch_size * 100 * STRATEGY.num_replicas_in_sync
- )
-
- logging.info('executions per trial: %s', args.executions_per_trial)
-
- # TODO: parameterize
- retries = 0
- num_retries = 5
- sleep_time = 5
- while retries < num_retries:
- try:
- tuner = RandomSearch(
- # tuner = Hyperband(
- create_model,
- objective='val_mae',
- # max_epochs=10,
- # hyperband_iterations=2,
- max_trials=args.max_trials,
- distribution_strategy=STRATEGY,
- executions_per_trial=args.executions_per_trial,
- directory=args.tuner_dir,
- project_name=args.tuner_proj
- )
- break
- except Exception as e:
- logging.warning(e)
- logging.info('sleeping %s seconds...', sleep_time)
- time.sleep(sleep_time)
- retries += 1
- sleep_time *= 2
-
- logging.info("search space summary:")
- logging.info(tuner.search_space_summary())
-
- logging.info("hp tuning model....")
- tuner.search(train_dataset,
- validation_data=eval_dataset,
- validation_steps=eval_batch_size,
- epochs=args.epochs,
- steps_per_epoch=steps_per_epoch,
- )
- best_hps = tuner.get_best_hyperparameters(args.num_best_hps)
- best_hps_list = [best_hps[i].values for i in range(args.num_best_hps)]
- logging.info('best_hps_list: %s', best_hps_list)
- best_hp_values = json.dumps(best_hps_list)
- logging.info('best hyperparameters: %s', best_hp_values)
-
- storage_client = storage.Client()
- logging.info('writing best results to %s', args.respath)
- bucket = storage_client.get_bucket(args.bucket_name)
- logging.info('using bucket %s: %s, path %s', args.bucket_name, bucket, args.respath)
- blob = bucket.blob(args.respath)
- blob.upload_from_string(best_hp_values)
-
- # uncomment to also save best model from hp search
- # OUTPUT_DIR = '{}/{}/{}/bwmodel/trained_model'.format(args.tuner_dir,
- # args.tuner_proj, args.tuner_num)
- # logging.info('Writing trained model to %s', OUTPUT_DIR)
-
- # best_model = tuner.get_best_models(1)[0]
- # logging.info('best model: %s', best_model)
-
- # ts = str(int(time.time()))
- # export_dir = '{}/export/bikesw/{}'.format(OUTPUT_DIR, ts)
- # logging.info('Exporting to %s', export_dir)
-
- # try:
- # logging.info("exporting model....")
- # tf.saved_model.save(best_model, export_dir)
- # except Exception as e: # retry once if error
- # logging.warning(e)
- # logging.info("retrying...")
- # time.sleep(10)
- # logging.info("again ... exporting model....")
- # tf.saved_model.save(best_model, export_dir)
-
-
-if __name__ == "__main__":
- main()
diff --git a/tensorboard/training_component/ml/bwmodel/__init__.py b/tensorboard/training_component/ml/bwmodel/__init__.py
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/tensorboard/training_component/ml/bwmodel/model.py b/tensorboard/training_component/ml/bwmodel/model.py
deleted file mode 100644
index 1f4e022bd6d4c18834b2b4ca92901cc1f191e4d9..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/ml/bwmodel/model.py
+++ /dev/null
@@ -1,111 +0,0 @@
-# Copyright 2020 Google Inc. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# https://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Adapted in part from:
-# https://github.com/GoogleCloudPlatform/data-science-on-gcp/blob/master/09_cloudml/flights_model_tf2.ipynb
-# by Valliappa Lakshmanan. (See that repo for more info about the accompanying book,
-# "Data Science on the Google Cloud Platform", from O'Reilly.)
-
-import tensorflow as tf
-
-
-CSV_COLUMNS = ('duration,end_station_id,bike_id,ts,day_of_week,start_station_id' +
- ',start_latitude,start_longitude,end_latitude,end_longitude' +
- ',euclidean,loc_cross,prcp,max,min,temp,dewp').split(',')
-LABEL_COLUMN = 'duration'
-DEFAULTS = [[0.0], ['na'], ['na'], [0.0], ['na'], ['na'],
- [0.0], [0.0], [0.0], [0.0],
- [0.0], ['na'], [0.0], [0.0], [0.0], [0.0], [0.0]]
-
-def load_dataset(pattern, batch_size=1):
- return tf.data.experimental.make_csv_dataset(pattern, batch_size, CSV_COLUMNS, DEFAULTS)
-
-def features_and_labels(features):
- label = features.pop('duration') # this is what we will train for
- features.pop('bike_id')
- return features, label
-
-def read_dataset(pattern, batch_size, mode=tf.estimator.ModeKeys.TRAIN, truncate=None):
- dataset = load_dataset(pattern, batch_size)
- dataset = dataset.map(features_and_labels, num_parallel_calls=tf.data.experimental.AUTOTUNE)
- if mode == tf.estimator.ModeKeys.TRAIN:
- dataset = dataset.repeat().shuffle(batch_size*10)
- # dataset = dataset.repeat()
- dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
- # dataset = dataset.prefetch(1)
- if truncate is not None:
- dataset = dataset.take(truncate)
- return dataset
-
-def get_layers():
-
- # duration,end_station_id,bike_id,ts,day_of_week,start_station_id,start_latitude,start_longitude,end_latitude,end_longitude,
- # euclidean,loc_cross,prcp,max,min,temp,dewp
- real = {
- colname : tf.feature_column.numeric_column(colname)
- for colname in
- # ('ts,start_latitude,start_longitude,end_latitude,end_longitude,euclidean,prcp,max,min,temp,dewp').split(',')
- # ('ts,euclidean,prcp,max,min,temp,dewp').split(',')
- ('euclidean,prcp,max,min,temp,dewp').split(',')
- }
- sparse = {
- 'day_of_week': tf.feature_column.categorical_column_with_vocabulary_list('day_of_week',
- vocabulary_list='1,2,3,4,5,6,7'.split(',')),
- 'end_station_id' : tf.feature_column.categorical_column_with_hash_bucket(
- 'end_station_id', hash_bucket_size=800),
- 'start_station_id' : tf.feature_column.categorical_column_with_hash_bucket(
- 'start_station_id', hash_bucket_size=800),
- 'loc_cross' : tf.feature_column.categorical_column_with_hash_bucket(
- 'loc_cross', hash_bucket_size=21000),
- # 'bike_id' : tf.feature_column.categorical_column_with_hash_bucket('bike_id', hash_bucket_size=14000)
- }
- inputs = {
- colname : tf.keras.layers.Input(name=colname, shape=(), dtype='float32')
- for colname in real.keys()
- }
- inputs.update({'ts': tf.keras.layers.Input(name='ts', shape=(), dtype='float64')})
- inputs.update({
- colname : tf.keras.layers.Input(name=colname, shape=(), dtype='string')
- for colname in sparse.keys()
- })
- # embed all the sparse columns
- embed = {
- 'embed_{}'.format(colname) : tf.feature_column.embedding_column(col, 10)
- for colname, col in sparse.items()
- }
- real.update(embed)
- # one-hot encode the sparse columns
- sparse = {
- colname : tf.feature_column.indicator_column(col)
- for colname, col in sparse.items()
- }
- return inputs, sparse, real
-
-# Build a wide-and-deep model.
-def wide_and_deep_classifier(inputs, linear_feature_columns, dnn_feature_columns,
- num_hidden_layers, dnn_hidden_units1, learning_rate):
- deep = tf.keras.layers.DenseFeatures(dnn_feature_columns, name='deep_inputs')(inputs)
- layers = [dnn_hidden_units1]
- if num_hidden_layers > 1:
- layers += [int(dnn_hidden_units1/(x*2)) for x in range(1, num_hidden_layers)]
- for layerno, numnodes in enumerate(layers):
- deep = tf.keras.layers.Dense(numnodes, activation='relu', name='dnn_{}'.format(layerno+1))(deep)
- wide = tf.keras.layers.DenseFeatures(linear_feature_columns, name='wide_inputs')(inputs)
- both = tf.keras.layers.concatenate([deep, wide], name='both')
- output = tf.keras.layers.Dense(1, name='dur')(both)
- model = tf.keras.Model(inputs, output)
- optimizer = tf.keras.optimizers.RMSprop(learning_rate)
- model.compile(loss='mse', optimizer=optimizer,
- metrics=['mse', 'mae', tf.keras.metrics.RootMeanSquaredError()])
- return model
diff --git a/tensorboard/training_component/ml/deploy_tuner.py b/tensorboard/training_component/ml/deploy_tuner.py
deleted file mode 100644
index a265a5f7c231a7274fc6ef873ecbc0bea4e1705d..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/ml/deploy_tuner.py
+++ /dev/null
@@ -1,156 +0,0 @@
-# Copyright 2020 Google Inc. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# https://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-import argparse
-import logging
-import os
-import subprocess
-import time
-
-from google.cloud import storage
-
-
-OUTPUT_PATH = '/tmp/hps.json'
-
-def main():
- parser = argparse.ArgumentParser(description='Keras distributed tuner')
- parser.add_argument(
- '--epochs', type=int, required=True)
- parser.add_argument(
- '--num-tuners', type=int, required=True)
- parser.add_argument(
- '--bucket-name', required=True)
- parser.add_argument(
- '--tuner-dir', required=True)
- parser.add_argument(
- '--tuner-proj', required=True)
- parser.add_argument(
- '--max-trials', type=int, required=True)
- parser.add_argument(
- '--namespace', default='default')
- parser.add_argument(
- '--executions-per-trial', type=int,
- default=2)
- parser.add_argument(
- '--num-best-hps', type=int,
- default=2)
- parser.add_argument('--deploy', default=False, action='store_true')
- parser.add_argument('--no-deploy', dest='deploy', action='store_false')
-
- args = parser.parse_args()
- logging.getLogger().setLevel(logging.INFO)
- tuner_path = 'gs://{}/{}'.format(args.bucket_name, args.tuner_dir)
- res_path = '{}/{}/{}'.format(args.bucket_name, args.tuner_dir, 'bp.txt')
- logging.info('tuner path: %s, res path %s', tuner_path, res_path)
-
- logging.info('Generating tuner deployment templates.')
- ts = int(time.time())
- KTUNER_CHIEF = 'ktuner{}-chief'.format(ts)
- logging.info('KTUNER_CHIEF: %s', KTUNER_CHIEF)
- KTUNER_DEP_PREFIX = 'ktuner{}-dep'.format(ts)
- logging.info('KTUNER_DEP_PREFIX: %s', KTUNER_DEP_PREFIX)
-
- template_file = os.path.join(
- os.path.dirname(os.path.realpath(__file__)), 'kchief_deployment_templ.yaml')
- chief_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'kchief_dep.yaml')
-
- with open(template_file, 'r') as f:
- with open(chief_file_path, "w") as target:
- data = f.read()
- changed = data.replace('EPOCHS', str(args.epochs)).replace(
- 'TUNER_DIR', tuner_path).replace('NAMESPACE', args.namespace).replace(
- 'TUNER_PROJ', args.tuner_proj).replace('TUNER_NUM', 'chief').replace(
- 'MAX_TRIALS', str(args.max_trials)).replace(
- 'KTUNER_CHIEF', KTUNER_CHIEF).replace('RES_PATH', res_path).replace(
- 'BUCKET_NAME', args.bucket_name).replace('NUM_BEST_HPS', str(args.num_best_hps)).replace(
- 'EXECS_PER_TRIAL', str(args.executions_per_trial))
- target.write(changed)
-
- tuner_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'ktuners_dep.yaml')
- logging.info("tuner file path: %s", tuner_file_path)
- if os.path.exists(tuner_file_path):
- os.remove(tuner_file_path)
- template_file = os.path.join(
- os.path.dirname(os.path.realpath(__file__)), 'ktuners_deployment_templ.yaml')
- logging.info("num tuners: %s", args.num_tuners)
- with open(template_file, 'r') as f:
- with open(tuner_file_path, "a") as target:
- data = f.read()
- for i in range(args.num_tuners):
- changed = data.replace('EPOCHS', str(args.epochs)).replace(
- 'TUNER_DIR', tuner_path).replace('NAMESPACE', args.namespace).replace(
- 'TUNER_PROJ', args.tuner_proj).replace('TUNER_NUM', str(i)).replace(
- 'KTUNER_CHIEF', KTUNER_CHIEF).replace(
- 'MAX_TRIALS', str(args.max_trials)).replace('RES_PATH', res_path).replace(
- 'BUCKET_NAME', args.bucket_name).replace(
- 'NUM_BEST_HPS', str(args.num_best_hps)).replace(
- 'EXECS_PER_TRIAL', str(args.executions_per_trial))
- changed = changed.replace(
- 'KTUNER_DEP_NAME', KTUNER_DEP_PREFIX +'{}'.format(i)).replace(
- 'KTUNER_ID', 'tuner{}'.format(i))
- target.write(changed)
-
- if args.deploy:
- logging.info('deploying chief...')
- subprocess.run(['kubectl', 'apply', '-f', chief_file_path])
- logging.info('pausing before tuner worker deployment...')
- time.sleep(120)
- logging.info('deploying tuners...')
- subprocess.run(['kubectl', 'apply', '-f', tuner_file_path])
- logging.info('finished deployments.')
-
- # wait for the tuner pods to be ready... if we're autoscaling the GPU pool,
- # this might take a while.
- for i in range(args.num_tuners):
- logging.info('waiting for tuner %s pod to be ready...', i)
- subprocess.run(['kubectl', '-n={}'.format(args.namespace), 'wait', 'pod',
- '--for=condition=ready', '--timeout=15m',
- '-l=job-name={}{}'.format(KTUNER_DEP_PREFIX, i)])
-
- # wait for all the tuner workers to complete
- for i in range(args.num_tuners):
- logging.info('waiting for completion of tuner %s...', i)
- # negative timeout value --> one week
- sp_res = subprocess.run(['kubectl', '-n={}'.format(args.namespace), 'wait',
- '--for=condition=complete', '--timeout=-1m', 'job/{}{}'.format(KTUNER_DEP_PREFIX, i)],
- capture_output=True)
- # In some cases the k8s api seems to be temporarily unavailable, which causes the
- # 'wait' to terminate prematurely. TODO: What's the best way to address this?
- if 'The connection to the server' in str(sp_res.stdout):
- # then try again after a pause
- logging.info('got connection error; sleeping for 20s')
- time.sleep(20)
- logging.info('waiting again for completion of tuner %s...', i)
- sp_res = subprocess.run(['kubectl', '-n={}'.format(args.namespace), 'wait',
- '--for=condition=complete', '--timeout=-1m', 'job/{}{}'.format(KTUNER_DEP_PREFIX, i)],
- capture_output=True)
-
-
- # get info on the best params once search has completed
- client = storage.Client()
- bucket = client.get_bucket(args.bucket_name)
- logging.info('using bucket %s: %s, path %s', args.bucket_name, bucket, res_path)
- blob = bucket.get_blob(res_path)
-
- results_string = blob.download_as_string()
- logging.info('got results info: %s', results_string)
- rs = results_string.decode("utf-8")
- logging.info('rs: %s', rs)
-
- with open(OUTPUT_PATH, 'w') as f:
- f.write(rs)
-
-if __name__ == "__main__":
- main()
diff --git a/tensorboard/training_component/ml/eval_metrics.py b/tensorboard/training_component/ml/eval_metrics.py
deleted file mode 100644
index 3f74e27df647cd9c4cca6f6ddd0c2e945cec066c..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/ml/eval_metrics.py
+++ /dev/null
@@ -1,55 +0,0 @@
-# Copyright 2020 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from typing import NamedTuple
-# from kfp.components import InputPath, OutputPath
-
-
-# An example of how the model eval info could be used to make decisions about whether or not
-# to deploy the model.
-def eval_metrics(
- metrics: str,
- thresholds: str
-) -> NamedTuple('Outputs', [('deploy', str)]):
-
- import json
- import logging
-
- def regression_threshold_check(metrics_info):
- # ...
- for k, v in thresholds_dict.items():
- logging.info('k {}, v {}'.format(k, v))
- if k in ['root_mean_squared_error', 'mae']:
- if metrics_info[k][-1] > v:
- logging.info('{} > {}; returning False'.format(metrics_info[k][0], v))
- return ('False', )
- return ('deploy', )
-
- logging.getLogger().setLevel(logging.INFO) # TODO: make level configurable
-
- thresholds_dict = json.loads(thresholds)
- logging.info('thresholds dict: {}'.format(thresholds_dict))
- logging.info('metrics: %s', metrics)
- metrics_dict = json.loads(metrics)
-
- logging.info("got metrics info: %s", metrics_dict)
- res = regression_threshold_check(metrics_dict)
- logging.info('deploy decision: %s', res)
- return res
-
-
-if __name__ == '__main__':
- import kfp
- kfp.components.func_to_container_op(eval_metrics,
- output_component_file='../../eval_metrics_component.yaml', base_image='gcr.io/deeplearning-platform-release/tf2-cpu.2-3:latest')
diff --git a/tensorboard/training_component/ml/kchief_deployment_templ.yaml b/tensorboard/training_component/ml/kchief_deployment_templ.yaml
deleted file mode 100644
index 8505d7a0a2cdd8dba69c5636243f42415051bae8..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/ml/kchief_deployment_templ.yaml
+++ /dev/null
@@ -1,66 +0,0 @@
----
- apiVersion: v1
- kind: Service
- metadata:
- labels:
- app: KTUNER_CHIEF
- apptype: ktuner-chief
- name: KTUNER_CHIEF
- namespace: NAMESPACE
- spec:
- ports:
- - name: grpc
- port: 9000
- targetPort: 9000
- selector:
- app: KTUNER_CHIEF
- type: ClusterIP
----
-apiVersion: batch/v1
-kind: Job
-metadata:
- labels:
- app: KTUNER_CHIEF
- apptype: ktuner-chief
- name: KTUNER_CHIEF-dep
- namespace: NAMESPACE
-spec:
- # replicas: 1
- template:
- metadata:
- labels:
- app: KTUNER_CHIEF
- apptype: ktuner-chief
- version: v1
- spec:
- containers:
- - args:
- - --epochs=EPOCHS
- - --tuner-dir=TUNER_DIR
- - --tuner-proj=TUNER_PROJ
- - --tuner-num=TUNER_NUM
- - --max-trials=MAX_TRIALS
- - --executions-per-trial=EXECS_PER_TRIAL
- - --num-best-hps=NUM_BEST_HPS
- - --respath=RES_PATH
- - --bucket-name=BUCKET_NAME
- image: gcr.io/google-samples/ml-pipeline-bikes-tuner
- env:
- - name: KERASTUNER_TUNER_ID
- value: chief
- - name: KERASTUNER_ORACLE_IP
- valueFrom:
- fieldRef:
- fieldPath: status.podIP
- - name: KERASTUNER_ORACLE_PORT
- value: "9000"
- imagePullPolicy: Always
- name: ktuner-chief
- ports:
- - name: tuner-port
- containerPort: 9000
- resources:
- limits:
- cpu: 1
- memory: 2Gi
- restartPolicy: Never
diff --git a/tensorboard/training_component/ml/ktuners_deployment_templ.yaml b/tensorboard/training_component/ml/ktuners_deployment_templ.yaml
deleted file mode 100644
index 4c56f9728726527715c0828209642f22395d8650..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/ml/ktuners_deployment_templ.yaml
+++ /dev/null
@@ -1,43 +0,0 @@
----
-apiVersion: batch/v1
-kind: Job
-metadata:
- labels:
- app: ktuner-tuner
- name: KTUNER_DEP_NAME
- namespace: NAMESPACE
-spec:
- # replicas: 1
- template:
- metadata:
- labels:
- app: ktuner-tuner
- version: v1
- spec:
- containers:
- - args:
- - --epochs=EPOCHS
- - --tuner-dir=TUNER_DIR
- - --tuner-proj=TUNER_PROJ
- - --tuner-num=TUNER_NUM
- - --max-trials=MAX_TRIALS
- - --executions-per-trial=EXECS_PER_TRIAL
- - --num-best-hps=NUM_BEST_HPS
- - --respath=RES_PATH
- - --bucket-name=BUCKET_NAME
- image: gcr.io/google-samples/ml-pipeline-bikes-tuner
- env:
- - name: KERASTUNER_TUNER_ID
- value: KTUNER_ID
- - name: KERASTUNER_ORACLE_IP
- value: KTUNER_CHIEF
- - name: KERASTUNER_ORACLE_PORT
- value: "9000"
- imagePullPolicy: Always
- name: kktuner-tuner
- ports:
- - name: tuner-port
- containerPort: 9000
- resources:
- limits: {nvidia.com/gpu: 1}
- restartPolicy: Never
diff --git a/tensorboard/training_component/training_comp.yaml b/tensorboard/training_component/training_comp.yaml
deleted file mode 100644
index 45a3d3c2e2f57eeca5753c6364a452afc5d2f0c5..0000000000000000000000000000000000000000
--- a/tensorboard/training_component/training_comp.yaml
+++ /dev/null
@@ -1,62 +0,0 @@
-# Copyright 2019 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Train bikes_weather model
-description: |
- A Kubeflow Pipeline component to train a Tensor2Tensor model
-metadata:
- labels:
- add-pod-env: 'true'
-inputs:
- - name: epochs
- type: Integer
- default: 1
- - name: steps_per_epoch
- type: Integer
- default: -1
- - name: data_dir
- type: String
- - name: workdir
- type: String
- - name: tb_dir
- type: String
- - name: hp_idx
- type: Integer
- - name: hptune_results
- type: String
-outputs:
- - name: train_output_path
- type: GCSPath
- - name: metrics_output_path
- type: String
- - name: MLPipeline UI metadata
- type: UI metadata
-implementation:
- container:
- image: gitlab-registry.cern.ch/ai-ml/kubeflow_images/training:7
- args: [
- --data-dir, {inputValue: data_dir},
- --epochs, {inputValue: epochs},
- --steps-per-epoch, {inputValue: steps_per_epoch},
- --workdir, {inputValue: workdir},
- --tb-dir, {inputValue: tb_dir},
- --train-output-path, {outputPath: train_output_path},
- --metrics-output-path, {outputPath: metrics_output_path},
- --hp-idx, {inputValue: hp_idx},
- --hptune-results, {inputValue: hptune_results}
- ]
- env:
- KFP_POD_NAME: "{{pod.name}}"
- fileOutputs:
- MLPipeline UI metadata: /mlpipeline-ui-metadata.json
diff --git a/tfjob-pipeline/.gitkeep b/tfjob-pipeline/.gitkeep
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/tfjob-pipeline/README.md b/tfjob-pipeline/README.md
deleted file mode 100644
index 9cbb71850f9bf5c7239faa084f7756a6ff0dc0fc..0000000000000000000000000000000000000000
--- a/tfjob-pipeline/README.md
+++ /dev/null
@@ -1,9 +0,0 @@
-## Example - TFJob from a Pipeline
-
-### What is it about?
-Run a distributed training job using TFJob, from a pipeline.
-
-### How to run?
-- Run notebook cells
-- Download created yaml file
-- Upload pipeline to https://ml.cern.ch/_/pipeline/#/pipelines
diff --git a/tfjob-pipeline/docker/.gitkeep b/tfjob-pipeline/docker/.gitkeep
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/tfjob-pipeline/docker/Dockerfile b/tfjob-pipeline/docker/Dockerfile
deleted file mode 100644
index 9609e9da29171dad85a71f174b5e67beb03eedca..0000000000000000000000000000000000000000
--- a/tfjob-pipeline/docker/Dockerfile
+++ /dev/null
@@ -1,29 +0,0 @@
-ARG IMAGE_TYPE="gpu"
-FROM gcr.io/kubeflow-images-public/tensorflow-2.1.0-notebook-gpu:1.0.0
-USER root
-
-ENV NB_PREFIX /
-
-RUN apt-get -qq update
-RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends apt-utils
-
-# KRB & EOS
-RUN apt-get -yqq install krb5-user libpam-krb5 && \
- apt-get -yqq clean && \
- mv /etc/krb5.conf /etc/krb5-backup.conf
-
-ENV SHELL /bin/bash
-COPY krb5.conf /etc/krb5.conf
-
-COPY requirements/requirements.txt /tmp/requirements.txt
-RUN pip3 install -r /tmp/requirements.txt
-
-COPY custom-code.py /
-
-ENV NB_USER jovyan
-ENV HOME /home/$NB_USER
-RUN chown -R ${NB_USER}:users $HOME
-
-RUN echo "jovyan ALL=(ALL:ALL) NOPASSWD:ALL" > /etc/sudoers.d/jovyan
-WORKDIR /home/jovyan
-USER jovyan
diff --git a/tfjob-pipeline/docker/README.md b/tfjob-pipeline/docker/README.md
deleted file mode 100644
index ae4b2572faaf8737f093aefd82ed620f66cf8336..0000000000000000000000000000000000000000
--- a/tfjob-pipeline/docker/README.md
+++ /dev/null
@@ -1,9 +0,0 @@
-## Example - TFJob
-
-### What is it about?
-Run a distributed training job using TFJob CRD and Katib UI.
-Using this example, training job is split across multiple GPUs.
-
-### How to run?
-- Build docker image with custom code: `docker build -f Dockerfile -t @username/custom-tfjob . --network=host`
-- Push docker image: `docker push @username/custom-tfjob`
diff --git a/tfjob-pipeline/docker/custom-code.py b/tfjob-pipeline/docker/custom-code.py
deleted file mode 100644
index 8e6294f451224bc6d4684976b7bf0fea523868e8..0000000000000000000000000000000000000000
--- a/tfjob-pipeline/docker/custom-code.py
+++ /dev/null
@@ -1,53 +0,0 @@
-import tensorflow as tf
-import random
-import numpy as np
-import tempfile
-import os, json
-import tensorflow_datasets as tfds
-import argparse
-import sys
-import time
-
-def main():
- strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy(tf.distribute.experimental.CollectiveCommunication.NCCL)
-
- with strategy.scope():
- config = tf.estimator.RunConfig(train_distribute=strategy, eval_distribute=strategy)
-
- parser = argparse.ArgumentParser()
- parser.add_argument('--learning_rate', type=float, default=0.001, help='Initial learning rate')
- FLAGS, unparsed = parser.parse_known_args()
- print('FLAGS:')
- print(FLAGS)
-
- model = tf.keras.models.Sequential([
- tf.keras.layers.Dense(2048, activation='relu', input_shape=(4,)),
- tf.keras.layers.Dense(1024),
- tf.keras.layers.Dense(512),
- tf.keras.layers.Dense(256),
- tf.keras.layers.Dropout(0.2),
- tf.keras.layers.Dense(3)
- ])
- model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), optimizer=tf.keras.optimizers.Adam(FLAGS.learning_rate))
- #model.summary()
-
- def input_fn():
- split = tfds.Split.TRAIN
- dataset = tfds.load('iris', split=split, as_supervised=True)
- dataset = dataset.map(lambda features, labels: ({'dense_input':features}, labels))
- dataset = dataset.batch(32).repeat()
- return dataset
-
- os.system('mkdir /tmp/model_outputs')
- model_dir = '/tmp/model_outputs'
- keras_estimator = tf.keras.estimator.model_to_estimator(keras_model=model, model_dir=model_dir)
-
- print('Training...')
- train_result = keras_estimator.train(input_fn=input_fn, steps=100)
- print('Train result: {}'.format(train_result))
- print('Evaluating...')
- eval_result = keras_estimator.evaluate(input_fn=input_fn, steps=100)
- print('Eval result: {}'.format(eval_result))
-
-if __name__ == '__main__':
- main()
diff --git a/tfjob-pipeline/docker/krb5.conf b/tfjob-pipeline/docker/krb5.conf
deleted file mode 100644
index 8598c71cf930a66ee712094f53f864011a518228..0000000000000000000000000000000000000000
--- a/tfjob-pipeline/docker/krb5.conf
+++ /dev/null
@@ -1,91 +0,0 @@
-; AD : This Kerberos configuration is for CERN's Active Directory realm.
-;
-; /etc/krb5.conf
-
-[libdefaults]
- default_realm = CERN.CH
- ticket_lifetime = 25h
- renew_lifetime = 120h
- forwardable = true
- proxiable = true
- default_tkt_enctypes = arcfour-hmac-md5 aes256-cts aes128-cts des3-cbc-sha1 des-cbc-md5 des-cbc-crc
- chpw_prompt = true
- rdns = false
-
-[appdefaults]
-pam = {
- external = true
- krb4_convert = false
- krb4_convert_524 = false
- krb4_use_as_req = false
-}
-
-[domain_realm]
-.cern.ch = CERN.CH
-.fnal.gov = FNAL.GOV
-.hep.man.ac.uk = HEP.MAN.AC.UK
-.in2p3.fr = IN2P3.FR
-# No default domain for KFKI.HU specified.
-
-[realms]
-# Start of puppet output for CERN.CH
- CERN.CH = {
- default_domain = cern.ch
- kpasswd_server = cerndc.cern.ch
- admin_server = cerndc.cern.ch
- kdc = cerndc.cern.ch
- v4_name_convert = {
- host = {
- rcmd = host
- }
- }
- }
-
-
-# Start of puppet output for FNAL.GOV
- FNAL.GOV = {
- default_domain = fnal.gov
- admin_server = krb-fnal-admin.fnal.gov
- kdc = krb-fnal-fcc3.fnal.gov:88
- kdc = krb-fnal-2.fnal.gov:88
- kdc = krb-fnal-3.fnal.gov:88
- kdc = krb-fnal-1.fnal.gov:88
- kdc = krb-fnal-4.fnal.gov:88
- kdc = krb-fnal-enstore.fnal.gov:88
- kdc = krb-fnal-fg2.fnal.gov:88
- kdc = krb-fnal-cms188.fnal.gov:88
- kdc = krb-fnal-cms204.fnal.gov:88
- kdc = krb-fnal-d0online.fnal.gov:88
- }
-
-
-# Start of puppet output for HEP.MAN.AC.UK
- HEP.MAN.AC.UK = {
- default_domain = hep.man.ac.uk
- kpasswd_server = afs4.hep.man.ac.uk
- admin_server = afs4.hep.man.ac.uk
- kdc = afs1.hep.man.ac.uk
- kdc = afs2.hep.man.ac.uk
- kdc = afs3.hep.man.ac.uk
- kdc = afs4.hep.man.ac.uk
- }
-
-
-# Start of puppet output for IN2P3.FR
- IN2P3.FR = {
- default_domain = in2p3.fr
- kpasswd_server = kerberos-admin.in2p3.fr
- admin_server = kerberos-admin.in2p3.fr
- kdc = kerberos-1.in2p3.fr
- kdc = kerberos-2.in2p3.fr
- kdc = kerberos-3.in2p3.fr
- }
-
-
-# Start of puppet output for KFKI.HU
- KFKI.HU = {
- admin_server = kerberos.kfki.hu
- kdc = kerberos.kfki.hu
- }
-
-
diff --git a/tfjob-pipeline/docker/requirements/.gitkeep b/tfjob-pipeline/docker/requirements/.gitkeep
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/tfjob-pipeline/docker/requirements/requirements.txt b/tfjob-pipeline/docker/requirements/requirements.txt
deleted file mode 100644
index 16a9ac460e439602675aaf22fec77fc8a0d00e89..0000000000000000000000000000000000000000
--- a/tfjob-pipeline/docker/requirements/requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-tensorflow.io
-tensorflow_datasets
\ No newline at end of file
diff --git a/tfjob-pipeline/tf_job_pipeline.ipynb b/tfjob-pipeline/tf_job_pipeline.ipynb
deleted file mode 100644
index c9bb6dc0e654b8ca5804a26fd95028ef60d2d50a..0000000000000000000000000000000000000000
--- a/tfjob-pipeline/tf_job_pipeline.ipynb
+++ /dev/null
@@ -1,134 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import kfp.dsl as dsl\n",
- "from string import Template\n",
- "import json"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "tfjobjson_template = Template(\"\"\"\n",
- "{\n",
- " \"apiVersion\": \"kubeflow.org/v1\",\n",
- " \"kind\": \"TFJob\",\n",
- " \"metadata\": {\n",
- " \"name\": \"tfjob-custom-code-{{workflow.uid}}\",\n",
- " \"namespace\": \"dejan-golubovic\"\n",
- " },\n",
- " \"spec\": {\n",
- " \"tfReplicaSpecs\": {\n",
- " \"Worker\": {\n",
- " \"replicas\": 1,\n",
- " \"template\": {\n",
- " \"spec\": {\n",
- " \"containers\": [\n",
- " {\n",
- " \"image\": \"registry.hub.docker.com/dejangolubovic/tfjob-test:customcode4\",\n",
- " \"name\": \"tensorflow\",\n",
- " \"command\": [\"python\", \"/custom-code.py\"]\n",
- " }\n",
- " ],\n",
- " \"restartPolicy\": \"Never\"\n",
- " },\n",
- " \"metadata\": {\n",
- " \"annotations\": {\n",
- " \"sidecar.istio.io/inject\": \"false\"\n",
- " }\n",
- " }\n",
- " }\n",
- " }\n",
- " }\n",
- " }\n",
- "}\n",
- "\"\"\")\n",
- "\n",
- "tfjobjson = tfjobjson_template.substitute()\n",
- "tfjob = json.loads(tfjobjson)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def tfjob_pipeline():\n",
- " tfjob_op = dsl.ResourceOp(\n",
- " name='test_tf_job',\n",
- " k8s_resource=tfjob,\n",
- " success_condition='status.replicaStatuses.Worker.succeeded == 1'\n",
- " )"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import kfp.compiler as compiler\n",
- "compiler.Compiler().compile(tfjob_pipeline, \"tfjob_pip.yaml\")"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "kubeflow_notebook": {
- "autosnapshot": false,
- "docker_image": "gitlab-registry.cern.ch/ai-ml/kubeflow_images/tensorflow-notebook-gpu-2.1.0:v0.6.1-30",
- "experiment": {
- "id": "",
- "name": ""
- },
- "experiment_name": "",
- "katib_metadata": {
- "algorithm": {
- "algorithmName": "grid"
- },
- "maxFailedTrialCount": 3,
- "maxTrialCount": 12,
- "objective": {
- "objectiveMetricName": "",
- "type": "minimize"
- },
- "parallelTrialCount": 3,
- "parameters": []
- },
- "katib_run": false,
- "pipeline_description": "",
- "pipeline_name": "",
- "snapshot_volumes": false,
- "steps_defaults": [],
- "volume_access_mode": "rwm",
- "volumes": []
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}