initial commit

README.md

@@ -0,0 +1 @@
+FLAVA fine-tuning

flava_finetuning_tutorial.ipynb

@@ -0,0 +1,325 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "oUL6DV1zCIlB"
+      },
+      "outputs": [],
+      "source": [
+        "%matplotlib inline\n",
+        "!nvidia-smi"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WmJySTGXCIlD"
+      },
+      "source": [
+        "\n",
+        "# TorchMultimodal Tutorial: Finetuning FLAVA\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ZJCb2uRyCIlE"
+      },
+      "source": [
+        "Multimodal AI has recently become very popular owing to its ubiquitous\n",
+        "nature, from use cases like image captioning and visual search to more\n",
+        "recent applications like image generation from text. **TorchMultimodal\n",
+        "is a library powered by Pytorch consisting of building blocks and end to\n",
+        "end examples, aiming to enable and accelerate research in\n",
+        "multimodality**.\n",
+        "\n",
+        "In this tutorial, we will demonstrate how to use a **pretrained SoTA\n",
+        "model called** [FLAVA](https://arxiv.org/pdf/2112.04482.pdf)_ **from\n",
+        "TorchMultimodal library to finetune on a multimodal task i.e. visual\n",
+        "question answering** (VQA). The model consists of two unimodal transformer\n",
+        "based encoders for text and image and a multimodal encoder to combine\n",
+        "the two embeddings. It is pretrained using contrastive, image text matching and \n",
+        "text, image and multimodal masking losses.\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0TjU3iQgCIlE"
+      },
+      "source": [
+        "## Installation\n",
+        "We will use TextVQA dataset and bert tokenizer from HuggingFace for this\n",
+        "tutorial. So you need to install datasets and transformers in addition to TorchMultimodal.\n",
+        "\n",
+        "<div class=\"alert alert-info\"><h4>Note</h4><p>When running this tutorial in Google Colab, install the required packages by\n",
+        "   creating a new cell and running the following commands:\n",
+        "\n",
+        "```\n",
+        "!pip install torchmultimodal-nightly\n",
+        "!pip install datasets\n",
+        "!pip install transformers</p></div>\n",
+        "```\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "LGuYfyaJCIlE"
+      },
+      "source": [
+        "## Steps \n",
+        "\n",
+        "1. Download the HuggingFace dataset to a directory on your computer by running the following command:\n",
+        "\n",
+        "```\n",
+        "wget http://dl.fbaipublicfiles.com/pythia/data/vocab.tar.gz \n",
+        "tar xf vocab.tar.gz\n",
+        "```\n",
+        "   .. note:: \n",
+        "      If you are running this tutorial in Google Colab, run these commands\n",
+        "      in a new cell and prepend these commands with an exclamation mark (!)\n",
+        "\n",
+        "\n",
+        "2. For this tutorial, we treat VQA as a classification task where\n",
+        "   the inputs are images and question (text) and the output is an answer class. \n",
+        "   So we need to download the vocab file with answer classes and create the answer to\n",
+        "   label mapping.\n",
+        "\n",
+        "   We also load the [textvqa\n",
+        "   dataset](https://arxiv.org/pdf/1904.08920.pdf)_ containing 34602 training samples\n",
+        "   (images,questions and answers) from HuggingFace\n",
+        "\n",
+        "We see there are 3997 answer classes including a class representing\n",
+        "unknown answers.\n",
+        "\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "b6c1oq0lCIlF"
+      },
+      "outputs": [],
+      "source": [
+        "with open(\"data/vocabs/answers_textvqa_more_than_1.txt\") as f:\n",
+        "  vocab = f.readlines()\n",
+        "\n",
+        "answer_to_idx = {}\n",
+        "for idx, entry in enumerate(vocab):\n",
+        "  answer_to_idx[entry.strip(\"\\n\")] = idx\n",
+        "print(len(vocab))\n",
+        "print(vocab[:5])\n",
+        "\n",
+        "from datasets import load_dataset\n",
+        "dataset = load_dataset(\"textvqa\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "kGCla9GgCIlF"
+      },
+      "source": [
+        "Lets display a sample entry from the dataset:\n",
+        "\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "GLS8HGYtCIlF"
+      },
+      "outputs": [],
+      "source": [
+        "import matplotlib.pyplot as plt\n",
+        "import numpy as np \n",
+        "idx = 5 \n",
+        "print(\"Question: \", dataset[\"train\"][idx][\"question\"]) \n",
+        "print(\"Answers: \" ,dataset[\"train\"][idx][\"answers\"])\n",
+        "im = np.asarray(dataset[\"train\"][idx][\"image\"].resize((500,500)))\n",
+        "plt.imshow(im)\n",
+        "plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "J1UO_daoCIlG"
+      },
+      "source": [
+        "3. Next, we write the transform function to convert the image and text into\n",
+        "Tensors consumable by our model - For images, we use the transforms from\n",
+        "torchvision to convert to Tensor and resize to uniform sizes - For text,\n",
+        "we tokenize (and pad) them using the BertTokenizer from HuggingFace -\n",
+        "For answers (i.e. labels), we take the most frequently occuring answer\n",
+        "as the label to train with:\n",
+        "\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "rO7lCn4DCIlG"
+      },
+      "outputs": [],
+      "source": [
+        "import torch\n",
+        "from torchvision import transforms\n",
+        "from collections import defaultdict\n",
+        "from transformers import BertTokenizer\n",
+        "from functools import partial\n",
+        "\n",
+        "def transform(tokenizer, input):\n",
+        "  batch = {}\n",
+        "  image_transform = transforms.Compose([transforms.ToTensor(), transforms.Resize([224,224])])\n",
+        "  image = image_transform(input[\"image\"][0].convert(\"RGB\"))\n",
+        "  batch[\"image\"] = [image]\n",
+        "\n",
+        "  tokenized=tokenizer(input[\"question\"],return_tensors='pt',padding=\"max_length\",max_length=512)\n",
+        "  batch.update(tokenized)\n",
+        "\n",
+        "\n",
+        "  ans_to_count = defaultdict(int)\n",
+        "  for ans in input[\"answers\"][0]:\n",
+        "    ans_to_count[ans] += 1\n",
+        "  max_value = max(ans_to_count, key=ans_to_count.get)\n",
+        "  ans_idx = answer_to_idx.get(max_value,0)\n",
+        "  batch[\"answers\"] = torch.as_tensor([ans_idx])\n",
+        "  return batch\n",
+        "\n",
+        "tokenizer=BertTokenizer.from_pretrained(\"bert-base-uncased\",padding=\"max_length\",max_length=512)\n",
+        "transform=partial(transform,tokenizer)\n",
+        "dataset.set_transform(transform)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "LOMy3UbpCIlG"
+      },
+      "source": [
+        "4. Finally, we import the flava_model_for_classification from\n",
+        "torchmultimodal. It loads the pretrained flava checkpoint by default and\n",
+        "includes a classification head.\n",
+        "\n",
+        "The model forward function passes the image through the visual encoder\n",
+        "and the question through the text encoder. The image and question\n",
+        "embeddings are then passed through the multimodal encoder. The final\n",
+        "embedding corresponding to the CLS token is passed through a MLP head\n",
+        "which finally gives the probability distribution over each possible\n",
+        "answers.\n",
+        "\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "drSfcYNCCIlG"
+      },
+      "outputs": [],
+      "source": [
+        "from torchmultimodal.models.flava.model import flava_model_for_classification\n",
+        "model = flava_model_for_classification(num_classes=len(vocab))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "976mlWvaCIlG"
+      },
+      "source": [
+        "5. We put together the dataset and model in a toy training loop to\n",
+        "demonstrate how to train the model for 3 iterations:\n",
+        "\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "0KvxQ4xaCIlH"
+      },
+      "outputs": [],
+      "source": [
+        "from torch import nn\n",
+        "BATCH_SIZE = 2\n",
+        "MAX_STEPS = 3\n",
+        "from torch.utils.data import DataLoader\n",
+        "\n",
+        "train_dataloader = DataLoader(dataset[\"train\"], batch_size= BATCH_SIZE)\n",
+        "optimizer = torch.optim.AdamW(model.parameters())\n",
+        "\n",
+        "\n",
+        "epochs = 1\n",
+        "for _ in range(epochs):\n",
+        "  for idx, batch in enumerate(train_dataloader):\n",
+        "    optimizer.zero_grad()\n",
+        "    out = model(text = batch[\"input_ids\"], image = batch[\"image\"], labels = batch[\"answers\"])\n",
+        "    loss = out.loss\n",
+        "    loss.backward()\n",
+        "    optimizer.step()\n",
+        "    print(f\"Loss at step {idx} = {loss}\")\n",
+        "    if idx > MAX_STEPS-1:\n",
+        "      break"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "A7An1sjZCIlH"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "This tutorial introduced the basics around how to finetune on a\n",
+        "multimodal task using FLAVA from TorchMultimodal. Please also check out\n",
+        "other examples from the library like\n",
+        "[MDETR](https://github.com/facebookresearch/multimodal/tree/main/torchmultimodal/models/mdetr)_\n",
+        "which is a multimodal model for object detection and\n",
+        "[Omnivore](https://github.com/facebookresearch/multimodal/blob/main/torchmultimodal/models/omnivore.py)_\n",
+        "which is multitask model spanning image, video and 3d classification.\n",
+        "\n",
+        "\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.9"
+    },
+    "colab": {
+      "provenance": []
+    },
+    "accelerator": "GPU",
+    "gpuClass": "standard"
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}