data/Pytorch_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sDv-N0TTz0VE"
   },
   "source": [
    "# What is Pytorch?\n",
    "PyTorch is an open-source library used in machine learning library developed using Torch library for python program. It is developed by Facebook’s AI Research lab and released in January 2016 as a free and open-source library mainly used in computer vision, deep learning, and natural language processing applications. Programmer can build a complex neural network with ease using PyTorch as it has a core data structure, Tensor, multi-dimensional array like Numpy arrays. PyTorch use is increasing in current industries and in the research community as it is flexible, faster, easy to get the project up and running, due to which PyTorch is one of the top deep learning tools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "irVOTYdzyNJJ"
   },
   "source": [
    "# How to install Pytorch\n",
    "In order to install Pytorch, you could do it using different package managers such as `conda` and `pip`. Visit [this link](https://pytorch.org/get-started/locally/) for information about the installation on your device."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YLQylD11zw4t"
   },
   "source": [
    "# Tensors\n",
    "Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters. Tensors are just like Numpy arrays, except that they can run on GPUs and other hardware accelerators. Tensors are also optimized for automatic differentiation (which is referred to autograd from now on). Tensor API is really similar with Numpy array API, so if you are familiar with Numpy arrays, you are probably not going to face any problem with Tensors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "FnosP4Qg1BuV"
   },
   "outputs": [],
   "source": [
    "# imporing packages\n",
    "\n",
    "import torch\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wnoHqqlj1iGO"
   },
   "source": [
    "## Initializing Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lDAnwOLG1hIu",
    "outputId": "dd68a255-49d8-4b0d-af30-031e56b13611"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor created directly from data: \n",
      " tensor([[1, 2],\n",
      "        [3, 4]]) \n",
      "\n",
      "Tensor created from a numpy array: \n",
      " tensor([[1, 2],\n",
      "        [3, 4]], dtype=torch.int32) \n",
      " \n",
      "Ones Tensor: \n",
      " tensor([[1, 1],\n",
      "        [1, 1]]) \n",
      "\n",
      "Random Tensor: \n",
      " tensor([[0.7931, 0.5523],\n",
      "        [0.8787, 0.1350]]) \n",
      "\n",
      "Random Tensor: \n",
      " tensor([[0.3151, 0.8994, 0.6220],\n",
      "        [0.1558, 0.1749, 0.5404]]) \n",
      "\n",
      "Ones Tensor: \n",
      " tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]]) \n",
      "\n",
      "Zeros Tensor: \n",
      " tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "# initializing directly from data\n",
    "data = [[1, 2],[3, 4]]\n",
    "x_data = torch.tensor(data)\n",
    "print(f\"Tensor created directly from data: \\n {x_data} \\n\")\n",
    "\n",
    "# from numpy arrays\n",
    "np_array = np.array(data)\n",
    "x_np = torch.from_numpy(np_array)\n",
    "print(f\"Tensor created from a numpy array: \\n {x_np} \\n \")\n",
    "\n",
    "# from another tensor\n",
    "x_ones = torch.ones_like(x_data) # retains the properties of x_data\n",
    "print(f\"Ones Tensor: \\n {x_ones} \\n\")\n",
    "\n",
    "x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data\n",
    "print(f\"Random Tensor: \\n {x_rand} \\n\")\n",
    "\n",
    "shape = (2,3,)\n",
    "rand_tensor = torch.rand(shape)\n",
    "ones_tensor = torch.ones(shape)\n",
    "zeros_tensor = torch.zeros(shape)\n",
    "\n",
    "print(f\"Random Tensor: \\n {rand_tensor} \\n\")\n",
    "print(f\"Ones Tensor: \\n {ones_tensor} \\n\")\n",
    "print(f\"Zeros Tensor: \\n {zeros_tensor}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1pS6uApz4gie"
   },
   "source": [
    "## Attributes of a Tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "udW5cVKlyIR7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of tensor: torch.Size([3, 4])\n",
      "Datatype of tensor: torch.float32\n",
      "Device tensor is stored on: cpu\n"
     ]
    }
   ],
   "source": [
    "tensor = torch.rand(3,4)\n",
    "\n",
    "print(f\"Shape of tensor: {tensor.shape}\")\n",
    "print(f\"Datatype of tensor: {tensor.dtype}\")\n",
    "print(f\"Device tensor is stored on: {tensor.device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y39VfFN55e_4"
   },
   "source": [
    "## Operations on Tensors\n",
    "Torch Tensors contain various operations including arithmetic, linear algebra, matrix manipulation (transposing, indexing, slicing), etc.\n",
    "\n",
    "By default, tensors are created on CPU. In order to get the most out of tensors, we can move these tensors to GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "Xu0Ltb464p14"
   },
   "outputs": [],
   "source": [
    "# We move our tensor to the GPU if available\n",
    "if torch.cuda.is_available():\n",
    "    tensor = tensor.to(\"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "Fd9ZbCgP6qcs"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First row: tensor([1., 1., 1., 1.])\n",
      "First column: tensor([1., 1., 1., 1.])\n",
      "Last column: tensor([1., 1., 1., 1.])\n",
      "tensor([[1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.]])\n"
     ]
    }
   ],
   "source": [
    "# Standard numpy-like indexing and slicing\n",
    "tensor = torch.ones(4, 4)\n",
    "print(f\"First row: {tensor[0]}\")\n",
    "print(f\"First column: {tensor[:, 0]}\")\n",
    "print(f\"Last column: {tensor[..., -1]}\")\n",
    "tensor[:,1] = 0\n",
    "print(tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "ruFKbyCR6xfk"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.]])\n"
     ]
    }
   ],
   "source": [
    "# Joining tensors\n",
    "t1 = torch.cat([tensor, tensor, tensor], dim=1)\n",
    "print(t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 1., 1., 1.])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2 = torch.ones((4,4,4))\n",
    "t2[:,1,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "8MWGwqPI67RU"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrix Multiplication: \n",
      " y1: tensor([[3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.]]) \n",
      "\n",
      " y2: tensor([[3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.]]) \n",
      "\n",
      " y3: tensor([[3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.],\n",
      "        [3., 3., 3., 3.]]) \n",
      "\n",
      "Element-wise Product: \n",
      " z1: tensor([[1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.]]) \n",
      "\n",
      " z2: tensor([[1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.]]) \n",
      "\n",
      " z3: tensor([[1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.],\n",
      "        [1., 0., 1., 1.]])\n"
     ]
    }
   ],
   "source": [
    "# Arithmatic operations\n",
    "# This computes the matrix multiplication between two tensors. y1, y2, y3 will have the same value\n",
    "y1 = tensor @ tensor.T\n",
    "y2 = tensor.matmul(tensor.T)\n",
    "\n",
    "y3 = torch.rand_like(y1)\n",
    "torch.matmul(tensor, tensor.T, out=y3)\n",
    "\n",
    "\n",
    "# This computes the element-wise product. z1, z2, z3 will have the same value\n",
    "z1 = tensor * tensor\n",
    "z2 = tensor.mul(tensor)\n",
    "\n",
    "z3 = torch.rand_like(tensor)\n",
    "torch.mul(tensor, tensor, out=z3)\n",
    "\n",
    "print(f\"Matrix Multiplication: \\n y1: {y1} \\n\\n y2: {y2} \\n\\n y3: {y3} \\n\")\n",
    "print(f\"Element-wise Product: \\n z1: {z1} \\n\\n z2: {z2} \\n\\n z3: {z3}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "XZvbDzkM7ayG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(12.) <class 'torch.Tensor'>\n",
      "12.0 <class 'float'>\n"
     ]
    }
   ],
   "source": [
    "# Single-element tensors\n",
    "agg = tensor.sum()\n",
    "print(agg, type(agg))\n",
    "agg_item = agg.item()\n",
    "print(agg_item, type(agg_item))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datasets and Dataloaders\n",
    "PyTorch provides two data primitives: `torch.utils.data.DataLoader` and `torch.utils.data.Dataset` that allow you to use pre-loaded datasets as well as your own data. `Dataset` stores the samples and their corresponding labels, and `DataLoader` wraps an iterable around the Dataset to enable easy access to the samples.\n",
    "\n",
    "Pytorch contains various pre-loaded subclasses of `Dataset` which you can load such as CIFAR10, MNIST, etc. Here is an example of loading MNIST test and train datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset\n",
    "from torchvision import datasets\n",
    "from torchvision.transforms import ToTensor\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "training_data = datasets.MNIST(\n",
    "    root=\"data\", # the root directory to save the downloaded dataset\n",
    "    train=True, # choose whether it is test or train\n",
    "    download=True, # choose whether to download the dataset or not \n",
    "    transform=ToTensor() # what transforms should be applied to the dataset, in this case it just \n",
    "                         # converts the images to torch.tensor\n",
    ")\n",
    "\n",
    "test_data = datasets.MNIST(\n",
    "    root=\"data\",\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=ToTensor()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of the image of the first data: torch.Size([1, 28, 28])\n",
      "Label of the first data: 5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGbCAYAAAAr/4yjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAPGElEQVR4nO3cfazX8//H8edHucjFN2pOjJ3IKS2SSOZa2GKYsvrDMoVlQ4YN4w81DDPXxshFuZq5KmYYE2FoyeVcLMmSlelCLo6LlPX+/uHnuW+Ln8/r43ROnW63zR8+ez/O511O3Xufo1etqqoqACAiNuvoGwBgwyEKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKbLS+/PLLqNVqccMNN7TZx3z11VejVqvFq6++2mYfEzYmokC7uv/++6NWq8U777zT0beywfj666/j1FNPjT333DO222672H777WPo0KHxwAMPhFNoaG9dO/oGYFO3fPnyWLRoUYwaNSqam5tj9erV8dJLL8W4cePis88+i2uuuaajb5FNiChAB9tnn33W+XLVhAkT4sQTT4zbbrstrrrqqujSpUvH3BybHF8+YoOzatWqmDhxYuy///7RvXv32GabbeKwww6LmTNn/u3m5ptvjt69e0e3bt3iiCOOiI8//nida+bOnRujRo2KHj16xFZbbRVDhgyJZ555Zn3+UP6V3XbbLX755ZdYtWpVR98KmxBPCmxwfvzxx7j33nvjlFNOifHjx0dra2vcd999MXz48Hj77bdj3333Xev6Bx98MFpbW+Pcc8+NlStXxq233hpHHXVUfPTRR9GrV6+IiPjkk0/ikEMOiV122SUuvfTS2GabbeLxxx+PESNGxLRp02LkyJFF97h69er44Ycf6rq2R48esdlm//znr19//TV+/vnn+Omnn+K1116LqVOnxkEHHRTdunUrujf4VypoR1OnTq0iopozZ87fXvP7779Xv/3221qvfffdd1WvXr2qM844I19bsGBBFRFVt27dqkWLFuXrs2fPriKiuvDCC/O1o48+uho4cGC1cuXKfG3NmjXVwQcfXPXt2zdfmzlzZhUR1cyZM//fH8ef19Xzz4IFC/7pp6Wqqqq69tpr19odffTR1VdffVXXFtqKJwU2OF26dMmvoa9Zsya+//77WLNmTQwZMiTee++9da4fMWJE7LLLLvnvQ4cOjQMPPDCef/75uOmmm2LFihXxyiuvxJVXXhmtra3R2tqa1w4fPjwmTZoUixcvXutj/JNBgwbFSy+9VNe1O+20U13XnXLKKTFkyJBYtmxZPPvss7FkyZL49ddf674naAuiwAbpgQceiBtvvDHmzp0bq1evztd33333da7t27fvOq/169cvHn/88YiImD9/flRVFZdffnlcfvnlf/l+S5cuLYrCDjvsEMccc0zd19ejd+/e0bt374j4IxBnnXVWHHPMMfHZZ5/5EhLtRhTY4Dz88MMxbty4GDFiRFx88cXR1NQUXbp0iWuvvTa++OKL4o+3Zs2aiIi46KKLYvjw4X95TUtLS9HHXLVqVaxYsaKua3fccceG/u+hUaNGxT333BOvv/763943tDVRYIPz5JNPRp8+fWL69OlRq9Xy9UmTJv3l9Z9//vk6r82bNy922223iIjo06dPRERsvvnmbfan+7feeiuGDRtW17ULFizIeynx55eO6v2GNrQFUWCD8+efqquqyijMnj07Zs2aFc3Nzetc//TTT6/1PYG33347Zs+eHRdccEFERDQ1NcWRRx4ZkydPjvPOOy923nnntfbLli2LHXfcsege2/J7Cn/3/vfdd1/UarXYb7/9iu4N/g1RoENMmTIlXnjhhXVeP//88+OEE06I6dOnx8iRI+P444+PBQsWxF133RUDBgyIn376aZ1NS0tLHHrooXH22WfHb7/9Frfcckv07NkzLrnkkrzmjjvuiEMPPTQGDhwY48ePjz59+sSSJUti1qxZsWjRovjwww+L7r8tv6dw9dVXx5tvvhnHHntsNDc3x4oVK2LatGkxZ86cOO+884q/tAX/hijQIe68886/fH3cuHExbty4+Oabb2Ly5Mnx4osvxoABA+Lhhx+OJ5544i8PqjvttNNis802i1tuuSWWLl0aQ4cOjdtvv32tJ4IBAwbEO++8E1dccUXcf//98e2330ZTU1MMHjw4Jk6cuL5+mHU5/vjj44svvogpU6bEsmXLYquttop99tknpk6dGmPHju3Qe2PTU6sqJ24B8AfHXACQRAGAJAoAJFEAIIkCAEkUAEh1/z2F/z1uAICNTz1/A8GTAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAUteOvgE2HXvssUdDu8suu6x4069fv+LNvHnzijdLly4t3tx9993Fm4iIL7/8sqEdlPCkAEASBQCSKACQRAGAJAoAJFEAIIkCAEkUAEiiAEASBQCSKACQRAGAVKuqqqrrwlptfd8LHWTMmDHFmyOPPLJ4c/jhhxdvIiJaWloa2rWHRn5dLF68uKH3GjZsWPFm/vz5Db0XnVM9v917UgAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQHIgXiez6667Fm/ef//94k3Pnj2LN9OmTSveREQ89NBDxZvu3bsXb0aOHFm8GTx4cPGmubm5eBMRcfvttxdvzj///Ibei87JgXgAFBEFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYDUtaNvgLa17bbbFm969OixHu5kXQceeGBDu9GjR7fxnfy1Rg7eGzZsWPFmxowZxZuIiP/85z8N7aCEJwUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACA5JbWTWbhwYfHmzDPPXA93sq5PP/20Xd6nPZ1zzjnt9l5PPfVUu70Xmy5PCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASLWqqqq6LqzV1ve9wEanzl8+a/n4448beq+BAwc2tIM/1fP56kkBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgCpa0ffAPyTLbfcsnjT0tJSvLnmmmuKN8uWLSvejB07tngD7cWTAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkgPxaDfHHXdcQ7uJEycWb3baaafiTVVVxZumpqbiDWzIPCkAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgDJKamdzJZbblm8mTBhQvHm5JNPLt4ccMABxZuIiC5duhRvFi5cWLwZPXp08QY6G08KACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABItaqqqrourNXW973QBvr371+8+eSTT4o3jXw+1PmptlF56623ijdXXHFFQ+81Y8aMhnbwp3p+DXpSACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAciBeJ7P99tsXb0aOHFm86devX/Fm3rx5xZtGNfLzMGnSpOJN9+7dizeNHgzYyIF406dPL9488sgjxZsff/yxeEP7cyAeAEVEAYAkCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgORAP/k9zc3O7bC677LLiTUTEUUcdVbzZYostijeNHLx30kknFW9WrlxZvOHfcSAeAEVEAYAkCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgORAPNhJDhw4t3txzzz3Fm7333rt48+ijjxZvxowZU7zh33EgHgBFRAGAJAoAJFEAIIkCAEkUAEiiAEASBQCSKACQRAGAJAoAJFEAIIkCAMkpqdCJ9e/fv3gzZ86c4s0WW2xRvNlrr72KNxER8+fPb2iHU1IBKCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgCpa0ffALD+zJ07t3jzwQcfFG8OOeSQ4s2JJ55YvImIuPnmmxvaUR9PCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASA7Eg05s0KBBxZuWlpbiTWtra/Hmo48+Kt6w/nlSACCJAgBJFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAciBeJ9O1a/l/0jFjxhRvHnvsseLNypUrized0Q477NDQbvjw4cWb66+/vnjT1NRUvHnllVeKNzNmzCjesP55UgAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQKpVVVXVdWGttr7vhTZw3XXXFW8uuuii4s3uu+9evPnqq6+KN43aa6+9ijctLS3Fm9NPP714M2zYsOJNRMS2225bvGltbS3eTJ48uXhzxx13FG/a8/OBP9Tz270nBQCSKACQRAGAJAoAJFEAIIkCAEkUAEiiAEASBQCSKACQRAGAJAoAJFEAIDkltZP5/PPPizd9+vQp3qxYsaJ4U+enWpvo1q1b8Wbrrbcu3jTy62L58uXFm4iId999t3hz9dVXF2/eeOON4g0bB6ekAlBEFABIogBAEgUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkgPxOplBgwYVbyZMmFC8Oeyww4o3ffv2Ld5ERCxcuLB48/LLLxdvnnvuueLN0qVLizeLFy8u3kQ09vMA/8uBeAAUEQUAkigAkEQBgCQKACRRACCJAgBJFABIogBAEgUAkigAkEQBgORAPIBNhAPxACgiCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQBIFAJIoAJBEAYAkCgAkUQAgiQIAqWu9F1ZVtT7vA4ANgCcFAJIoAJBEAYAkCgAkUQAgiQIASRQASKIAQBIFANJ/AQmIi12V3zs8AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Choosing one data and visualizing it\n",
    "print(f\"Shape of the image of the first data: {training_data[0][0].shape}\")\n",
    "print(f\"Label of the first data: {training_data[0][1]}\")\n",
    "\n",
    "# Visualizing it\n",
    "idx = np.random.randint(0,10000)\n",
    "img, label = training_data[idx]\n",
    "plt.title(f\"Label = {label}\")\n",
    "plt.imshow(img.squeeze(), cmap=\"gray\")\n",
    "plt.axis(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Custom Dataset\n",
    "Most of the time, you might now work with pre-loaded datasets (such as some of the questions of your homework:)). In these case you must create your own custom dataset. In order to do so you should create a child from the class `Dataset` as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ASUS\\AppData\\Local\\Temp\\ipykernel_18972\\1050226701.py:2: DeprecationWarning: \n",
      "Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),\n",
      "(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)\n",
      "but was not found to be installed on your system.\n",
      "If this would cause problems for you,\n",
      "please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466\n",
      "        \n",
      "  import pandas as pd\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from torchvision.io import read_image\n",
    "\n",
    "class CustomImageDataset(Dataset):\n",
    "    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):\n",
    "        self.img_labels = pd.read_csv(annotations_file)\n",
    "        self.img_dir = img_dir\n",
    "        self.transform = transform\n",
    "        self.target_transform = target_transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.img_labels)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])\n",
    "        image = read_image(img_path)\n",
    "        label = self.img_labels.iloc[idx, 1]\n",
    "        if self.transform:\n",
    "            image = self.transform(image)\n",
    "        if self.target_transform:\n",
    "            label = self.target_transform(label)\n",
    "        return image, label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing your data for training with DataLoader\n",
    "using `DataLoader`, you can shuffle data, created minibatches from it, and iterate through does minibatches just by one command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "train_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)\n",
    "test_dataloader = DataLoader(test_data, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First iter: \n",
      " tensor([5, 1, 7, 1, 6, 3, 5, 4, 0, 3, 2, 2, 2, 7, 7, 3, 4, 8, 5, 0, 7, 9, 3, 1,\n",
      "        7, 1, 4, 2, 2, 7, 4, 3, 1, 0, 2, 4, 9, 3, 8, 1, 5, 8, 7, 7, 7, 9, 7, 0,\n",
      "        2, 3, 2, 6, 6, 8, 9, 0, 5, 4, 1, 6, 1, 4, 1, 8]) \n",
      " \n",
      "Second iter: \n",
      " tensor([9, 1, 3, 3, 7, 0, 6, 1, 7, 9, 8, 6, 7, 3, 8, 1, 9, 4, 7, 5, 3, 7, 7, 1,\n",
      "        9, 6, 8, 1, 5, 7, 4, 3, 8, 6, 8, 6, 3, 4, 6, 5, 5, 9, 3, 1, 2, 6, 7, 2,\n",
      "        0, 2, 9, 7, 7, 2, 9, 8, 3, 0, 3, 9, 1, 9, 4, 2])\n",
      "Features batch shape: \n",
      " torch.Size([64, 1, 28, 28]) \n",
      "\n",
      "Labels batch shape: \n",
      " torch.Size([64]) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# iterating on train_dataloader and printing just the labels in order to see the difference\n",
    "first_batch = next(iter(train_dataloader))\n",
    "second_batch = next(iter(train_dataloader))\n",
    "print(f\"First iter: \\n {first_batch[1]} \\n \")\n",
    "print(f\"Second iter: \\n {second_batch[1]}\")\n",
    "\n",
    "print(f\"Features batch shape: \\n {first_batch[0].shape} \\n\")\n",
    "print(f\"Labels batch shape: \\n {first_batch[1].shape} \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transforms\n",
    "Data does not always come in its final processed form that is required for training machine learning algorithms. We use transforms to perform some manipulation of the data and make it suitable for training.\n",
    "\n",
    "All TorchVision datasets have two parameters -`transform` to modify the features and `target_transform` to modify the labels - that accept callables containing the transformation logic. For more information about transform you can visit [torchvision.transform](https://pytorch.org/vision/stable/transforms.html).\n",
    "\n",
    "Here is an example of using transfomrs on `torchvision.datasets.MNIST`. MNIST datasets contains PIL images which should be converted tensors normalized between 0 to 1(using `ToTensor()`)and labels which are integers and should be converted to one-hot encoded values as shown below.\n",
    "\n",
    "In the example below, we also use `Lambda` transforms which apply any user-defined lambda function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.transforms import ToTensor, Lambda\n",
    "ds = datasets.MNIST(\n",
    "    root=\"data\",\n",
    "    train=True,\n",
    "    download=True,\n",
    "    transform=ToTensor(),\n",
    "    target_transform = Lambda(lambda y: torch.zeros(10, dtype=torch.float).scatter_(dim=0, index=torch.tensor(y),\n",
    "                                                                                    value=1))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features tensor: \n",
      " tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.0706, 0.0706, 0.0706,\n",
      "          0.4941, 0.5333, 0.6863, 0.1020, 0.6510, 1.0000, 0.9686, 0.4980,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.1176, 0.1412, 0.3686, 0.6039, 0.6667, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.9922, 0.8824, 0.6745, 0.9922, 0.9490, 0.7647, 0.2510,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1922,\n",
      "          0.9333, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.9843, 0.3647, 0.3216, 0.3216, 0.2196, 0.1529, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0706,\n",
      "          0.8588, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.7765, 0.7137,\n",
      "          0.9686, 0.9451, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.3137, 0.6118, 0.4196, 0.9922, 0.9922, 0.8039, 0.0431, 0.0000,\n",
      "          0.1686, 0.6039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0549, 0.0039, 0.6039, 0.9922, 0.3529, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.5451, 0.9922, 0.7451, 0.0078, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0431, 0.7451, 0.9922, 0.2745, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.1373, 0.9451, 0.8824, 0.6275,\n",
      "          0.4235, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3176, 0.9412, 0.9922,\n",
      "          0.9922, 0.4667, 0.0980, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1765, 0.7294,\n",
      "          0.9922, 0.9922, 0.5882, 0.1059, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0627,\n",
      "          0.3647, 0.9882, 0.9922, 0.7333, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.9765, 0.9922, 0.9765, 0.2510, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1804, 0.5098,\n",
      "          0.7176, 0.9922, 0.9922, 0.8118, 0.0078, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.1529, 0.5804, 0.8980, 0.9922,\n",
      "          0.9922, 0.9922, 0.9804, 0.7137, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0941, 0.4471, 0.8667, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.7882, 0.3059, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0902, 0.2588, 0.8353, 0.9922, 0.9922, 0.9922, 0.9922, 0.7765,\n",
      "          0.3176, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0706, 0.6706,\n",
      "          0.8588, 0.9922, 0.9922, 0.9922, 0.9922, 0.7647, 0.3137, 0.0353,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.2157, 0.6745, 0.8863, 0.9922,\n",
      "          0.9922, 0.9922, 0.9922, 0.9569, 0.5216, 0.0431, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.5333, 0.9922, 0.9922, 0.9922,\n",
      "          0.8314, 0.5294, 0.5176, 0.0627, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000]]]) \n",
      "\n",
      "Labels tensor: \n",
      " tensor([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "print(f\"Features tensor: \\n {ds[0][0]} \\n\")\n",
    "print(f\"Labels tensor: \\n {ds[0][1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural networks comprise of layers/modules that perform operations on data. The `torch.nn` namespace provides all the building blocks you need to build your own neural network. Every module in PyTorch subclasses the `nn.Module`. A neural network is a module itself that consists of other modules (layers). This nested structure allows for building and managing complex architectures easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cuda device\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "# setting the device to cuda if available\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(f\"Using {device} device\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the class\n",
    "We define our neural network by subclassing `nn.Module`, and initialize the neural network layers in `__init__`. Every `nn.Module` subclass implements the operations on input data in the `forward` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.linear_relu_stack = nn.Sequential(\n",
    "            nn.Linear(28*28, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 10),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.flatten(x)\n",
    "        logits = self.linear_relu_stack(x)\n",
    "        return logits\n",
    "    \n",
    "class NeuralNetwork1(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.linear1 = nn.Linear(28*28, 512)\n",
    "        self.relu1 = nn.ReLU()\n",
    "        self.linear2 = nn.Linear(512, 512)\n",
    "        self.relu2 = nn.ReLU()\n",
    "        self.linear3 = nn.Linear(512, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.flatten(x)\n",
    "        x = self.linear1(x)\n",
    "        x = self.relu1(x)\n",
    "        x = self.linear2(x)\n",
    "        x = self.relu2(x)\n",
    "        logits = self.linear3(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NeuralNetwork(\n",
      "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
      "  (linear_relu_stack): Sequential(\n",
      "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "    (3): ReLU()\n",
      "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Creating an instance of the class `NeuralNetwork` and moving it to device\n",
    "model = NeuralNetwork().to(device)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "whenever we want to get the model's output on an input data `X`, we pass the data to the model directly using `model(X)`. This automatically runs `forward` method with some other background calculations. **Do not run `model.forward()` directly.**\n",
    "\n",
    "Note that the device of the data passed to the model should be the same device in which the model is saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model's output: \n",
      " tensor([[-0.0256, -0.0040, -0.0285, -0.0217,  0.0594,  0.0672, -0.0618, -0.0377,\n",
      "         -0.0170, -0.0272]], device='cuda:0', grad_fn=<AddmmBackward0>) \n",
      "\n",
      "Prediction probabilities: \n",
      " tensor([[0.0983, 0.1005, 0.0981, 0.0987, 0.1071, 0.1079, 0.0948, 0.0972, 0.0992,\n",
      "         0.0982]], device='cuda:0', grad_fn=<SoftmaxBackward0>) \n",
      "\n",
      "Predicted label: \n",
      " 5 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "X = torch.rand(1, 28, 28, device=device) # creating a random input with the same device\n",
    "\n",
    "# passing the input to the model\n",
    "logits = model(X)\n",
    "print(f\"Model's output: \\n {logits} \\n\")\n",
    "\n",
    "# calculating model predictions by applying a softmax\n",
    "probs = nn.Softmax(dim=1)(logits) # finding the probabilities\n",
    "print(f\"Prediction probabilities: \\n {probs} \\n\")\n",
    "\n",
    "y_pred = probs.argmax(dim = 1)\n",
    "print(f\"Predicted label: \\n {y_pred.item()} \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each Layer in the previous models works as explained below:\n",
    "- `nn.Flatten()`: Flattens the data except for dim=0. For example a data with the shape of (1, 10, 10) is converted to a data with the shape of (1, 100)\n",
    "- `nn.Linear(in_features, out_features)`: A single linear layer with `in_features` inputs and `out_features` outputs\n",
    "- `nn.ReLU()`: Simply applies relu function on each element of its input\n",
    "- `nn.Sequential(module1, module2, ...)`: It is a container of modules. It simply passes the data through the modules with the given order\n",
    "- `nn.Softmax(dim)`: Applies a softmax function on the given dimension of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Parameters\n",
    "Many layers inside a neural network are parameterized, i.e. have associated weights and biases that are optimized during training. Subclassing `nn.Module` automatically tracks all fields defined inside your model object, and makes all parameters accessible using your model’s `parameters()` or `named_parameters()` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model structure: NeuralNetwork(\n",
      "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
      "  (linear_relu_stack): Sequential(\n",
      "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "    (3): ReLU()\n",
      "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
      "  )\n",
      ")\n",
      "\n",
      "\n",
      "Layer: linear_relu_stack.0.weight | Size: torch.Size([512, 784]) | Values : tensor([[-0.0227, -0.0286,  0.0266,  ...,  0.0331,  0.0057, -0.0345],\n",
      "        [-0.0298,  0.0025, -0.0354,  ...,  0.0126,  0.0340, -0.0043]],\n",
      "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
      "\n",
      "Layer: linear_relu_stack.0.bias | Size: torch.Size([512]) | Values : tensor([ 0.0288, -0.0333], device='cuda:0', grad_fn=<SliceBackward0>) \n",
      "\n",
      "Layer: linear_relu_stack.2.weight | Size: torch.Size([512, 512]) | Values : tensor([[ 0.0246,  0.0265,  0.0128,  ..., -0.0425,  0.0385, -0.0296],\n",
      "        [-0.0200,  0.0194,  0.0015,  ...,  0.0097, -0.0058, -0.0273]],\n",
      "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
      "\n",
      "Layer: linear_relu_stack.2.bias | Size: torch.Size([512]) | Values : tensor([ 0.0084, -0.0268], device='cuda:0', grad_fn=<SliceBackward0>) \n",
      "\n",
      "Layer: linear_relu_stack.4.weight | Size: torch.Size([10, 512]) | Values : tensor([[ 4.0648e-02, -6.4324e-03, -1.7283e-02,  ...,  1.1251e-02,\n",
      "          3.5044e-02, -7.0807e-05],\n",
      "        [ 1.9634e-02,  3.4438e-03, -4.1713e-02,  ..., -1.3394e-02,\n",
      "         -4.3820e-02, -4.2858e-03]], device='cuda:0', grad_fn=<SliceBackward0>) \n",
      "\n",
      "Layer: linear_relu_stack.4.bias | Size: torch.Size([10]) | Values : tensor([0.0323, 0.0086], device='cuda:0', grad_fn=<SliceBackward0>) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(f\"Model structure: {model}\\n\\n\")\n",
    "\n",
    "for name, param in model.named_parameters():\n",
    "    print(f\"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic Differentiation with `torch.autograd`\n",
    "The most frequently used algorithm in neural networks is `back propagation`. One of the fundumental benefits of `Pytorch` and other deep learning frameworks is the implementation of automatic differetiaion using back propagation. This means that in order to find the gradients of a model, you can simply call `backward()` method and torch will automatically calculate the gradients of the pararmeters for you.\n",
    "\n",
    "For example, we try to calculate the gradients of a given function $y = exp(x^Tw)$ where $x \\in \\mathcal{R}^{10}$ is a constant vector and $w \\in \\mathcal{R}^{10}$ is our variable. Using back propagation, we can take $z = x^Tw$. Then $\\frac{d\\exp(z)}{dz} = \\exp(z)$, $\\frac{dz}{dw_i} = x_i$, and consequently $\\frac{dy}{dw_i} = \\frac{dy}{dz} \\frac{dz}{dw_i} = x_i . \\exp(x^Tw)= x_i . y $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected gradients: \n",
      " tensor([ 3269017.2500,  6538034.5000,  9807052.0000, 13076069.0000,\n",
      "        16345086.0000]) \n",
      "\n",
      "Calculated gradients: \n",
      " tensor([ 3269017.2500,  6538034.5000,  9807052.0000, 13076069.0000,\n",
      "        16345086.0000])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([1.,2.,3.,4.,5.])\n",
    "w = torch.tensor([1.,1.,1.,1.,1.], requires_grad=True)\n",
    "z = torch.matmul(x,w)\n",
    "y = torch.exp(z)\n",
    "y.backward()\n",
    "\n",
    "\n",
    "expected_grads = (y * x).detach() # setting requires_grad to False\n",
    "\n",
    "print(f\"Expected gradients: \\n {expected_grads} \\n\")\n",
    "print(f\"Calculated gradients: \\n {w.grad}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First method's requires_grad before: True\n",
      "First method's requires_grad after: False\n",
      "\n",
      "Second method's requires_grad before: True\n",
      "Second method's requires_grad after: False\n"
     ]
    }
   ],
   "source": [
    "# how to set requires grad to False\n",
    "# there are two ways to do so\n",
    "# 1\n",
    "z1 = x * w\n",
    "print(\"First method's requires_grad before:\", z1.requires_grad)\n",
    "with torch.no_grad():\n",
    "    z1 = x * w\n",
    "print(\"First method's requires_grad after:\", z1.requires_grad)\n",
    "\n",
    "# 2\n",
    "print()\n",
    "z2 = x * w\n",
    "print(\"Second method's requires_grad before:\", z2.requires_grad)\n",
    "\n",
    "z2 = z2.detach()\n",
    "print(\"Second method's requires_grad after:\", z2.requires_grad)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimization\n",
    "Now that we have the model, dataset and parameters, we should optimize our model on the given dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NeuralNetwork(\n",
       "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
       "  (linear_relu_stack): Sequential(\n",
       "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
       "    (1): ReLU()\n",
       "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
       "    (3): ReLU()\n",
       "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creating an instance of our model\n",
    "model = NeuralNetwork().to(device)\n",
    "\n",
    "# setting models mode to train mode\n",
    "model.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters\n",
    "We define the following hyperparameters for training:\n",
    "- **Number of Epochs** - the number times to iterate over the dataset\n",
    "- **Batch Size** - the number of data samples propagated through the network before the parameters are updated\n",
    "- **Learning Rate** - how much to update models parameters at each batch/epoch. Smaller values yield slow learning speed, while large values may result in unpredictable behavior during training.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 1e-3\n",
    "batch_size = 64\n",
    "epochs = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization loop\n",
    "Each iteration of the optimization loop is called **epoch**. In each epoch these two main parts should be implemented:\n",
    "- **The Train Loop** - Iterate over the batches and try to converge to optimal parameters\n",
    "- **The Validation/Test Loop** - Iterate over test/validation dataset to see whether the model is improving or not\n",
    "\n",
    "We also need a loss function. There are various loss functions implemented in `torch.nn`. In this example, we use cross entropy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the loss function\n",
    "loss_fn = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also have to add an optimizer to the loop. The objective of an optimizer is to use the gradien of the parameters and optimize them based on the selected optimization algorithm such as SGD, Adam, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inside the training loop for each batch, we first call `optimizer.zero_grad()` to make all gradients equal to zero. Then we call `loss.backward()` to calculate the gradients (using the autograd which was explained before), and finally, we call `optimizer.step()` to adjust the parameters based on their gradients.\n",
    "\n",
    "The final implementation of an optimization loop should be something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_loop(dataloader, model, loss_fn, optimizer):\n",
    "    size = len(dataloader.dataset)\n",
    "    for batch, (X, y) in enumerate(dataloader):\n",
    "        # Compute prediction and loss\n",
    "        pred = model(X)\n",
    "        loss = loss_fn(pred, y)\n",
    "\n",
    "        # Backpropagation\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        if batch % 100 == 0:\n",
    "            loss, current = loss.item(), batch * len(X)\n",
    "            print(f\"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]\")\n",
    "\n",
    "\n",
    "def test_loop(dataloader, model, loss_fn):\n",
    "    size = len(dataloader.dataset)\n",
    "    num_batches = len(dataloader)\n",
    "    test_loss, correct = 0, 0\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for X, y in dataloader:\n",
    "            pred = model(X)\n",
    "            test_loss += loss_fn(pred, y).item()\n",
    "            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n",
    "\n",
    "    test_loss /= num_batches\n",
    "    correct /= size\n",
    "    print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1\n",
      "-------------------------------\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32me:\\Sharif University\\Sharif Courses\\TA Files\\Social Robotics TA\\Pytorch_Tutorial.ipynb Cell 51\u001b[0m line \u001b[0;36m7\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m \u001b[39mfor\u001b[39;00m t \u001b[39min\u001b[39;00m \u001b[39mrange\u001b[39m(epochs):\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=5'>6</a>\u001b[0m     \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mEpoch \u001b[39m\u001b[39m{\u001b[39;00mt\u001b[39m+\u001b[39m\u001b[39m1\u001b[39m\u001b[39m}\u001b[39;00m\u001b[39m\\n\u001b[39;00m\u001b[39m-------------------------------\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m----> <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=6'>7</a>\u001b[0m     train_loop(train_dataloader, model, loss_fn, optimizer)\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=7'>8</a>\u001b[0m     test_loop(test_dataloader, model, loss_fn)\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=8'>9</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39mDone!\u001b[39m\u001b[39m\"\u001b[39m)\n",
      "\u001b[1;32me:\\Sharif University\\Sharif Courses\\TA Files\\Social Robotics TA\\Pytorch_Tutorial.ipynb Cell 51\u001b[0m line \u001b[0;36m5\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m size \u001b[39m=\u001b[39m \u001b[39mlen\u001b[39m(dataloader\u001b[39m.\u001b[39mdataset)\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39mfor\u001b[39;00m batch, (X, y) \u001b[39min\u001b[39;00m \u001b[39menumerate\u001b[39m(dataloader):\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m     \u001b[39m# Compute prediction and loss\u001b[39;00m\n\u001b[1;32m----> <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m     pred \u001b[39m=\u001b[39m model(X)\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=5'>6</a>\u001b[0m     loss \u001b[39m=\u001b[39m loss_fn(pred, y)\n\u001b[0;32m      <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=7'>8</a>\u001b[0m     \u001b[39m# Backpropagation\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\ASUS\\.conda\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\module.py:1518\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1516\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_compiled_call_impl(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)  \u001b[39m# type: ignore[misc]\u001b[39;00m\n\u001b[0;32m   1517\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m-> 1518\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_call_impl(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n",
      "File \u001b[1;32mc:\\Users\\ASUS\\.conda\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\module.py:1527\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1522\u001b[0m \u001b[39m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[0;32m   1523\u001b[0m \u001b[39m# this function, and just call forward.\u001b[39;00m\n\u001b[0;32m   1524\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m (\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_pre_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_pre_hooks\n\u001b[0;32m   1525\u001b[0m         \u001b[39mor\u001b[39;00m _global_backward_pre_hooks \u001b[39mor\u001b[39;00m _global_backward_hooks\n\u001b[0;32m   1526\u001b[0m         \u001b[39mor\u001b[39;00m _global_forward_hooks \u001b[39mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[1;32m-> 1527\u001b[0m     \u001b[39mreturn\u001b[39;00m forward_call(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n\u001b[0;32m   1529\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m   1530\u001b[0m     result \u001b[39m=\u001b[39m \u001b[39mNone\u001b[39;00m\n",
      "\u001b[1;32me:\\Sharif University\\Sharif Courses\\TA Files\\Social Robotics TA\\Pytorch_Tutorial.ipynb Cell 51\u001b[0m line \u001b[0;36m1\n\u001b[0;32m     <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=12'>13</a>\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mforward\u001b[39m(\u001b[39mself\u001b[39m, x):\n\u001b[0;32m     <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=13'>14</a>\u001b[0m     x \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mflatten(x)\n\u001b[1;32m---> <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=14'>15</a>\u001b[0m     logits \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mlinear_relu_stack(x)\n\u001b[0;32m     <a href='vscode-notebook-cell:/e%3A/Sharif%20University/Sharif%20Courses/TA%20Files/Social%20Robotics%20TA/Pytorch_Tutorial.ipynb#Y101sZmlsZQ%3D%3D?line=15'>16</a>\u001b[0m     \u001b[39mreturn\u001b[39;00m logits\n",
      "File \u001b[1;32mc:\\Users\\ASUS\\.conda\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\module.py:1518\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1516\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_compiled_call_impl(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)  \u001b[39m# type: ignore[misc]\u001b[39;00m\n\u001b[0;32m   1517\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m-> 1518\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_call_impl(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n",
      "File \u001b[1;32mc:\\Users\\ASUS\\.conda\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\module.py:1527\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1522\u001b[0m \u001b[39m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[0;32m   1523\u001b[0m \u001b[39m# this function, and just call forward.\u001b[39;00m\n\u001b[0;32m   1524\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m (\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_pre_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_pre_hooks\n\u001b[0;32m   1525\u001b[0m         \u001b[39mor\u001b[39;00m _global_backward_pre_hooks \u001b[39mor\u001b[39;00m _global_backward_hooks\n\u001b[0;32m   1526\u001b[0m         \u001b[39mor\u001b[39;00m _global_forward_hooks \u001b[39mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[1;32m-> 1527\u001b[0m     \u001b[39mreturn\u001b[39;00m forward_call(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n\u001b[0;32m   1529\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m   1530\u001b[0m     result \u001b[39m=\u001b[39m \u001b[39mNone\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\ASUS\\.conda\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\container.py:215\u001b[0m, in \u001b[0;36mSequential.forward\u001b[1;34m(self, input)\u001b[0m\n\u001b[0;32m    213\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mforward\u001b[39m(\u001b[39mself\u001b[39m, \u001b[39minput\u001b[39m):\n\u001b[0;32m    214\u001b[0m     \u001b[39mfor\u001b[39;00m module \u001b[39min\u001b[39;00m \u001b[39mself\u001b[39m:\n\u001b[1;32m--> 215\u001b[0m         \u001b[39minput\u001b[39m \u001b[39m=\u001b[39m module(\u001b[39minput\u001b[39;49m)\n\u001b[0;32m    216\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39minput\u001b[39m\n",
      "File \u001b[1;32mc:\\Users\\ASUS\\.conda\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\module.py:1518\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1516\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_compiled_call_impl(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)  \u001b[39m# type: ignore[misc]\u001b[39;00m\n\u001b[0;32m   1517\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m-> 1518\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_call_impl(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n",
      "File \u001b[1;32mc:\\Users\\ASUS\\.conda\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\module.py:1527\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1522\u001b[0m \u001b[39m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[0;32m   1523\u001b[0m \u001b[39m# this function, and just call forward.\u001b[39;00m\n\u001b[0;32m   1524\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m (\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_pre_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_pre_hooks\n\u001b[0;32m   1525\u001b[0m         \u001b[39mor\u001b[39;00m _global_backward_pre_hooks \u001b[39mor\u001b[39;00m _global_backward_hooks\n\u001b[0;32m   1526\u001b[0m         \u001b[39mor\u001b[39;00m _global_forward_hooks \u001b[39mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[1;32m-> 1527\u001b[0m     \u001b[39mreturn\u001b[39;00m forward_call(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n\u001b[0;32m   1529\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m   1530\u001b[0m     result \u001b[39m=\u001b[39m \u001b[39mNone\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\ASUS\\.conda\\envs\\torch\\lib\\site-packages\\torch\\nn\\modules\\linear.py:114\u001b[0m, in \u001b[0;36mLinear.forward\u001b[1;34m(self, input)\u001b[0m\n\u001b[0;32m    113\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mforward\u001b[39m(\u001b[39mself\u001b[39m, \u001b[39minput\u001b[39m: Tensor) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m Tensor:\n\u001b[1;32m--> 114\u001b[0m     \u001b[39mreturn\u001b[39;00m F\u001b[39m.\u001b[39;49mlinear(\u001b[39minput\u001b[39;49m, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mweight, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mbias)\n",
      "\u001b[1;31mRuntimeError\u001b[0m: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)"
     ]
    }
   ],
   "source": [
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n",
    "\n",
    "epochs = 4\n",
    "for t in range(epochs):\n",
    "    print(f\"Epoch {t+1}\\n-------------------------------\")\n",
    "    train_loop(train_dataloader, model, loss_fn, optimizer)\n",
    "    test_loop(test_dataloader, model, loss_fn)\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving and loading the model\n",
    "After training the model, we should save it to be able to use it afterwards. There are two ways to do this:\n",
    "- Saving just the weights of the model\n",
    "- Saving the weights and the structure "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving & Loading the weigths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NeuralNetwork(\n",
       "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
       "  (linear_relu_stack): Sequential(\n",
       "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
       "    (1): ReLU()\n",
       "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
       "    (3): ReLU()\n",
       "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# saving\n",
    "torch.save(model.state_dict(), 'model_weights.pth')\n",
    "\n",
    "# loading\n",
    "model.load_state_dict(torch.load('model_weights.pth'))\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving & Loading the model and its weigths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NeuralNetwork(\n",
       "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
       "  (linear_relu_stack): Sequential(\n",
       "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
       "    (1): ReLU()\n",
       "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
       "    (3): ReLU()\n",
       "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# saving\n",
    "torch.save(model, 'model.pth')\n",
    "\n",
    "# loading\n",
    "model = torch.load('model.pth')\n",
    "model.eval()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9.13 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "vscode": {
   "interpreter": {
    "hash": "304d98a18a6597b5074573a35a99b631fbbf66bbb48b57fc984d20c778d1912e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}