From 94e88d31fa7309d393f1e970cfe7c556ba542e71 Mon Sep 17 00:00:00 2001 From: Niklas Eberts <99niklas@gmx.de> Date: Wed, 22 Nov 2023 21:47:28 +0100 Subject: [PATCH] Deleted duplicate initial jupyter files --- exercises/exercise2/Exercise_2(4).ipynb | 579 ------------------------ exercises/exercise2/Exercise_2(5).ipynb | 444 ------------------ 2 files changed, 1023 deletions(-) delete mode 100644 exercises/exercise2/Exercise_2(4).ipynb delete mode 100644 exercises/exercise2/Exercise_2(5).ipynb diff --git a/exercises/exercise2/Exercise_2(4).ipynb b/exercises/exercise2/Exercise_2(4).ipynb deleted file mode 100644 index 9152881..0000000 --- a/exercises/exercise2/Exercise_2(4).ipynb +++ /dev/null @@ -1,579 +0,0 @@ -{ - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.9" - }, - "colab": { - "provenance": [], - "include_colab_link": true - }, - "accelerator": "GPU" - }, - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "view-in-github", - "colab_type": "text" - }, - "source": [ - "<a href=\"https://colab.research.google.com/github/mindgarage/very-deep-learning-wise2324/blob/main/exercises/Exercise_2(4).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>" - ] - }, - { - "cell_type": "markdown", - "source": [ - "# Programming Exercise 2(4): CIFAR Challenge\n", - "\n", - "## Very Deep Learning (VDL) - Winter Semester 2023/24\n", - "\n", - "---\n", - "\n", - "### Group Details:\n", - "\n", - "- **Group Name:** \\[Enter OLAT Group Name Here\\]\n", - "\n", - "### Members:\n", - "\n", - "- \\[Participant 1 Name\\], \\[Matrikel-Nr 1\\]\n", - "- \\[Participant 2 Name\\], \\[Matrikel-Nr 1\\]\n", - "- ...\n", - "\n", - "---\n", - "\n", - "**Instructions**: The tasks in this notebook are a part of Sheet 1. Look for `TODO` tags throughout the notebook and complete the sections with missing code. Once done, ensure all outputs are visible and correctly displayed. Save your notebook and submit the `.ipynb` file together with the exercise sheet PDF in a single ZIP file." - ], - "metadata": { - "id": "994jSp9_dTAp" - } - }, - { - "cell_type": "code", - "metadata": { - "id": "jCXXHGdAtdHp" - }, - "source": [ - "%matplotlib inline" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "JL5-EqeYtdHs" - }, - "source": [ - "import torch\n", - "\n", - "if torch.cuda.is_available():\n", - " device = torch.device('cuda')\n", - "else:\n", - " device = torch.device('cpu')\n", - "\n", - "print(device)" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "73l8LqGetdHu" - }, - "source": [ - "Note: Tasks in this notebook can be solved without a GPU as well" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "5BANL3zstdHv" - }, - "source": [ - "__Objective of this notebook is to give hands on experience in training a Convolutional Neural Network(CNN) from scratch using Pytorch framework. Further the tasks will also give an insight on how to change the network architecture and hyperparameter settings to achieve better results.__\n", - "\n", - "Fill the code in the notebook where it is explicitly mentioned and solve all tasks to complete the assignment" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "UJDuAZI8tdHv" - }, - "source": [ - "We divide our task in followind steps:\n", - "\n", - "1- Load the CIFAR-10 dataset and look at some properties of the dataset like:\n", - " * size of the images\n", - " * number of images in the dataset\n", - " * number of channels in the image (RGB=3, grayscale=1)\n", - " * number of classes in the dataset\n", - " \n", - "2- Define a network architecture which takes batches of the images in the form of a 4-dimensional tensor and compute the output scores for each class in the dataset.\n", - "\n", - "3- Define a loss function for training the network, and an optimizer (SGD, Adam, or Adagrad) which defines the strategy to update weights of the network.\n", - "\n", - "4- Finally, train the network and find out the accuracy on validation set\n", - "\n", - "5- Improve the network architecture to get better accuracy." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "PN6B0Rc3tdHw" - }, - "source": [ - "---\n", - "\n", - "__Step-1__ : Loading the CIFAR-10 dataset is very easy using ``torchvision`` package, which has readily availabe famous datasets and API to load these datasets into batches known as dataloaders via ``torchvision.Datasets``and ``torch.utils.data.DataLoader'``" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "Y_gKOz9otdHy" - }, - "source": [ - "###########################################################################################\n", - "## Define transform variable ##\n", - "## ##\n", - "## ##\n", - "## First define a transform to convert RGB images into tensors and another transform to ##\n", - "## normalize them with the channel-wise mean and standard deviation of the CIFAR-10 ##\n", - "## dataset ##\n", - "## ##\n", - "###########################################################################################\n", - "import torchvision\n", - "import torchvision.transforms as transforms\n", - "\n", - "transform = transforms.Compose([\n", - " transforms.ToTensor(),\n", - " transforms.Normalize(mean=[0.5, 0.5, 0.5], # approx. mean and std of R, G, and B channels of\n", - " std=[0.25, 0.25, 0.25]) # all the images in CIFAR-10. You can find out the\n", - " ]) # exact values and replace these with them." - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "7Y3opFMJtdH0" - }, - "source": [ - "## TODO ##\n", - "###########################################################################################\n", - "## Use above transform as an argument to load the training and the test set for CIFAR-10 ##\n", - "## https://pytorch.org/vision/stable/generated/torchvision.datasets.CIFAR10.html ##\n", - "###########################################################################################\n", - "trainset =\n", - "testset =" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "xSQFwGMttdH2" - }, - "source": [ - "## TODO ##\n", - "###########################################################################################\n", - "## Write code to print below mentioned statistics: ##\n", - "## ##\n", - "## - size of the images (height, width and number of channels) ##\n", - "## - size of the training images ##\n", - "## - size of the test images ##\n", - "## - classes in CIFAR-10 dataset ##\n", - "## ##\n", - "###########################################################################################\n", - "print(\"CIFAR-10 DATASET STATS\")\n", - "print(\" Image size (h, w, c):\")\n", - "print(\" # of training images:\")\n", - "print(\" # of test images:\")\n", - "print(\" # of classes:\")" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "Wg-6GWRptdH4" - }, - "source": [ - "###########################################################################################\n", - "## Define your data loaders and split training dataset into train and validation sets ##\n", - "## ##\n", - "###########################################################################################\n", - "import torch\n", - "\n", - "batch_size = 64\n", - "\n", - "# Create dataloader for testset\n", - "test_loader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=2)\n", - "\n", - "# Split the trainset into train_data and val_data with 1000 images in validation set and\n", - "# rest in the training set (use random_split in pytorch to split the dataset)\n", - "val_data, train_data = torch.utils.data.random_split(trainset, [1000, 49000])\n", - "\n", - "# Create dataloaders for train and validation sets\n", - "trainloader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, shuffle=True, num_workers=2)\n", - "valloader = torch.utils.data.DataLoader(val_data, batch_size=batch_size, shuffle=True, num_workers=2)" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "aisKXV5stdH6" - }, - "source": [ - "## TODO ##\n", - "###########################################################################################\n", - "## Write code to load a batch of images and visualize them ##\n", - "## Hint: Use plt.imshow and torchvision.utils.make_grid ##\n", - "###########################################################################################\n", - "%matplotlib inline\n", - "import matplotlib.pyplot as plt\n", - "import numpy as np\n", - "\n", - "\n", - "def convert_img(img):\n", - " mean = np.array([0.5, 0.5, 0.5])\n", - " std = np.array([0.25, 0.25, 0.25])\n", - "\n", - " np_img = img.numpy()\n", - " np_img = np.transpose(np_img, (1, 2, 0))\n", - " np_img = (np_img*std) + mean\n", - " return np_img" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "qlXPkuuEtdH8" - }, - "source": [ - "---\n", - "__Step-2:__ Define a simple convolutional Neural network with two convolutional layers and 2 fully-connected layers\n", - "\n", - "\n", - " (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1)) \\\n", - " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) \\\n", - " (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1)) \\\n", - " (fc1): Linear(in_features=400, out_features=120, bias=True) \\\n", - " (fc2): Linear(in_features=120, out_features=84, bias=True) \\\n", - " (fc3): Linear(in_features=84, out_features=10, bias=True)\n" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "pNtxLznQtdH8" - }, - "source": [ - "## TODO ##\n", - "###########################################################################################\n", - "## Define forward pass of the network ##\n", - "## ##\n", - "###########################################################################################\n", - "import torch.nn as nn\n", - "import torch.nn.functional as F\n", - "\n", - "\n", - "class Net(nn.Module):\n", - " def __init__(self):\n", - " super(Net, self).__init__()\n", - "\n", - " def forward(self, x):\n", - " return x" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ctugwZvitdH-" - }, - "source": [ - "__Step-3:__ Defining the loss function" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "32Vu678etdH_" - }, - "source": [ - "## TODO ##\n", - "## Define the loss function here\n", - "criterion =" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1tx67-5ytdIB" - }, - "source": [ - "__Step 4:__ Train the network, for 10 epochs and calculate the accuracy on the validation set\n", - "\n", - "Define some utility functions and then functions to train the model on the test set and validate it on the validation set" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "WDZlUhw5tdIB" - }, - "source": [ - "###########################################################################################\n", - "## DO NOT CHANGE THE CODE HERE!! ##\n", - "## ##\n", - "###########################################################################################\n", - "def weight_reset(m):\n", - " if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):\n", - " m.reset_parameters()" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "yte8dzR8tdID" - }, - "source": [ - "###########################################################################################\n", - "## DO NOT CHANGE THE CODE HERE!! ##\n", - "## ##\n", - "###########################################################################################\n", - "def get_accuracy(output, target, topk=(1,)):\n", - " \"\"\"Computes the accuracy over the\n", - " k top predictions for the specified values of k\"\"\"\n", - " with torch.no_grad():\n", - " maxk = max(topk)\n", - " batch_size = target.size(0)\n", - "\n", - " _, pred = output.topk(maxk, 1, True, True)\n", - " pred = pred.t()\n", - " correct = pred.eq(target.view(1, -1).expand_as(pred))\n", - "\n", - " res = []\n", - " for k in topk:\n", - " correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)\n", - " res.append(correct_k.mul_(100.0 / batch_size))\n", - " return res" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "hmm7U-AotdIF" - }, - "source": [ - "###########################################################################################\n", - "## DO NOT CHANGE THE CODE HERE!! ##\n", - "## ##\n", - "###########################################################################################\n", - "def validate(net, valoader, criterion):\n", - " val_loss = 0\n", - " val_accuracy = 0\n", - " net.eval()\n", - " with torch.no_grad():\n", - " for batch_idx, (inputs, targets) in enumerate(valloader):\n", - " inputs, targets = inputs.to(device), targets.to(device)\n", - " outputs = net(inputs)\n", - " loss = criterion(outputs, targets)\n", - " val_loss += loss.item()\n", - " accuracy = get_accuracy(outputs, targets)\n", - " val_accuracy += accuracy[0].item()\n", - " avg_loss = val_loss/(batch_idx+1)\n", - " avg_acc = val_accuracy/(batch_idx+1)\n", - "\n", - " return avg_loss, avg_acc" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "Pt0Ro1FytdIG" - }, - "source": [ - "###########################################################################################\n", - "## DO NOT CHANGE THE CODE HERE!! ##\n", - "## ##\n", - "###########################################################################################\n", - "def train_one_epoch(model, trainloader, optimizer, criterion):\n", - " model.train()\n", - " train_loss = 0\n", - " train_accuracy = 0\n", - " for batch_idx, (inputs, targets) in enumerate(trainloader):\n", - " inputs, targets = inputs.to(device), targets.to(device)\n", - " optimizer.zero_grad()\n", - " outputs = model(inputs)\n", - " loss = criterion(outputs, targets)\n", - " loss.backward()\n", - " optimizer.step()\n", - " train_loss += loss.item()\n", - " accuracy = get_accuracy(outputs, targets)\n", - " train_accuracy += accuracy[0].item()\n", - " avg_train_loss = train_loss/(batch_idx+1)\n", - " avg_train_acc = train_accuracy/(batch_idx+1)\n", - " return avg_train_loss, avg_train_acc, model" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "NdtZ0lf9tdII" - }, - "source": [ - "## TODO ##\n", - "###########################################################################################\n", - "## Define an optimizer (SGD, learning rate = 0.01 and momentum 0.9) and ##\n", - "## train the model for 10 epochs ##\n", - "## ##\n", - "###########################################################################################\n", - "import torch.optim as optim\n", - "\n", - "model = Net().to(device)\n", - "model.apply(weight_reset)\n", - "optimizer =\n", - "\n", - "num_epochs = 10\n", - "for epoch in range(num_epochs):\n", - " train_loss, train_acc, model = train_one_epoch(model, trainloader, optimizer, criterion)\n", - " val_loss, val_acc = validate(model, valloader, criterion)\n", - " message = (f\"epoch:{epoch+1} train_acc:{train_acc:.2f} \"\n", - " f\"val_acc:{val_acc:.2f}\")\n", - " print(message)" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "LsoL3ZxStdIL" - }, - "source": [ - "### Task: Define your own network, such that the validation acccuracy of the model is greater than 70%. Following constraints should be strictly followed: ###\n", - "\n", - "1. The network should be trained from scratch (randomly initialized weigths), transfer learning is not allowed\n", - "\n", - "Note: Apply __model.apply(weight_reset)__ method over your model before training to make sure model is randomly initialized\n", - "2. Use the training methods as defined above (train_one_epoch() and validate() should be used and not changed)\n", - "3. Your network should reach atleast 70% accuracy on validatin set under 10 epochs" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "m0oXbr_4tdIM" - }, - "source": [ - "## TODO ##\n", - "###########################################################################################\n", - "## Write the implementation of the model here according to the task defined above. ##\n", - "## Define any other changes you made in optimizer or transforms (if required) in this ##\n", - "## cell only. Train the model for 10 epochs and print validation accuracy after the ##\n", - "## 10th epoch ##\n", - "###########################################################################################\n", - "import torch.nn as nn\n", - "import torch.nn.functional as F\n", - "\n", - "from collections import OrderedDict\n", - "\n", - "\n", - "class GhostModel(nn.Module):\n", - " def __init__(self):\n", - " super(GhostModel, self).__init__()\n", - "\n", - " def forward(self, x):\n", - " return x" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "C8pIlH4utdIQ" - }, - "source": [ - "__Hint:__ One might try following ways to obtain the required results:\n", - "\n", - "1) __Filter Size:__ The filter size can be decreased from 5 to 3, this can increase efficiency and reduce overfitting\n", - "\n", - "2) __Number of Filters:__ Increase the number of filters in each convoltuional layer. More convolutional filters gives a network higher representation power\n", - "\n", - "3) __Add more convolutions:__ Increase from two to three convolutional layers. This indeed helps in better receptive fields of the filters and increase represtation power of the network.\n", - "\n", - "4) __Apply Batch Normalizations:__ Batch Normalization will help your model to train faster. Hence it can help the network to converge to better accuracy in less number of epochs\n", - "\n", - "5) __Global Average Pooling:__ Get rid of all the layers and replace them by average pooling and a linear layer for classification. This idea is from the Inception network. \n", - "\n", - "6) __Regularization:__ Apply L2 regularization and dropout, if you see validation accuracy going much lower than the training accuracy\n", - "\n", - "7) __Different Optimizer:__ Instead of SGD, optimizers like Adam and Adagrad can be tried" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "qwkbsl6-tdIQ" - }, - "source": [ - "import torch.optim as optim\n", - "\n", - "\n", - "model = GhostModel().to(device)\n", - "model.apply(weight_reset)\n", - "\n", - "optimizer =\n", - "\n", - "num_epochs = 10\n", - "for epoch in range(num_epochs):\n", - " train_loss, train_acc, model = train_one_epoch(model, trainloader, optimizer, criterion)\n", - " val_loss, val_acc = validate(model, valloader, criterion)\n", - " message = (f\"epoch:{epoch+1} train_acc:{train_acc:.2f} \"\n", - " f\"val_acc:{val_acc:.2f}\")\n", - " print(message)" - ], - "execution_count": null, - "outputs": [] - } - ] -} \ No newline at end of file diff --git a/exercises/exercise2/Exercise_2(5).ipynb b/exercises/exercise2/Exercise_2(5).ipynb deleted file mode 100644 index fe6fa2b..0000000 --- a/exercises/exercise2/Exercise_2(5).ipynb +++ /dev/null @@ -1,444 +0,0 @@ -{ - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "provenance": [], - "include_colab_link": true - }, - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - }, - "language_info": { - "name": "python" - }, - "accelerator": "GPU", - "gpuClass": "standard" - }, - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "view-in-github", - "colab_type": "text" - }, - "source": [ - "<a href=\"https://colab.research.google.com/github/mindgarage/very-deep-learning-wise2324/blob/main/exercises/Exercise_2(5).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>" - ] - }, - { - "cell_type": "markdown", - "source": [ - "# Programming Exercise 2(5): Depthwise Separable Convolutions\n", - "\n", - "## Very Deep Learning (VDL) - Winter Semester 2023/24\n", - "\n", - "---\n", - "\n", - "### Group Details:\n", - "\n", - "- **Group Name:** \\[Enter OLAT Group Name Here\\]\n", - "\n", - "### Members:\n", - "\n", - "- \\[Participant 1 Name\\], \\[Matrikel-Nr 1\\]\n", - "- \\[Participant 2 Name\\], \\[Matrikel-Nr 1\\]\n", - "- ...\n", - "\n", - "---\n", - "\n", - "**Instructions**: The tasks in this notebook are a part of Sheet 1. Look for `TODO` tags throughout the notebook and complete the sections with missing code. Once done, ensure all outputs are visible and correctly displayed. Save your notebook and submit the `.ipynb` file together with the exercise sheet PDF in a single ZIP file." - ], - "metadata": { - "id": "3xdlD0FNSmKm" - } - }, - { - "cell_type": "markdown", - "source": [ - "### Define imports" - ], - "metadata": { - "id": "CsaYjGxwpYn7" - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "qMHks9r3axuw" - }, - "outputs": [], - "source": [ - "# Define necessary imports\n", - "import torch\n", - "from torch import nn,save,load\n", - "from torch.utils.data import DataLoader\n", - "from torch.optim import Adam\n", - "from torchvision import datasets\n", - "from torchvision.transforms import ToTensor" - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Download the data and define the Dataloader" - ], - "metadata": { - "id": "DN-8uNQ_phna" - } - }, - { - "cell_type": "code", - "source": [ - "# Download the dataset\n", - "train_dataset = datasets.MNIST(root=\"data\", train=True, download=True, transform=ToTensor())\n", - "test_dataset = datasets.MNIST(root=\"data\", train=False, download=True, transform=ToTensor())\n", - "\n", - "# Define corresponding Data Loaders\n", - "train = DataLoader(train_dataset, batch_size=32, shuffle=True)\n", - "test = DataLoader(test_dataset, 32)" - ], - "metadata": { - "id": "eLWzQyjPlOK0" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "### Convolutional Neural Network\n", - "\n", - "**Definition:** Convolutional neural networks are a specialized type of artificial neural networks that use a mathematical operation called convolution in place of general matrix multiplication. They are specifically designed to process pixel data and are used in image recognition and processing. (source wikipedia)\n", - "\n", - "**Pooling:** Pooling is a technique to down sample feature maps by summarizing the presence of features in patches of the feature map. They can be various types such as maxpooling, average pooling etc. It decreases the number of parameters and therefor the necessary computation.\n", - "\n" - ], - "metadata": { - "id": "f7Tc33hpqMJ6" - } - }, - { - "cell_type": "markdown", - "source": [ - "" - ], - "metadata": { - "id": "y8cIjLEa1wo3" - } - }, - { - "cell_type": "markdown", - "source": [ - "#### **Task**\n", - "\n", - "Create a Neural Network model as shown in the above image to train MNIST dataset.\n", - "\n", - "*Convolution operation*\n", - "\n", - "> Note that each convolution operation must be followed by a **ReLU activation**. Please use a **3x3 kernal for convolution** operation. Assume the zero padding and a stride of 1 for convolution operations.\n", - "\n", - "\n", - "*Pooling operation*\n", - "\n", - "> Also assume zero padding for pooling operation. Rest of the necessary infomation (parameters) for pooling are described in the above image.\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "One need not code for FC (Fully Connected Layer and output) as it is already mentioned in the below code." - ], - "metadata": { - "id": "fbl3kbxlpwh2" - } - }, - { - "cell_type": "code", - "source": [ - "# TODO: Complete the below network\n", - "class ImageClassifier(nn.Module):\n", - " def __init__(self):\n", - " super().__init__()\n", - " self.model = nn.Sequential(\n", - "\n", - "\n", - "\n", - " nn.Flatten(),\n", - " nn.Linear(64*5*5, 10)\n", - " )\n", - " def forward(self, x):\n", - " return self.model(x)\n", - "\n" - ], - "metadata": { - "id": "f2IHEYVznOq_" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "### Summary of the model\n", - "\n", - "Its time to explore the **output shape** of each layer and the **number of parametres** that we obtain from each layer of the Neural Network. At present there are two options to print the summary of your model.\n", - "\n", - "*Option 1:* Use torchsummary package [corresponding git repo](https://github.com/sksq96/pytorch-summary)\n", - "\n", - "*Option 2:* Use torchinfo package. [corresponding git repo](https://github.com/TylerYep/torchinfo)" - ], - "metadata": { - "id": "WURH9LjELijh" - } - }, - { - "cell_type": "markdown", - "source": [ - "#### **Task**\n", - "\n", - "Use one of the above options to print the your model summary as output." - ], - "metadata": { - "id": "sXMcfNnbB81y" - } - }, - { - "cell_type": "code", - "source": [ - "# TODO: write code to print summary of your model.\n", - "# If you are stuck, please check your input data size format and also the device (cpu/gpu) your data is loaded into.\n" - ], - "metadata": { - "id": "8x7mKdbA1OA0" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "source": [ - "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")" - ], - "metadata": { - "id": "NoGxvsDNakI7" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "### Define training and evaluation" - ], - "metadata": { - "id": "1hak6Ev_rUsk" - } - }, - { - "cell_type": "code", - "source": [ - "# Training\n", - "'''\n", - "classifier (model), loss function and optimiser must be passed as an argument\n", - "'''\n", - "def train_epoch(clf, loss_fn, opt):\n", - " no_epochs = 15\n", - " for epoch in range(no_epochs):\n", - " for batch in train:\n", - " X, y = batch\n", - " X, y = X.to(device), y.to(device)\n", - " # predict\n", - " y_hat = clf(X)\n", - " #loss\n", - " loss = loss_fn(y_hat, y)\n", - " #back prop steps\n", - " opt.zero_grad()\n", - " loss.backward()\n", - " # update\n", - " opt.step()\n", - " print(f\"Epoch:{epoch+1} loss is {loss.item()}\")\n", - "\n" - ], - "metadata": { - "id": "yjVVGs7zrbzD" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "source": [ - "# Evaluation\n", - "'''\n", - "classifier (model) and loss function must be passed as an argument.\n", - "It returns accuracy of the model.\n", - "'''\n", - "def eval(clf, loss_fn):\n", - " total_accuracy = 0.0\n", - " for batch in test:\n", - " X, y = batch\n", - " X, y = X.to(device), y.to(device)\n", - " y_hat = clf(X)\n", - " loss = loss_fn(y_hat, y)\n", - " total_accuracy += (y_hat.max(dim=1)[1] == y).sum().item();\n", - " return total_accuracy / len(test.dataset)" - ], - "metadata": { - "id": "bFNXedtkt4Kj" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "### Perform training and evaluation" - ], - "metadata": { - "id": "oLG4ZiL5ufqw" - } - }, - { - "cell_type": "code", - "source": [ - "# Train and evaluate the model that uses standard convolution operations\n", - "# on MNIST Dataset.\n", - "\n", - "clf = ImageClassifier().to(device)\n", - "loss_fn = nn.CrossEntropyLoss()\n", - "opt= Adam(clf.parameters(), lr=1e-3)\n", - "\n", - "train_epoch(clf, loss_fn, opt)\n", - "accuracy = eval(clf, loss_fn)\n", - "print(\"\\nAccuracy of the model that uses standard convolution operations:\", accuracy*100)\n", - "\n" - ], - "metadata": { - "id": "_kVhtJtBu3y8" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "### **Depthwise Separable Convolution**\n", - "\n", - "It is composed of two steps:\n", - "\n", - "> First perform **Depthwise convolution**, it means applying a single convolutional filter per each input channel. Then perform **Pointwise convolution**, it means to create a linear combination of the output of the depthwise convolution.\n", - "\n", - "**Task:**\n", - "\n", - "Modify the above model (ImageClassifier) that uses Standard 2D Convolution to Depthwise Separable Convolution.\n", - "\n", - "\n", - "hint: a) use groups parameter to achieve Depthwise convolution, b) use 1x1 size kernel to achieve Pointwise convolution.\n", - "\n" - ], - "metadata": { - "id": "L3dX1nJgI88I" - } - }, - { - "cell_type": "code", - "source": [ - "# TODO: Complete the below model with Depthwise separable convolutions\n", - "class ImageClassifierDepthwiseSeparable(nn.Module):\n", - " def __init__(self):\n", - " super().__init__()\n", - " self.model = nn.Sequential(\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - " nn.Flatten(),\n", - " nn.Linear(64*5*5, 10)\n", - " )\n", - " def forward(self, x):\n", - " return self.model(x)" - ], - "metadata": { - "id": "x0f2vlC-qnKu" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "### Summary of your new model.\n" - ], - "metadata": { - "id": "-sEWRSYVO7vD" - } - }, - { - "cell_type": "code", - "source": [ - "# TODO write code to print summary of the new model.\n", - "\n" - ], - "metadata": { - "id": "7lM4ECgf2Xmx" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "source": [ - "# Depthwise separable convolution\n", - "clf_new = ImageClassifierDepthwiseSeparable().to(device)\n", - "loss_fn = nn.CrossEntropyLoss()\n", - "opt= Adam(clf_new.parameters(), lr=1e-3)\n", - "\n", - "train_epoch(clf_new, loss_fn, opt)\n", - "depthwise_sep_acc = eval(clf_new, loss_fn)\n", - "print(\"Accuracy of new model with Depthwise separable convolution operationst:\", depthwise_sep_acc*100)" - ], - "metadata": { - "id": "FZVZ6DapyrQk" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "### Analysing the two models.\n", - "\n", - "#### **Task**\n", - "\n", - "**TODO:** Answer the below questions:\n", - "\n", - "\n", - "1) The total number of parameters present in first model (ImageClassifier) are ??\n", - "\n", - "\n", - "2) The total number of parameters present in new model (ImageClassifierDepthwiseSeparable) are ??\n", - "\n" - ], - "metadata": { - "id": "kqXuW5XUSG1i" - } - }, - { - "cell_type": "markdown", - "source": [ - "The first model that uses **Standard convolutions has more number of parameters compared to the new model with Depthwise separable convolution**s for the same neural network architecture that is trained on MNIST dataset to classify images from zero to nine.\n", - "\n", - "Thus Depthwise separable is computationally efficient and also it prevents the model from overfitting." - ], - "metadata": { - "id": "TP5l_vAqVf8Y" - } - } - ] -} \ No newline at end of file -- GitLab