{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d13df7ea",
   "metadata": {
    "origin_pos": 1
   },
   "source": [
    "# Synthetic Regression Data\n",
    ":label:`sec_synthetic-regression-data`\n",
    "\n",
    "\n",
    "Machine learning is all about extracting information from data.\n",
    "So you might wonder, what could we possibly learn from synthetic data?\n",
    "While we might not care intrinsically about the patterns \n",
    "that we ourselves baked into an artificial data generating model,\n",
    "such datasets are nevertheless useful for didactic purposes,\n",
    "helping us to evaluate the properties of our learning \n",
    "algorithms and to confirm that our implementations work as expected.\n",
    "For example, if we create data for which the correct parameters are known *a priori*,\n",
    "then we can check that our model can in fact recover them.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b9773b7e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:11.561594Z",
     "iopub.status.busy": "2023-08-18T19:36:11.560983Z",
     "iopub.status.idle": "2023-08-18T19:36:15.344149Z",
     "shell.execute_reply": "2023-08-18T19:36:15.342706Z"
    },
    "origin_pos": 3,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import random\n",
    "import torch\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b65a3a2",
   "metadata": {
    "origin_pos": 6
   },
   "source": [
    "## Generating the Dataset\n",
    "\n",
    "For this example, we will work in low dimension\n",
    "for succinctness.\n",
    "The following code snippet generates 1000 examples\n",
    "with 2-dimensional features drawn \n",
    "from a standard normal distribution.\n",
    "The resulting design matrix $\\mathbf{X}$\n",
    "belongs to $\\mathbb{R}^{1000 \\times 2}$. \n",
    "We generate each label by applying \n",
    "a *ground truth* linear function, \n",
    "corrupting them via additive noise $\\boldsymbol{\\epsilon}$, \n",
    "drawn independently and identically for each example:\n",
    "\n",
    "(**$$\\mathbf{y}= \\mathbf{X} \\mathbf{w} + b + \\boldsymbol{\\epsilon}.$$**)\n",
    "\n",
    "For convenience we assume that $\\boldsymbol{\\epsilon}$ is drawn \n",
    "from a normal distribution with mean $\\mu= 0$ \n",
    "and standard deviation $\\sigma = 0.01$.\n",
    "Note that for object-oriented design\n",
    "we add the code to the `__init__` method of a subclass of `d2l.DataModule` (introduced in :numref:`oo-design-data`). \n",
    "It is good practice to allow the setting of any additional hyperparameters. \n",
    "We accomplish this with `save_hyperparameters()`. \n",
    "The `batch_size` will be determined later.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e174e8b1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.351059Z",
     "iopub.status.busy": "2023-08-18T19:36:15.350010Z",
     "iopub.status.idle": "2023-08-18T19:36:15.358156Z",
     "shell.execute_reply": "2023-08-18T19:36:15.357035Z"
    },
    "origin_pos": 7,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "class SyntheticRegressionData(d2l.DataModule):  #@save\n",
    "    \"\"\"Synthetic data for linear regression.\"\"\"\n",
    "    def __init__(self, w, b, noise=0.01, num_train=1000, num_val=1000,\n",
    "                 batch_size=32):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        n = num_train + num_val\n",
    "        self.X = torch.randn(n, len(w))\n",
    "        noise = torch.randn(n, 1) * noise\n",
    "        self.y = torch.matmul(self.X, w.reshape((-1, 1))) + b + noise"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60afd383",
   "metadata": {
    "origin_pos": 8
   },
   "source": [
    "Below, we set the true parameters to $\\mathbf{w} = [2, -3.4]^\\top$ and $b = 4.2$.\n",
    "Later, we can check our estimated parameters against these *ground truth* values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "38a83404",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.362285Z",
     "iopub.status.busy": "2023-08-18T19:36:15.361503Z",
     "iopub.status.idle": "2023-08-18T19:36:15.390526Z",
     "shell.execute_reply": "2023-08-18T19:36:15.389339Z"
    },
    "origin_pos": 9,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "data = SyntheticRegressionData(w=torch.tensor([2, -3.4]), b=4.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4105d024",
   "metadata": {
    "origin_pos": 10
   },
   "source": [
    "[**Each row in `features` consists of a vector in $\\mathbb{R}^2$ and each row in `labels` is a scalar.**] Let's have a look at the first entry.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "43e267cb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.395708Z",
     "iopub.status.busy": "2023-08-18T19:36:15.394509Z",
     "iopub.status.idle": "2023-08-18T19:36:15.405078Z",
     "shell.execute_reply": "2023-08-18T19:36:15.402629Z"
    },
    "origin_pos": 11,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "features: tensor([0.9026, 1.0264]) \n",
      "label: tensor([2.5148])\n"
     ]
    }
   ],
   "source": [
    "print('features:', data.X[0],'\\nlabel:', data.y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf8516d3",
   "metadata": {
    "origin_pos": 12
   },
   "source": [
    "## Reading the Dataset\n",
    "\n",
    "Training machine learning models often requires multiple passes over a dataset, \n",
    "grabbing one minibatch of examples at a time. \n",
    "This data is then used to update the model. \n",
    "To illustrate how this works, we \n",
    "[**implement the `get_dataloader` method,**] \n",
    "registering it in the `SyntheticRegressionData` class via `add_to_class` (introduced in :numref:`oo-design-utilities`).\n",
    "It (**takes a batch size, a matrix of features,\n",
    "and a vector of labels, and generates minibatches of size `batch_size`.**)\n",
    "As such, each minibatch consists of a tuple of features and labels. \n",
    "Note that we need to be mindful of whether we're in training or validation mode: \n",
    "in the former, we will want to read the data in random order, \n",
    "whereas for the latter, being able to read data in a pre-defined order \n",
    "may be important for debugging purposes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1686e6b2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.409740Z",
     "iopub.status.busy": "2023-08-18T19:36:15.408327Z",
     "iopub.status.idle": "2023-08-18T19:36:15.417911Z",
     "shell.execute_reply": "2023-08-18T19:36:15.416944Z"
    },
    "origin_pos": 13,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "@d2l.add_to_class(SyntheticRegressionData)\n",
    "def get_dataloader(self, train):\n",
    "    if train:\n",
    "        indices = list(range(0, self.num_train))\n",
    "        # The examples are read in random order\n",
    "        random.shuffle(indices)\n",
    "    else:\n",
    "        indices = list(range(self.num_train, self.num_train+self.num_val))\n",
    "    for i in range(0, len(indices), self.batch_size):\n",
    "        batch_indices = torch.tensor(indices[i: i+self.batch_size])\n",
    "        yield self.X[batch_indices], self.y[batch_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2437653f",
   "metadata": {
    "origin_pos": 14
   },
   "source": [
    "To build some intuition, let's inspect the first minibatch of\n",
    "data. Each minibatch of features provides us with both its size and the dimensionality of input features.\n",
    "Likewise, our minibatch of labels will have a matching shape given by `batch_size`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d5af1472",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.424253Z",
     "iopub.status.busy": "2023-08-18T19:36:15.423639Z",
     "iopub.status.idle": "2023-08-18T19:36:15.430119Z",
     "shell.execute_reply": "2023-08-18T19:36:15.429099Z"
    },
    "origin_pos": 15,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X shape: torch.Size([32, 2]) \n",
      "y shape: torch.Size([32, 1])\n"
     ]
    }
   ],
   "source": [
    "X, y = next(iter(data.train_dataloader()))\n",
    "print('X shape:', X.shape, '\\ny shape:', y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c485f93e",
   "metadata": {
    "origin_pos": 16
   },
   "source": [
    "While seemingly innocuous, the invocation \n",
    "of `iter(data.train_dataloader())` \n",
    "illustrates the power of Python's object-oriented design. \n",
    "Note that we added a method to the `SyntheticRegressionData` class\n",
    "*after* creating the `data` object. \n",
    "Nonetheless, the object benefits from \n",
    "the *ex post facto* addition of functionality to the class.\n",
    "\n",
    "Throughout the iteration we obtain distinct minibatches\n",
    "until the entire dataset has been exhausted (try this).\n",
    "While the iteration implemented above is good for didactic purposes,\n",
    "it is inefficient in ways that might get us into trouble with real problems.\n",
    "For example, it requires that we load all the data in memory\n",
    "and that we perform lots of random memory access.\n",
    "The built-in iterators implemented in a deep learning framework\n",
    "are considerably more efficient and they can deal\n",
    "with sources such as data stored in files, \n",
    "data received via a stream, \n",
    "and data generated or processed on the fly. \n",
    "Next let's try to implement the same method using built-in iterators.\n",
    "\n",
    "## Concise Implementation of the Data Loader\n",
    "\n",
    "Rather than writing our own iterator,\n",
    "we can [**call the existing API in a framework to load data.**]\n",
    "As before, we need a dataset with features `X` and labels `y`. \n",
    "Beyond that, we set `batch_size` in the built-in data loader \n",
    "and let it take care of shuffling examples  efficiently.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d5ae674e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.434272Z",
     "iopub.status.busy": "2023-08-18T19:36:15.433429Z",
     "iopub.status.idle": "2023-08-18T19:36:15.441792Z",
     "shell.execute_reply": "2023-08-18T19:36:15.439267Z"
    },
    "origin_pos": 18,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "@d2l.add_to_class(d2l.DataModule)  #@save\n",
    "def get_tensorloader(self, tensors, train, indices=slice(0, None)):\n",
    "    tensors = tuple(a[indices] for a in tensors)\n",
    "    dataset = torch.utils.data.TensorDataset(*tensors)\n",
    "    return torch.utils.data.DataLoader(dataset, self.batch_size,\n",
    "                                       shuffle=train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "617242ed",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.448335Z",
     "iopub.status.busy": "2023-08-18T19:36:15.447832Z",
     "iopub.status.idle": "2023-08-18T19:36:15.457888Z",
     "shell.execute_reply": "2023-08-18T19:36:15.456920Z"
    },
    "origin_pos": 19,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "@d2l.add_to_class(SyntheticRegressionData)  #@save\n",
    "def get_dataloader(self, train):\n",
    "    i = slice(0, self.num_train) if train else slice(self.num_train, None)\n",
    "    return self.get_tensorloader((self.X, self.y), train, i)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b36d404",
   "metadata": {
    "origin_pos": 20
   },
   "source": [
    "The new data loader behaves just like the previous one, except that it is more efficient and has some added functionality.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2f5d57c3",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "4"
    },
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.464003Z",
     "iopub.status.busy": "2023-08-18T19:36:15.462740Z",
     "iopub.status.idle": "2023-08-18T19:36:15.474793Z",
     "shell.execute_reply": "2023-08-18T19:36:15.473623Z"
    },
    "origin_pos": 21,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X shape: torch.Size([32, 2]) \n",
      "y shape: torch.Size([32, 1])\n"
     ]
    }
   ],
   "source": [
    "X, y = next(iter(data.train_dataloader()))\n",
    "print('X shape:', X.shape, '\\ny shape:', y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e8e09f0",
   "metadata": {
    "origin_pos": 22
   },
   "source": [
    "For instance, the data loader provided by the framework API \n",
    "supports the built-in `__len__` method, \n",
    "so we can query its length, \n",
    "i.e., the number of batches.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "790cbdfb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:36:15.479797Z",
     "iopub.status.busy": "2023-08-18T19:36:15.478884Z",
     "iopub.status.idle": "2023-08-18T19:36:15.489245Z",
     "shell.execute_reply": "2023-08-18T19:36:15.488320Z"
    },
    "origin_pos": 23,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.train_dataloader())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5e73e34",
   "metadata": {
    "origin_pos": 24
   },
   "source": [
    "## Summary\n",
    "\n",
    "Data loaders are a convenient way of abstracting out \n",
    "the process of loading and manipulating data. \n",
    "This way the same machine learning *algorithm* \n",
    "is capable of processing many different types and sources of data \n",
    "without the need for modification. \n",
    "One of the nice things about data loaders \n",
    "is that they can be composed. \n",
    "For instance, we might be loading images \n",
    "and then have a postprocessing filter \n",
    "that crops them or modifies them in other ways. \n",
    "As such, data loaders can be used \n",
    "to describe an entire data processing pipeline. \n",
    "\n",
    "As for the model itself, the two-dimensional linear model \n",
    "is about the simplest we might encounter. \n",
    "It lets us test out the accuracy of regression models \n",
    "without worrying about having insufficient amounts of data \n",
    "or an underdetermined system of equations. \n",
    "We will put this to good use in the next section.  \n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. What will happen if the number of examples cannot be divided by the batch size. How would you change this behavior by specifying a different argument by using the framework's API?\n",
    "1. Suppose that we want to generate a huge dataset, where both the size of the parameter vector `w` and the number of examples `num_examples` are large.\n",
    "    1. What happens if we cannot hold all data in memory?\n",
    "    1. How would you shuffle the data if it is held on disk? Your task is to design an *efficient* algorithm that does not require too many random reads or writes. Hint: [pseudorandom permutation generators](https://en.wikipedia.org/wiki/Pseudorandom_permutation) allow you to design a reshuffle without the need to store the permutation table explicitly :cite:`Naor.Reingold.1999`. \n",
    "1. Implement a data generator that produces new data on the fly, every time the iterator is called. \n",
    "1. How would you design a random data generator that generates *the same* data each time it is called?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a6d8f2c",
   "metadata": {
    "origin_pos": 26,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/6663)\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "required_libs": []
 },
 "nbformat": 4,
 "nbformat_minor": 5
}