01a_test_your_setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "70f43429",
   "metadata": {},
   "source": [
    "<img src='./fig/vertical_COMILLAS_COLOR.jpg' style= 'width:70mm'>\n",
    "<h1 style='font-family: Optima;color:#ecac00'>\n",
    "Máster en Big Data. Tecnología y Analítica Avanzada (MBD).\n",
    "</h1>\n",
    "\n",
    "<h1 style='font-family: Optima;color:#ecac00'>\n",
    "Fundamentos Matemáticos del Análisis de Datos (FMAD). 2022-2023.\n",
    "</h1>\n",
    "\n",
    "<h1 style='font-family: Optima;color:#ecac00'>\n",
    "01a Python Setup Checks\n",
    "</h1> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eb2245e",
   "metadata": {},
   "source": [
    "\n",
    "**To test that your python setup is looking good please select the `Cell` menu above and click on `Run All`**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3678d957",
   "metadata": {},
   "source": [
    "## Check your Conda environment and Python version\n",
    "\n",
    "Look for the active environment in the output of the following command. It should be `fmad`. Also check the Python version you are using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dad8a685",
   "metadata": {},
   "outputs": [],
   "source": [
    "!conda info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba5cfe67",
   "metadata": {},
   "source": [
    "## Check the Basic Libraries for Data Science with Python\n",
    "\n",
    "### NumPy \n",
    "\n",
    "Check the numpy version and make sure that no errors appear in the output of the code cells below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9bba367",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "print(np.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "134ec87b",
   "metadata": {},
   "source": [
    "**Note about standard imports:**\n",
    "\n",
    "+ Many of the Python libraries we will use have standard import names. Even though these are not official names, we strongly recommend their use, in order to make your code readable and compatible. For example, as we have just seen, NumPy should always be imported as `np`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00922089",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.random.rand(1000)\n",
    "%timeit a @ a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64b5aaf9",
   "metadata": {},
   "source": [
    "### Pandas\n",
    "\n",
    "Similarly for Pandas and the rest of the libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9456079",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard import for Pandas\n",
    "import pandas as pd\n",
    "print(pd.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "714f588f",
   "metadata": {},
   "source": [
    "### MatplotLib and Seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c925a7ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard import for Matplotlib\n",
    "import matplotlib as mpl \n",
    "print(mpl.__version__)\n",
    "\n",
    "# But we will more frequently use this import sequence\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47db9749",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard import for Seaborn\n",
    "import seaborn as sns\n",
    "print(sns.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f2f12f6",
   "metadata": {},
   "source": [
    "### Scikit-learn\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41695c80",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard import for Scikit-learn\n",
    "import sklearn as sk \n",
    "print(sk. __version__) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dfe2feb",
   "metadata": {},
   "source": [
    "## Working with Pandas DataFrame, First Example.\n",
    "\n",
    "+ For our first example we will use a data set called `titanic`, which is contained in the Seaborn library that you should have already installed. The data set contains information about the passengers of the Titanic, such as their age, gender, class in which they were traveling, whether they survived the ship shinking, etc.  \n",
    "\n",
    "+ We will soon see how to use Python to read data from different sources: csv and Excel files, urls, databases, APIs, etc. But for now we just want to run some tests and get an overview of the data structures we will be working with.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aac7024a",
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic = sns.load_dataset('titanic')\n",
    "print(type(titanic))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9faf9c16",
   "metadata": {},
   "source": [
    "As you can see, `titanic` is now a Pandas DataFrame. This is the object that we will most frequently meet when working with data tables (alongside with NumPy arrays). To see the first lines of the `titanic` data set we can use the `head` method. The optional `n` argument determines the number of rows in the output (the default is `n = 5`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a4bdf16",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(titanic.head(n = 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "491f5b42",
   "metadata": {},
   "source": [
    "**Exercise:** try running the above with other values of `n`. Also run it as `print(titanic.head(4))`. Use the *Help* menu to open the *Reference* link and look up the information about the `head` method for Pandas DataFrames. \n",
    "\n",
    "**Note:** you may have noticed sone `NaN` values in the `deck` column of the table. This are *missing data*. We will see how to deal with missing data later in the course. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e523580",
   "metadata": {},
   "source": [
    "### Basic properties of a DataFrame\n",
    "\n",
    "How many rows and columns of data are there in this data set? We get the answer with `shape`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a4945a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(titanic.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f155f59",
   "metadata": {},
   "source": [
    "If you just use\n",
    "\n",
    "```python\n",
    "print(titanic)\n",
    "```\n",
    "\n",
    "you will get a summary of the table and the shape information will appear below. \n",
    "\n",
    "To see the *column names* we use, quite naturally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67c438b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(titanic.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b02dedd0",
   "metadata": {},
   "source": [
    "As we see, the column names are stored in an `Index` object. We will learn more about indices later in the course. \n",
    "\n",
    "Sometimes the row names can contain information as well. They are also stored in an index which can be accessed with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21b1c92d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(titanic.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a363bcb6",
   "metadata": {},
   "source": [
    "### Accessing the data\n",
    "\n",
    "+ We can get any element in the data table using brackets, the `iloc` method and row/column pairs. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3d3f7b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.iloc[1, 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6bd0f5c",
   "metadata": {},
   "source": [
    "shows that the element of the second row and fourth column in the table is `38.0`. \n",
    "\n",
    "**Always keep in mind that Python counts are zero-based!!** Thus index zero corresponds to the first element in an ordered set.\n",
    "\n",
    "In a data table the columns usually correspond to variables and the rows correspond to observations. And we should have a very good reason to do otherwise!! In the above example, the fourth column corresponds to the `age` variable of the Titanic passengers. It is often better to refer to variables by their names. We can do it with the `loc` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04e3a8c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.loc[1, 'age']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2630f767",
   "metadata": {},
   "source": [
    "In this cases the row indices are numeric (in fact, consecutive zero-based integers), so we can use 1 with `loc` or `iloc` to select a row. \n",
    "\n",
    "We could also first extract the `age` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a72db84",
   "metadata": {},
   "outputs": [],
   "source": [
    "ttnc_age = titanic['age']\n",
    "print(ttnc_age)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b26b9854",
   "metadata": {},
   "source": [
    "What kind of object is this column?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f41bf3d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(ttnc_age))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddb053ba",
   "metadata": {},
   "source": [
    "As we see, it is a *Pandas Series* object. A DataFrame can be considered as a collection of Series (columns) with a common (row) index.  \n",
    "\n",
    "**Note:** the `age` column can also be accessed as an attribute of the table, with `titanic.age`. Usually we prefer the bracket method, but this *attribute notation* can be handy when we want to shorten our code sentences. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ba3cd06",
   "metadata": {},
   "source": [
    "### Modifying data\n",
    "\n",
    "+ The `loc` and `iloc` methods can also be used to modify elements of the table. Let us modify the age of that passenger and check the result with head:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29dd1b6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.iloc[1, 3] = 19.0\n",
    "\n",
    "print(titanic.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1064e016",
   "metadata": {},
   "source": [
    "**Exercise:** now use `iloc` to return the age of the passenger to the original `38.0` value and check your work with `head`. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "863df897",
   "metadata": {},
   "source": [
    "### Accessing larger portions of the DataFrame (indexing and slicing)\n",
    "\n",
    "\n",
    "+ If we want to access several rows and or columns we can use explicit *indexing* (note that the order is relevant) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47dbed2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.loc[[0, 1, 2, 3, 4] , ['sex', 'age', 'survived', 'pclass']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1ba2316",
   "metadata": {},
   "source": [
    "or we can use *slicing* with the colon`:` to get consecutive rows or columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efeafe2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.loc[:4 , 'survived':'age']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5127d122",
   "metadata": {},
   "source": [
    "**Warning:** contrary to the usual Python (and NumPy) convention, the final values 4 and age of the indices are included in the output for `loc`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01f94b67",
   "metadata": {},
   "source": [
    "+ In the above we have used `loc` but you can also use numeric positions with `iloc`. **Note**, however, that the final values of the numeric indexes are excluded for `iloc` output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ff6538f",
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.iloc[:4, :3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "363fdf5a",
   "metadata": {},
   "source": [
    "### Condition based filtering\n",
    "\n",
    "+ One of the most powerful tools in data analysis is the ability to filter only those rows of a data table that meet some condition, usually expressed as a *boolean* condition. For example, we will filter the `titanic` DataFrame, keeping only the rows corresponding to female passengers with ages equal or greater than 25 years (observe the use of the attribute notation we mentioned before).   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cde1c08f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ttnc_female_25plus = titanic.loc[(titanic.age >= 25) &  (titanic.sex == 'female')]\n",
    "\n",
    "print(ttnc_female_25plus.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a5bae87",
   "metadata": {},
   "source": [
    "## Some final fun for the session\n",
    "\n",
    "+ Create a subfolder called `data` in the same folder that contains your Jupyter notebooks for this session  Download the [data set in this link](https://raw.githubusercontent.com/mbdfmad/fmad2223/main/data/gapminder.csv)  to that folder ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2172d5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "gapminder = pd.read_csv('data/gapminder.csv', index_col = 0)\n",
    "print(gapminder.head())\n",
    "gdp_cap = gapminder.iloc[:, -1].tolist()\n",
    "life_exp = gapminder.iloc[:, -2].tolist()\n",
    "pop = gapminder.iloc[:, 2].tolist()\n",
    "cont = gapminder.iloc[:, 3].tolist()\n",
    "continentColors = {\n",
    "    'Asia':'red',\n",
    "    'Europe':'green',\n",
    "    'Africa':'blue',\n",
    "    'Americas':'yellow',\n",
    "    'Oceania':'black'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3adefbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams_old = plt.rcParams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09c43dfa",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "sns.set()\n",
    "\n",
    "plt.rcParams['figure.figsize'] = [12, 8]\n",
    "plt.rcParams['figure.dpi'] = 100\n",
    "\n",
    "col = [continentColors[item] for item in cont]\n",
    "\n",
    "# Specify c and alpha inside plt.scatter()\n",
    "plt.scatter(x = gdp_cap, y = life_exp, s = np.array(pop) /1000000,\n",
    "c = col, alpha = 0.4)\n",
    "\n",
    "# Customizations\n",
    "plt.xscale('log')\n",
    "plt.xlabel('GDP per Capita [in USD]')\n",
    "plt.ylabel('Life Expectancy [in years]')\n",
    "plt.title('World Development in 2007')\n",
    "plt.xticks([1000,10000,100000], ['1k','10k','100k'])\n",
    "plt.grid(True)\n",
    "\n",
    "# Show the plot\n",
    "plt.show()\n",
    "plt.clf()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25f39dc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyleaflet import Map\n",
    "\n",
    "Map(center=[40.43000779017108, -3.7125999200687105], zoom=17)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fde1262e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db963509",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "144px",
    "left": "1166px",
    "right": "20px",
    "top": "121px",
    "width": "350px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}